Compiler FE: testing with models (1.9.1)

[seanshpark]

This issue is to describe compiler frontend typical positve usage and negative usage.

• positive usage: typical usage of our compiler frontend with supported models.
• negative usage: compiler frontend results of unsupported models or invalid files

This issue assumes

• description is based on recent as-of-today ~1.9.0~ 1.9.1 release
• run in Ubuntu 16.04 or 18.04 running on x86-64

Similar issue: #1179 Revision of #3864

[seanshpark]

Download and install

• download from https://github.com/Samsung/ONE/releases/tag/1.9.1
  • https://github.com/Samsung/ONE/releases/download/1.9.1/nncc-1.9.1.tar.gz

• extract files. this will look like this

  • ``` $ tree . ├── bin │   ├── circle2circle │   ├── circle-quantizer │   ├── generate_bcq_output_arrays │   ├── model2nnpkg.sh │   ├── one-codegen │   ├── one-import │   ├── one-import-bcq │   ├── one-import-tf │   ├── one-import-tflite │   ├── one-optimize │   ├── one-pack │   ├── one-prepare-venv │   ├── one-quantize │   ├── one-version │   ├── preserve_bcq_info │   ├── record-minmax │   ├── tf2nnpkg │   ├── tf2tfliteV2.py │   └── tflite2circle ├── doc │   ├── how-to-prepare-virtualenv.txt │   └── how-to-use-one-commands.txt ├── lib │   ├── libloco.so │   ├── libluci_env.so │   ├── libluci_export.so │   ├── libluci_import.so │   ├── libluci_interpreter.so │   ├── libluci_lang.so │   ├── libluci_logex.so │   ├── libluci_log.so │   ├── libluci_pass.so │   └── libluci_service.so └── nncc-1.9.1.tar.gz ```
• add above bin folder to your PATH envrionment
• prepare python virtual environment. for more information, read how-to-prepare-virtualenv.txt file in doc folder.

  $ one-prepare-venv

[seanshpark]

Test case 1: Typical positive usage

Convert to .circle file

one-import tf \
--input_path ./inception_v3.pb \
--output_path ./inception_v3.circle \
--input_arrays input --input_shapes "1,299,299,3" \
--output_arrays InceptionV3/Predictions/Reshape_1

On success, there will be no (error) messages and two files

• inception_v3.circle
• inception_v3.circle.log
  • inception_v3.circle.log will show internal conversion log of internal commands used.

Optimize .circle file

one-optimize --all \
--input_path ./inception_v3.circle  \
--output_path ./inception_v3-opt.circle

On suceess, there will be no (error) messages and two new files

• inception_v3-opt.circle
• inception_v3-opt.circle.log
  • inception_v3-opt.circle.log will show optimization log.

FYI,

one-optimize --help

will show current provided optimization algorithms.

Create package for runtime

one-pack -i ./inception_v3-opt.circle -o nnpack

On suceess, there will be no (error) messages and one new folder nnpack

Your working directory should have these files with tree command

$ tree
.
├── inception_v3_2018_04_27.tgz
├── inception_v3.circle
├── inception_v3.circle.log
├── inception_v3-opt.circle
├── inception_v3-opt.circle.log
├── inception_v3-opt.pack.log
├── inception_v3.pb
├── inception_v3.tflite
├── labels.txt
└── nnpack
    └── inception_v3-opt
        ├── inception_v3-opt.circle
        └── metadata
            └── MANIFEST

[seanshpark]

Test case 2: Typical negative usage

Let's try conversion with normal file but inappropriate input

one-import tf \
--input_path ./inception_v3.tflite \
--output_path ./inception_v3.circle \
--input_arrays input --input_shapes "1,299,299,3" \
--output_arrays InceptionV3/Predictions/Reshape_1

This will drop TensorFlow logs and hint codes about the error and the error message

ValueError: Invalid tensors 'input' were found.

Let's optimize with normal file but inappropriate input

one-optimize --all \
--input_path ./inception_v3.pb \
--output_path ./inception_v3-opt.circle

Error messages something like

/home/.../bin/circle2circle --all ./inception_v3.pb ./inception_v3-opt.circle ERROR: Invalid input file './inception_v3.pb'

[seanshpark]

Test case 3: Typical negative usage with unsupported dynamic tensor

Download model: while_3.zip

one-import tf \
--input_path ./while_3.pbtxt \
--output_path ./while_3.circle \
--input_arrays Hole,Hole_2 --input_shapes "1,1:1,1" \
--output_arrays Output

This will drop error something like this

tensorflow.lite.python.convert.ConverterError: :0: error: loc("While/LoopCond"): body function result type tensor<1x2xi32> is incompatible with result type tensor<1x1xi32> at index 0

[jinevening]

Test case for one-quantize, which is a tool to quantize floating point circle model to integer model.

Test case 1. Positive case

Let's assume we have inception_v3.circle (generated from this guide) whose weights are float32.

one-quantize performs post-training static quantization, which requires a representative dataset (hdf5 file) for calibration (See this link for detailed explanation. You can also get a general idea about post-training static quantization here).

The representative dataset for inception_v3.circle can be created by downloading Imagenet dataset, preprocessing the images, and packaging some of those images into an hdf5 file.

We know this is a laborious job, so we provide a test dataset which has 10 pre-processed image data (inception_v3_test_data.zip). You will get inception_v3_test_data.h5 after unzipping the file. (You can print the contents of inception_v3_test_data.h5 using a tool like h5dump) (md5sum: 6faf460b6d3b5ffddbeacdd2e21014b9)

With the representative dataset, you can quantize the model.

./one-quantize \
--input_dtype float32 \
--quantized_dtype uint8 \
--input_path ./inception_v3.circle \
--input_data ./inception_v3_test_data.h5 \
--output_path ./inception_v3.quantized.circle

This will generate inception_v3.quantized.circle, whose tensors are quantized to uint8 values.

Test case 2. Negative case (Unsupported input dtype)

Currently we only support quantization from float32 -> uint8 model. If you run one-quantize with other data types like this,

./one-quantize \
--input_dtype float64 \
--quantized_dtype uint8 \
--input_path ./inception_v3.circle \
--input_data ./inception_v3_test_data.h5 \
--output_path ./inception_v3.quantized.circle

You will see this message.

ERROR: Unsupported input type. List of supported input type: float32

If you give a data type other than uint8 to --quantized_dtype, you will see the below message.

ERROR: Unsupported output type. List of supported output type: uint8

Test case 3. Negative case (Wrong representative dataset)

The input shape of inception_v3.circle is [1, 299, 299, 3]. If you give a representative dataset with a different shape, you will see the below message.

ERROR: Input shape mismatch.

You can test this case by running below command with mobilenet_test_data.zip, which is a dataset of image size [1, 224, 224, 3]. (md5sum: 771bdccf0806e812fc7592b18892dce0)

./one-quantize \
--input_dtype float32 \
--quantized_dtype uint8 \
--input_path ./inception_v3.circle \
--input_data ./mobilenet_test_data.h5 \
--output_path ./inception_v3.quantized.circle

[llFreetimell]

one-import-bcq

Followings are test cases for one-import-bcq, which is a tool to generated BCQ applied circle model. bcq.pb file is in bcq.pb.zip.

Test case 1. Positive case

Following scripts is typical positive case, and therefore it will generate bcq.circle file which includes BCQ information nodes.

• Scripts

  ./one-import-bcq \
  --input_path ./bcq.pb \
  --output_path ./bcq.circle \
  --input_arrays Placeholder \
  --output_arrays MatMul

• Expected Results
  • No errors.

Test case 2. Negative case (Invalid Input)

When input tensor is wrong, error will be occurred.

• Scripts

  ./one-import-bcq \
  --input_path ./bcq.pb \
  --output_path ./bcq.circle \
  --input_arrays Placeholder_null \
  --output_arrays MatMul

• Expected Results
  • In ./bcq.circle.log

    (omit)
    ...
    ValueError: Invalid tensors 'Placeholder_null' were found.

    Test case 3. Negative case (Invalid Output)

When output tensor is wrong, error will be occurred.

• Scripts

  ./one-import-bcq \
  --input_path ./bcq.pb \
  --output_path ./bcq.circle \
  --input_arrays Placeholder \
  --output_arrays MatMul_null

• Expected Results
  • In ./bcq.circle.log

    (omit)
    ...
    ValueError: Invalid tensors 'MatMul_null' were found.

Test case 4. Negative case (Invalid Input Path)

When output tensor is wrong, error will be occurred.

• Scripts

  ./one-import-bcq \
  --input_path ./bcq_null.pb \
  --output_path ./bcq.circle \
  --input_arrays Placeholder \
  --output_arrays MatMul

• Expected Results

  Error: input model not found

[lemmaa]

• tc-from-sqe.xlsx

Script	TC Scenario	TC Origin	TC Steps	Output (1.9.1)
one-pack
	1. negative usage with invalid input path	SRBD	$ one-pack \ -i ./inception_v2.circle \ -o nnpack	Error: input model not found
	2. negative usage with  filename without extension	SRBD	$ touch sample $ one-pack \ -i sample \ -o nnpack	error: modelfile does not have extension. Please provide extension so that model2nnpkg.sh can identify what type of model you use.
	3. negative usage without output folder path	SRBD	$ one-pack \ -i ./inception_v3.circle	Error: output path not set
one-import-tf
	1. Positive usage	HQ	$ one-import tf \ --input_path ./inception_v3.pb \ --output_path ./inception_v3.circle \ --input_arrays input --input_shapes "1,299,299,3" \ --output_arrays InceptionV3/Predictions/Reshape_1	On success, there will be no (error) messages and two files - inception_v3.circle -  inception_v3.circle.log         - inception_v3.circle.log will show internal conversion log of internal commands used.
			$ one-optimize --all \ --input_path ./inception_v3.circle  \ --output_path ./inception_v3-opt.circle	On suceess, there will be no (error) messages and two new files - inception_v3-opt.circle - inception_v3-opt.circle.log      - inception_v3-opt.circle.log will show optimization log.
			$ one-pack -i ./inception_v3-opt.circle -o nnpack	On suceess, there will be no (error) messages and one new folder nnpack
	2. Typical negative usage	HQ	$ one-import tf \ --input_path ./inception_v3.tflite \ --output_path ./inception_v3.circle \ --input_arrays input --input_shapes "1,299,299,3" \ --output_arrays InceptionV3/Predictions/Reshape_1	ValueError: Invalid tensors 'input' were found.
			$ one-optimize --all \ --input_path ./inception_v3.pb \ --output_path ./inception_v3-opt.circle	ERROR: Invalid input file './inception_v3.pb'
	3. Typical negative usage with unsupported dynamic tensor	HQ	$ one-import tf \ --input_path ./while_3.pbtxt \ --output_path ./while_3.circle \ --input_arrays Hole,Hole_2 --input_shapes "1,1:1,1" \ --output_arrays Output	tensorflow.lite.python.convert.ConverterError: :0: error: loc("While/LoopCond"): body function result type tensor<1x2xi32> is incompatible with result type tensor<1x1xi32> at index 0
	4. negative usage with invalid output array	SRBD	$ one-import tf \ --input_path ./inception_v3.pb \ --output_path ./inception_v3.circle \ --input_arrays input --input_shapes "1,299,299,3" \ --output_arrays InceptionV3/Predictions/Reshape_2	ValueError: Invalid tensors 'InceptionV3/Predictions/Reshape_2' were found.
	5. negative usage with  invalid input shape	SRBD	$ one-import tf \ --input_path ./inception_v3.pb \ --output_path ./inception_v3.circle \ --input_arrays input --input_shapes "1,299,299,1" \ --output_arrays InceptionV3/Predictions/Reshape_1	ValueError: The shape of tensor 'input' cannot be changed from (None, 299, 299, 3) to [1, 299, 299, 1]. Dimension 3 in both shapes must be equal, but are 3 and 1. Shapes are [?,299,299,3] and [1,299,299,1].
		SRBD	$ one-import tf \ --input_path ./inception_v3.pb \ --output_path ./inception_v3.circle \ --input_arrays input --input_shapes "1,299,299" \ --output_arrays InceptionV3/Predictions/Reshape_1	ValueError: The shape of tensor 'input' cannot be changed from (None, 299, 299, 3) to [1, 299, 299]. Shapes must be equal rank, but are 4 and 3
		SRBD	$ one-import tf \ --input_path ./inception_v3.pb \ --output_path ./inception_v3.circle \ --input_arrays input --input_shapes "0,299,299,3" \ --output_arrays InceptionV3/Predictions/Reshape_1	tensorflow.lite.python.convert.ConverterError: :0: error: loc("InceptionV3/InceptionV3/Mixed_7c/concat"): 'tfl.concatenation' op failed to verify that values and output must have same element type :0: note: loc("InceptionV3/InceptionV3/Mixed_7c/concat"): see current operation: %0 = "tfl.concatenation"() {axis = 3 : i32, fused_activation_function = "NONE"} : () -> tensor<0x8x8x2048xf32>
		SRBD	$ one-import tf \ --input_path ./inception_v3.pb \ --output_path ./inception_v3.circle \ --input_arrays input --input_shapes "None,299,299,3" \ --output_arrays InceptionV3/Predictions/Reshape_1	ValueError: invalid literal for int() with base 10: 'None'
		SRBD	$ one-import tf \ --input_path ./inception_v3.pb \ --output_path ./inception_v3.circle \ --input_arrays input,InceptionV3/Predictions/Shape --input_shapes "1,299,299,3" \ --output_arrays InceptionV3/Predictions/Reshape_1	ValueError: --input_shapes and --input_arrays must have the same number of items
	6. negative usage with invalid output path	SRBD	$ one-import tf \ --input_path ./inception_v3.pb \ --output_path . \ --input_arrays input --input_shapes "1,299,299,3" \ --output_arrays InceptionV3/Predictions/Reshape_1	ERROR: Failed to write circle '.'
	7. negative usage with invalid input_arrays	SRBD	$ one-import tf  \ --input_path ./inception_v3.pb \ --output_path ./inception_v3.circle \ --input_arrays input2 --input_shapes "1,299,299,3" \ --output_arrays InceptionV3/Predictions/Reshape_1	ValueError: Invalid tensors 'input2' were found.
one-quantize
	1. Positive case	HQ	$ one-quantize \ --input_dtype float32 \ --quantized_dtype uint8 \ --input_path ./inception_v3.circle \ --input_data ./inception_v3_test_data.h5 \ --output_path ./inception_v3.quantized.circle	This will generate inception_v3.quantized.circle, whose tensors are quantized to uint8 values.
	2. Negative case (Unsupported input dtype)	HQ	$ one-quantize \ --input_dtype float64 \ --quantized_dtype uint8 \ --input_path ./inception_v3.circle \ --input_data ./inception_v3_test_data.h5 \ --output_path ./inception_v3.quantized.circle	ERROR: Unsupported input type. List of supported input type: float32
			$ one-quantize \ --input_dtype float32 \ --quantized_dtype uint16 \ --input_path ./inception_v3.circle \ --input_data ./inception_v3_test_data.h5 \ --output_path ./inception_v3.quantized.circle	ERROR: Unsupported output type. List of supported output type: uint8
	3. Negative case (Wrong representative dataset)	HQ	$ one-quantize \ --input_dtype float32 \ --quantized_dtype uint8 \ --input_path ./inception_v3.circle \ --input_data ./mobilenet_test_data.h5 \ --output_path ./inception_v3.quantized.circle	ERROR: Input shape mismatch.
	4. negative usage with invalid output path	SRBD	$ one-quantize \ --input_dtype float32 \ --quantized_dtype uint8 \ --input_path ./inception_v3.circle \ --input_data ./inception_v3_test_data.h5 \ --output_path .	ERROR: Failed to export '.'
	5. negative usage with invalid input path	SRBD	$ one-quantize \ --input_dtype float32 \ --quantized_dtype uint8 \ --input_path ./while_3.pbtxt \ --input_data ./inception_v3_test_data.h5 \ --output_path ./inception_v3.quantized.circle	ERROR: Invalid input file './while_3.pbtxt'
			$ one-quantize \ --input_dtype float32 \ --quantized_dtype uint8 \ --input_path ./inception_v2.circle \ --input_data ./inception_v3_test_data.h5 \ --output_path ./inception_v3.quantized.circle	Error: input model not found
	6. negative usage with invalid input_data	SRBD	$ one-quantize \ --input_dtype float32 \ --quantized_dtype uint8 \ --input_path ./inception_v3.circle \ --input_data ./inception_v3.circle \ --output_path ./inception_v3.quantized.circle	ERROR: Error: Given data file is not HDF5
	7. negative usage with invalid mode	SRBD	$ one-quantize \ --input_dtype float32 \ --quantized_dtype uint8 \ --input_path ./inception_v3.circle \ --input_data ./inception_v3_test_data.h5 \ --mode average \ --output_path ./inception_v3.quantized.circle	ERROR: Unsupported mode
	8. negative usage with invalid max_percentile	SRBD	$ one-quantize \ --input_dtype float32 \ --quantized_dtype uint8 \ --input_path ./inception_v3.circle \ --input_data ./inception_v3_test_data.h5 \ --max_percentile 101 \ --output_path ./inception_v3.quantized.circle	ERROR: Percentile must be ranged from 0 to 100
			$ one-quantize \ --input_dtype float32 \ --quantized_dtype uint8 \ --input_path ./inception_v3.circle \ --input_data ./inception_v3_test_data.h5 \ --max_percentile -1 \ --output_path ./inception_v3.quantized.circle	ERROR: Percentile must be ranged from 0 to 100
	9. negative usage with invalid min_percentile	SRBD	$ one-quantize \ --input_dtype float32 \ --quantized_dtype uint8 \ --input_path ./inception_v3.circle \ --input_data ./inception_v3_test_data.h5 \ --min_percentile 101 \ --output_path ./inception_v3.quantized.circle	ERROR: Percentile must be ranged from 0 to 100
			$ one-quantize \ --input_dtype float32 \ --quantized_dtype uint8 \ --input_path ./inception_v3.circle \ --input_data ./inception_v3_test_data.h5 \ --min_percentile -1 \ --output_path ./inception_v3.quantized.circle	ERROR: Percentile must be ranged from 0 to 100
	10. negative usage with invalid granularity	SRBD	$ one-quantize \ --input_dtype float32 \ --quantized_dtype uint8 \ --input_path ./inception_v3.circle \ --input_data ./inception_v3_test_data.h5 \ --granularity layered \ --output_path ./inception_v3.quantized.circle	ERROR: Unsupported granularity. List of supported granularity: layer, channel
one-import-bcq
	1. Positive case	HQ	$ one-import-bcq \ --input_path ./bcq.pb \ --output_path ./bcq.circle \ --input_arrays Placeholder \ --output_arrays MatMul	bcq.cirle file is generated
	2. Negative case (Invalid Input)	HQ	$ one-import-bcq \ --input_path ./bcq.pb \ --output_path ./bcq.circle \ --input_arrays Placeholder_null \ --output_arrays MatMul	ValueError: Invalid tensors 'Placeholder_null' were found.
	3. Negative case (Invalid Output)	HQ	$ one-import-bcq \ --input_path ./bcq.pb \ --output_path ./bcq.circle \ --input_arrays Placeholder \ --output_arrays MatMul_null	ValueError: Invalid tensors 'MatMul_null' were found.
	4. Negative case (Invalid Input Path)	HQ	$ one-import-bcq \ --input_path ./bcq_null.pb \ --output_path ./bcq.circle \ --input_arrays Placeholder \ --output_arrays MatMul	Error: input model not found
	5. negative usage with invalid input	SRBD	$ one-import-bcq \ --input_path ./while_3.pbtxt \ --output_path ./bcq.circle \ --input_arrays Placeholder \ --output_arrays MatMul	google.protobuf.message.DecodeError: Error parsing message
	6. Invalid usage with invalid output_path	SRBD	$ one-import-bcq \ --input_path ./bcq.pb \ --output_path . \ --input_arrays Placeholder \ --output_arrays MatMul	ERROR: Failed to write circle '.'
	7. negative usage with  invalid input shape	SRBD	$ one-import bcq \ --input_path ./bcq.pb \ --output_path ./bcq.circle \ --input_arrays Placeholder --input_shapes "1,32,32" \ --output_arrays MatMul	ValueError: The shape of tensor 'Placeholder' cannot be changed from (32, 32) to [1, 32, 32]. Shapes must be equal rank, but are 2 and 3.
			$ one-import bcq \ --input_path ./bcq.pb \ --output_path ./bcq.circle \ --input_arrays Placeholder --input_shapes "30,30" \ --output_arrays MatMul	ValueError: The shape of tensor 'Placeholder' cannot be changed from (32, 32) to [30, 30]. Dimension 0 in both shapes must be equal, but are 32 and 30. Shapes are [32,32] and [30,30].
			$ one-import bcq \ --input_path ./bcq.pb \ --output_path ./bcq.circle \ --input_arrays Placeholder --input_shapes "32,O" \ --output_arrays MatMul	ValueError: invalid literal for int() with base 10: 'O'
			$ one-import bcq \ --input_path ./bcq.pb \ --output_path ./bcq.circle \ --input_arrays Placeholder --input_shapes "32,32:1" \ --output_arrays MatMul	ValueError: --input_shapes and --input_arrays must have the same number of items
one-optimize
	1. Invalid usage with invalid input path	SRBD	$ one-optimize --all \ --input_path ./inception_v3.circletxt  \ --output_path ./inception_v3-opt.circle	Error: input model not found
	2. negative usage without output folder path	SRBD	$ one-optimize --all \ --input_path  ./inception_v3.circle	ERROR: Failed to export ''