[Good First Issue]: Remove the remaining uses of `test-generator` package

p-wysocki commented 7 months ago

Context

Model Optimizer is currently using an extension to unittest called test-generator. When you take a look at its PyPI page it turns out that the package hasn't been updated in a while.

It needs to be removed and all of its usages replaced with unittest functionality. This ticket is a follow up to https://github.com/openvinotoolkit/openvino/issues/17619 - most of the work has been done there, all there is left is some edge cases.

Do not hesitate to ask for help!

What needs to be done?

Find test-generator usage in openvino/tools/mo by searching the codebase for @generator decorators in tests
Replace the generator's functionality with unittest's subTest
Confirm the tests work correctly
Remove test-generator dependency from openvino/tools/constraints.txt and openvino/tools/mo/requirements.txt
Create a Pull Request

Example Pull Requests

Resources

Contribution guide - start here!
Running MO tests
unittest's subTest functionality documentation
Python requirements management in OpenVINO
Blog post on contributing to OpenVINO

Contact points

@rkazants

Ticket

110705

kus123123 commented 7 months ago

Context

Model Optimizer is currently using an extension to unittest called test-generator. When you take a look at its PyPI page it turns out that the package hasn't been updated in a while.

It needs to be removed and all of its usages replaced with unittest functionality. This ticket is a follow up to #17619 - most of the work has been done there, all there is left is some edge cases.

Do not hesitate to ask for help!

What needs to be done?

Find test-generator usage in openvino/tools/mo by searching the codebase for @generator decorators in tests

Replace the generator's functionality with unittest's subTest

Confirm the tests work correctly

Remove test-generator dependency from openvino/tools/constraints.txt and openvino/tools/mo/requirements.txt

Create a Pull Request

Example Pull Requests

Remove test-generator dependency #18512

pytest special test case 3 #20313

generator to pytest #19487

Resources

Contribution guide - start here!

Running MO tests

unittest's subTest functionality documentation

Python requirements management in OpenVINO

Blog post on contributing to OpenVINO

Contact points

@rkazants

Ticket

110705

hello p-wysocki , i am trying to remove testgenerator dependency from constrains.txt and replace it with unitest== 1.4.34 . (this is latest update ) . kindly let me know if this version is okay or any specific version is required

p-wysocki commented 7 months ago

Hello @kus123123, unittest is built into Python, so there's no need to version it - whatever Python the user will be using, it will work, no reason to worry about it, it doesn't have to be added to requirements or constraints either.

Thank you for taking a look at the issue, please let us know if you have any questions!

taho9802 commented 7 months ago

Good Afternoon @p-wysocki I have worked on couple of files already that has @generator. I just need a guide on how to test these test cases. so far I have worked on custom_replacement_config_test.py, mo_fallback_test_tf_fe.py, and ir_engine_test.py

p-wysocki commented 6 months ago

Hello, sorry for the late reply, the holiday season just ended. :)

It looks like two people are working on the same task, and @kus123123 was first - are you still working on the issue?

Also, I am happy to announce that we have created a channel dedicated to Good First Issues support on our Intel DevHub Discord server! Join it to receive support, engage in discussions, ask questions and talk to OpenVINO developers.

p-wysocki commented 6 months ago

The task is now open for pickup - @taho7766 feel free to do so if you like it. :)

taho9802 commented 6 months ago

.take

github-actions[bot] commented 6 months ago

Thank you for looking into this issue! Please let us know if you have any questions or require any help.

p-wysocki commented 6 months ago

Hi @taho7766, can we help you with anything?

taho9802 commented 5 months ago

@p-wysocki, Thank you for the follow up, and yes actually. I have written the code and I was wondering how to test if my tests work properly.

on the side note, Is discord preferred for communications?

p-wysocki commented 5 months ago

In order to test the changes you could just run the tests which you modified - if they're passing and test-generator package is removed, the work is done. :)

on the side note, Is discord preferred for communications?

That's entirely up to you - whichever is more convenient for you. We're present in both channels.

p-wysocki commented 5 months ago

Hi @taho7766, are you still working on that issue?

taho9802 commented 5 months ago

Hello @p-wysocki ,

Yes, yes I am. when I try to run the tests in MacOS, the ZSH gets killed without any explanation so I tried with bootcamp windows 10 and this is the error log I am running into.

This test attempt is before I apply my code changes that removes @generator package.

coverage run -m unittest discover -p "mo_fallback_test_tf_fe.py"

ERROR: test_transform_config_fallback_tf_fe_pb[(True, False, 'mo_legacy', None)] (mo_fallback_test_tf_fe.TestMoFallback)

Traceback (most recent call last): File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/utils/class_registration.py", line 286, in apply_transform replacer.find_and_replace_pattern(graph) File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/load/loader.py", line 14, in find_and_replace_pattern self.load(graph) File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/load/tf/loader.py", line 43, in load if argv.tensorflow_custom_layer_libraries: AttributeError: 'Namespace' object has no attribute 'tensorflow_custom_layer_libraries'

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/generator/init.py", line 19, in wrapper_func return func(test_case, arg) File "/home/taho9802/devStorage/openvino/tools/mo/unit_tests/mo/utils/mo_fallback_test_tf_fe.py", line 118, in test_transform_config_fallback_tf_fe_pb prepare_ir(args) File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/convert_impl.py", line 432, in prepare_ir graph = unified_pipeline(argv) File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/pipeline/unified.py", line 13, in unified_pipeline class_registration.apply_replacements(graph, [ File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/utils/class_registration.py", line 338, in apply_replacements apply_replacements_list(graph, replacers_order) File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/utils/class_registration.py", line 324, in apply_replacements_list apply_transform( File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/utils/logger.py", line 124, in wrapper function(args, **kwargs) File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/utils/class_registration.py", line 312, in apply_transform raise Exception('Exception occurred during running replacer "{} ({})": {}'.format( Exception: Exception occurred during running replacer "REPLACEMENT_ID (<class 'openvino.tools.mo.load.tf.loader.TFLoader'>)": 'Namespace' object has no attribute 'tensorflow_custom_layer_libraries'

====================================================================== ERROR: test_transform_config_fallback_tf_fe_saved_model[(False, False, 'mo_legacy', None)] (mo_fallback_test_tf_fe.TestMoFallback)

Traceback (most recent call last): File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/utils/class_registration.py", line 286, in apply_transform replacer.find_and_replace_pattern(graph) File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/load/loader.py", line 14, in find_and_replace_pattern self.load(graph) File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/load/tf/loader.py", line 43, in load if argv.tensorflow_custom_layer_libraries: AttributeError: 'Namespace' object has no attribute 'tensorflow_custom_layer_libraries'

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/generator/init.py", line 19, in wrapper_func return func(test_case, arg) File "/home/taho9802/devStorage/openvino/tools/mo/unit_tests/mo/utils/mo_fallback_test_tf_fe.py", line 147, in test_transform_config_fallback_tf_fe_saved_model prepare_ir(args) File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/convert_impl.py", line 432, in prepare_ir graph = unified_pipeline(argv) File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/pipeline/unified.py", line 13, in unified_pipeline class_registration.apply_replacements(graph, [ File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/utils/class_registration.py", line 338, in apply_replacements apply_replacements_list(graph, replacers_order) File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/utils/class_registration.py", line 324, in apply_replacements_list apply_transform( File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/utils/logger.py", line 124, in wrapper function(args, **kwargs) File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/utils/class_registration.py", line 312, in apply_transform raise Exception('Exception occurred during running replacer "{} ({})": {}'.format( Exception: Exception occurred during running replacer "REPLACEMENT_ID (<class 'openvino.tools.mo.load.tf.loader.TFLoader'>)": 'Namespace' object has no attribute 'tensorflow_custom_layer_libraries'

====================================================================== ERROR: test_transform_config_fallback_tf_fe_saved_model[(True, False, 'mo_legacy', None)] (mo_fallback_test_tf_fe.TestMoFallback)

Traceback (most recent call last): File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/utils/class_registration.py", line 286, in apply_transform replacer.find_and_replace_pattern(graph) File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/load/loader.py", line 14, in find_and_replace_pattern self.load(graph) File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/load/tf/loader.py", line 43, in load if argv.tensorflow_custom_layer_libraries: AttributeError: 'Namespace' object has no attribute 'tensorflow_custom_layer_libraries'

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/generator/init.py", line 19, in wrapper_func return func(test_case, arg) File "/home/taho9802/devStorage/openvino/tools/mo/unit_tests/mo/utils/mo_fallback_test_tf_fe.py", line 147, in test_transform_config_fallback_tf_fe_saved_model prepare_ir(args) File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/convert_impl.py", line 432, in prepare_ir graph = unified_pipeline(argv) File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/pipeline/unified.py", line 13, in unified_pipeline class_registration.apply_replacements(graph, [ File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/utils/class_registration.py", line 338, in apply_replacements apply_replacements_list(graph, replacers_order) File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/utils/class_registration.py", line 324, in apply_replacements_list apply_transform( File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/utils/logger.py", line 124, in wrapper function(args, **kwargs) File "/home/taho9802/devStorage/openvino/.env3/lib/python3.10/site-packages/openvino/tools/mo/utils/class_registration.py", line 312, in apply_transform raise Exception('Exception occurred during running replacer "{} ({})": {}'.format( Exception: Exception occurred during running replacer "REPLACEMENT_ID (<class 'openvino.tools.mo.load.tf.loader.TFLoader'>)": 'Namespace' object has no attribute 'tensorflow_custom_layer_libraries'

Ran 4 tests in 11.506s

p-wysocki commented 5 months ago

@rkazants could you please take a look?

rkazants commented 5 months ago

Hi @taho7766, please run unit-test without code coverage:

python -m unittest discover -p mo_fallback_test_tf_fe.py

Also, check documentation how to run unit-test: https://github.com/openvinotoolkit/openvino/blob/master/tools/mo/README.md. Check that prerequisites are installed:

pip install - r requirements_dev.txt

Best regards, Roman

p-wysocki commented 4 months ago

Hello @taho7766, are you still working on this?

taho9802 commented 4 months ago

@p-wysocki good afternoon, After re-cloning entire repository and running command below, I am still running into same issue where namespace object has no attribute called tensorflow_custom_layer_libraries here is my command history and pip list in my current python3.8 v-environment

(.env3.8) taewoohong@Taewoos-MacBook-Pro mo % history 1144 git submodule update --init --recursive 1145 python3.8 -m venv .env3.8 1146 source .env3.8/bin/activate 1147 pip install openvino-dev[all] 1148 cd tools/mo 1149 pip install -e . 1150 pip install -r "requirements.txt" "requirements_dev.txt" 1151 pip install -r "requirements.txt" -r "requirements_dev.txt" 1152 pip install -e ".[all]" 1153 python setup.py develop 1154 pip install -r "requirements_onnx.txt" -r "requirements_tf2.txt" 1155 pip install -r "requirements_mnxet.txt" 1156 pip install -r "requirements_mxnet.txt" 1157 python -m unittest discover -p openvino/tools/mo/unit_tests/mo/utils/mo_fallback_test_tf_fe.py 1158 pytest openvino/tools/mo/unit_tests/mo/utils/mo_fallback_test_tf_fe.py 1159 pytest /Users/taewoohong/dev/openSource/openvino/tools/mo/unit_tests/mo/utils/mo_fallback_test_tf_fe.py (.env3.8) taewoohong@Taewoos-MacBook-Pro mo % pip list Package Version Editable project location

absl-py 2.1.0 addict 2.4.0 astroid 3.0.3 astunparse 1.6.3 attrs 23.2.0 cachetools 5.3.2 certifi 2024.2.2 charset-normalizer 3.3.2 coverage 7.0.5 defusedxml 0.7.1 dill 0.3.8 exceptiongroup 1.2.0 fastjsonschema 2.17.1 flatbuffers 23.5.26 gast 0.4.0 google-auth 2.28.0 google-auth-oauthlib 1.0.0 google-pasta 0.2.0 graphviz 0.8.4 grpcio 1.60.1 h5py 3.10.0 idna 3.6 importlib-metadata 7.0.1 iniconfig 2.0.0 isort 5.13.2 jstyleson 0.0.2 keras 2.13.1 libclang 16.0.6 Markdown 3.5.2 MarkupSafe 2.1.5 mccabe 0.7.0 mxnet 1.9.1 networkx 3.1 numpy 1.23.5 oauthlib 3.2.2 onnx 1.15.0 opencv-python 4.9.0.80 openvino 2023.3.0 openvino-dev 2023.3.0 openvino-mo 0.0.0 /Users/taewoohong/dev/openSource/openvino/tools/mo openvino-telemetry 2023.2.1 opt-einsum 3.3.0 packaging 23.2 pillow 10.2.0 pip 23.0.1 platformdirs 4.2.0 pluggy 1.4.0 protobuf 3.20.3 py 1.11.0 pyasn1 0.5.1 pyasn1-modules 0.3.0 pyenchant 3.2.2 pylint 3.0.3 pytest 7.2.2 PyYAML 6.0.1 requests 2.31.0 requests-oauthlib 1.3.1 rsa 4.9 scipy 1.10.1 setuptools 56.0.0 six 1.16.0 tensorboard 2.13.0 tensorboard-data-server 0.7.2 tensorflow 2.13.1 tensorflow-estimator 2.13.0 tensorflow-io-gcs-filesystem 0.34.0 termcolor 2.4.0 test-generator 0.1.1 texttable 1.7.0 tomli 2.0.1 tomlkit 0.12.3 tqdm 4.66.2 typing_extensions 4.5.0 urllib3 2.2.1 Werkzeug 3.0.1 wheel 0.42.0 wrapt 1.16.0 zipp 3.17.0 WARNING: There was an error checking the latest version of pip. (.env3.8) taewoohong@Taewoos-MacBook-Pro mo %

Thank you for working with me, I apologize for the lack of update and I will be more proactive moving forward.

p-wysocki commented 4 months ago

Hi @taho7766, unfortunately I haven't worked much with that part of the codebase. @rkazants could you please take a look?

rkazants commented 4 months ago

Hi @taho7766, it can be the test issue. Could you please add into base_args_config in mo_fallback_test_tf_fe.py:

args.tensorflow_custom_layer_libraries= None

Did you manage to avoid generator in other tests?

Best regards, Roman

taho9802 commented 4 months ago

Good Evening, @rkazants After setting args.tensorflow_custom_layer_libraries= None Another exception occurs: Exception occurred during running replacer "REPLACEMENT_ID (<class 'openvino.tools.mo.load.tf.loader.TFLoader'>)": 'Namespace' object has no attribute 'tensorboard_logdir'

On your question, I was able to successfully update these files with no issues: ir_engine_test.py

conversion_incorrect_models_test.py

loader_test.py

On these files, I managed to avoid generator, however, these tests may be outdated: custom_replacement_config_test.py - function get_json_configs(mo_root_dir) pointing to directories that no longer exists

conversion_with_layout_test.py - Encountered an error during unit testing with existing test cases with using the Model Optimizer to convert a TensorFlow model into OpenVINO format, specifically within test_model_with_convolution_dynamic_rank. The error relates to the handling of the batch dimension during conversion.

At last, conversion_basic_models_test.py file is running into this issue: Error: Attribute error indicating a 'NoneType' object has no attribute 'values' during model conversion. Error: OpConversionFailure for a Multiply operation, indicating a type mismatch between arguments (int32 vs float32).

p-wysocki commented 4 months ago

cc @rkazants

rkazants commented 1 month ago

PR: https://github.com/openvinotoolkit/openvino/pull/24576

openvinotoolkit / openvino