search

krai / axs2mlperf

Automated KRAI X workflows for reproducing MLPerf Inference submissions
https://krai.ai
MIT License
1 stars 1 forks source link

Update all entries to support of `entry_creator` #40

Open maria-18-git opened 8 months ago

maria-18-git commented 8 months ago

All axs entries should use entry_creator for object creation. Now different types of save are used for it. For example, set_path(entry_name setting) is used in case of record_entry.

First entries without dependencies on others should be updated and then it should be tested. Also we need to check whether we can move pipeline parameter , e.g.

    "pipeline": [ "^^", "execute", [[
        [ "run" ],
        [],
        [ "get", "stored_newborn_entry" ]
    ]] ],

to entry_creator or not.

maria-18-git commented 6 months ago

Updated loadgen_wheel_recipe

on chai:

 axs byquery package_wheel,package_name=mlperf_loadgen
['^', 'byname', 'loadgen_package_wheel_for_python3.9']

Structure of created dicrectory:

maria@chai ~/work_collection/axs2mlperf/loadgen_wheel_recipe (master *=)$ tree -a `axs byquery package_wheel,package_name=mlperf_loadgen , get_path`
/home/maria/work_collection/loadgen_package_wheel_for_python3.9
├── data_axs.json
├── mlperf_loadgen-4.0-cp39-cp39-linux_x86_64.whl
└── mlperf_loadgen.egg-info
    ├── dependency_links.txt
    ├── PKG-INFO
    ├── SOURCES.txt
    └── top_level.txt

1 directory, 6 files
axs byquery python_package,package_name=mlperf_loadgen,desired_python_version===3.9
...
        "/home/maria/bin/python3" -m pip install /home/maria/work_collection/loadgen_package_wheel_for_python3.9/mlperf_loadgen-4.0-cp39-cp39-linux_x86_64.whl --prefix=/home/maria/work_collection/mlperf_loadgen_package_for_python3.9/install --ignore-installed
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /home/maria/work_collection/loadgen_package_wheel_for_python3.9/mlperf_loadgen-4.0-cp39-cp39-linux_x86_64.whl                                                                         DEPRECATION: devscripts 2.22.1ubuntu1 has a non-standard version number. pip 24.0 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of devscripts or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
Installing collected packages: mlperf-loadgen
Successfully installed mlperf-loadgen-4.0

[notice] A new release of pip is available: 23.3.2 -> 24.0
[notice] To update, run: python3 -m pip install --upgrade pip
INFO:root:Matched Rule #1/2 produced an entry, which matches the original query.

['^', 'byname', 'mlperf_loadgen_package_for_python3.9']

Structure of directory:

maria@chai ~/work_collection/axs2mlperf/loadgen_wheel_recipe (master *=)$ tree -a /home/maria/work_collection/mlperf_loadgen_package_for_python3.9/                                              /home/maria/work_collection/mlperf_loadgen_package_for_python3.9/
├── data_axs.json
└── install
    └── lib
        └── python3.9
            └── site-packages
                ├── mlperf_loadgen-4.0.dist-info
                │   ├── direct_url.json
                │   ├── INSTALLER
                │   ├── METADATA
                │   ├── RECORD
                │   ├── REQUESTED
                │   ├── top_level.txt
                │   └── WHEEL
                └── mlperf_loadgen.cpython-39-x86_64-linux-gnu.so

5 directories, 9 files
maria-18-git commented 6 months ago

Check correct work with mlperf_loadgen_package_for_python3.9:

maria-18-git commented 6 months ago 
maria@chai ~/work_collection/axs2mlperf/loadgen_wheel_recipe (master *=)$  echo "Accuracy: $ACCURACY_OUTPUT"
Accuracy: 22.852

maria@chai ~/work_collection/axs2mlperf/loadgen_wheel_recipe (master *=)$ echo `axs func round $ACCURACY_OUTPUT 0`
23.0
maria-18-git commented 6 months ago

maria@chai ~/work_collection/axs2mlperf (master *=)$ echo "Accuracy: $ACCURACY_OUTPUT" Accuracy: 52.98

maria-18-git commented 6 months ago 
maria@chai ~/work_collection/axs2mlperf (master *=)$ export ACCURACY_OUTPUT=`axs byquery loadgen_output,task=image_classification,framework=onnxrt,loadgen_scenario=Offline,loadgen_mode=AccuracyOnly,model_name=resnet50,loadgen_dataset_size=20,loadgen_buffer_size=100,execution_device=cpu , get accuracy `  

maria@chai ~/work_collection/axs2mlperf (master *=)$  echo "Accuracy: $ACCURACY_OUTPUT"
Accuracy: 85.0
maria-18-git commented 6 months ago

maria@chai ~/work_collection/axs2mlperf (master *=)$ echo "Accuracy: $ACCURACY_OUTPUT" Accuracy: 75.0

maria-18-git commented 6 months ago

Install mlperf_loadgen_package_for_python3.9 automatically

axs byquery python_package,package_name=mlperf_loadgen,desired_python_version===3.9 --- , remove

INFO:root:Matched Rule #1/1: ['package_wheel', 'package_name=mlperf_loadgen'] from Entry 'loadgen_wheel_recipe'... INFO:root:Pipeline: [['get', 'pipeline']], Cumulative params: {'query': 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9', 'package_name': 'mlperf_loadgen', 'desired_python_version': '3.9', 'tags': ['package_wheel'], 'cumulative_param_names': ['__query', 'package_name', 'desired_python_version', 'tags']} INFO:root:[work_collection] byquery(package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9) did not find anything, but there are tags: {'package_wheel'} , trying to find a producer... INFO:root:[work_collection] A total of 1 matched rules found.

INFO:root:Matched Rule #1/1: ['package_wheel', 'package_name=mlperf_loadgen'] from Entry 'loadgen_wheel_recipe'... INFO:root:Pipeline: [['get', 'pipeline']], Cumulative params: {'query': 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9', 'package_name': 'mlperf_loadgen', 'desired_python_version': '3.9', 'tags': ['package_wheel'], 'cumulative_param_names': ['__query', 'package_name', 'desired_python_version', 'tags']} WARNING:root:[loadgen_wheel_recipe -> loadgen_wheel_recipe] parameter 'pipeline' is contained here, but BLOCKED by this entry -- all blockers: {'loadgen_wheel_recipe'} INFO:root:Matched Rule #1/1 didn't produce a result, 0 more matched rules to try... ------------------------------------------------------------------------------------------------------------------------ While computing nested_calls in ['^^', 'execute', [[['get_kernel'], ['byquery', ['^^', 'substitute', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===#{desired_python_version}#']], ['find_file', 'mlperf_loadgen.\.whl', {'return_full': True, 'abs_paths': True}]]]] the following exception was raised: In pipeline [ ['get_kernel'] ['byquery', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9'] ['find_file', 'mlperf_loadgen.\.whl', {'return_full': True, 'abs_paths': True}] ] step ['find_file', 'mlperf_loadgen.\.whl', {'return_full': True, 'abs_paths': True}] cannot be executed on value (None) produced by ['byquery', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9'] ======================================================================================================================== ------------------------------------------------------------------------------------------------------------------------ While computing nested_calls in [[['get', 'collection_object'], ['attached_entry', [['^^', 'get', 'newborn_entry_name'], {'tags': ['^^', 'get', ['tags', ['^^', 'get', 'newborn_entry_tags']]]}]], ['plant', ['_parent_entries', ['^^', 'parent_list_from_names']]], ['plant', ['^^', 'slice', ['^^', 'get', ['__cumulative_param_names', []]], {'plantable': True, 'skip_missing': True}]], ['plant', ['^^', 'slice', ['^^', 'get', 'newborn_entry_param_names'], {'plantable': True, 'skip_missing': True}]], ['plant', ['^^', 'substitute', [['^^', 'get', 'newborn_entry_param_listmap']]]]]] the following exception was raised: In pipeline [ ['get_kernel'] ['byquery', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9'] ['find_file', 'mlperf_loadgen.\.whl', {'return_full': True, 'abs_paths': True}] ] step ['find_file', 'mlperf_loadgen.\.whl', {'return_full': True, 'abs_paths': True}] cannot be executed on value (None) produced by ['byquery', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9'] ... ] step ['find_file', 'mlperf_loadgen.\.whl', {'return_full': True, 'abs_paths': True}] cannot be executed on value (None) produced by ['byquery', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9']

While computing nested_calls in {'query': ['python_package', 'package_name=mlperf_loadgen', ['desired_python_version', '3.9']], 'installable': ['^^', 'execute', [[['get_kernel'], ['byquery', ['^^', 'substitute', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===#{desired_python_version}#']], ['find_file', 'mlperf_loadgen.*\.whl', {'return_full': True, 'abs_paths': True}]]]], 'package_name': 'mlperf_loadgen', 'desired_python_version': '3.9', 'tags': ['python_package'], 'cumulative_param_names': ['__query', 'installable', 'package_name', 'desired_python_version', 'tags']} the following exception was raised: In pipeline [ ['get_kernel'] ['byquery', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9'] ['find_file', 'mlperf_loadgen.\.whl', {'return_full': True, 'abs_paths': True}] ] step ['find_file', 'mlperf_loadgen.\.whl', {'return_full': True, 'abs_paths': True}] cannot be executed on value (None) produced by ['byquery', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9']

While computing nested_calls in [['^^', 'python_sync_pip_package', [['^^', 'get', 'tensorflow_query']]], ['^^', 'python_sync_pip_package', [['^^', 'get', 'onnxruntime_query']]], ['^^', 'python_sync_pip_package', [['^^', 'get', 'loadgen_query']]]] the following exception was raised: In pipeline [ ['get_kernel'] ['byquery', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9'] ['find_file', 'mlperf_loadgen.\.whl', {'return_full': True, 'abs_paths': True}] ] step ['find_file', 'mlperf_loadgen.\.whl', {'return_full': True, 'abs_paths': True}] cannot be executed on value (None) produced by ['byquery', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9']

While computing nested_calls in ['^^', 'ext_use_python_deps'] the following exception was raised: In pipeline [ ['get_kernel'] ['byquery', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9'] ['find_file', 'mlperf_loadgen.\.whl', {'return_full': True, 'abs_paths': True}] ] step ['find_file', 'mlperf_loadgen.\.whl', {'return_full': True, 'abs_paths': True}] cannot be executed on value (None) produced by ['byquery', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9']

While computing nested_calls in ['^^', 'execute', [[['run'], [], ['get', 'output_entry']]]] the following exception was raised: In pipeline [ ['get_kernel'] ['byquery', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9'] ['find_file', 'mlperf_loadgen.\.whl', {'return_full': True, 'abs_paths': True}] ] step ['find_file', 'mlperf_loadgen.\.whl', {'return_full': True, 'abs_paths': True}] cannot be executed on value (None) produced by ['byquery', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9']

ERROR:root:RuntimeError: In pipeline [ ['get_kernel'] ['byquery', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9'] ['find_file', 'mlperf_loadgen.\.whl', {'return_full': True, 'abs_paths': True}] ] step ['find_file', 'mlperf_loadgen.\.whl', {'return_full': True, 'abs_paths': True}] cannot be executed on value (None) produced by ['byquery', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9'] None

maria-18-git commented 6 months ago

axs byquery package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9 doesn't work correctly with entry_creator.

maria-18-git commented 6 months ago

If axs byquery package_wheel,package_name=mlperf_loadgen --- , remove then

axs byquery python_package,package_name=mlperf_loadgen,desired_python_version===3.9
INFO:root:[work_collection] byquery(python_package,package_name=mlperf_loadgen,desired_python_version===3.9) did not find anything, but there are tags: {'python_package'} , trying to find a producer...
INFO:root:[work_collection] A total of 2 matched rules found.                                                                                                                                    
INFO:root:Matched Rule #1/2: ['python_package', 'package_name=mlperf_loadgen', 'desired_python_version.'] from Entry 'loadgen_wheel_recipe'...                                                   INFO:root:Pipeline: [['get_kernel'], ['byname', 'pip'], ['install']], Cumulative params: {'__query': 'python_package,package_name=mlperf_loadgen,desired_python_version===3.9', 'installable': ['^^', 'execute', [[['get_kernel'], ['byquery', ['^^', 'substitute', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===#{desired_python_version}#']], ['find_file', 'mlperf_loadgen.*\\.whl', {'return_full': True, 'abs_paths': True}]]]], 'package_name': 'mlperf_loadgen', 'desired_python_version': '3.9', 'tags': ['python_package'], '__cumulative_param_names': ['__query', 'installable', 'package_name', 'desired_python_version', 'tags']}
INFO:root:[work_collection] byquery(package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9) did not find anything, but there are tags: {'package_wheel'} , trying to find a producer...
INFO:root:[work_collection] A total of 1 matched rules found.

INFO:root:Matched Rule #1/1: ['package_wheel', 'package_name=mlperf_loadgen'] from Entry 'loadgen_wheel_recipe'...                                                                               INFO:root:Pipeline: [['get', 'pipeline']], Cumulative params: {'__query': 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9', 'package_name': 'mlperf_loadgen', 'desired_python_version': '3.9', 'tags': ['package_wheel'], '__cumulative_param_names': ['__query', 'package_name', 'desired_python_version', 'tags']}                                                      INFO:root:[work_collection] byquery(package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9) did not find anything, but there are tags: {'package_wheel'} , trying to find a producer...
INFO:root:[work_collection] A total of 1 matched rules found.                                                                                                                                    
INFO:root:Matched Rule #1/1: ['package_wheel', 'package_name=mlperf_loadgen'] from Entry 'loadgen_wheel_recipe'...                                                                               INFO:root:Pipeline: [['get', 'pipeline']], Cumulative params: {'__query': 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9', 'package_name': 'mlperf_loadgen', 'desired_python_version': '3.9', 'tags': ['package_wheel'], '__cumulative_param_names': ['__query', 'package_name', 'desired_python_version', 'tags']}                                                      WARNING:root:[loadgen_wheel_recipe -> loadgen_wheel_recipe] parameter 'pipeline' is contained here, but BLOCKED by this entry -- all blockers: {'loadgen_wheel_recipe'}                          INFO:root:Matched Rule #1/1 didn't produce a result, 0 more matched rules to try...

------------------------------------------------------------------------------------------------------------------------                                                                         While computing nested_calls in ['^^', 'execute', [[['get_kernel'], ['byquery', ['^^', 'substitute', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===#{desired_python_version}#']], ['find_file', 'mlperf_loadgen.*\\.whl', {'return_full': True, 'abs_paths': True}]]]] the following exception was raised: In pipeline [
        ['get_kernel']                                                                                                                                                                                   ['byquery', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9']
        ['find_file', 'mlperf_loadgen.*\\.whl', {'return_full': True, 'abs_paths': True}]
] step ['find_file', 'mlperf_loadgen.*\\.whl', {'return_full': True, 'abs_paths': True}] cannot be executed on value (None) produced by ['byquery', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9']
...
While computing nested_calls in {'__query': 'python_package,package_name=mlperf_loadgen,desired_python_version===3.9', 'installable': ['^^', 'execute', [[['get_kernel'], ['byquery', ['^^', 'substitute', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===#{desired_python_version}#']], ['find_file', 'mlperf_loadgen.*\\.whl', {'return_full': True, 'abs_paths': True}]]]], 'package_name': 'mlperf_loadgen', 'desired_python_version': '3.9', 'tags': ['python_package'], '__cumulative_param_names': ['__query', 'installable', 'package_name', 'desired_python_version', 'tags']} the following exception was raised: In pipeline [
        ['get_kernel']
        ['byquery', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9']                                                                                                            ['find_file', 'mlperf_loadgen.*\\.whl', {'return_full': True, 'abs_paths': True}]
] step ['find_file', 'mlperf_loadgen.*\\.whl', {'return_full': True, 'abs_paths': True}] cannot be executed on value (None) produced by ['byquery', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9']
========================================================================================================================                                                                         ERROR:root:RuntimeError: In pipeline [
        ['get_kernel']
        ['byquery', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9']
        ['find_file', 'mlperf_loadgen.*\\.whl', {'return_full': True, 'abs_paths': True}]                                                                                                        ] step ['find_file', 'mlperf_loadgen.*\\.whl', {'return_full': True, 'abs_paths': True}] cannot be executed on value (None) produced by ['byquery', 'package_wheel,package_name=mlperf_loadgen,desired_python_version===3.9']
None

It is the same issue.

maria-18-git commented 6 months ago

It is an issue with entry_creator in loadgen_wheel_recipe/data_axs.json. We don't use entry_creator many times.