Post-merge Validator error

[ppebay]

Steps to reproduce:

python LBAF_app.y --conf conf.yaml

(from Applications in develop branch).

Observed error:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/pppebay/Documents/Git/LB-analysis-framework/src/lbaf/Applications/LBAF_app.py", line 79, in <module>
    from lbaf.Applications.rank_object_enumerator import compute_min_max_arrangements_work
  File "/Users/pppebay/Documents/Git/LB-analysis-framework/src/lbaf/Applications/rank_object_enumerator.py", line 16, in <module>
    from lbaf.IO.lbsVTDataReader import LoadReader
  File "/Users/pppebay/Documents/Git/LB-analysis-framework/src/lbaf/IO/lbsVTDataReader.py", line 9, in <module>
    from ..imported.JSON_data_files_validator import SchemaValidator
ImportError: cannot import name 'SchemaValidator' from 'lbaf.imported.JSON_data_files_validator' (/Users/pppebay/Documents/Git/LB-analysis-framework/src/lbaf/imported/JSON_data_files_validator.py)

[marcinwrobel1986]

@ppebay Hello Philippe, I assume:

• first potential cause could have something to do you are not up to date with develop branch. Please pull the changes that have been merged recently.
• the other potential cause would be you were not connected to the internet. Now in src/lbaf an imported directory is created and inside that directory JSON_data_files_validator.py is downloaded from VT, before execution of LBAF.
• the third potential cause could be that the configuration file with command line argument has to be passed like below:

  python LBAF_app.y --config conf.yaml
  # or
  python LBAF_app.y --config=conf.yaml

  so the command line argument is config not conf.

Please eliminate the all causes I mentioned above and let me know if that help. When that would be true, then please close the issue, otherwise please let me know if still the same or maybe sm other than that and I will take a look once more. Thanks!

[ppebay]

@marcinwrobel1986 this one persists even after #295 was resolved (I also pushed further changes from time to load). Can you look into it? Thanks.

[marcinwrobel1986]

@ppebay that have to be something on your side. Please take a look HERE this is working on our CI and on my PC.

Here is my terminal:

Please send here the whole output of command python LBAF_app.py --config=conf.yaml then I could help more.

You should see that line first, before any import error: [LBAF_app] Saved SchemaValidator to: /home/marcin/Dokumenty/NGA/projects/LB-analysis-framework/src/lbaf/imported/JSON_data_files_validator.py

This is because this saving Schema from VT is happening before anything else is imported.

[ppebay]

[pppebay@localhost]~/Documents/Git/LB-analysis-framework/src/lbaf/Applications$ python LBAF_app.py --conf conf.yaml
[LBAF_app] Saved SchemaValidator to: /Users/pppebay/Documents/Git/LB-analysis-framework/src/lbaf/imported/JSON_data_files_validator.py
[LBAF_app] Executing LBAF version 0.1.0rc1
[LBAF_app] Executing with Python 3.8.13
[LBAF_app] Found configuration file /Users/pppebay/Documents/Git/LB-analysis-framework/src/lbaf/Applications/conf.yaml
[configurationValidator] Skeleton schema is valid
[configurationValidator] Reading from data was chosen
[configurationValidator] from_data schema is valid
[configurationValidator] Checking algorithm schema of: InformAndTransfer
[configurationValidator] Algorithm: InformAndTransfer schema is valid
[LBAF_app] Logging level: info
[LBAF_app] Output directory: /Users/pppebay/Documents/Git/LB-analysis-framework/output
[LBAF_app] Saved SchemaValidator to: /Users/pppebay/Documents/Git/LB-analysis-framework/src/lbaf/imported/JSON_data_files_validator.py
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/pppebay/Documents/Git/LB-analysis-framework/src/lbaf/Applications/LBAF_app.py", line 79, in <module>
    from lbaf.Applications.rank_object_enumerator import compute_min_max_arrangements_work
  File "/Users/pppebay/Documents/Git/LB-analysis-framework/src/lbaf/Applications/rank_object_enumerator.py", line 16, in <module>
    from lbaf.IO.lbsVTDataReader import LoadReader
  File "/Users/pppebay/Documents/Git/LB-analysis-framework/src/lbaf/IO/lbsVTDataReader.py", line 9, in <module>
    from ..imported.JSON_data_files_validator import SchemaValidator
ImportError: cannot import name 'SchemaValidator' from 'lbaf.imported.JSON_data_files_validator' (/Users/pppebay/Documents/Git/LB-analysis-framework/src/lbaf/imported/JSON_data_files_validator.py)

[marcinwrobel1986]

@ppebay Philippe I posted the right command:

python LBAF_app.py --config conf.yaml

or

python LBAF_app.y --config=conf.yaml

[marcinwrobel1986]

But the funny part is, that even if I used wrong command it is still working for me. It takes the default config file, which I think we should remove, to not confuse anyone...

[ppebay]

Exact same error, does not change anything

[pppebay@localhost]~/Documents/Git/LB-analysis-framework/src/lbaf/Applications$ python LBAF_app.py --config=conf.yaml
[LBAF_app] Saved SchemaValidator to: /Users/pppebay/Documents/Git/LB-analysis-framework/src/lbaf/imported/JSON_data_files_validator.py
[LBAF_app] Executing LBAF version 0.1.0rc1
[LBAF_app] Executing with Python 3.8.13
[LBAF_app] Found configuration file /Users/pppebay/Documents/Git/LB-analysis-framework/src/lbaf/Applications/conf.yaml
[configurationValidator] Skeleton schema is valid
[configurationValidator] Reading from data was chosen
[configurationValidator] from_data schema is valid
[configurationValidator] Checking algorithm schema of: InformAndTransfer
[configurationValidator] Algorithm: InformAndTransfer schema is valid
[LBAF_app] Logging level: info
[LBAF_app] Output directory: /Users/pppebay/Documents/Git/LB-analysis-framework/output
[LBAF_app] Saved SchemaValidator to: /Users/pppebay/Documents/Git/LB-analysis-framework/src/lbaf/imported/JSON_data_files_validator.py
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/opt/local/Library/Frameworks/Pyth
...

[marcinwrobel1986]

I am afraid something is messed up with you machine. Why? You have twice:

[LBAF_app] Saved SchemaValidator to: /Users/pppebay/Documents/Git/LB-analysis-framework/src/lbaf/imported/JSON_data_files_validator.py

which is not really possible :/

[marcinwrobel1986]

Do you have in our lbsVTDataReader.py same first 16 lines:

import json
from logging import Logger
from multiprocessing import Pool
import os
import sys

import brotli

from ..imported.JSON_data_files_validator import SchemaValidator
from ..Model.lbsObject import Object
from ..Model.lbsObjectCommunicator import ObjectCommunicator
from ..Model.lbsRank import Rank
from ..Utils.exception_handler import exc_handler

class LoadReader:

[ppebay]

I am getting the same error on all 3 Macs. Here's what I have:

import json
from logging import Logger
from multiprocessing import Pool
import os
import sys

import brotli

from ..imported.JSON_data_files_validator import SchemaValidator
from ..Model.lbsObject import Object
from ..Model.lbsObjectCommunicator import ObjectCommunicator
from ..Model.lbsRank import Rank
from ..Utils.exception_handler import exc_handler

class LoadReader:

Seems identical to me

[marcinwrobel1986]

I am having:

python LBAF_app.py --config conf.yaml
[LBAF_app] Saved SchemaValidator to: /home/marcin/Dokumenty/NGA/projects/LB-analysis-framework/src/lbaf/imported/JSON_data_files_validator.py
[LBAF_app] Executing LBAF version 0.1.0rc1
[LBAF_app] Executing with Python 3.8.13
[LBAF_app] Found configuration file /home/marcin/Dokumenty/NGA/projects/LB-analysis-framework/src/lbaf/Applications/conf.yaml
[configurationValidator] Skeleton schema is valid
[configurationValidator] Reading from data was chosen
[configurationValidator] from_data schema is valid
[configurationValidator] Checking algorithm schema of: InformAndTransfer
[configurationValidator] Algorithm: InformAndTransfer schema is valid
[LBAF_app] Logging level: info
[LBAF_app] Output directory: /home/marcin/Dokumenty/NGA/projects/LB-analysis-framework/output
[lbsVTDataReader] Reading /home/marcin/Dokumenty/NGA/projects/LB-analysis-framework/data/synthetic_lb_data/data.0.json VT object map
[lbsVTDataReader] Reading /home/marcin/Dokumenty/NGA/projects/LB-analysis-framework/data/synthetic_lb_data/data.2.json VT object map
[lbsVTDataReader] Reading /home/marcin/Dokumenty/NGA/projects/LB-analysis-framework/data/synthetic_lb_data/data.1.json VT object map
[lbsVTDataReader] Reading /home/marcin/Dokumenty/NGA/projects/LB-analysis-framework/data/synthetic_lb_data/data.3.json VT object map

That's why I am asking what do you have in your data reader, because it has already downloaded files and now should start reading data.

[marcinwrobel1986]

Wow man, this problem could be deeper. By deeper I mean it could have something to do with the multiprocessing Python module. I have found that: multiprocessing's default start method is spawn instead of fork on macOS in 3.8.

I will try to find a solution tomorrow.

[ppebay]

Ok thanks for looking into this.

TTYL

[marcinwrobel1986]

Please, pull the code and try now.

[ppebay]

Perfect, it's fixed! Thank you @marcinwrobel1986