Problem with the fake memory data step

[ppebay]

It seems that there is a missing key in this dataset.

Steps to reproduce:

1. pull 279-Rename-Time-and-Load-and-add-capabilities-to-handle-Memory branch
2. in Applications execute python LBAF_app.py --config=memory.yaml

Observed error:

[lbsVTDataReader] Reading /Users/pppebay/Documents/Git/LB-analysis-framework/data/fake-4x-block-overdecomp-3-uncompressed/fake-4x-block-overdecomp-3.8.json VT object map
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/Users/pppebay/Documents/Git/LB-analysis-framework/src/lbaf/IO/lbsVTDataReader.py", line 68, in _load_vt_file
    raise TypeError("JSON data is missing 'type' key")
TypeError: JSON data is missing 'type' key
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "LBAF_app.py", line 385, in <module>
    LBAFApp().main()
  File "LBAF_app.py", line 245, in main
    reader = LoadReader(file_prefix=self.params.data_stem, n_ranks=self.params.n_ranks, logger=self.logger,
  File "/Users/pppebay/Documents/Git/LB-analysis-framework/src/lbaf/IO/lbsVTDataReader.py", line 48, in __init__
    self.vt_files = self._load_vt_files()
  File "/Users/pppebay/Documents/Git/LB-analysis-framework/src/lbaf/IO/lbsVTDataReader.py", line 87, in _load_vt_files
    for rank, decompressed_dict in results:
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/pool.py", line 868, in next
    raise value
TypeError: JSON data is missing 'type' key

[marcinwrobel1986]

@ppebay unfortunately you haven't pushed the data to the repository. By the data I mean: data/fake-4x-block-overdecomp-3-uncompressed/fake-4x-block-overdecomp-3.8.json The problem you are facing is due to the fact that this data is missing a type key. There should be a type key with value LBDatafile or LBStatsfile. The type key should be on the same level as phases. Two solutions:

1. You push the data to the repo and I could add that
2. You need to decompress the file, add "type":"LBDatafile" or "type":"LBStatsfile" depending on data type and after that compress again.

[ppebay]

@lifflander @nlslatt is it alright to push the "fake dataset" (fake-4x-block-overdecomp-3-uncompressed) to the repo?

Please advise. Thanks

[marcinwrobel1986]

@ppebay please do as follows in your terminal:

• pull the data from GitHub
• backup the data you will be changing
• go to <LBAF_path>/src/lbaf/Utils
• make sure the name of the files are:
  • fake-4x-block-overdecomp-3.0.json
  • fake-4x-block-overdecomp-3.1.json
  • fake-4x-block-overdecomp-3.2.json
  • and so on ....
  • when not, then you need use --file_path instead of --dir_path
• run python3 data_stat_files_updater.py --dir_path=../../../data/fake-4x-block-overdecomp-3-uncompressed --schema_type=LBDatafile

[ppebay]

The updater works great, thanks Marcin! Keeping a note of that.