search

InstituteforDiseaseModeling / idmtools

https://docs.idmod.org/projects/idmtools/en/latest/
Other
4 stars 4 forks source link

Performance issue with get_item in slurm platform when there are lot of items in job directory #2388

Closed shchen-idmod closed 1 week ago

shchen-idmod commented 1 week ago

Try code before, it take forever to get last print statement for experiment.id. I have lot of suites under dest dir, and each suite, there is experiment, and each experiment, there are hundreds or thousands of simulations. run: python get_id.py in NU environment (note, you need to replace your exp_id and job_directory values to yours

from idmtools.core.platform_factory import Platform
from idmtools.core import ItemType

dest="/home/scj6369/example_emodpy_malaria/burnin_create_and_use_sweep_larval_habitat1"

platform = Platform("SLURM_LOCAL", job_directory=dest, partition='b1139', time='6:00:00',
                            account='b1139')
exp_id = "d673fbb3-b4bd-4c3f-ab9a-8dd2533b0917"
exp = platform.get_item(exp_id, ItemType.EXPERIMENT, raw=False)
print("exp:", exp.id)

I have to cancel this run, and got stack message like this:

filepath: /home/scj6369/example_emodpy_malaria/burnin_create_and_use_sweep_larval_habitat1/Suite_cc8d162f-0816-48bb-89fb-f64c993ae33a/4EIR_pickup_PvAd_4b4b72c4-0499-4855-86ac-1c82d7051841/abb94364-d5e7-4d40-86b8-69459ba3cb7e/metadata.json
^CTraceback (most recent call last):
  File "/home/scj6369/github/emodpy-malaria/examples/burnin_create_and_use_sweep_larval_habitat1/get_id.py", line 12, in <module>
    exp = platform.get_item(exp_id, ItemType.EXPERIMENT, raw=False)
  File "/home/scj6369/venv/malaria_idmtools_201/lib/python3.10/site-packages/idmtools/entities/iplatform.py", line 235, in get_item
    return_object = self._convert_platform_item_to_entity(ce, **kwargs)
  File "/home/scj6369/venv/malaria_idmtools_201/lib/python3.10/site-packages/idmtools/entities/iplatform.py", line 527, in _convert_platform_item_to_entity
    return getattr(self, interface).to_entity(platform_item, **kwargs)
  File "/home/scj6369/github/idmtools/idmtools_platform_slurm/idmtools_platform_slurm/platform_operations/experiment_operations.py", line 200, in to_entity
    parent = self.platform.get_item(slurm_exp.parent_id, ItemType.SUITE, force=True)
  File "/home/scj6369/venv/malaria_idmtools_201/lib/python3.10/site-packages/idmtools/entities/iplatform.py", line 235, in get_item
    return_object = self._convert_platform_item_to_entity(ce, **kwargs)
  File "/home/scj6369/venv/malaria_idmtools_201/lib/python3.10/site-packages/idmtools/entities/iplatform.py", line 527, in _convert_platform_item_to_entity
    return getattr(self, interface).to_entity(platform_item, **kwargs)
  File "/home/scj6369/github/idmtools/idmtools_platform_slurm/idmtools_platform_slurm/platform_operations/suite_operations.py", line 126, in to_entity
    suite.experiments = self.get_children(slurm_suite, parent=suite, raw=False)
  File "/home/scj6369/github/idmtools/idmtools_platform_slurm/idmtools_platform_slurm/platform_operations/suite_operations.py", line 102, in get_children
    exp = self.platform._experiments.to_entity(slurm_exp, parent=parent)
  File "/home/scj6369/github/idmtools/idmtools_platform_slurm/idmtools_platform_slurm/platform_operations/experiment_operations.py", line 216, in to_entity
    exp.simulations = self.get_children(slurm_exp, parent=exp, raw=False)
  File "/home/scj6369/github/idmtools/idmtools_platform_slurm/idmtools_platform_slurm/platform_operations/experiment_operations.py", line 98, in get_children
    slurm_sim.status = self.platform._op_client.get_simulation_status(slurm_sim.id)
  File "/home/scj6369/github/idmtools/idmtools_platform_slurm/idmtools_platform_slurm/slurm_operations/local_operations.py", line 251, in get_simulation_status
    sim_dir = self.get_directory_by_id(sim_id, ItemType.SIMULATION)
  File "/home/scj6369/github/idmtools/idmtools_platform_slurm/idmtools_platform_slurm/slurm_operations/local_operations.py", line 101, in get_directory_by_id
    metas = self.platform._metas.filter(item_type=item_type, property_filter={'id': str(item_id)})
  File "/home/scj6369/github/idmtools/idmtools_platform_slurm/idmtools_platform_slurm/platform_operations/json_metadata_operations.py", line 257, in filter
    meta_items = self.get_all(item_type)
  File "/home/scj6369/github/idmtools/idmtools_platform_slurm/idmtools_platform_slurm/platform_operations/json_metadata_operations.py", line 211, in get_all
    meta = self.load_from_file(meta_file)
  File "/home/scj6369/github/idmtools/idmtools_platform_slurm/idmtools_platform_slurm/platform_operations/json_metadata_operations.py", line 138, in load_from_file
    meta = self._read_from_file(metadata_filepath)
  File "/home/scj6369/github/idmtools/idmtools_platform_slurm/idmtools_platform_slurm/platform_operations/json_metadata_operations.py", line 41, in _read_from_file
    metadata = json.load(f)
  File "/software/python/3.10.1/lib/python3.10/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/software/python/3.10.1/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/software/python/3.10.1/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/software/python/3.10.1/lib/python3.10/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)