Mutated paths in config?

[pbelange]

Hi,

I keep running into this error when submitting to HTC:

Traceback (most recent call last):
  File "/afs/cern.ch/work/p/phbelang/abp/NLResidual/studies/LHC_2025/trees/DA_2025_octupole_scan/ID_0_generation_1/ID_01_distribution_file_00_qx_62.305_qy_60.31_i_oct_b1_-100.0_i_oct_b2_-100.0/generation_2.py", line 219, in <module>
    collider, fingerprint = configure_collider(full_configuration)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/afs/cern.ch/work/p/phbelang/abp/NLResidual/studies/LHC_2025/trees/DA_2025_octupole_scan/ID_0_generation_1/ID_01_distribution_file_00_qx_62.305_qy_60.31_i_oct_b1_-100.0_i_oct_b2_-100.0/generation_2.py", line 52, in configure_collider
    collider = xc.load_collider()
               ^^^^^^^^^^^^^^^^^^
  File "/usr/local/DA_study/miniforge_docker/lib/python3.12/site-packages/study_da/generate/master_classes/xsuite_collider.py", line 224, in load_collider
    return self._load_collider(self.path_collider_file_for_configuration_as_input)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/DA_study/miniforge_docker/lib/python3.12/site-packages/study_da/generate/master_classes/xsuite_collider.py", line 204, in _load_collider
    return xt.Multiline.from_json(path_collider)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/DA_study/miniforge_docker/lib/python3.12/site-packages/xtrack/multiline/multiline.py", line 161, in from_json
    return cls.from_dict(json_utils.from_json(file), **kwargs)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/DA_study/miniforge_docker/lib/python3.12/site-packages/xtrack/json_utils.py", line 32, in from_json
    fh, close = open(file, "r"), True
                ^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '../collider_file_for_configuration.json'

I am trying to figure out where to look for to tackle this problem...:

1. Is the config file supposed to be mutated to convert relative paths into absolute paths? My final config file still contains:

   path_collider_file_for_configuration_as_input: ../collider_file_for_configuration.json

2. Or is the collider_file_for_configuration.json supposed to be copied on the HTCondor node?

[pbelange]

@ColasDroin

[ColasDroin]

Hi, If the collider_file_for_configuration.json file was produced at the end of the first generation, it looks like a path problem indeed.

Are you sure you properly declared the dic_dependencies_per_gen when submitting? E.g.

# Dependencies for the executable of each generation. Only needed if one uses HTC or Slurm.
dic_dependencies_per_gen = {
    1: ["acc-models-lhc"],
    2: ["path_collider_file_for_configuration_as_input", "path_distribution_folder_input"],
}

If yes, the path should be mutated to be absolute (granted that you submit from the root of the study as recommended in the doc, this might not work otherwise), such that your collider file should be accessible from the node (the json file itself is not copied on the node, that would be too heavy to do for every job).

[pbelange]

Ah! Indeed, I think my problem is here: https://github.com/ColasDroin/study-DA/blob/902b25eedeef14445fd51e884fb1cb1acabdab16/study_da/submit/submit_scan.py#L259C13-L260C81

            # Build l_dependencies and add to the kwargs
            l_dependencies = dic_dependencies_per_gen.get(generation_number, [])

I had already seen that the generation number is wrong when you dont submit from the root... I'll see what I can do and/or simply submit from the root

[ColasDroin]

Yes indeed! I didn't want to handle the dependencies when submitting from any folder as this might create a cascade of unexpected issues.

[pbelange]

@ColasDroin what do you think of a solution like this? https://github.com/pbelange/study-DA/commit/1ad6d84b0441514028bd912104ce0566577f44a3

It allowed me to submit properly for the first time...

this is because I have a file structure like this:

-rw-rw-r--. 1 120232 phbelang 1610 Nov 27 16:25 01_config_tree.yaml
-rw-rw-r--. 1 120232 phbelang 1563 Nov 27 16:24 02_build_tree.py
-rw-rw-r--. 1 120232 phbelang 2182 Nov 27 16:24 03_submit_tree.py
drwxrwxr-x. 2 120232 phbelang 2048 Nov 12 16:08 configs
drwxrwxr-x. 2 120232 phbelang 2048 Nov 12 10:00 fillingschemes
drwxrwxr-x. 2 120232 phbelang 2048 Nov 27 16:21 scripts
drwxrwxr-x. 5 120232 phbelang 2048 Nov 27 20:27 trees

where the tree.yaml is found under trees/DA_2025_octupole_scan/tree.yaml. The reason for this file structure is that some additional scripts/dependencies are found in my script folder, and I might want to generate several different trees which use the same additionnal script (such that if I need to modify something, I do it only in the main script)...

I will let you know if everything goes as planned, but for now the submission is running fine and the tracking is ongoing... just need to see if the second script will run after the DA tracking...

[ColasDroin]

Ok nice, this seems like a non-invasive solution. Tell me how it goes, and you can do a PR if everything is fine. Cheers