pepkit / looper

A job submitter for Portable Encapsulated Projects
http://looper.databio.org
BSD 2-Clause "Simplified" License
20 stars 7 forks source link

looper variable namespaces should be updated to contain what is in the `.looper` config file #423

Closed donaldcampbelljr closed 6 months ago

donaldcampbelljr commented 8 months ago

Original comment: looper variable namespaces need to be updated; in particular, does the looper namespace get what's in the .looper config file now?

donaldcampbelljr commented 8 months ago

Some notes: -Looper reads the config file and sets arguments based on the config file: https://github.com/pepkit/looper/blob/6523e4b525570aa0f7b3013d19a049d1866a28df/looper/cli_looper.py#L593-L619

However, when Looper creates a Project, not all of these lines from the config file become attributes on the Project.

Specific example is with the addition of pipestat in the looper config file:

pep_config: ./project/project_config.yaml
output_dir: ./results
pipeline_interfaces:
  sample:  ./pipeline_pipestat/pipeline_interface.yaml
pipestat:
  results_file_path: results.yaml

The project does not contain the results_file_path. It does, however, contain a boolean pipestat_configured = True.

Further in the process, SubmissionConductor builds the looper namespace: https://github.com/pepkit/looper/blob/6523e4b525570aa0f7b3013d19a049d1866a28df/looper/conductor.py#L462-L500

It appears that only select items are added to the Looper namespace. The config file that is added is the project_config NOT the looper_config.

donaldcampbelljr commented 8 months ago

Question: Is the desire to have everything in the config files (looper and project and pep configs) be an attribute of the looper namespace such that it could be accessed if desired?

donaldcampbelljr commented 8 months ago

I believe looper namespace is being built properly now. In the above example, I mentioned results_file_path as an example but that Is actually for pipestat and is contained within the pipestat namespace.