Code packaging files in parent directories

[spitz-dan-l]

I have a codebase that is structured as follows:

/my_project
├── __init__.py
├── library
│   ├── __init__.py
│   └── library_1.py
└── workflows
    ├── __init__.py
    ├── workflow_1
    │   ├── .redun
    │   ├── __init__.py
    │   └── main.py
    └── workflow_2
        ├── .redun
        ├── __init__.py
        └── main.py

6 directories, 8 files

When working on either /my_project/workflows/workflow_1/ or /my_project/workflows/workflow_2/, I may need to make changes to the library code in /my_project/library/. Redun's code packaging feature seems not to work on patterns that contain absolute paths or relative paths that contain ... It seems to be correctly zipping up the matched code, but not unzipping it in a location that makes it usable when executing the task.

Simply moving the .redun/ directory to the project root is not a good option because different workflows may need different redun.ini files, so a single such file at the root can't really work.

A workaround I have identified is to cd to the root of the project, and then use -c to explicitly reference the redun config dir to use:

cd /my_project
redun -c workflows/workflow_1/.redun run workflows/workflow_1/main.py main

But this makes the command excessively verbose in my opinion.

Overall the code packaging feature is a huge productivity win over rebuilding a docker image every time. Cheers!

[mattrasmus]

You raise good points here. I have thought about how to support code-packaging of parent directories. I haven't come up great solutions at the moment, but raising this issue encourages me to continue to search for improvements. There are some tricky aspects of recreating exactly the right directory structure in the docker container that have been a blocker thus far.

However, there is one workaround that might help you. You can use the environment variable REDUN_CONFIG to avoid having to specify the config on the command line all the time.

cd /my_project/workflows/workflow_1/
export REDUN_CONFIG=../../.redun
redun run main.py main

I think one remaining challenge for your case is that code_includes is interpreted relative to the CWD and likely the more general solution is to interpret it relative to the redun config dir.

You likely already consider this, but if the common code doesn't change much, you can install it directly in the docker image to avoid relying on redun's code packaging.

[spitz-dan-l]

Thank you for the suggestions!

I do agree that code_includes probably ought to be interpreted relative to the redun config dir (or really the immediate parent).

Perhaps the following could work as a proper fix.

• New config var code_root- the path to the base directory to use for code packaging. Interpreted as relative to the redun config dir. The code_includes and code_excludes patterns will be interpreted relative to code_root.
• Disallow absolute paths and relative paths containing .. in code_includes and code_excludes.

This would at least solve my issue, I could just set code_root = ../.. in redun.ini. I'm sure there are other ways, and use cases I haven't considered.