Support custom color and description files on Terra runs

[huddlej]

Current Behavior

Custom color and description files used for ncov builds are expected to exist as local files (e.g., in nextstrain_profiles/, etc.) based on the paths in the builds YAML file. However, these files are not part of the ncov workflow itself (and shouldn't be), so they do not exist on local storage when running the workflow on Terra.

Expected behavior

We should be able to support custom files like colors, includes, etc. that are stored in remote paths or in a separate GitHub repository.

Possible solutions

• Define ncov workflow inputs for colors, descriptions, and other user-defined paths with the path_or_url function, so users could upload their custom files to any valid remote URLs and create builds YAML files that reference these paths
• Allow users to provide the WDL workflow an optional git URL for a repo with these files that gets cloned into the ncov directory on Terra.
• Ask users to upload custom files to Terra and then specify an optional directory path in the WDL inputs to get localized for each job. The builds YAML would have to reference this localized path somehow. Not sure how this would work in practice.

[j23414]

I'm still thinking, but for option 3 the implementation would look like:

mv ~{localized_dir} ncov/custom_profile

And the build.yaml would use local paths, like colors: custom_profile/colors/algeria_colors.tsv and assume it'll be moved into the ncov directory. Yeah, this can get messy fast. I'll need to think.

[huddlej]

After discussing this issue today as a group, we agreed that the solution to this issue should include a solution to the broader issue of how to allow users to bring their own "pile of files" to the ncov workflow. We discussed two related approaches to handle this problem:

1. Place ncov inside an outer directory that includes the user's data and config files. This approach emphasizes that ncov is a piece of software, but it doesn't work yet with the ncov workflow.
2. Place the user's data and config files inside a subdirectory of ncov. This is one approach we've used within the group already and one that works with the existing ncov workflow. This approach relies more on the existence of ncov as a Git repo that you work from instead of a piece of software you use, though.

Regardless of the nesting pattern we decide above, we discussed allowing users to define a path to a zip file as an optional input to the ncov WDL workflow. The workflow would then fetch the user's zip file and extract it in the appropriate place before running the workflow. This solution would support linking to GitHub branches/releases/etc through GitHub's auto-generated zip files, but it would also support manually created zip files that were uploaded to Google Storage through Terra.

This solution should address the specific color and description files issues here without requiring users to modify their builds config to include remote paths, etc. It should also support any other additional files users would want to reference for their build (e.g., custom rules, markdown files, etc.).

[j23414]

Sorry it took me a while. Implemented solution 2 and tested it on the Chad dataset: https://nextstrain.org/staging/ncov_chad .

• Feel free to run a test on Terra

If the run successfully completes, the result is deployed to a url similar to:

• https://nextstrain.org/staging/ncov_[build_id]

Additional columns in the input Tables/builds allows for each location to:

• be associated with a different config file
• be fine-tuned for resource use
• and be pushed to different s3 endpoint.

Happy to set up a meeting if that works better. Otherwise, I figure you're busy this week so feel free to explore. Suggestions welcome.

[huddlej]

This will be closed by the wdl/optionals branch.

[j23414]

Closed by merging the the wdl/optionals branch.