Open fedorov opened 8 months ago
workflows currently refer to the notebooks from "main" branch; instead we should use git tags and refer to tagged notebooks parameterize papermill with the variable to pass the tag, and refer to the tagged versions of the artifacts from the notebook
@vkt1414 I thought about this, and here's how I think we could do it:
If you have better ideas, let's discuss here before implementing!
I do not have a complete plan yet but two things worry about the workflow you suggested. We use URLs at a lot of places. Links to image sources and config files in notebooks, and links to the notebooks in the wdl and cwl files. Chaning them manually I think poses a lot of room for errors, especially when I revisit the repo after a few months of no activity. Secondly, I'm wary about cloning the entire git repo as we will be running thousands of workflows in parallel and that would mean we rely even more on github. We never faced an issue so far with individual artifacts but I do not think it may be a good idea.
What I have in mind currently is to use a github action that can take care of updating the branch/tag references. We already have bits and pieces of this in several repos now (idc-index, browser, and tutorials) and it should not take a while to come up with one.
The gha should recognize tag and run a python script that will use a regex express to identify main branch and replace them with the tag. Otherwise, we always refer to main branch as you also suggested.
What are your thoughts?
Chaning them manually I think poses a lot of room for errors
I agree - I didn't mean to do it manually, it can be done with a script.
What I have in mind currently is to use a github action that can take care of updating the branch/tag references. We already have bits and pieces of this in several repos now (idc-index, browser, and tutorials) and it should not take a while to come up with one.
For this kind of tasks, I think it is easier to have a script that can be run locally, and then whoever ran it an easily review all the changes and confirm that modifications make sense.
We can also consider encoding the specific version of the workflow in the SEG/SR objects created, but I want to discuss this with David C first.