config file needed from CLI and also on azure blob storage. Want to remove duplicative dependence on config because if they differ naming will be incorrectly mapped to whats running.
Requirements
[ ] refactor R scripts to take in the arguments from the config that need to be passed to the script, parse them in python and pass the config in via the command line
Context
Essentially, we want to rewrite setup_job.py to parse the config that it gets passed and to pass those as arguments in a loop to the command line R script. This way, we will never end up having a mismatch between what we pass to azure from the CLI and whats on blob storage.
Might end up being a lot of args though, and they all have to be in the correct order I think to work properly?
@damonbayer tagging you for awareness, I don't think this is an urgent task but part of the general updates to azure pipelining
Problem
config file needed from CLI and also on azure blob storage. Want to remove duplicative dependence on config because if they differ naming will be incorrectly mapped to whats running.
Requirements
Context
Essentially, we want to rewrite setup_job.py to parse the config that it gets passed and to pass those as arguments in a loop to the command line R script. This way, we will never end up having a mismatch between what we pass to azure from the CLI and whats on blob storage.
Might end up being a lot of args though, and they all have to be in the correct order I think to work properly?
@damonbayer tagging you for awareness, I don't think this is an urgent task but part of the general updates to azure pipelining