projectglow / glow

An open-source toolkit for large-scale genomic analysis
https://projectglow.io
Apache License 2.0
263 stars 110 forks source link

Improve handling of environment variables in the pipe transformer #480

Closed Hoeze closed 1 year ago

Hoeze commented 2 years ago

From https://github.com/projectglow/glow/blob/master/docs/source/tertiary/pipe-transformer.rst:

Options beginning with env_ are interpreted as environment variables. Like other options, the environment variable name is converted to lower snake case. For example, providing the option env_aniMal=MONKEY results in an environment variable with key ani_mal and value MONKEY being provided to the piped program.

Would it be possible to have some argument jsonEnv that can decode json-encoded environment variables? E.g.:

jsonEnv='{"PATH": "/usr/bin"}'

This would make it significantly easier to adjust environment variables IMHO.

Hoeze commented 2 years ago

The snake-case parsing of arguments makes it impossible to pass uppercase environment variables, right? E.g. env_TEST would get translated to t_e_s_t.

Interestingly, passing env_test_path='asdf' results in an environment variable test_path="asdf". So actually there is no need to parse them as snake-case.

williambrandler commented 2 years ago

An alternative approach is to define environment variables via JSON and put those variables into your script, which is then run through the pipe transformer.

Here's an example,

Screen Shot 2022-02-01 at 1 28 27 PM

Screen Shot 2022-02-01 at 1 28 36 PM

I have never used the environment variable feature in glow. Probably because I hit the same limitations you describe above!

Hoeze commented 2 years ago

Ah, OK, I did not think of that solution :grin:

williambrandler commented 2 years ago

will keep the issue open for now until this is properly documented

should delete those environment variable options from the docs, they are somewhat confusing

williambrandler commented 2 years ago

updating the example in the docs to avoid this issue in future:

https://github.com/projectglow/glow/pull/490