Environment variable substitution

fdaguin commented 3 years ago

Is your feature request related to a problem?

I want to define environment variables for tasks with values that will be substituted with already defined environment variables.

Describe the solution you'd like

Let's consider this dummy task:

[tool.poe.tasks]
"path" = { cmd = "echo $MY_PATH", env = { MY_PATH = "$PATH" }}

When a run poe path, I want the value of $PATH to be resolved instead of only echoing $PATH as a string.

Describe alternatives you've considered

Using shell tasks by:

defining the environment variable inline: no success

[tool.poe.tasks]
"path" = { shell = "MY_PATH=$PATH echo $MY_PATH" }

defining the environment variable with Poe: no success

[tool.poe.tasks]
"path" = { shell = "echo $MY_PATH", env = { MY_PATH = "$PATH" }}

nat-n commented 3 years ago

Hi @fdaguin, thanks for the suggestion.

Could you help me better understand your use case? If you know there's already a variable with the value you want... why not just use that? Why alias it?

If your objective is to take an externally defined environment variable and make it available to another executable as something else then your first approach should actually work e.g. path = "MY_PATH=$PATH python -c \"import os; print(os.environ['MY_PATH'])\"". The key difference is that in this case python is consuming the variable set at the start of the line, instead of the host shell which doesn't get variables set in that way.

If you just want to alias the variable within the shell as in your example (maybe to make it shorter?), then putting a ; after the assignment should achieve that i.e. "path" = { shell = "MY_PATH=$PATH; echo $MY_PATH" }

Another work around exists in the latest pre-release (though it's probably not ideal), which is to use the task dependency feature like so:

"_path" = { shell = "MY_PATH=$PATH echo $MY_PATH" }
"path" = { shell = "MY_PATH=$PATH echo $MY_PATH" }

Unfortunately the approach that you suggest would be a breaking change which I'm not certain is worth it. I think a new dedicated option would be required to handle it cleanly e.g.

[tool.poe.tasks.path]
cmd = "echo $MY_PATH"
alias = { MY_PATH = "PATH" }

Though I'm not yet convinced it's worth it.

fdaguin commented 3 years ago

Hi @nat-n, thanks for this detailed answer!

If your objective is to take an externally defined environment variable and make it available to another executable

This is indeed what I'm trying to achieve, however I end up with some limitations. To provide you more context, I need to dive into the specifics, sorry for the long read.

My use case

Here is what I'm trying to achieve:

I need to run a task on Jenkins with Poe the Poet
the task will run on a host where 3 environments variables are set: PLAYWRIGHT_BROWSER_CHROMIUM_PATH, PLAYWRIGHT_BROWSER_FIREFOX_PATH and PLAYWRIGHT_BROWSER_WEBKIT_PATH
the task will launch an executable that needs the values of the 3 previous environment variables (one at a time) but which only accept the environment variable RF_BROWSER_EXECUTABLE_PATH; An example to make it work is to define RF_BROWSER_EXECUTABLE_PATH=$PLAYWRIGHT_BROWSER_CHROMIUM_PATH, same with the others two
this task also needs to use credentials from the Jenkins plugin Credentials Binding (in short, it "allows credentials to be bound to environment variables"); In my case, Jenkins is setting the environment variable $SECRET_PATH

The command I'd like to run is invoke run --env-file '$SECRET_PATH' where:

the environment variable RF_BROWSER_EXECUTABLE_PATH=$PLAYWRIGHT_BROWSER_CHROMIUM_PATH needs to be defined
$SECRET_PATH is an environment variable set by Jenkins

My experiments

with this shell task:

[tool.poe.tasks]
"test:acceptance" = { shell = "RF_BROWSER_EXECUTABLE_PATH=$PLAYWRIGHT_BROWSER_CHROMIUM_PATH invoke run --env-file '$SECRET_PATH'" }

for some obscure reasons I end up with Jenkins not replacing $SECRET_PATH with the value of its vault ❌

with this cmd task:
```
[tool.poe.tasks]
"test:acceptance" = { cmd = "RF_BROWSER_EXECUTABLE_PATH=$PLAYWRIGHT_BROWSER_CHROMIUM_PATH invoke run --env-file '$SECRET_PATH'" }
```
- $SECRET_PATH is correctly replaced by Jenkins ✔️
- $PLAYWRIGHT_BROWSER_CHROMIUM_PATH is correctly substituted ✔️
- however I end up with an invalid command thrown by Python: FileNotFoundError: [Errno 2] No such file or directory: 'RF_BROWSER_EXECUTABLE_PATH=/ms-playwright/chromium-888113/chrome-linux/chrome' ❌

with this cmd task:

[tool.poe.tasks]
"test:acceptance" = { cmd = "invoke run --env-file '$SECRET_PATH'", env = { RF_BROWSER_NAME = "chromium", RF_BROWSER_RUN_HEADLESS = "true", RF_BROWSER_EXECUTABLE_PATH = "$PLAYWRIGHT_BROWSER_CHROMIUM_PATH" }}

$SECRET_PATH is correctly replaced by Jenkins ✔️
$PLAYWRIGHT_BROWSER_CHROMIUM_PATH is not substituted ❌

If you just want to alias the variable within the shell as in your example (maybe to make it shorter?), then putting a ; after the assignment should achieve that i.e. "path" = { shell = "MY_PATH=$PATH; echo $MY_PATH" }

While this is a good idea, based on my observations, I can't make use of a shell task because of the Jenkins limitation I encounter.

To summarize:

I can't use a shell task
if you have any tips on how to make a cmd task work, I'm more than interested
based on your previous message, a new feature for this use case is indeed not worth it, I will deal with it differently

nat-n commented 3 years ago

I'm curious what's going on that stops the $SECRET_PATH working in the shell task. It might be useful to know that the shell task should in principle work exactly like a bash script. So whatever you can get to work in a normal shell should transfer.

Have you tried removing the single quotes (or using double quotes) around the $SECRET_PATH? or do you really intend to pass the string '$SECRET_PATH' to the invoke process?

The only way I can think of to make the cmd task do what you want is by making it depend on another task that just echos the other variable as a I mentioned above (not great).

It looks like there isn't anything more to try/fix for this issue so I'll close it for now.

Good luck!

fdaguin commented 3 years ago

Have you tried removing the single quotes (or using double quotes) around the $SECRET_PATH? or do you really intend to pass the string '$SECRET_PATH' to the invoke process?

Yes for both 😁

The only way I can think of to make the cmd task do what you want is by making it depend on another task that just echos the other variable as a I mentioned above (not great).

I will keep it in my mind. Thanks a lot for your time @nat-n !

nat-n commented 3 years ago

Note to self: the failure of the invoke command to find the $SECRET_PATH value in its environment could be indicative of shell tasks no propagating all given environment variables as expected.

nat-n commented 2 years ago

@fdaguin

As of v0.13.0 you can now alias env vars via POSIX style templating into a new env var. e.g. ,

env = { RF_BROWSER_EXECUTABLE_PATH = "${PLAYWRIGHT_BROWSER_CHROMIUM_PATH}" }

I hope this is still useful for you.

fdaguin commented 2 years ago

Thanks a lot for this notification @nat-n, I will try this ASAP! 👍

nat-n / poethepoet