Support profile environment with dotenv

eblondel commented 1 year ago

Description of the new feature: Management of dotenv file through the configuration profile

Example of new feature (profile):

"profile": {
    "environment": {
        "file": ".env",
        "hide_env_vars": ["PASSWORD", "MOTDEPASSE", "TOKEN", "JWT"]
    }
}

As shown in the example, the environment will be enabled through at minimum a file property (name of the file), completed eventually by one or more regular expressions (using hide_env_vars) used to detect environment variables that should be hidden (based on their name) from the workflow job logs. A value of "*" will hide all environment variables:

"profile": {
    "environment": {
        "file": ".env",
        "hide_env_vars": "*"
    }
}

* A list of default regular expressions is set (can be extended): "PASSWORD", "PWD", "TOKEN"

As indicated, the workflow environment variables will be listed in the logs. If a variable is meant to be hidden, its content will be replaced by '**' in the logs.

In addition, this feature will contribute to consolidate:

initWorkflow where dotenv::load_dot_env function will be triggered. The #227 eval_variable_expressions function will be replaced by a function call load_workflow_environment, run after dotenv logic, considering the fact environment might not be inherited only from variables defined in an .env file, but just with variables inherited with the session environment.
closeWorkflow where a function unload_workflow_environment will be triggered to achieve 2 tasks at the end of the workflow 1) unset the environment variables defined in the .env; 2) reset environment variables that would have been overwriten by the env file.

kikislater commented 1 year ago

Example available here : https://github.com/IRDG2OI/geoflow-g2oi/

eblondel commented 1 year ago

yes, indeed I already use dotenv in geoflow to pass environment variables, since i had added the support in geoflow to actually parse these environment variables (see https://github.com/r-geoflow/geoflow/issues/227), but being not part of the workflow, it's common to forget loading the env with dot env :-) so I will make it part of the config (as profile capability).

You will have the possibility to define an "environment" in the profile part of the config, where you give the env file path. geoflow will try to trigger dotenv::load_dot_env for the user.

eblondel commented 1 year ago

@kikislater @juldebar I've updated above the issue description so you can see what's available. Since you extensively use .env in https://github.com/IRDG2OI/geoflow-g2oi/, i would appreciate if you could give a try to this new feature.

kikislater commented 1 year ago

It works, thank you But I found one behavior not important for me but to be know : passwords could be shown event with hide_env_vars. Example Sys.getenv() after executeWorkflow :

> Sys.getenv()
CLICOLOR_FORCE                              1
DISPLAY                                     :0
EDITOR                                      vi
geonapi@https://<url>/geonetwork:user_pwd.                                            mypassword
geonapi@https://<url>/geonetwork:user_token                                            xxxx-xxxx-xxxxx-xxxx-xxxx
geosapi@https://<url>/geoserver:user                                            mypassword
geosapi@https://<url>/geoserver/rest:user                                            mypassword

eblondel commented 1 year ago

hum normally this should be reset with unload_workflow_environment when workflow is closed. Clearly, the workflow should set the env only for the duration of the execution, afterwhat it should be back to same env as before.

eblondel commented 1 year ago

@kikislater when you have chance, if you can retry this, I suspect that your env was somehow polluted, I had an issue with the closeWorkflow function. I hope that it works now. Closing the workflow should unload the workflow environment.

eblondel commented 1 year ago

@kikislater I see the problem now. The software managers (geosapi, geonapi, etc) use by default keyring 'env' storage internally (that can be modified btw as software parameter in most of cases), but these managers dot not have proper "close" methods to destroy the keyring pwd/token storage. I'm going to close this ticket. This works, but needs to be completed by proper software closures when closing the workflow

kikislater commented 1 year ago

Thanks, you're right this one coule be close

kikislater commented 1 year ago

Dotenv now internal: https://github.com/r-geoflow/geoflow/commit/42a57f3ee6117418271d9261f476ec7fdb4a9268

eblondel commented 1 year ago

Yes, actually it doesnt change anything but i was calling internal functions with ::: , ie not exported by dotenv package. From CRAN check perspective, this is forbidden. I contacted the author to make the funs exported, but since they are small he recommended to copy them. Note that dotenv is still required as package.

Le ven. 3 mars 2023 à 05:45, Sylvain POULAIN @.***> a écrit :

Dotenv now internal: 42a57f3 https://github.com/r-geoflow/geoflow/commit/42a57f3ee6117418271d9261f476ec7fdb4a9268

— Reply to this email directly, view it on GitHub https://github.com/r-geoflow/geoflow/issues/295#issuecomment-1452962361, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKDK3CFIBG7H4YZGCUVTX3W2FZOLANCNFSM6AAAAAASKCUAFE . You are receiving this because you modified the open/close state.Message ID: @.***>

r-geoflow / geoflow

Support profile environment with dotenv #295