search

treasure-data / digdag

Workload Automation System
https://www.digdag.io/
Apache License 2.0
1.31k stars 222 forks source link

[feature request] Add an option to hide parameters of digdag.env.store from web ui #968

Open sonots opened 5 years ago

sonots commented 5 years ago

My final purpose is to use secrets in embulk liquid files and in docker containers. If my final purpose can be achieved, another approach is acceptable for me.

To use secrets in embulk liquids or in docker containers, I currently need to map all secrets to environment variables. We can use digdag.env.store with python API to do it.

I wrote details at https://qiita.com/sonots/items/cb71834857235f965225 (Japanese)

For this purpose, I feel good if following is supported.

  1. Add a python API to get secrets (so that I do not need to write _env) ref. https://github.com/treasure-data/digdag/issues/942
  2. Add an option to digdag.env.store to hide from web ui of the digdag server. (this feature request)
muga commented 5 years ago

Thank you for suggesting @sonots, for 1st one, let's keep discussion on #942. We couldn't get appropriate response back to you and then, ping us on this ticket. so, sorry to be late response. I will update my thought on the ticket soon.

For 2nd one, as mentioned by me on #944, I still not sure that the option might work for you and us if we could introduce the option. I have the following 2 reasons. The 1st one is my bigger concern instead of 2nd.

  1. store params are written back to tasks table as plain text to keep the current status of the tasks in Digdag concept. We (and probably you may) not want someone to see the exact values of the store params from the database instead of secrets table.

  2. store params can show and help to know the current status of the task and workflow progress. if they will be hidden on the UI, it may make hard for you and us to check the status and investigate some issues.

What do you think?

sonots commented 5 years ago

Although I did not know 1, I also have concerns about it, and do not like to be seen as plain texts.

For the reason of 2, I want to hide only some paramers, not all parameters.

I supposed that the API would be

digdag.env.store({key: value}, secret=True)

The parameter is not shown or masked on Web UI and stored as encoded values into DB.

muga commented 5 years ago

Ah I saw your qiita article and got your use case now. I will discuss about _env scope with @yoyama and update.

muga commented 5 years ago

We will start designing secrets enablement in export parameters. That has been discussed on https://github.com/treasure-data/digdag/issues/926. Still not 100% sure that we could finish implementing it though.