search

treasure-data / digdag

Workload Automation System
https://www.digdag.io/
Apache License 2.0
1.31k stars 222 forks source link

[feature request] python and ruby API to get secrets #942

Open sonots opened 5 years ago

sonots commented 5 years ago

I want to have python (and ruby) API to get secrets.

We currently have to map them into environment variables to get in python (or ruby) as https://blog.torut.tokyo/entry/2017/11/22/233844 says. To do it, we again suffer with _env as we state at https://github.com/treasure-data/digdag/issues/937.

muga commented 5 years ago

There is actual REST API endpoint for getting secrets on digdag-server as following. We don't have specific python or ruby client library to talk to Digdag server though. https://github.com/treasure-data/digdag/blob/master/digdag-server/src/main/java/io/digdag/server/rs/ProjectResource.java#L727

muga commented 5 years ago

Your suggestion makes sense. If we use secrets in py> / rb> operator, we'd better to get them from project secrets API directly. To pass secrets to the Python/Ruby program, the operator create a JSON file and write the exact values of the secrets in the file as plain text to communicate inter-processes between operator process (java) and external process (Python/Ruby).

My thought is that it's good to introduce Python/Ruby API defined in runner.py/runner.rb. But the spec is almost similar to http client to support various types of authentication methods. I don't have other better idea. ref: https://github.com/treasure-data/digdag/blob/master/digdag-standards/src/main/resources/digdag/standards/py/runner.py

@sonots @yoyama @frsyuki If you have any good idea, please leave your comment in this ticket.

muga commented 5 years ago

As talked with @yoyama about this ticket, I was missing that Digdag doesn't have an endpoint for getting the secret value(s) of specific key(s) instead of endpoint to list secret keys from the perspective of security. My above idea doesn't work.