Env vars and secrets for dataset flows

[sergiimk]

Tasks

• Root datasets can have key-value pairs associated with it to pass secrets and params to ingest tasks
  • On the polling ingest level we already support paramterization via Env vars - this task will introduce a layer of indirection between ingest and the source of values
• These kv pairs are stored in DB
  • Consider how secrets encrypted in DB (salted?)
• They can be created via web UI dataset settings tab
• Introduce kv store object that will be passed down to ingest stage

[sergiimk]

Re-opening the issue as it is still in acceptance testing phase

[sergiimk]

Suggestions after acceptance testing of env vars:

1) Rename Add row button to Add variable/secret ("row" is confusing)

2) In the add dialog let's move the "Secret value" checkbox up. When checked the "Value" input should switch into the masked mode. This way during a demo / video recording it will be possible to paste a real secret in without revealing it to the viewers.

3) In place of Key-value pairs associated with dataset I think we should better explain the purpose of vars, where they can be used, and link to docs. For now you can link to: https://docs.kamu.dev/cli/ingest/input-formats/?highlight=env#dealing-with-api-keys

4) Variable and secrets menu should not be available on derivative datasets - there is no place that uses them in derivative data flows (yet)

5) Since our secrets are extractable - you can see them when editing and can even switch a checkbox to convert a secret into a variable - the difference between a secret and a variable is unclear.

Personally I feel that non-extractable secrets is a safer bet, but willing to give current implementation a try and see how it will interact with multi-user / organization sharing.

But for now I suggest adding a (?) tooltip for the "Secret" checkbox and explaining the difference. Even if the difference is as small as "secrets will be masked in all outputs like task logs"

[sergiimk]

Issues after next round of testing:

1) Visual artifacts:

2) "Update now" button should use fetch_uncacheable: true

Before changing this we should also figure out why pressing this button now in our test scenario doesn't show the warning status.

3) (Not directly related but what I thought was fixed) "Add data" button should not be shown for root datasets with polling source defined

4) Another tricky scenario:

• dataset is uncacheable
• I enable scheduled updates (force: false)
• one tick passes - I get the warning
• I click "force update" link
• the next round of ingest was already scheduled, so my click just adds just an additional manual trigger
• next tick passess - dataset is still not updated and I get the warning

In other words, my manual request for force update did not effect the config of the flow that was already scheduled:

(I suggest to ticket up this problem but not address it immediately)

[sergiimk]

Blocker found: https://github.com/kamu-data/kamu-web-ui/issues/404

[sergiimk]

Acceptance passed with only one minor issue found - closing kamu-data/kamu-web-ui#427