ArroyoSystems / arroyo

Distributed stream processing engine in Rust
https://arroyo.dev
Apache License 2.0
3.45k stars 189 forks source link

Allow env var substitution in UDF dependencies #608

Closed benjamin-awd closed 2 months ago

benjamin-awd commented 2 months ago

This PR adds support for environment variable substitution in UDF dependencies, which allows private GitHub repositories to be cloned using Rust's native git library.

For example,

[dependencies]
my-repo = { git = "https://{{ GITHUB_USER }}:{{ GITHUB_TOKEN }}@github.com/{{ GITHUB_ORG }}/my-repo.git" }

will resolve to something like:

[dependencies]
my-repo = { git = "https://user:ghp123@github.com/my-github-org/my-repo.git" }

This allows UDFs to be stored in version control without exposing any credentials, and avoids the need to install the Git CLI.