Data connections

[cisaacstern]

Closes #32 #33

[cisaacstern]

TODO:

• [x] Fix test failures
• [x] Omit all clients/connections from get-params introspection
• [x] Add CLI subparser for keystore that lets us create keystores for a spec
• [x] Tests for this PR

[cisaacstern]

I have presented the proposed solution to the Pydantic community for feedback here: https://github.com/pydantic/pydantic/discussions/3205#discussioncomment-9799424

[cisaacstern]

I have now a draft of an interactive local "keystore" generator (i.e. .env file). Remaining TODO:

• [ ] programmatic keystore interface, for use in web app
• [ ] think about naming of this interface
• [ ] do we actually support local .env files not named ".env"?
• [x] fix failing tests
• [ ] docs

[cisaacstern]

@Yun-Wu @atmorling @walljcg, this is ready to go IMHO.

It's obviously a bit of a hard PR to review given its size and relative lack of documentation 🤭 . But it's also an important one, so before merging if you all want to / are able to give it a quick skim, perhaps something will jump out at you that you'd like to ask a question about.

I'll leave this open for now and move on to other work, and then we can merge early next week once anyone who'd like to has given it a quick read.

To clarify though, while I'm very open to criticism on this, and the possibility that I've overlooked something, I'm also quite confident in the behavior at this point, and would feel comfortable merging without review as well, if you happen to be busy with other things!

To list a few things here for our collective future reference:

• Currently connections config is only stored locally (or via env vars), but the pydantic settings framework gives us the option to fetch config from other sources as well (e.g., Google Secrets Manager, etc.). The code I've vendored here in _settings._gsm.py I found out about through this video, which is actually a really nice, short watch, if you want to see how that switching between local and remote settings looks IRL. Of course, once we get to that step, we also need to add our chosen remote storage (GSM or otherwise) as an optional backend to store connections config, i.e.:

  $ ecoscope-workflows connections create \
  --type earthranger --name $NAME \
  --store gsm 

• I am following the example of rclone, aws-cli, and other established projects to store credentials in clear text in a dot file in $HOME, but users can also opt out of that if they prefer to just use env vars (or mix and match), as shown in the tests.
• The only implemented connection is EarthRanger, but the general interface can support any connection that inherits from and implements the ecoscope_workflows.connections.DataConnection ABC.
• As we use more surface area on the earthranger client API (e.g. #29 #42), we'll just need to add the interfaces we use to this protocol, the purpose of which is to provide some static + runtime type checking of the EarthRanger client, without actually taking on a hard dependency on ecoscope
• The ecoscope-workflows connections create command runs interactively by default, but if the --fields option is used (which takes a json string), then it can run without interactivity. I am imagining we will dispatch to this command from the frontend when users want to create data connections (or when we want to auto-create EarthRanger connections for them based on SSO).

[Yun-Wu]

Thank you Charles! It looks great! Added some minor comments for clarification here.

[cisaacstern]

Thanks so much for the review Yun! I really appreciate it!! Left you responses inline above. 🙏

[atmorling]

I really like this implementation :partying_face: Do you see the gsm solution being tenable when we get to storing user configured data connections (and therefore potentially user credentials to those data connections) via ecoscope-web and then being retrieved wherever the task is executed?

[cisaacstern]

I really like this implementation 🥳 Do you see the gsm solution being tenable when we get to storing user configured data connections (and therefore potentially user credentials to those data connections) via ecoscope-web and then being retrieved wherever the task is executed?

Great question @atmorling. Short answer: I'm not sure. Slightly longer answer: the GSM solution here is at least a reference for how a GCP-based backed can be plugged into this PR eventually. At most, it's the actual solution we'd want to use, but that part I'm not sure about.

I've opened https://github.com/wildlife-dynamics/ecoscope-workflows/issues/48 to track this further, please add to the discussion there if you have further thoughts or if I haven't captured the questions correctly there!

[cisaacstern]

Thank you @Yun-Wu and @atmorling for the reviews!!