Closed aaronsteers closed 2 months ago
Quick update. I've added create+delete tests for a source and destination. Both tests are passing. π
This isn't the final API, but you can see how it works in the below excerpt:
@bindipankhudi - FYI, I just cleaned up the docs a bit more and put a link to a recent docs snapshot in the PR description.
I found it very helpful to review the docs and you might find so also (especially the secrets
and cloud
submodules):
Resolves: https://github.com/airbytehq/PyAirbyte/issues/33 Resolves: https://github.com/airbytehq/PyAirbyte/issues/104 Resolves: #54
What's new in the
secrets
modulesecrets
module. The previoussecrets
code is moved from a singlesecrets.py
file, to submodules per topic/retriever. Most of the core code is unchanged, except...SecretSource
toSecretSourceEnum
, to better distinguish from the newSecretManager
classes.SecretString
- inherits fromstr
, but adds some security measures:repr()
will be masked. This includes printing a class or a dict that may contain secret properties.parse_json()
implementation streamlines cast todict
(a frequent user case), and ensures the exceptions don't inadvertently print the value.SecretManager
- Has theget_secret(<name>)
method.CustomSecretManager
- Abstract base class for custom implementations. Custom secret managers have the additional behavior to register themselves into the list of secrets sources that are automatically checked whenab.get_secret()
is called.SecretHandle
- A pointer to a (not-yet-retrieved) secret. Allow streamlined iteration over a list of secrets and secret names without the cost/risk of retrieving locally.GoogleGSMSecretManager
class.ci_credentials
. It inherits fromSecretManager
and also addsfetch_secrets
,fetch_secrets_by_label
andfetch_connector_secrets
.fetch_connector_secrets
code is what we use in integration tests to replaceci_credentials
dependency`. This can now also be used at runtime to test any source, without requiring an import of any external libraries.The above changes are performed in a backwards compatible way, so that
ab.get_secret()
still works just as before.What's new in the
exceptions
moduleAll existing exceptions are the same except that
AirbyteLib
prefix is replaced byPyAirbyte
and more care has been placed to ensure a distinction betweenAirbyte*
exceptions (protocol) andPyAirbyte*
exceptions (Python-specific). A group of newAirbyteCloud
exceptions (from Cloud) have also been added with their own prefix.Note:
PyAirbyte*
has caused a large number of files to show as modified. (Sorry! π)Reviewing Docs
I would highly recommend reviewing the autogenerated docs for both the
secrest
module and thecloud
module. Those can be found fromActions
here in github, as a downloadable artifact from thedocs-generate
job.Here is a recent copy:
generated-docs.zip
To use, simply download the zip, double-click to decompress, then double-click to open the included "index.html" file.
Pre-Merge TODO
experimental
status if we don't need them - specifically theCloud*
classes.deploy*()
anddelete*()
methods.Optionally merge thePin to a specific commit instead of a branch ref.airbyte-api
rename in the other repo - or else