chore: add auth to s3 destination test - Githubissues

Unstructured-IO / unstructured

Open source libraries and APIs to build custom preprocessing pipelines for labeling, training, or production machine learning pipelines.

https://www.unstructured.io/

Apache License 2.0

8.66k stars 707 forks source link

chore: add auth to s3 destination test #3122

Closed ryannikolaidis closed 4 months ago

ryannikolaidis commented 4 months ago

We should be validating the S3 Destination with authenticated requests, with credentials from a limited test user.

Changes

Updates s3 destination test to point to a bucket that requires authentication.
Adds authentication to the s3 destination test request
Bonus: fix deserialization of S3ConnectionConfig for s3 V2 destination
Bonus: fix S3ConnectionConfig never registered for s3 V2 destination
Bonus: repair version and changelog version for consistency with -dev convention

Testing

Validated by changes to S3 destination ingest test