latitude-dev / latitude-llm

Work in progress

37 stars 3 forks source link

Infra for setting S3 bucket for storing datasets #147

Closed andresgutgon closed 1 week ago

andresgutgon commented 1 week ago

What?

Set S3 bucket for storing datasets in latitude workspaces

TODO

[x] Setup Pulumi config and AWS secrets for AWS access and secret
[x] ~~Setup S3 Policy in bucket~~ Not necessary. User from AWS_ACCESS_KEY already has these permissions.
[x] Setup bucket in the eu-central-1 region
[x] Run pulumi up to create the bucket
[x] Ask Gerard how we can avoid removing the bucket even if we run pulumi destroy

Next PR

[ ] Test the files are uploaded to S3 in development by setting the Driver as S3. Not working doing the fix in another PR to merge the Pulumi state