Feature Request: s3 as storage and serverless DB

[Eli-Meitner]

Is there any way (or future plan) to use S3 as storage for modules/providers ? And any plans to support serveless DB suchas DynamoDB or DocumentDB ? This is very important cost wise.

[MatthewJohn]

Hey :)

Whilst I could foresee the storage being pushed to s3, I feel (at least from a majority of use-cases that I've seen), people are generally using Git for modules, which requires no storage, (and Github for the storage of providers is still a requirement) - though I wouldn't too be adverse at supporting s3.

DynamoDB, on the other hand, would affect a lot of the code-base (if anything, the ORM system that we use doesn't support it). Porting the search functionality would be very difficult and, as a base rule of thumb, I prefer to err away from having requirements on a cloud provider (whilst s3 is quite different, in that minio does a great job at replicating s3 for local development). Though, DocumentDB (and mongo) support would solve the cloud provider requirement.

AWS RDS aurora serverless v2 can be run serverless and cost (if running 24/7) at $1051.2/yr. RDS db.t4g.micro will run at $148.92/yr. So if you're looking for less than this, then maybe an all-in-one solution on another provider might be best (running MySQL locally with terrareg in a docker-compose setup)? Or the default SQLite support?

[Eli-Meitner]

Thank you for the quick response, I was sure I needed storage to store the modules (so I can fully migrate away from git), as for providers is there a plan to migrate them as well (right now from documentation it was not quite clear on how to add them to Terrareg)

[MatthewJohn]

Hey @Eli-Meitner

Yep, that's correct, if you do not wish to use Git for the modules, then they will need to be stored. I expect the interaction with s3 should be fairly simple to implement.

Out of interest, when you say about moving away from git, do you mean for the actual source code or just to serve Terraform from the registry without using git? (Not sure if you mean this scenario, but have created a ticket for being able to "index" from Git and serve the modules from local storage).

In terms of provider support, it is really not a production-ready function, so I wouldn't recommend using it (it was only really merged to main to avoid horrendous merge issues!). But providers work with a tight Github integration (following a similar approach to Hashicorp's registry) and the binaries are not stored by Terrareg (at least, this isn't an option at the moment) and Terraform downloads it straight from Github, so no storage is required on the Terrareg side (this does mean that private providers are not yet supported, as well). Update: Actually, I realise that we do store the provider binaries locally, but they are not currently used.

Yeh, I kind of realise the documentation for providers isn't very clear, and it's somewhere between not wanting to advertise this feature too much and also expecting that it might change a bit in the future, as the feature is improved :roll_eyes:

[Eli-Meitner]

Our main objective is to move to a private registry so we won't have to expose our modules (and their code) publicly, as for provider we are OK with it as we have only one that we use for something very specific so we probably can move away from it. our objective is either moving the modules to private repo in bitbucket or keeping them in github but switching to private repo, once again thanks for the answer :-)

[MatthewJohn]

Hey

That makes sense - are you expecting the users of the modules to have access to the private repository - if so, there's no stopping you using the git repository in Terrareg - all that happens is that it provides the user with a link to the repo during a Terraform init and, assuming the user is able to git clone the repos, Terraform will clone the repo like normal, using the user's SSH key (or basic auth over HTTPS).

In our setup, we only use Terrareg with private git repositories :) (If you specifically want to not use git for the modules, then no problem - just wanted to make sure you're aware :) )

Either way, I'll try to start looking at the s3 support some time this week and see how it goes.

[Eli-Meitner]

Thanks

[MatthewJohn]

Hey @Eli-Meitner ,

Just to keep you updated - I've got a change prepared for adding S3 support: https://gitlab.dockstudios.co.uk/pub/terrareg/-/merge_requests/380

Just doing some final checks and trying to fix a CI configuration problem

Thanks Matt

[MatthewJohn]

Hey @Eli-Meitner ,

The support for s3 was released in v3.1.0

Thanks Matt

[MatthewJohn]

@Eli-Meitner

Since S3 support has been added and, unfortunately, the realistic possibility of supporting both SQL and Dynamodb/DocumentDB is going to be low, are you happy with this issue being closed?

Matt

[MatthewJohn]

Created gitlab issue: https://gitlab.dockstudios.co.uk/pub/terrareg/-/issues/507 gitlab-issue-id:507