Feature Request: terraform provider #32

Closed whoracle closed 7 months ago

Are there any plans for a terraform provider for terrareg?

I am currently debating whether to deploy terrareg in my org to move away from our current GitLab module registry.

Since we create our module repositories via terraform anyways, it would be nice to shine up our repo modules by automatically creating the corresponding terrareg module and the namespaces along with our gitlab groups.

If this is not currently in the pipeline, would you be amendable to support me if I find the time to create a provider for this?

Hey :)

Thank you for the suggestion - it's a really good idea. I've created an upstream issue: https://gitlab.dockstudios.co.uk/pub/terrareg/-/issues/486

The only concern I'd have is around authentication - the only static authentication is the admin password. Otherwise, finding a way to handle SSO and/or API keys. If you have any suggestions for this, it would be great.

Initially, I could only imagine it being able to create namespaces and modules - not sure how much other functionality could be handled via Terraform

Hey :)

Initially modules and namespaces should be enough, I agree. Next step would be providers, and as the cherry on top I'd say terrareg config like OIDC and user mgmt, but that can come WAY later IMHO

As for authentication I'd go with more or less the same method as with the PUBLISH_KEY and the UPLOAD_KEY. For a PoC creating a static token is enough, next iteration would be to nail some kind of auth token to a user. If I remember correctly from the last time I implemented OAUTH (yes, 1, not 2/OIDC) in flask, this was relatively simple - terrareg would of course need to keep track of the token in the DB, but since you already have a Role/Permission model in place (with OIDC compat, even), this should not be too hard:

On API calls, check for the Header X-Terrareg-Auth-Token and then run through the same auth code as e.g. the upload token with X-TerraReg-ApiKey.

I'm a bit unsure about how one would match a token to a user, though, especially since the token should not be saved plain text in the DB and is thus not recoverable. But I'm sure there are industry best practises. I'll see if I can find some code to steal in one of the other SSO-Enabled and terraformable Flask Apps I use.

Is it OK if we keep talking here, or do you want me to make an account on upstream for the further discussion? I'd prefer here for the visibility if someone searches for the same feature tbh.

Hey :D

Initially modules and namespaces should be enough, I agree. Next step would be providers, and as the cherry on top I'd say terrareg config like OIDC and user mgmt, but that can come WAY later IMHO

Cool - sounds like a good first step. Yes, providers... that's certainly in early days and needs properly dealing with anyway (the "feature", irelevant of the actual terraform provider for providers :D )

As for authentication I'd go with more or less the same method as with the PUBLISH_KEY and the UPLOAD_KEY. For a PoC creating a static token is enough

Sounds good!

next iteration would be to nail some kind of auth token to a user. If I remember correctly from the last time I implemented OAUTH (yes, 1, not 2/OIDC) in flask, this was relatively simple - terrareg would of course need to keep track of the token in the DB, but since you already have a Role/Permission model in place (with OIDC compat, even), this should not be too hard:

Yep - just, at the moment, we don't hold any user information for SSO users, except audit logs (not that they're relevant) and sessions. Having said that, we do have the Terraform IDP now - I wonder if we could do something with that and re-use that for API authentication :thinking:

On API calls, check for the Header X-Terrareg-Auth-Token and then run through the same auth code as e.g. the upload token with X-TerraReg-ApiKey.

Yep, I wonder if re-using the pre-existing X-Terrareg-ApiKey, rathern than switching to a new X-Terrareg-Auth-Token header?

I'm a bit unsure about how one would match a token to a user, though, especially since the token should not be saved plain text in the DB and is thus not recoverable.

I'm assume it could be safely hashes to achieve this? I assume it doesn't need to be recoverable and could be temporary (at least have an expiration). Though lots of applications do use API keys and API secrets (one being identifiable and, I assume, the other being hashed or such, so it's not recoverable!

Is it OK if we keep talking here, or do you want me to make an account on upstream for the further discussion? I'd prefer here for the visibility if someone searches for the same feature tbh.

Sure - completely up to you :)

Hey :D

Cool - sounds like a good first step. Yes, providers... that's certainly in early days and needs properly dealing with anyway (the "feature", irelevant of the actual terraform provider for providers :D )

Yep - just, at the moment, we don't hold any user information for SSO users, except audit logs (not that they're relevant) and sessions.

We would definitely need to store SSO user info in the DB - at the very least an identifier coupled with an uniqoe SSO identifier. That can of course just be a numeric ID and an SSO UUID, so no actual compromisable userdata gets stored.

Having said that, we do have the Terraform IDP now - I wonder if we could do something with that and re-use that for API authentication 🤔

I know precisely zilch about the Terraform IdP, but it's at least somewhere to start looking, even if it isn't completely reusable. I'd personally not get too attached to that, though, if only for the simple reason that that IdP is bound to the TF Registry API Spec (as far as I know), and you want to decouple your logic from that. Else they'll release, say, tf-registry-api v5, and change fundamental things, and BAM your tf provider and the whole auth token stack won't be usable anymore.

Yep, I wonder if re-using the pre-existing X-Terrareg-ApiKey, rathern than switching to a new X-Terrareg-Auth-Token header?

I wouldn't worry too much about the header name :) just look at how that logic is implemented, slap a corresponding decorator in front of the other API endpoints and call it v0.1. Afterwards, you can branch out into different headers for different use cases, but I'd personally take a hard look at the token logic and rework that: Instead of having a PUBLISH and an UPLOAD and an API token, have a TOKEN that has the roles/permissions publish, upload and orapi (and possibly admin so admin users could create tokens that are for "regular" API calls, but that may be too complicated).

That way you can have static tokens in the first step that are properly scoped, and that would not lead to breaking changes. After that you can look at how to nail the tokens to the users.

I'm assume it could be safely hashes to achieve this? I assume it doesn't need to be recoverable and could be temporary (at least have an expiration). Though lots of applications do use API keys and API secrets (one being identifiable and, I assume, the other being hashed or such, so it's not recoverable!

Right, API key IDs. Totally forgot about that. So an http header would look like X-Terrareg-Token: 0xDEADBEEF X-Terrareg-TokenID: 12345 for example.

Sure - completely up to you :)

Then I'll stay here for now, until/unless I start contributing Code :)

Hey @whoracle ,

I've begun working on this. Unfortunately it's taken a little longer as I'm trying to use the "new" framework (rather than SDK v2) to build it, which I haven't used before. Currently have the boilerplate there, so things should speed up now (currently able to create a namespace). Progress can be seen here: https://gitlab.dockstudios.co.uk/pub/terra/terraform-provider-terrareg/-/merge_requests/1

Thanks Matt

Way faster than I would have expected :)

If you want me to test something, let me know.

The namespace is now working - should be able to create, update, destroy and supports name/display name.

Auth is just the Admin token for now - looking to implement as much as possible without Terrareg changes - then can design and test/implement those and update the provider to reflect them (hopefully that seems sensible).

Currently working on trying to get unit tests working - I've written a couple of providers before, but, embarrassingly, never written tests for them :eyes:

Feel free to pull it down, if you do fancy giving it a try. The README should outline a way to use it - though, worth noting, a terraform init will fail (due to the provider not being published to Hashicorp's registry yet), but you can ignore the error and simply run a plan/apply after and it should work (as long as the built provider binary is in the directory, as specified in the updated .terraformrc file)

Hey @whoracle

I've created a base provider now and the latest release has:

namespace resource
module resource
git providers data source
git provider data source

The code is at:

and published at:

https://registry.terraform.io/providers/DockStudios/terrareg/latest

I'll close this issue now, if that's okay.

Maybe new issues can be created for:

Changes to authentication (API keys - probably needs a Terrareg ticket and then provider ticket after to support the outcome)
New resources, data sources (e.g. user groups and user group mappings)

Many thanks Matt

Update: For the API keys, I've created https://gitlab.dockstudios.co.uk/pub/terrareg/-/issues/488 and https://gitlab.dockstudios.co.uk/pub/terrareg/-/issues/489 - I suspect 489 could be more beneficial thanks to the Terraform provider, so maybe 488 could be ignored?

Created gitlab issue: https://gitlab.dockstudios.co.uk/pub/terrareg/-/issues/506 gitlab-issue-id:506

Quick feedback: Due to Life getting in the way I won't be having time to test this for the next few weeks or months, but I still have this tracked and will get back to you. Don't want to be that guy(tm) ;)

Hey @whoracle

No issue at all! I hope all is well! No rush at all - apologies - I spent this morning writing a script to keep issues in-sync with Gitlab (because I was losing track of cross-posting comments), so the comment was from a script that ran across all issues :)

Many thanks Matt

MatthewJohn / terrareg

Feature Request: terraform provider #32