Discussion on adding custom resources types to manage tokens, users, groups, projects

[tuxmea]

I am considering adding custom types and providers to manage gitlab internal configs.

What is the best way to interact from a custom type/provider with gitlab?

At the moment I see two ways:

1. executing gitlab-rails runner
2. communicate with api

gitlab-rails command

The gitlab-rails command can be run as root user. No further authentication required. This allows to check existing and add new tokens, users, groups, projects.

But: the command is very slow.

api

Communication with api needs a token. root token can be managed using gitlab-rails only. other user tokens can be managed using the root token.

all other resource types will need an autorequire on the gitlab_token type.

But: where to store the root api token itself so it can be used by all other resource types? In puppet cache dir? or is it possible to pass a variable from one provider to another one without again running the gitlab-rails command?

[bastelfreak]

@baurmatt can you take a look at well? Do you have any opinions on that topic?

[baurmatt]

I've never really used the gitlab-rails runner command so I had to check it out on our system:

$ time gitlab-rails runner 'puts Rails.env'
production

real    0m44.396s
user    0m36.290s
sys 0m5.594s

This is our production Gitlab (8 CPU/32GB RAM). To be honest, I don't see how this can be successfully integrated. Even if all resource creation requests would share one gitlab-rails runner session this would be to long IMHO.

So from my point of view the only option is the API - even thought that the API token handling might be a manual step.

Regarding the storage location of the token: I don't like to put relevant things in the cache dir. I would prefer to just store it in /etc/gitlab with the restrictive permissions. This might be "ok" because an attacker who can access this file will also be able to just use gitlab-rails runner/console. From a design perspective it would be also nice to store the token in the PuppetDB because in a "classic" web setup this would be the place where state is persistent. Though I don't know how to implement this.

[tuxmea]

@baurmatt many thanks for your feedback.

How about the following idea: gitlab_token type/provider check if the token file exists in /etc/gitlab/. If the file exists it does not do anything. If the file is missing, the token will be created using gitlab-rails runner command and written to the file.

This will allow us to do a complete automated setup.

Usage of the gitlab_token type will be optional. All other types/providers use the token file and API calls.

[baurmatt]

Sounds good! Looking forward to the implementation! :)

[tuxmea]

@baurmatt List of types and properties:

1. gitlab_token:
   • newproperty: user - String
   • newproperty: key - String
   • newproperty: scope - Array[String]
2. gitlab_user
   • newproperty: username - String
   • newproperty: realname - String
   • newproperty: password (optional) - String
   • newproperty: confirmed - Boolean
   • newproperty email - String
3. gitlab_group
   • newproperty: name - Strig
   • newproperty: member_role - Hash (user: role mapping)
4. gitlab_project
   • newproperty: name - String
   • newproperty: group - String
   • newproperty: origin_url - Optional[String]

[baurmatt]

@tuxmea You probably want email - String for gitlab_user as well.

Beside this it looks like a good start! 👍

[alexjfisher]

If using the API, could these types be used to manage projects/groups etc. on gitlab.com ? That might be useful to some people? (In which case you probably don't want to implement instances?? and you shouldn't expect gitlab-rails command to be available?)

For the api settings file, (assuming you don't want to make the credentials parameters of each of the individual resources), you can do confine :exists => '/path/to/api_settings.yaml' in the providers and all types can also have

autorequire(:file) do
 '/path/to/api_settings.yaml'
end

I'd then probably provide a class that users can use to create this file. eg. the user would do something like...

class { 'gitlab::api_settings':
  server => 'https://gitlab.com',
  token => Sensitive($my_api_token),
}

gitlab_project { 'my_project':
  group => 'some/group',
}

# etc.

(With the gitlab_group type, dunno if groups need to know about their parent groups - perhaps so they can autorequire them?)

Finally, I've used the gitlab gem before with good results. Maybe the provider should make use of this? (The module could offer to install it and the providers can confine themselves based on whether it's installed yet)

[tuxmea]

@alexjfisher sounds like a good idea. But: where can we receive the API token from? Or how to set the token explicitly? Using gitlab gem also came into my mind. But the same question: how to set or generate the API token?

[baurmatt]

The token needs to be provided manually. If you don't want this, you could automate it with a local command as described here. This token would than be saved to e.g. /etc/gitlab/$user-token and loaded by the provider.

[alexjfisher]

I guess for use with gitlab.com it would have to be manually provided, but if using the module to deploy your own gitlab installation, then being able to automatically generate it would be better.

[baurmatt]

Is there an advantage to using these types/providers instead of the official Gitlab Terraform Provider on gitlab.com? While thinking about it... Is it worth developing custom types/provider at all? What would be the benefit?