Gitlab Support

[m-q-t]

Hi team,

Please find attached in this PR the following tasks which are responsible for supporting Gitlab:

• Two new entities have been created:

  • GitlabAccount
  • GitlabProject

• Each entity has their own respective enrichment task:

  • lib/tasks/enrich/gitlab_account.rb
  • lib/tasks/enrich/gitlab_project.rb

  lib/tasks/enrich/gitlab_project.rb follows a similar pattern to its GithubRepository counterpart in terms of creating metadata regarding the Gitlab Project.

• lib/tasks/gather_gitlab_projects.rb

  • This task will return a specified user's Gitlab Projects and accepts a GitlabAccount entity. This supports both SaaS and self-managed instances. The returned projects will be created as GitlabProject entities.

• lib/tasks/gitleaks.rb

  • This task has been modified to support GitlabProject entities. Similar to the task above it supports both SaaS and self-managed instances.

• lib/tasks/helpers/gitlab.rb

  • This contains all the helper methods that are associated with the Gitlab tasks. Found in here are the typical methods such as retrieving the access token from the task.json, verifying the access token exists, etc.

  However one interesting method is parse_gitlab_uri(). It appears to be far more common for organizations to have self-managed instances with Gitlab. As such, a technique was needed to be implemented that would be able to make the appropriate API calls to gather information. Luckily, the routes for the API calls are the same for all instances. Furthermore Gitlab has some quirky naming conventions such as being able to have subgroups within groups thus a project such as the following is valid: https://gitlab.intrigue.io/mgroup/random/one/two/abc/b/intrigue-core

  Gitlab Groups are analogous to Github Organizations.

  The parse_gitlab_uri() method was written to be able to deal with such quirky instances and it extracts the following information:

  • host
  • account (this could be a group containing subgroups)
  • project

  Running on the URI shown above, it returns:

  >> parse_gitlab_uri('https://gitlab.intrigue.io/mgroup/random/one/two/abc/b/intrigue-core')
  => {"host"=>"https://gitlab.intrigue.io", "account"=>"mgroup/random/one/two/abc/b", "project"=>"intrigue-core"}

  Furthermore a spec is provided to verify this method behaves as intended -> spec/integration/gitlab_parse_uri_spec.rb

• lib/tasks/search_searchcode.rb

  • As Gitlab only allows code to be searched across self-hosted instances (and requiring the instance to be integrated with ElasticSearch), it was decided to fall back on a 3rd party API to find any repositories containing instances of unique keywords on https://gitlab.com. Searchcode provides this service for Gitlab and several other repository hosting services (Github, Bitbucket, etc.) At the current moment this task will only search across the Gitlab results however its planned in the future to expand the scope of this task to support the different types of repository hosting services.

09/08/2021 Updates

• gather_gitlab_projects task was refactored and enhancements implemented to add additional exception handling such as checking whether rate limiting is in place, etc.

• spec/integration/gitlab_gather_projects.rb was added as a spec. This spec is not aimed to behave as a traditional spec but rather as a smoke test. Furthermore it has the added value in providing additional context for the different use cases covered which helps both the reviewer and myself. Will probably not be a good idea to run this spec in the pipeline due to various reasons such as maybe the amount of repositories a user has is subject to change (all the accounts belong to me) or the chance of someone registering an invalid account. I suggest the spec be removed once the PR is confirmed to be working.

Thank you.

Best regards, Maxim

[m-q-t]

Note: Setting this as a draft while I add the associated task to enable searching across Gitlab repositories.

[jcran]

This looks good to merge here.

[m-q-t]

Converting to draft as there's still some additional refactoring and testing to be done.

[m-q-t]

@shpendk PR is ready to go - updated the initial description to add more information regarding changes.