Write a project crawler

[marcingrzejszczak]

in order to properly generate pipelines we need to crawl through the most common git storages like

• github
• gitlab
• bitbucket

and allow hooking in whatever else you have. We need to allow

• logging in to that server's API via username / password / token
• print the list of repos for a given org
• print the contents of the sc-pipelines.yml or whatever the PIPELINE_DESCRIPTOR points to
• parse the file and basing on that build the proper pipeline

this will work really smoothly with Jenkins Job DSL and kind of well with jenkinsfile (for both we use a seed job) for concourse we need to access the concourse API somehow

GITHUB

Getting a list of repos

We used to call such code (worth adding authentication to it)

def organization = binding.variables['ORGANIZATION_NAME'] ?: 'microhackathon-test'
def reposApi = new URL("https://api.github.com/orgs/${organization}/repos")

Getting the file

we can reuse the same abstraction we have in release tools (https://github.com/spring-cloud/spring-cloud-release-tools/blob/master/spring-cloud-release-tools-core/src/main/java/org/springframework/cloud/release/internal/git/GithubMilestones.java)

boolean descriptorExists = new RtGithub("user", "pass").repos().get(new Coordinates.Simple("org", "repo")).contents().exists("sc-pipelines.yml", "master");
if (descriptorExists) {
String descriptorContents = new java.util.Scanner(new RtGithub("user", "pass").repos().get(new Coordinates.Simple("org", "repo")).contents().get("sc-pipelines.yml").raw())).useDelimiter("\\A").next()
}

possible parameterizations:

• PIPELINE_DESCRIPTOR
• APPLICATION_BRANCH
• GIT_USERNAME
• GIT_PASSWORD
• GIT_TOKEN