Monitor bitbucket project/repo - auto create build plans

[sbhakta99]

Can this plugin monitor a Bitbucket Project, so whenever a new repo is checked in, it will auto create all build plans and deployment plans relative to that project?

We want to have Bamboo DSL job scan a Bitbucket Project for new repos, once a new repo is checked it we want DSL job to run to create that repos Bamboo Plans.

[mrueegg]

Hi,

Thanks for your feature request.

This is currently not possible, but I think it would be a very interesting feature. There is one question: which DSL file would you like to use for a new repository of a Bitbucket project? Would you like to configure a specific DSL file to use for every new repository of a project ?

I also think that the DSL plug-in would need to export some build variables so that you are able to configure your repository to your liking, e.g. bamboo.repository.url. What do you think?

Best regards, Michael

[sbhakta99]

Ideally we want to use concept of Pipeline as Code Table of Contents by Jenkins https://jenkins.io/doc/book/pipeline-as-code/ We are trying to figure how to do that in bamboo, we think that code that scans the project should go in the seed job.

So we would have one bamboo dsl job per bitbucket project type. So example if Bitbucket Project was tomcat, all application repos for tomcat would go under that. Assuming everyone is following some code structure to build their code then that Bamboo DSL job would create plans with that new repo name etc.

We've been trying to use API calls but not sure if that is supported.

Example: import groovyx.net.http.HTTPBuilder import groovyx.net.http.Method

def http = new HTTPBuilder('https://bitbucket.company.com') authSite.auth.basic 'bamboouser', 'bamboouserpassword' def html = http.get(path : '/rest/api/1.0/projects/OURPROJECTTYPE/repos', query : [q:'slug']) println(html)

[mrueegg]

That makes perfectly sense. I could think of a new feature for the plug-in where you could specify which project to watch for (e.g., "Tomcat") in a DSL seed task and the plug-in will trigger the job where this task is located whenever a new repository in this project is detected. We should also export some build variables like NEW_REPO_NAME so that the DSL can process it. What do you think?

[sbhakta99]

Yes we agree with this. So we are trying to standardize our applications by technology. So any applications which sit on Tomcat Platform they would get put under "app_tomcat". The repo name would obviously just be whatever that application is. For NodeJs Platform it would be "app_nodejs". For Weblogic it would be "app_weblogic", etc. As application teams checkin their repo into those projects "app_tomcat" or "app_nodejs" the DSL job should pickup the new repo and process its Build Plans accordingly.

[sbhakta99]

Would you guys be able to fulfill this in your next release?

[muchino]

Hi together just saw now you conservation and as an heavy plandsl user I like to put my two cents to this conversation. We have over 300 bitbucket repos and all configured by one single plan dsl plan (with switches within configuration.) This one plan dsl config is running every 3 hours and configures the existing (maybe changes) and new repos from bitbucket.

What we are doing is to use the Bitbucket API and loop over all repos. depending on the content we decide what we do. As you can handle a repo information as json object this is pretty easy :)

getting the repos and configure every single one could be dine by this groovy script:

def authString = MyUtility.getAuth(env(BambooGlobalVariables.GIT_USER), env(BambooGlobalVariables.GIT_PASSWORD))
def connection = BitbucketAPI.getBitbucketRestAPIResponse("/projects?limit=2000", authString)
if (connection.responseCode == 200) {

    def jsonProjects = MyUtility.parseJSONConnection(connection)
    jsonProjects.values.each { projectRepo ->

        if (projectRepo.key != "BVMPSERV") {
            println("Found projet Plan for ${projectRepo.name} - ${projectRepo.key}")
            def connectionProject = BitbucketAPI.getBitbucketRestAPIResponse("/projects/${projectRepo.key}/repos?limit=2000", authString)
            if (connectionProject.responseCode == 200) {

                def json = MyUtility.parseJSONConnection(connectionProject)
                json.values.each { repo ->

                    if (BitbucketAPI.isMavenRepository(repo.project.key, repo.slug, authString)) {

                        PlanInformation planInformation = MyUtility.getProjectInformation(repo)
                        println("Create Plan for ${repo.project.name} - ${repo.name}")
                        def project = project(key: "${planInformation.projectKey}", name: planInformation.projectName) {
                        }
                        MyProjects.createAtlassianAddonPlan(project, planInformation, repo, authString)
                    }
                }
            }
        }
    }
}

and here some special utils and other useful stuff (we still have everything as lib: jobs, tasks, project types, deployments...)

static def HttpURLConnection getBitbucketRestAPIResponse(String url, String authString) {
    return getBitbucketResponse(BambooSettings.GIT_REST_ENDPOINT + url, authString)

}

static def HttpURLConnection getBitbucketResponse(String url, String authString) {

    def connection = new URL(url)
            .openConnection() as HttpURLConnection

// set some headers

    connection.setRequestProperty('User-Agent', 'groovy-2.4.4')
    // connection.setRequestProperty('Accept', 'application/json')
    connection.setRequestProperty("Authorization", "Basic " + authString)
    return connection

}

static def Boolean isMavenRepository(String projectKey, String reploSlug, String authString) {
    def connection = BitbucketAPI.getBitbucketRestAPIResponse("/projects/" + projectKey + "/repos/" + reploSlug + "/browse/pom.xml", authString);
    return connection.responseCode == 200
}

static def Object getAllBranches(String projectKey, String reploSlug, String authString) {

    def connection = BitbucketAPI.getBitbucketRestAPIResponse("/projects/" + projectKey + "/repos/" + reploSlug + "/branches", authString);
    if (connection.responseCode == 200) {
        return MyUtility.parseJSONConnection(connection);
    }
    return null
}

static def Object parseJSONConnection(HttpURLConnection connection) {
    // get the JSON response
    return connection.inputStream.withCloseable { inStream ->
        new JsonSlurper().parse(inStream as InputStream)
    }
}

static def GPathResult parseXMLConnection(HttpURLConnection connection) {
    // get the XML response

    return new XmlSlurper().parseText(connection.content.text)

}

static def String getAuth(String username, String password) {
    String auth = username + ":" + password
    return auth.getBytes().encodeBase64().toString()
}

Btw Michael if you see some ugly or old stuff in our code, just give me an hint ;) The idea for some support by the add-on is cool, but just "there is a new project" is in our case not helpful, because we adapt the plans every month to extend them and sometimes smth in the repo changed, influencing the plan. This is the reason why we loop over all repos every time.

We moved to a clean bamboo instance 1 month ago and it was pretty easy to get everything moved with this scripts, everything is scripted in our instance - thanks to this add-on

Thank you Mibex :)

[mrueegg]

@sbhakta99 While we have several ideas and plans how to move the plug-in forward in terms of pipeline as code, nothing is concrete yet and I cannot commit to any releases at this point. I'll let you know as soon as I know more. Thanks in advance for your patience!

@muchino You're very welcome and thanks a lot for sharing your code samples. I think they are very useful for others as well. Nothing I could spot that I would see a necessity to change :-)

[sbhakta99]

Hello, was there ever and update or solution for this?

[mrueegg]

@sbhakta99 Hi. Please excuse my late response. No, there was no new release of the app targeting this feature request. We still have this on our roadmap, but it is not something we can provide in the very near future. Thank you for your understanding.