[aws-codepipeline-actions] fetchSubmodules support for all Sources

jpSimkins commented 3 years ago

BitBucketSourceAction is required to setup a pipeline using BitBucket. It's easy enough to setup codestar to work with this for private repos. The issue I am running into, is that submodules are not being included in the source code. This makes it impossible to use the pipeline in it's current state.

Please add the ability to specify submodule in the action and ideally, make it match what is possible in the console. I cannot even choose the events I want to trigger either which is a big issue as well but one I have been dealing with for a while now.

Use Case

Submodules are a necessary part of all git projects I am working on. This limitation is making using pipelines extremely difficult because now the only solution is to use keys and checkout the repo in codebuild. This is not only redundant, it makes this action useless for anything other than a simple git project. Again, you cannot even control the events so you cannot stop building on every commit to the repo. No ability to add a nobuild comment or control the events you want to trigger the build.

Proposed Solution

Ideal solution would be to allow the same props as the Codebuild BitBucketSourceProps: https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-codebuild.BitBucketSourceProps.html

It seems you cannot use that for the pipeline with the CDK and that really stinks because it has everything I need...

Other

I am currently using:

          new BitBucketSourceAction({
            actionName: 'BitBucket',
            branch: this._props.git.branch,
            codeBuildCloneOutput: true,
            connectionArn: this._props.git.codeStartArn,
            output: this._artifactBaseSource,
            owner: this._props.git.owner,
            repo: this._props.git.repo,
          }),

This is what I used to use before I figured out this was not possible to use with code pipeline:

        const bbSource = codebuild.Source.bitBucket({
          branchOrRef: this._props.git.branch,
          cloneDepth: 1,
          fetchSubmodules: true,
          owner: this._props.git.owner,
          repo: this._props.git.repo,
          webhook: true,
          webhookFilters: [
            // Only on PUSH and PR Merged do we build (no support for message yet to allow for no builds)
            codebuild.FilterGroup.inEventOf(codebuild.EventAction.PUSH, codebuild.EventAction.PULL_REQUEST_MERGED).andBranchIs(GIT_REPO_BRANCH)
          ]
        });

The codebuild props allow more control. Ideally, I would prefer the ability to use that over the BitBucketSourceAction but this is not possible.

It should also be noted that codeBuildCloneOutput is not very clear on what it does. I cannot find any examples of how to use this properly so not really sure if I even need this.

I cannot select full clone for the repo even though it is an option when I go to edit the source in the pipeline. I bring this up because there is a work-a-round here: https://stackoverflow.com/a/64163023/650206

Basically, I cannot use CDK pipelines yet due to this limitation with the CDK. This seems like a pretty serious issue and moving from BitBucket is not an option. May need to go back to Terraform for this and that is not ideal.

[ ] :wave: I may be able to implement this feature request
[ ] :warning: This feature might incur a breaking change

This is a :rocket: Feature Request

skinny85 commented 3 years ago

Hey @jpSimkins ,

thanks for opening the issue. Unfortunately, I'm pretty sure this is a CodePipeline limitation, and there's not much CDK can do here. If you have premium support, I would suggest opening a feature request through there.

One trick I see a lot of customers use is to have a CodeBuild Project that watches the BitBucket repository with a webhook and fetchSubmodules: true, which then publishes the contents as an artifact to S3, and then a CodePipeline gets triggered from that S3 Bucket. A little roundabout, but should work.

Thanks, Adam

jpSimkins commented 3 years ago

Hello @skinny85,

I think this is a CDK issue because I can do this all manually without an issue. I just cannot do IaC for this using the CDK due to construct limitations. I already have a working prototype that I setup using the console. So the limitation seems to be with the CDK which may be due to CloudFortmation limitations.

I will go ahead and test out your method above. I just wanted to make note that this is not a limitation with CodePipeline as I already having this working when I build it VIA the console.

I do have premium support on a few accounts but I was not sure if they support the CDK as there are not support options there for CDK. Assuming you meant CodePipeline feature request, well this is already possible with CodePipeline.

One suggestion that would make this a lot easier:

BitBucketSourceAction does not allow the option to select Full Clone which is available when you go to edit the source action in the console. If I was able to use this, then it would be only a minor issue to get this to work using the link I posted in the initial report. At minimum, adding support for Full Clone would be enough to get this working for me. For now, I will just backup the construct and work on a new one to see if I can get this working.

Thanks for your time!

UPDATE: I checked my prototype and you are correct. The Source cannot do this in CodePipeline. I have CodeBuild configured this way. I will go ahead with your suggestion and contact support to ask for a feature request. I think I may have done this last year but I will do it again. Thanks again

skinny85 commented 3 years ago

@jpSimkins no problem, glad I could help 🙂

AliaksandrBadretdzinau commented 3 years ago

Hello @skinny85,

I think this is a CDK issue because I can do this all manually without an issue. I just cannot do IaC for this using the CDK due to construct limitations. I already have a working prototype that I setup using the console. So the limitation seems to be with the CDK which may be due to CloudFortmation limitations.

I will go ahead and test out your method above. I just wanted to make note that this is not a limitation with CodePipeline as I already having this working when I build it VIA the console.

I do have premium support on a few accounts but I was not sure if they support the CDK as there are not support options there for CDK. Assuming you meant CodePipeline feature request, well this is already possible with CodePipeline.

One suggestion that would make this a lot easier:

BitBucketSourceAction does not allow the option to select Full Clone which is available when you go to edit the source action in the console. If I was able to use this, then it would be only a minor issue to get this to work using the link I posted in the initial report. At minimum, adding support for Full Clone would be enough to get this working for me. For now, I will just backup the construct and work on a new one to see if I can get this working.

Thanks for your time!

UPDATE: I checked my prototype and you are correct. The Source cannot do this in CodePipeline. I have CodeBuild configured this way. I will go ahead with your suggestion and contact support to ask for a feature request. I think I may have done this last year but I will do it again. Thanks again

Is AWS CDK CodePipeline Source action supporting the "full clone" option now?

skinny85 commented 3 years ago

It does not yet, no. This is the feature request for it, and it's open 🙂.

luxaritas commented 3 years ago

Per my ill-formed issue above, this is also an issue for other Git sources, such as GitHub - perhaps worth updating the issue title?

skinny85 commented 3 years ago

Per my ill-formed issue above, this is also an issue for other Git sources, such as GitHub - perhaps worth updating the issue title?

Done! 😉

teuteuguy commented 3 years ago

Any updates on this ?

skinny85 commented 3 years ago

Still not supported by CodePipeline, so not much the CDK can do here for the time being.

lschierer commented 2 years ago

I am hitting this with cdk 2.10.0 with codecommit as the source repository.

lschierer commented 2 years ago

as a work around, in the commands section of the ShellStep, I included 'git submodule init' and 'git submodule update' after the 'npm run build' which seems to have worked. It is a bit fragile though, as it seems to not work if you put it before that command, even though it shouldn't matter.

mavogel commented 1 year ago

Thank you for the hint @lschierer :) However, I needed to do even more and re-initialize the git repo and submodules fully.

const repository = codecommit.Repository.fromRepositoryName(this, 'hugo-blog-repo', 'hugo-blog');
    const pipepline = new pipelines.CodePipeline(this, 'hugo-blog-pipeline', {
      synth: new pipelines.ShellStep('Synth', {
        input: pipelines.CodePipelineSource.codeCommit(repository, 'development'),
        // not implemented on 2022-12-28: https://github.com/aws/aws-cdk/issues/11399
        // so we clone submodules manually
        commands: [
          'npm ci',
          'git init && git add .',
          'git -c user.name="John Doe" -c user.email="jd@mail.org" commit -m"chore: init"',
          'rm -rf frontend/themes/blist/',
          'git submodule add https://github.com/apvarun/blist-hugo-theme.git frontend/themes/blist',
          'git submodule set-branch --branch v2.2.0 frontend/themes/blist',
          'npm run build',
          'npm run synth',
        ],
      }),
      dockerEnabledForSynth: true,
      codeBuildDefaults: {
        timeout: Duration.minutes(20),
      },
    });

Does anyone spot even more improvements on this? Feel free to let me know. (.. and the cloneSubmodules feature will be highly appreciated. I'll also open a support / feature request ticket for this)

Update and working example below: https://github.com/aws/aws-cdk/issues/11399#issuecomment-1367180696

lschierer commented 1 year ago

git submodule add and git submodule set-branch should only be necessary once each. That suggests at you have never run a git commit after git submodule add in your working directory on your local working copy. If you correctly configure the local working copy with the submodule committed, then when you push the repository to the master, future clones of the master repository need only do the git submodule init.

mavogel commented 1 year ago

That suggests at you have never run a git commit after git submodule add in your working directory on your local working copy.

Valid point, I double-checked and I did commit the submodule. What happened to me was that git submodule init did nothing as described here as well.

The problem is that in the synth step mentioned above there is no .git folder. I checked with ls -lash. This is why I re-initialized git via git init to be able to add the submodules.

Running command ls -lash
8.0K -rw-r--r-- 1 root root 4.2K Dec 28 23:00 .eslintrc.json
4.0K -rw-r--r-- 1 root root 582 Dec 28 23:00 .gitattributes
4.0K -rw-r--r--   1 root root  663 Dec 28 23:00 .gitignore
4.0K -rw-r--r--   1 root root  138 Dec 28 23:00 .gitmodules 
4.0K -rw-r--r-- 1 root root 324 Dec 28 23:00 .npmignore

...

Running command git submodule update --init --recursive
fatal: not a git repository (or any parent up to mount point /codebuild)

TL:DR working example

Thank you @lschierer for pointing me in the right direction. I got it working by cloning the repo instead of copying it from S3, which removes the .git folder.

const pipepline = new pipelines.CodePipeline(this, 'hugo-blog-pipeline', {
      synth: new pipelines.ShellStep('Synth', {
        input: pipelines.CodePipelineSource.codeCommit(repository, 'development', {
          codeBuildCloneOutput: true, // <= we need this to preserve the git history
        }),
        // not implemented on 2022-12-28: https://github.com/aws/aws-cdk/issues/11399
        // so we clone submodules manually
        commands: [
          'npm ci',
          'git submodule update --init',
          'npm run build',
          'npm run synth',
        ],
      }),
      dockerEnabledForSynth: true,
      codeBuildDefaults: {
        timeout: Duration.minutes(20),
      },
    });

aws / aws-cdk