search

awslabs / aws-service-catalog-factory

This is a framework where you define a Service Catalog portfolio, products and versions using YAML. For versions of your products you specify where the source code for them can be found and the framework publishes the portfolio, products and versions in every* AWS Region after validating, linting and testing them.
Apache License 2.0
136 stars 44 forks source link

Help Defining Product Version from GitHub #269

Closed TannerIsBender closed 2 years ago

TannerIsBender commented 2 years ago

Hello,

I'm having issues adding a Product Version to my Portfolio and Product. When I do NOT include the version everything runs as expected. My portfolio is creating as well as the defined products. The issue happens when I try to add a product from GitHub. The error message points to line 77 in create_version_pipeline_task, however, its not clear to me which parameter is invalid. )

Steps to reproduce

The docs aren't clear to me on the configuration required for Github. The repo my-repo-name has a file in the root called ssm-automation_v1.template.yaml there is also an identical template file product.template.yaml

Here is my portfolios/ccoe.yaml

Schema: factory-2019-04-01
Portfolios:
  - DisplayName: shared-services-portfolio
    Description: A place for self service products ready for your account
    ProviderName: ccoe
    Associations:
      - arn:aws:iam::123456789012:role/servicecatalog-puppet/PuppetOrgRoleForExpands123456789012
    Tags:
    - Key: provider
      Value: ccoe
    Products:
      - Name: ssm-automation
        Owner: Tanner
        Description: The iam roles needed for you to do your jobs
        Distributor: ccoe
        SupportDescription: Contact us 
        SupportEmail: support@example.com
        SupportUrl: https://google.com
        Tags:
        - Key: Name
          Value: "Service Catalog Factory"
        - Key: custom:CostCenter
          Value: "4415"
        - Key: custom:ProjectName
          Value: "Service Catalog Factory"
        - Key: custom:Owner
          Value: "me@example.com"
        Versions:
          - Name: v1
            Description: The ssm automation self service application
            Source:
              Provider: Github
              Configuration:
                Owner: my-github-org
                Repo: my-repo-name
                Branch: main
                PollForSourceChanges: false

Expected results

The docs were not clear to me on how the CloudFormation is pulled from GitHub in the specified repo.

  1. The expected results is that a portfolio is created [WORKING]
  2. A product is created and shared to the portfolio [WORKING]
  3. v1 version pulled the CloudFormation template from the repo [NOT WORKING]

Actual results

Pipeline: servicecatalog-factory-pipeline

Name Last executed
servicecatalog-puppet-pipeline Succeeded
servicecatalog-factory-pipeline Failed 


===== Luigi Execution Summary =====

Scheduled 23 tasks of which:
* 22 ran successfully:
    - 4 AssociateProductWithPortfolioTask(...)
    - 4 CreatePortfolioAssociationTask(...)
    - 4 CreatePortfolioTask(...)
    - 4 CreateProductTask(...)
    - 1 CreateVersionPipelineTemplateTask(...)
    ...
* 1 failed:
    - 1 CreateVersionPipelineTask(...)

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

INFO MainThread 
===== Luigi Execution Summary =====

Scheduled 23 tasks of which:
* 22 ran successfully:
    - 4 AssociateProductWithPortfolioTask(...)
    - 4 CreatePortfolioAssociationTask(...)
    - 4 CreatePortfolioTask(...)
    - 4 CreateProductTask(...)
    - 1 CreateVersionPipelineTemplateTask(...)
    ...
* 1 failed:
    - 1 CreateVersionPipelineTask(...)

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

+---------+------------------------------------+----------------------------------------------------+---------------------+
| Result  | Task                               | Significant Parameters                             | Duration            |
+---------+------------------------------------+----------------------------------------------------+---------------------+
| Success | CreatePortfolioTask                | display_name: shared-services-portfolio            | 0.136979341506958   |
|         |                                    | portfolio_group_name: ccoe                         |                     |
|         |                                    | region: us-east-1                                  |                     |
|         |                                    |                                                    |                     |
| Success | CreatePortfolioTask                | display_name: shared-services-portfolio            | 0.2655336856842041  |
|         |                                    | portfolio_group_name: ccoe                         |                     |
|         |                                    | region: us-east-2                                  |                     |
|         |                                    |                                                    |                     |
| Success | GetBucketTask                      | region: us-east-1                                  | 0.23003101348876953 |
|         |                                    |                                                    |                     |
| Success | CreatePortfolioTask                | display_name: shared-services-portfolio            | 0.4723856449127197  |
|         |                                    | portfolio_group_name: ccoe                         |                     |
|         |                                    | region: us-west-2                                  |                     |
|         |                                    |                                                    |                     |
| Success | CreatePortfolioTask                | display_name: shared-services-portfolio            | 0.4981570243835449  |
|         |                                    | portfolio_group_name: ccoe                         |                     |
|         |                                    | region: us-west-1                                  |                     |
|         |                                    |                                                    |                     |
| Success | CreateProductTask                  | name: ssm-automation                               | 0.2301468849182129  |
|         |                                    | region: us-east-1                                  |                     |
|         |                                    | uid: ccoe-shared-services-portfolio-ssm-automation |                     |
|         |                                    |                                                    |                     |
| Success | CreateProductTask                  | name: ssm-automation                               | 0.288907527923584   |
|         |                                    | region: us-east-2                                  |                     |
|         |                                    | uid: ccoe-shared-services-portfolio-ssm-automation |                     |
|         |                                    |                                                    |                     |
| Success | CreatePortfolioAssociationTask     | display_name: shared-services-portfolio            | 0.6340596675872803  |
|         |                                    | portfolio_group_name: ccoe                         |                     |
|         |                                    | region: us-east-1                                  |                     |
|         |                                    |                                                    |                     |
| Success | AssociateProductWithPortfolioTask  | portfolio: ccoe-shared-services-portfolio          | 0.1951005458831787  |
|         |                                    | product: ssm-automation                            |                     |
|         |                                    | region: us-east-1                                  |                     |
|         |                                    |                                                    |                     |
| Success | AssociateProductWithPortfolioTask  | portfolio: ccoe-shared-services-portfolio          | 0.20890545845031738 |
|         |                                    | product: ssm-automation                            |                     |
|         |                                    | region: us-east-2                                  |                     |
|         |                                    |                                                    |                     |
| Success | EnsureProductVersionDetailsCorrect | product: ssm-automation                            | 0.11521220207214355 |
|         |                                    | region: us-east-1                                  |                     |
|         |                                    | version: v1                                        |                     |
|         |                                    |                                                    |                     |
| Success | CreateProductTask                  | name: ssm-automation                               | 0.529327392578125   |
|         |                                    | region: us-west-2                                  |                     |
|         |                                    | uid: ccoe-shared-services-portfolio-ssm-automation |                     |
|         |                                    |                                                    |                     |
| Success | EnsureProductVersionDetailsCorrect | product: ssm-automation                            | 0.14854907989501953 |
|         |                                    | region: us-east-2                                  |                     |
|         |                                    | version: v1                                        |                     |
|         |                                    |                                                    |                     |
| Success | CreateProductTask                  | name: ssm-automation                               | 0.5606236457824707  |
|         |                                    | region: us-west-1                                  |                     |
|         |                                    | uid: ccoe-shared-services-portfolio-ssm-automation |                     |
|         |                                    |                                                    |                     |
| Success | CreateVersionPipelineTemplateTask  | product: ssm-automation                            | 0.2816281318664551  |
|         |                                    | type: CloudFormation                               |                     |
|         |                                    | version: v1                                        |                     |
|         |                                    |                                                    |                     |
| Success | AssociateProductWithPortfolioTask  | portfolio: ccoe-shared-services-portfolio          | 0.40169191360473633 |
|         |                                    | product: ssm-automation                            |                     |
|         |                                    | region: us-west-2                                  |                     |
|         |                                    |                                                    |                     |
| Success | AssociateProductWithPortfolioTask  | portfolio: ccoe-shared-services-portfolio          | 0.42559289932250977 |
|         |                                    | product: ssm-automation                            |                     |
|         |                                    | region: us-west-1                                  |                     |
|         |                                    |                                                    |                     |
| Success | EnsureProductVersionDetailsCorrect | product: ssm-automation                            | 0.3400893211364746  |
|         |                                    | region: us-west-2                                  |                     |
|         |                                    | version: v1                                        |                     |
|         |                                    |                                                    |                     |
| Success | CreatePortfolioAssociationTask     | display_name: shared-services-portfolio            | 1.0064573287963867  |
|         |                                    | portfolio_group_name: ccoe                         |                     |
|         |                                    | region: us-west-1                                  |                     |
|         |                                    |                                                    |                     |
| Success | EnsureProductVersionDetailsCorrect | product: ssm-automation                            | 0.339397668838501   |
|         |                                    | region: us-west-1                                  |                     |
|         |                                    | version: v1                                        |                     |
|         |                                    |                                                    |                     |
| Success | CreatePortfolioAssociationTask     | display_name: shared-services-portfolio            | 0.5486633777618408  |
|         |                                    | portfolio_group_name: ccoe                         |                     |
|         |                                    | region: us-east-2                                  |                     |
|         |                                    |                                                    |                     |
| Success | CreatePortfolioAssociationTask     | display_name: shared-services-portfolio            | 0.9967682361602783  |
|         |                                    | portfolio_group_name: ccoe                         |                     |
|         |                                    | region: us-west-2                                  |                     |
|         |                                    |                                                    |                     |
+---------+------------------------------------+----------------------------------------------------+---------------------+
CreateVersionPipelineTask failed
parameters:
  all_regions:
  - us-east-1
  - us-west-1
  - us-east-2
  - us-west-2
  factory_version: 0.81.0
  product: <FrozenOrderedDict OrderedDict([('Name', 'ssm-automation'), ('Owner', 'Tanner'),
    ('Description', 'ssm-automation'), ('Distributor', 'ccoe'), ('SupportDescription',
    'Contact us on the CCOE Request Portal for help'), ('SupportEmail', 'support@example.com'),
    ('SupportUrl', 'https://google.com'),
    ('Tags', (<FrozenOrderedDict OrderedDict([('Key', 'Name'), ('Value', 'Service
    Catalog Factory')])>, <FrozenOrderedDict OrderedDict([('Key', 'custom:CostCenter'),
    ('Value', '4415')])>, <FrozenOrderedDict OrderedDict([('Key', 'custom:ProjectName'),
    ('Value', 'Service Catalog Factory')])>, <FrozenOrderedDict OrderedDict([('Key',
    'custom:Owner'), ('Value', 'me@example.com')])>)), ('Versions', (<FrozenOrderedDict
    OrderedDict([('Name', 'v1'), ('Description', 'v1'), ('Source', <FrozenOrderedDict
    OrderedDict([('Provider', 'Github'), ('Configuration', <FrozenOrderedDict OrderedDict([('Owner',
    'my-github-org'), ('Repo', 'my-repo-name'), ('Branch', 'main'), ('PollForSourceChanges',
    False)])>)])>)])>,))])>
  products_args_by_region: <FrozenOrderedDict OrderedDict([('us-east-1', <FrozenOrderedDict
    OrderedDict([('region', 'us-east-1'), ('name', 'ssm-automation'), ('owner', 'Tanner'),
    ('description', 'ssm-automation'), ('distributor', 'ccoe'), ('support_description',
    'Contact us on the CCOE Request Portal for help'), ('support_email', 'support@example.com'),
    ('support_url', 'https://gtechcorp.sharepoint.com/sites/CloudCenterofExcellence/SitePages/request.aspx?selected_option=services'),
    ('tags', (<FrozenOrderedDict OrderedDict([('Key', 'Name'), ('Value', 'Service
    Catalog Factory')])>, <FrozenOrderedDict OrderedDict([('Key', 'custom:CostCenter'),
    ('Value', '4415')])>, <FrozenOrderedDict OrderedDict([('Key', 'custom:ProjectName'),
    ('Value', 'Service Catalog Factory')])>, <FrozenOrderedDict OrderedDict([('Key',
    'custom:Owner'), ('Value', 'me@example.com')])>)), ('uid', 'ccoe-shared-services-portfolio-ssm-automation')])>),
    ('us-west-1', <FrozenOrderedDict OrderedDict([('region', 'us-west-1'), ('name',
    'ssm-automation'), ('owner', 'Tanner'), ('description', 'ssm-automation'), ('distributor',
    'ccoe'), ('support_description', 'Contact us on the CCOE Request Portal for help'),
    ('support_email', 'support@example.com'), ('support_url', 'https://gtechcorp.sharepoint.com/sites/CloudCenterofExcellence/SitePages/request.aspx?selected_option=services'),
    ('tags', (<FrozenOrderedDict OrderedDict([('Key', 'Name'), ('Value', 'Service
    Catalog Factory')])>, <FrozenOrderedDict OrderedDict([('Key', 'custom:CostCenter'),
    ('Value', '4415')])>, <FrozenOrderedDict OrderedDict([('Key', 'custom:ProjectName'),
    ('Value', 'Service Catalog Factory')])>, <FrozenOrderedDict OrderedDict([('Key',
    'custom:Owner'), ('Value', 'me@example.com')])>)), ('uid', 'ccoe-shared-services-portfolio-ssm-automation')])>),
    ('us-east-2', <FrozenOrderedDict OrderedDict([('region', 'us-east-2'), ('name',
    'ssm-automation'), ('owner', 'Tanner'), ('description', 'ssm-automation'), ('distributor',
    'ccoe'), ('support_description', 'Contact us on the CCOE Request Portal for help'),
    ('support_email', 'support@example.com'), ('support_url', 'https://gtechcorp.sharepoint.com/sites/CloudCenterofExcellence/SitePages/request.aspx?selected_option=services'),
    ('tags', (<FrozenOrderedDict OrderedDict([('Key', 'Name'), ('Value', 'Service
    Catalog Factory')])>, <FrozenOrderedDict OrderedDict([('Key', 'custom:CostCenter'),
    ('Value', '4415')])>, <FrozenOrderedDict OrderedDict([('Key', 'custom:ProjectName'),
    ('Value', 'Service Catalog Factory')])>, <FrozenOrderedDict OrderedDict([('Key',
    'custom:Owner'), ('Value', 'me@example.com')])>)), ('uid', 'ccoe-shared-services-portfolio-ssm-automation')])>),
    ('us-west-2', <FrozenOrderedDict OrderedDict([('region', 'us-west-2'), ('name',
    'ssm-automation'), ('owner', 'Tanner'), ('description', 'ssm-automation'), ('distributor',
    'ccoe'), ('support_description', 'Contact us on the CCOE Request Portal for help'),
    ('support_email', 'support@example.com'), ('support_url', 'https://gtechcorp.sharepoint.com/sites/CloudCenterofExcellence/SitePages/request.aspx?selected_option=services'),
    ('tags', (<FrozenOrderedDict OrderedDict([('Key', 'Name'), ('Value', 'Service
    Catalog Factory')])>, <FrozenOrderedDict OrderedDict([('Key', 'custom:CostCenter'),
    ('Value', '4415')])>, <FrozenOrderedDict OrderedDict([('Key', 'custom:ProjectName'),
    ('Value', 'Service Catalog Factory')])>, <FrozenOrderedDict OrderedDict([('Key',
    'custom:Owner'), ('Value', 'me@example.com')])>)), ('uid', 'ccoe-shared-services-portfolio-ssm-automation')])>)])>
  provisioner: <FrozenOrderedDict OrderedDict([('Type', 'CloudFormation')])>
  region: us-east-1
  tags:
  - <FrozenOrderedDict OrderedDict([('Key', 'Name'), ('Value', 'Service Catalog Factory')])>
  - <FrozenOrderedDict OrderedDict([('Key', 'custom:CostCenter'), ('Value', '4415')])>
  - <FrozenOrderedDict OrderedDict([('Key', 'custom:ProjectName'), ('Value', 'Service
    Catalog Factory')])>
  - <FrozenOrderedDict OrderedDict([('Key', 'custom:Owner'), ('Value', 'me@example.com')])>
  template: <FrozenOrderedDict OrderedDict()>
  version: <FrozenOrderedDict OrderedDict([('Name', 'v1'), ('Description', 'v1'),
    ('Source', <FrozenOrderedDict OrderedDict([('Provider', 'Github'), ('Configuration',
    <FrozenOrderedDict OrderedDict([('Owner', 'my-github-org'), ('Repo', 'my-repo-name'),
    ('Branch', 'main'), ('PollForSourceChanges', False)])>)])>)])>

Traceback (most recent call last):

  File "/root/.pyenv/versions/3.7.13/lib/python3.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()

  File "/root/.pyenv/versions/3.7.13/lib/python3.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()

  File "/root/.pyenv/versions/3.7.13/lib/python3.7/site-packages/servicecatalog_factory/workflow/portfolios/create_version_pipeline_task.py", line 77, in run
    StackName=friendly_uid, TemplateBody=template_contents, Tags=tags,

  File "/root/.pyenv/versions/3.7.13/lib/python3.7/site-packages/betterboto/cloudformation.py", line 53, in create_or_update
    self.create_stack(**kwargs)

  File "/root/.pyenv/versions/3.7.13/lib/python3.7/site-packages/botocore/client.py", line 388, in _api_call
    return self._make_api_call(operation_name, kwargs)

  File "/root/.pyenv/versions/3.7.13/lib/python3.7/site-packages/botocore/client.py", line 708, in _make_api_call
    raise error_class(parsed_response, operation_name)

botocore.exceptions.ClientError: An error occurred (ValidationError) when calling the CreateStack operation: Given input did not match expected format
``
TannerIsBender commented 2 years ago

The issue seems to be with referencing a template for a Product that uses GitHub as the source.

What is the correct way to setup product.template.yaml in a GitHub Repo?

Ultimately, I think my issue is getting the correct file path that hosts the version for the my-repo-name template.yaml during the CodeBuild servicecatalog-product-factory-build I've tried these files in the repo's root directory:

  1. product.template.yaml
  2. ssm-automation.template.yaml
  3. ssm-automation_v1.template.yaml
Versions:
          - Name: v1
            Description: The ssm automation self service application
            Source:
              Provider: Github
              Configuration:
                Owner: my-github-org
                Repo: my-repo-name
                Branch: main
                PollForSourceChanges: false
eamonnfaherty commented 2 years ago

When using Github we recommend using CodeStar pipelines: https://service-catalog-tools-workshop.com/every-day-use/100-creating-a-product/305-creating-codestar-pipelines.html

Let me know if this doc needs more details.

TannerIsBender commented 2 years ago

Thank you @eamonnfaherty that fixed my issue. I am now using GitHub via my Codestar connection.