[feature] allow versioning via S3 paths

[AndreiBarbuOz]

Describe the Feature Allow taskcat upload to handle subpaths in the .project.name key would enable versioning of templates via subpaths.

aws-quickstart-project
├── v1.1
│   ├── templates
│   │   ├── main.template.yaml
│   │   └── nested1.template.yaml
│   └── scripts
└── v1.2
    ├── templates
    │   ├── main.template.yaml
    │   └── nested1.template.yaml
    └── scripts

To Reproduce

1. Are you testing a QuickStart or Custom template? -> QuickStart
2. Attach or link a copy of the template if possible (remove any sensitive info) -> https://github.com/aws-quickstart/quickstart-uipath-automation-suite
3. Provide the parameters that you passed. (remove any sensitive info) -> N/A
4. How did you install taskcat? (docker or pip3) -> pip3
5. Are you using a profile, an instance role or access keys to run taskcat? -> no
6. Is your AWS environment configured via aws configure? -> yes

Expected behavior For an existing stack, if a user uses the update-stack functionality, the nested templates are "pulled" or "downloaded" from the S3 bucket. This means that a main template can be updated using the content at the point of deployment, but it will reference the updated versions during the update-stack process.

Screenshots

andrei@DESKTOP:aws-quickstart-project $ taskcat upload
 _            _             _
| |_ __ _ ___| | _____ __ _| |_
| __/ _` / __| |/ / __/ _` | __|
| || (_| \__ \   < (_| (_| | |_
 \__\__,_|___/_|\_\___\__,_|\__|

version 0.9.30
[ERROR  ] : ValidationError 'aws-quickstart-project/v1' does not match '^[a-z0-9-]*$'

Failed validating 'pattern' in schema['properties']['project']['properties']['name']:
    {'description': 'Project name, used as s3 key prefix when uploading '
                    'objects',
     'examples': ['my-project-name'],
     'pattern': '^[a-z0-9-]*$',
     'type': 'string'}

On instance['project']['name']:
    'aws-quickstart-project/v1'

**Version (Please make sure you are running the latest version of taskcat)

• Taskcat Version: 0.9.30

Additional context Add any other context about the problem here.

[andrew-glenn]

For the moment this can be (somewhat) worked around via the -k flag under the upload subcommand.

Ex: taskcat upload -p /path/to/project -k key-prefix-in-s3

With that in mind, I'm hesitant to modify the project_name regex because of the blast radius. However I'm totally open to a project_version config key.

Any concerns?

Edit: I see your use case here.

Would each version have its own config file, or are you looking to have a central top-level config with multiple project roots under (v1, v2, etc)?

[AndreiBarbuOz]

we have a 2 step release process:

1. Perform a taskcat test run in an AWS dev account, using the config file in the root of the project. This will upload and deploy the current branch, using S3 infrastructure which is internal only. Not interested in versioning this step.
2. Perform taskcat upload for an external customer audience. This step will upload to public S3 infrastructure.

The "versions" would reside on different branches. I'm open to have some metadata file in each branch, containing the version akin to https://github.com/torvalds/linux/blob/7e57714cd0ad2d5bb90e50b5096a0e671dec1ef3/Makefile#L1-L6 or even dynamically modify the command line or config file during the publishing step

[AndreiBarbuOz]

this diagram represent the tree containing both nestedstacks and basic infrastructure objects

our problem statement is:

during update-stack operations, the nested stack templates are downloaded from the S3 buckets and may not be compatible with the initial deployment or with the main template

so we have proposed the above solution which involves versioning on subpaths. we are happy to look at other possible solutions 😁

[himanshu219]

We are facing similar issue while testing multiple versions with taskcat, is it possible to just fix the regex for name or introduce upload-artifacts-prefix as a new parameter for uploading artifacts ?