codeartifact copy-package-versions should allow copying only the last (or select) version asset

[gh-andre]

Describe the feature

CodeArtifact is a development package repository, so for packages that allow multiple version assets, such as Python wheels with a build number in the package name, a single version would have multiple wheel packages listed as assets. All of those assets, except one, would have failed verification and the one that passed all tests would be expected to be promoted to the next-level repository.

This way candidate packages from a CI pipeline can be published into a CI repository after passing unit and integration tests, then upon completing staging tests promoted to a staging one, then to the production one, and then to a public repository, possibly with the build number stripped of, depending on how people want their package seen publicly.

However, current implementation of copy-package-versions copies all version assets, good and bad (those that failed the verification) to the destination repository. Only the selected version asset, which typically is the last one that passed all the tests, should be copied into the destination repository. Doing so leaves a good tracking record of what packages made it to each repository and can be linked to work items associated with that build.

Use Case

Today, higher level package repositories keep receiving version assets that failed some tests because copy-package-versions copies all assets for a given version revision. This muddies the waters in those repositories in that one cannot see only the good package assets for any given version revision.

My typical build/publish/promote steps would be like this, where 24 is the build number:

python3 -m build
python3 -m wheel tags --build 024 dist/PyABC-1.0.0-py3-none-any.whl
python3 -m twine upload --repository codeartifact dist/PyABC-1.0.0-24-py3-none-any.whl

# if package passes all tests

aws codeartifact copy-package-versions
        --domain abc --allow-overwrite
        --source-repository AbcDev
        --destination-repository AbcStaging
        --format pypi --package pyabc
        --version-revision 1.0.0=123456

That last bit with version revision is problematic in that it will fail if the repository was updated since the package was pulled for testing, as there's no version revision snapshot and there is no way to promote the exact version asset that passed the tests.

Proposed Solution

What would be helpful if instead of the version revision in copy-package-versions I could specify a version asset, similar to how I can list it today, along with a version revision, with this command for the last one:

aws codeartifact list-package-version-assets
        --domain abc --repository AbcDev
        --format pypi
        --package abc --package-version 1.0.0
        --query "[assets[-1].name,versionRevision]"

This way the destination repository will have only version assets that passed the tests needed for each promoted level. Besides just being cleaner in the sense that it will not have as many packages as a CI repository, it will also provide an added benefit of being able to pick some previous package in a rare case when the latest package has some unexpected bug.

Other Information

I use version revisions to ensure that the package asset that was downloaded from a repository for testing is the same as when copy-package-versions runs, but version revision has no history, so if there is another version asset uploaded in between, version revision changes and copy-package-versions just fails, so I have to start over. If I could promote specifically the package (version asset) I just tested, this would make the process more straightforward.

Acknowledgements

• [ ] I may be able to implement this feature request
• [ ] This feature might incur a breaking change

CLI version used

aws-cli/2.15.49

Environment details (OS name and version, etc.)

Ubuntu 22

[tim-finnigan]

Thanks for the feature request. I'll share this with the CodeArtifact team, who owns and maintains the CopyPackageVersions API (ref: P138888390). In the meantime I'll transfer this to our cross-SDK repository for tracking, since service APIs like this are used across AWS SDKs.

[gh-andre]

@tim-finnigan Thank you for following up. Much appreciated.

Also, I re-read my post and realized that I wasn't explicit in that all of these promotions are just steps on the way to releasing a single package version. It is probably clear from the context, but I wanted to mention anyway :)

[gh-andre]

@tim-finnigan I was trying to use generic packages to maintain DevOps scripts and stumbled upon package assets in generic versions behaving completely differently, compared to pypi packages. After some experimentation, I realized that twine uploads use CodeArtifact in a creative way to allow Python's build numbers in pypi repositories, which may reflect on how this issue is interpreted by the backend dev team, so I wanted to add a note in the context of this feature request.

In generic packages, package assets can be uploaded only to unfinished versions, so a version may have a bunch of assets that are considered as a complete asset set for that version. This is distinctly different from pypi uploads, in which each package with the same version and a different build number is uploaded as a package asset. So, for generic packages, a version is comprised of all of its assets, while for pypi packages, only one package asset is active, which is the one with the highest build number sequence, as described in the wheel package format.

It is almost as if the twine tool (i.e. the one in CodeArtifact, not the actual twine) first downloads all of package version assets, deletes the version, adds the new build and then uploads them all, so version revision changes and the new package is included in the version as a new package asset.

If this is how it works, then copy-package-versions with pypi repositories should copy only the last or selected asset, as this feature request describes, given that only one of pypi packages is considered as the current one that represents the version in question.

In the broader sense, my suggestion would be for the backend team to introduce package asset sets, so an unfinished version would add new package assets to the current unfinished package asset set, but would maintain multiple sets per version, which would allow CI/CD pipelines to keep adding build artifacts to a version, as it matures towards a release, so only the last (or select) package asset set is downloaded when a given package version is requested from the repository.

This would allow all CodeArtifact repositories to behave as development repositories, not just pypi ones. That is, current npm repository isn't much of a development repository, as I have to delete the version every time another CI/CD build runs and reupload it with the new package build. What is described above would allow managing package maturity towards the upcoming release with multiple package asset sets per version and just one of them being active, which is the one that would be promoted via copy-package-version between repositories in different CI/CD environments.

[tim-finnigan]

Thanks again for the feature request. I don't have any updates as of now, but the CodeArtifact team is continuing to track this in their backlog for consideration. We're going to close this on our end as the service team would need to take the next steps here. Please refer to the blog or CHANGELOG for updates, or feel free to reach out through support if you have a support plan.

[github-actions[bot]]

This issue is now closed.

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.