snapshot switch - unable to publish: unable to process packages: error putting file - file already exists and is different

[yaleman]

I'm trying to host multiple repositories for internally-packaged files and I can't find a consistent set of commands to update them as they change.

I think? I might be doing it wrong by not having the full version number in the filename, but I thought the snapshot switch would get around that to sign/overwrite with the current state of the packages.

I don't want endless old versions lying around, I'm building new iterations of the packages as I'm testing/fixing them, but the deb internal package numbers are updating, so that should be fine?

Detailed Description

I'm hosting packages for multiple OS distribution/releases/architectures:

• ubuntu bionic
• ubuntu focal
• debian buster (and a bunch of others, but you get the idea)

Each of them (at this point) has arm and amd64 packages.

I create my repos:

aptly -config=aptly.conf repo create -distribution="bionic" -component=main "kanidm-bionic"
aptly -config=aptly.conf repo create -distribution="focal" -component=main "kanidm-focal"

I have my deb files locally:

debs/bionic/kanidm-latest-amd64.deb
debs/bionic/kanidm-latest-arm64.deb
debs/bionic/kanidm-pamnss-latest-amd64.deb
debs/bionic/kanidm-pamnss-latest-arm64.deb
debs/focal/kanidm-latest-amd64.deb
debs/focal/kanidm-latest-arm64.deb
debs/focal/kanidm-pamnss-latest-amd64.deb
debs/focal/kanidm-pamnss-latest-arm64.deb

I add my files to the repos:

# find debs/buster -type f -name '*.deb' -exec aptly -config=aptly.conf repo add "kanidm-buster" {} \;
Loading packages...
[+] kanidm_1.1.0-alpha.61633574164_amd64 added
Loading packages...
[+] kanidm_1.1.0-alpha.51633020420_arm64 added
Loading packages...
[+] kanidm-pamnss_1.1.0-alpha.61633574321_amd64 added
Loading packages...
[+] kanidm-pamnss_1.1.0-alpha.51633021003_arm64 added
# find debs/focal -type f -name '*.deb' -exec aptly -config=aptly.conf repo add "kanidm-focal" {} \;
Loading packages...
[+] kanidm_1.1.0-alpha.61633574164_amd64 added
Loading packages...
[+] kanidm_1.1.0-alpha.51633020420_arm64 added
Loading packages...
[+] kanidm-pamnss_1.1.0-alpha.61633574321_amd64 added
Loading packages...
[+] kanidm-pamnss_1.1.0-alpha.51633021003_arm64 added

I create my initial snapshot:

aptly -config=aptly.conf snapshot create "bionic-init" from repo "kanidm-bionic"
aptly -config=aptly.conf snapshot create "focal-init" from repo "kanidm-focal"

Then I publish them to s3 (changing the

# aptly -config=aptly.conf publish snapshot bionic-init s3:minio.internal:bionic
Loading packages...
Generating metadata files and linking package files...
Finalizing metadata files...
Signing file 'Release' with gpg, please enter your passphrase when prompted:
Clearsigning file 'Release' with gpg, please enter your passphrase when prompted:

Snapshot bionic-init has been successfully published.
Now you can add following line to apt sources:
  deb http://your-server/bionic/ bionic main
Don't forget to add your GPG key to apt with apt-key.

You can also use `aptly serve` to publish your repositories over HTTP quickly.
# aptly -config=aptly.conf publish snapshot focal-init s3:minio.internal:focal
Loading packages...
Generating metadata files and linking package files...
Finalizing metadata files...
Signing file 'Release' with gpg, please enter your passphrase when prompted:
Clearsigning file 'Release' with gpg, please enter your passphrase when prompted:

Snapshot focal-init has been successfully published.
Now you can add following line to apt sources:
  deb http://your-server/focal/ focal main
Don't forget to add your GPG key to apt with apt-key.

You can also use `aptly serve` to publish your repositories over HTTP quickly.

When I compile new iterations of the packages, I place them in their respective debs dirs, then add them again (same commands as above) and create new snapshots, then try to publish switch them to the new snapshots per the published instructions.

The filenames are staying the same, but the versions are changing internally - we're trying to iterate on builds of the same version to get things working, this is why I thought the snapshot was... the way to go with this?

# aptly -config=aptly.conf snapshot create bionichousenet-2021-10-07-072938 from repo "kanidm-bionic"

Snapshot bionichousenet-2021-10-07-072938 successfully created.
You can run 'aptly publish snapshot bionichousenet-2021-10-07-072938' to publish snapshot as Debian repository.

# aptly -config=aptly.conf snapshot list | grep bionichousenet
 * [bionichousenet-2021-10-07-072938]: Snapshot from local repo [kanidm-bionic]

# aptly -config=aptly.conf publish switch buster s3:minio.internal:bionic bionichousenet-2021-10-07-072938
Loading packages...
Generating metadata files and linking package files...
ERROR: unable to publish: unable to process packages: error putting file to bionic/pool/main/k/kanidm/kanidm-latest-amd64.deb: file already exists and is different: S3: us-east-1:aptly/

Your Environment

aptly version: 1.4.0+ds1-4+b4 os: Debian Bullseye docker container s3 software: minio

[r4co0n]

I suppose that is just how Debian packages and many repository managers, including aptly, work (by default), This is because APT does the same on the client side, only upgrading packages with a higher version than the previously installed version - aptly tries to save you from creating an inconsistent repository and installation base.

My first question would be how you build your packages, there are many ways to automatically bump your version on development builds, which seems to be what you want. For example, look at git-buildpackage's documentation. This is a tool I have found to be highly reliable, though you will have to familiarize yourself with how gbp and Debian Packaging in general works, as opposed to some "It all just works, throw a tarball at me, I spit out a deb"-tools.

More info:

• Debian Wiki - Packaging with Git
• The rest of the docu at honk.sigxcpu.org linked above
• Debian New Maintainers' Guide
• Debian Policy Manual

This should help you quite some way, I still refer to those documents, the wiki-entry excluded, fairly often.

To summarize, you cannot republish a package with the same name, version and architecture, but a different hashsum, to the same aptly publish endpoint and subdirectory and distribution.

Happy to help you along further if necessary.

[yaleman]

The version numbers are already being bumped on the package metadata, I solved that to make apt upgrades work - and it shows up when aptly adds the new versions.

I’ve worked around it after logging this ticket by (for each update) deleting the .aptly dir entirely, recreating the repo and publishing locally then using “aws —delete s3 sync” to the bucket… I couldn’t work out a trivial way of parsing aptly’s output in a way I could leverage to successfully automate this.

I’d really rather find a way which wasn’t so “scorched earth” every time however as it feels like a bug I’m hitting somewhere.

James

On 2021-10-08 00:42 Simon Hollenbach wrote:

I suppose that is just how Debian packages and many repository managers, including aptly, work (by default), This is because APT does the same on the client side, only upgrading packages with a higher version than the previously installed version - aptly tries to save you from creating an inconsistent repository and installation base.

My first question would be how you build your packages, there are many ways to automatically bump your version on development builds, which seems to be what you want. For example, look at git-buildpackage's documentation http://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.snapshots.html. This is a tool I have found to be highly reliable, though you will have to familiarize yourself with how gbp and Debian Packaging in general works, as opposed to some "It all just works, throw a tarball at me, I spit out a deb"-tools.

More info:

• Debian Wiki - Packaging with Git https://wiki.debian.org/PackagingWithGit
• The rest of the docu at honk.sigxcpu.org linked above
• Debian New Maintainers' Guide https://www.debian.org/doc/manuals/maint-guide/
• Debian Policy Manual https://www.debian.org/doc/debian-policy/index.html This should help you quite some way, I still refer to those documents, the wiki-entry excluded, fairly often.

To summarize, you cannot republish a package with the same name, version and architecture, but a different hashsum, to the same aptly publish endpoint and subdirectory and distribution.

Happy to help you along further if necessary.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aptly-dev/aptly/issues/984#issuecomment-937860789, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABJB7FMN4SCUPKDT5RKHH3UFWWVBANCNFSM5FQUH4XA.

[r4co0n]

If you bump the version already, you should only need to rename the package to kanidm_<version>_arch.deb, e.g. kanidm_1.1.0-alpha.61633574164_amd64.deb.

Additionally, you should be able to force-publish your snapshot using aptly -config=aptly.conf publish switch -force [...], but I would recommend renaming the files. If you have difficulty determining the version from the deb before import, try dpkg-deb -f package.deb Version, this command should be available on any APT-based distribution with package dpkg.

I think a mv kanidm-latest-amd64.deb kanidm_$(dpkg-deb -f kanidm-latest-amd64.deb)_amd64.deb (and similar for each architecture) before importing will fix your problem. You used - as delimiter in your package, instead of _ as is conventional, and somehow aptly preserves your package name when importing the package to a pool, instead of reconstructing <package_name>_<version>_<arch>.deb from the package's control data. (On import, the packages are correctly identified, as seen in your log.) Maybe there's a good reason for this, you should be fine as long as the packages you produce follow Debian Policy as linked above.

So yeah, this might actually be a bug, or at least an issue with documenting edge-cases better, I suppose.

EDIT: I just found dpkg-name(1), which might do exactly what you want.

[yaleman]

Thanks for the responses, I'll see if I can work it into the build process. The biggest problem with all of this is that each of these distributions seems to assume you're packaging a c-style package and that you're only doing it for their things, so I'm working around workarounds with all of this...

(and force-switching fails because somehow it pushes new Releases files without touching the .debs so all the repo clients start screaming about size mismatches etc.)

[r4co0n]

@yaleman, your observation about aptly snapshot switch -force not updating the packages according to the specified snapshot is quite obviously a bug, this seems opposed to documentation and reasonable user expectations.

Purposefully unique names are (in my opinion, this is not authoritative, but observative) an engineering choice that prevents the usual APT client (and probably your friend's one-off-script) from running into hashsum and size mismatches because of simultaneous repository synchronisation. (With a simplified sync consisting of fetching upstream release file and referenced /pool packages, switching out the release file last.)

Best of luck with your packaging pipeline.