aptly publish switch causes thousands of hard links in public/pool to disappear

[epifeny]

Dozens of thousands of hard links in public/pool disappear every day

I run a cron job that executes once a day, iterates all my mirrors, updates them, creates a snapshot with today's date and uses aptly publish switch to switch with the new snapshot. On a daily basis, this causes dozens of thousands of hard links to disappear. I've been unsuccessful thus far to understand why.

Has anyone experienced this before? I would love your input.

I'm using aptly version 1.5.0.

[epifeny]

Something I discovered is, that I have a flat repository in my list of mirrors, that when I use aptly publish switch on, breaks a the hard links as I mentioned originally. I don't know why, but I was able to repeatedly publish switch all my snapshots and all the hard links were fine, and when I went and did the same with that flat repo, all the hard links broke.

I hope to get some clues from the community, on what I might be able to look out for, that is wrong with this repository that could cause this.

[r4co0n]

I would really be like to be able to reproduce this issue. Can you share your mirror configuration, or try to recreate a minimal extract of it that still shows the same problem, preferably in a pristine environment (E.g., try to recreate it in a disposable container.), and share that?

When you are talking about "flat" repositories, you mean an aptly mirror item you created that follows what is called flat here, right?: https://www.aptly.info/doc/aptly/mirror/create/ This is what Debian Wiki has to say about this repository/aptly mirror format: https://wiki.debian.org/DebianRepository/Format#Flat_Repository_Format

Do I understand you correctly, this flat mirror only causes problems when its snapshots are published, corrupting the published pool? If that is the case, please report if publishing with -distribution=unusedtestdist alleviates your issue.

[epifeny]

Hello @r4co0n, Thanks for your response.

To recreate the entire scenario would take a bit more effort than I have time for at the moment. However, I wrote a small wrapper for Aptly to figure this out. What I did was count the number of hard links before and after running aptly mirror update. It turned out that two flat repositories (as you correctly understood) were the cause.

For instance, I added a ZeroMQ flat repository with the source set as ./:

# aptly publish show zeromq
Prefix: .
Distribution: zeromq
Architectures: amd64
Sources:
  ./: zeromq-20230624_065712 [snapshot]

After forcefully removing it with the -force option, I recreated it:

# aptly publish show zeromq
Prefix: .
Distribution: zeromq
Architectures: amd64
Sources:
  main: zeromq-20230624_174032 [snapshot]

This fixed my issue with broken hard links. However, once I re-created both flat repositories, they both now show main under Sources, and I haven't encountered broken links again.

Perhaps the information I provided above could help you recreate the scenario. The reason why this caused all the hard links to break is still a mystery.