Aptly 1.4.0+ds1-6ubuntu0.1 breaks mirror handling: Error decoding remote repo: EOF

[robin-checkmk]

Detailed Description

After aptly was updated by unattended upgrades from 1.4.0+ds1-6 to 1.4.0+ds1-6ubuntu0.1 mirror management was broken. The symptom were constant messages like this: Error decoding remote repo: EOF

Context

We use aptly in our CI, and it creates several mirrors and snapshots in one go. After the upgrade, we saw this behavior:

Mirror [base-bookworm]: http://ftp.de.debian.org/debian/ bookworm successfully added.
You can run 'aptly mirror update base-bookworm' to download repository contents.
2024/11/27 09:06:28 Error decoding remote repo: EOF
2024/11/27 09:06:28 Error decoding remote repo: EOF
2024/11/27 09:06:28 Error decoding remote repo: EOF
ERROR: unable to update: mirror with name base-bookworm not found
Execute: 'aptly -config mirror/aptly.conf mirror list'
List of mirrors:
 * 
 * [base-buster]: http://ftp.de.debian.org/debian/ buster
 * [base]: http://ftp.de.debian.org/debian/ stretch
 * [updates-bookworm]: http://ftp.de.debian.org/debian/ bookworm-updates
 * [updates-buster]: http://ftp.de.debian.org/debian/ buster-updates
 * [updates]: http://ftp.de.debian.org/debian/ stretch-updates

The remarkable things from my point of view:

• Error decoding remote repo: EOF
• The empty mirror entry (above the others, only consisting of *

We downgraded back to 1.4.0+ds1-6 and the commands now finish again successfully. However, we still see the aforementioned messages.

Quite frankly, I do not really know where to look, so I am happy for any hints as to how to fix this.

Possible Implementation

Your Environment

Ubuntu 22.04.05 LTS

The update notes for 1.4.0+ds1-6ubuntu0.1 merely mention this:

aptly (1.4.0+ds1-6ubuntu0.1) jammy-security; urgency=medium

  * No change rebuild due to golang-1.18 update. Note that this package
    was built with golang-1.17.

 -- Allen Huang <email address hidden>  Mon, 18 Nov 2024 15:24:14 +0000

[neolynx]

since this is not an aptly upstream build, I think it is best to report this to ubuntu. maybe some build dependency does not match ?

[robin-checkmk]

Thanks for the blazing fast response!

I was afraid you said that, and it makes sense, but I figured bringing it up here as well, as users might pop by. I will look into reporting it to Ubuntu, and will report back, if/what I can learn. If anyone else has ideas and thoughts, do let me know.

P.S.: @neolynx While I have your attention: Any hints towards how I might be able to remove those "empty" mirror entries, which seem to cause the Error decoding remote repo: EOF error? I am afraid, I have to mess with the database directly, but maybe I overlooked something obvious.

[neolynx]

aptly 1.4.0 is already quite old. one way might be to use the latest 1.6.0 CI build of aptly, which has a lot of issues fixed. Otherwise go back to the working version from ubuntu.

With a known good version, you can try running aptly db cleanup which compacts the levedb and mayb also fixes your EOF problem. Maybe removing and recreating the mirrors does the trick. There are also official go LevelDB tools for manipulating databases, maybe this can help.

It would be a good idea to backup the .aptly/db directory, in case things go wrong. if you have such a backup already, you might be able to compare the 2 dbs.

hope this helps !

[robin-checkmk]

Thanks for the hints, we will look into them (already thought about some, actually).

I appreciate you taking the time, thank you!

[ju2wheels]

Im hitting the same issue on 22.04 with 1.6.0 CI build. Im going to double check I used the right packages but it still seems to be present in the latest release and its an issue that came out of nowhere since it was working fine before a few days ago even with outdated aptly packages.

Maybe removing and recreating the mirrors does the trick.

Can confirm that does work. I still have the old copy of the aptly dir, so ill put it back and try the db cleanup.

[edit] db cleanup/recover commands didnt help, it still produces the same EOF error.

[robin-checkmk]

@ju2wheels I am glad it is reproducible across releases, which means, it is most certainly an Ubuntu issue. Do you have means to file a bug report with them? I currently lack both the time and an account to do so.

[ju2wheels]

Its reproducible once it happens and you have a corrupted leveldb, but I really dont think it originated with the aptly side (aside from the fact that it maybe could have been caused by an unhandled exception). I have a few thousand devices running aptly in Docker on container start running on 18.04, 20.04, and 22.04 base with the default aptly version for each upstream distro. I have aptly on RPi 4 18.04 that hasnt yet corrupted its leveldb and its still perfectly able to mirror packages from the upstream Ubuntu 18.04 mirror but I did have this device off for most of last week.

Only 10 of my devices started hitting this EOF issue all starting on Nov 25.

My current theory is something happened to upstream repos last week and caused an unhandled exception in aptly which led to the corrupted leveldb but I dont know anything about this code base to be able to confirm that. But it looks like from your comment that you were mirroring Debian. Im still looking through my logs to see if i can identify the first occurrence on one of my 10 devices and see if it raised a more useful exception before the leveldb EOF errors started showing up.

Just to add that in my case im getting two EOF errors:

Error decoding mirror: EOF

and

Error decoding remote repo: EOF

I do have a Launchpad account so I can open a ticket later if you guys want.

[neolynx]

is this happening on arm4 devices only ? or also on amd64 ?

[ju2wheels]

The 10 devices im currently hitting the issue on are all amd64 and mirroring only amd64.