iofq
commented
2 months ago @cheeeee If you would be willing to attach your database backups here while you have them, they could potentially be of use in troubleshooting. Though without a log of commands run against the aptly it would be difficult to pinpoint exactly what caused the corruption, if anything.
You mentioned this is running in a container, was the container potentially killed by a docker rm -f or a pod rollover? I could see SIGKILL potentially causing some db corruption if something wasn't written to disk.
Tbh we are just running high-level goleveldb API commands here and aren't well equipped to troubleshoot the database corruption. Maybe upstream leveldb repos could offer more support?
neolynx
Detailed Description
Found inconsistency between cli output and REST API output:
Also, the mirror update does not work for this mirror:
Note the last line: error locating part of the DB. I tried to cleanup and recover, but the error is the same, part of DB is missing.
Aptly running in the docker container, most operations are done using cli, and some operations are done with REST API. I also have db backups if this will help.
I do not have particular steps to reproduce, since I do not know exactly when this inconsistency started. Please let me know how to approach this problem. Mirror re-creation is an option, but I would like to avoid it if possible.
To the best of my understanding, the issue is caused by opening db during the mirror update on this line.
Context
I want consistent data about the same object returned via REST API and CLI.
Possible Implementation
Your Environment
Aptly 1.5.0 Docker container with mounted volume for data.