search

openwrt / openwrt

This repository is a mirror of https://git.openwrt.org/openwrt/openwrt.git It is for reference only and is not active for check-ins. We will continue to accept Pull Requests here. They will be merged via staging trees then into openwrt.git.
Other
20.71k stars 10.61k forks source link

apk fails to read its own database: FDB format error #16794

Closed hnyman closed 1 month ago

hnyman commented 1 month ago

Describe the bug

I have compiled an apk enabled version of main/master snapshot r27902-6835ff8cbc (with apk-openssl) for mediatek MT6000, but using apk fails as it seems unable to read its own database of already installed packages (in the image itself).

root@router6000:/lib/apk/db# apk update
ERROR: FDB format error (line 6860, entry 'F')
ERROR: Unable to read database: v2 database format error
ERROR: Failed to open apk database: v2 database format error

root@router6000:/lib/apk/db# apk info
ERROR: FDB format error (line 6860, entry 'F')
ERROR: Unable to read database: v2 database format error
ERROR: Failed to open apk database: v2 database format error

Looking at the line number, it seems to fail on parsing package info for openssl, roughly at the middle of the 11300 lines in the file:

root@router6000:~# less -N /lib/apk/db/installed
   6854 U:https://www.openssl.org/
   6855 L:Apache-2.0
   6856 o:feeds/base/libs/openssl
   6857 m:Eneas U de Queiroz <cotequeiroz@gmail.com>
   6858 c:
   6859 D:libc libopenssl3
   6860 F:etc
   6861 F:etc/config
   6862 R:openssl
   6863 Z:Q247DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=
   6864 F:etc/init.d
   6865 R:openssl
   6866 a:0:0:755
   6867 Z:Q2K3nUsjzYTwbSQIAcrvqq7gqnCnrl1bW4fk9q3gw1Zv8=

But apk is able to e.g. verify the package index downloaded from apracar's test site, so in general, it should be ok?

root@router6000:~# apk verify packages.adb
packages.adb: OK

I am using apk-openssl:

root@router6000:/etc# grep apk build.config
# CONFIG_PACKAGE_apk-mbedtls is not set
CONFIG_PACKAGE_apk-openssl=y

The same happens also with RT3200 at the same place:

 OpenWrt SNAPSHOT, r27909-d7ed4c77b5
 -----------------------------------------------------
root@router4:~# apk info
ERROR: FDB format error (line 6904, entry 'F')
ERROR: Unable to read database: v2 database format error
ERROR: Failed to open apk database: v2 database format error

   6901 m:Eneas U de Queiroz <cotequeiroz@gmail.com>
   6902 c:
   6903 D:libc libopenssl3
   6904 F:etc
   6905 F:etc/config
   6906 R:openssl

OpenWrt version

r27902-6835ff8cbc

OpenWrt release

SNAPSHOT

OpenWrt target/subtarget

mediatek/filogic and mediatek/7622

Device

GL.iNet GL-MT6000 and Belkin RT3200

Image kind

Self-built image

Steps to reproduce

No response

Actual behaviour

No response

Expected behaviour

No response

Additional info

No response

Diffconfig

No response

Terms

github-actions[bot] commented 1 month ago

Invalid Target/Subtarget reported. mediatek/filogic and mediatek/7622 Is this from a supported device?

aparcar commented 1 month ago

@Ansuel wdyt

Ansuel commented 1 month ago

Feel like a bug with APK but the bisect from @hnyman made me understand the problem better... I also had this error but I hoped it was caused by my downstream patch and not from that.

By the looks of it, it seems empty entry with .apk file.

hnyman commented 1 month ago

it seems empty entry with .apk file.

If you refer to the "c:" field, it seems to be empty for all packages, not only for openssl.

By looking at the apk-tools sources, the field seems to be for a commit hash. MAP('c', ADBI_PI_REPO_COMMIT), ADB_FIELD(ADBI_PI_REPO_COMMIT, "repo-commit", scalar_hexblob),

Ansuel commented 1 month ago

No the error talks about entry F.

Entry can be empty... Problem is if they are apk_blob... They still needs to be allocated and properly init.

Or maybe someone in APK fkup up and entry F was dropped but never actually dropped from the schema. Can you check what should be the usage of entry F ?

hnyman commented 1 month ago

Can you check what should be the usage of entry F ?

Well, looking at the apk sources for first time, it seems pretty hard to find. At least here it is said that it is about directories: https://github.com/alpinelinux/apk-tools/blob/master/src/app_convdb.c#L154

And indeed, looking at the "installed" file contents, F is about defining directories for the files to be installed (and subsequent R is then the file).

   1319 m:Jo-Philipp Wich <jo@mein.io>, Hannu Nyman <hannu.nyman@iki.fi>
   1320 c:
   1321 D:collectd libc
   1322 F:lib
   1323 F:lib/apk
   1324 F:lib/apk/packages
   1325 R:collectd-mod-interface.list
   1326 Z:Q2Vq/ENS/oHulypWu7+3ZR061rPWXZ5YhnC3PpSGryECE=
   1327 F:usr
   1328 F:usr/lib
   1329 F:usr/lib/collectd
   1330 R:interface.so
   1331 a:0:0:755
   1332 Z:Q29yfaitEuJplzvXaV4+bA+1A8tgnAfdujZhzjnF+u/sw=
Ansuel commented 1 month ago

It's in database.c that is just an app to convert the db

https://github.com/alpinelinux/apk-tools/blob/master/src/database.c

But why the format error.... strange....

hnyman commented 1 month ago

I wonder if we are encountering the problem just because we are using apk from a commit in a feature branch, which has then later been merged into the mainline apk, maybe in a slightly different way or with some earlier/later fixes?

We are using the apk commit 54caa31b (from early August) as our apk source, but that commit is not in the official apk commit history. (Apparently 7972a780 of 21 August is the same commit in the official apk history)

@aparcar refers to branch "index-trust" in his commit ef8c1adb "apk: switch to index-trust branch" here, but there is no such branch in apk any more, as the feature has been adopted into the official apk.

Maybe we are encountering something that has already been fixed in apk? Should we bump our apk to be from the official apk mainline, and maybe something newer than the current 6 Aug 2024 commit from a feature branch?

(I will test the most recent apk commit)

EDIT: Sadly, using apk commit from 23.10.2024 didn't help.

PKG_SOURCE_DATE:=2024-10-23
PKG_SOURCE_VERSION:=cef30b61c1a4c870f23f905423d76a287c22bf02
PKG_MIRROR_HASH:=39e9e649d608977da5947c239c71660bff30fee92494dff8be9c9663d513d550

stills leads into


 OpenWrt SNAPSHOT, r27913-a2aabc9a7a
 -----------------------------------------------------
root@router6000:~# apk --version
apk-tools 3.0.0_pre20241023, compiled for aarch64.
root@router6000:~# apk info
ERROR: FDB format error (line 6910, entry 'F')
ERROR: Unable to read database: v2 database format error
ERROR: Failed to open apk database: v2 database format error
Ansuel commented 1 month ago

@hnyman ok I discovered the culprit :D

If you notice openssl adds a new line and destroy the database... Check the T entry... This is totally a fragility of APK and I will produce a patch to handle this.

Aka

hnyman commented 1 month ago

Great that you figured it out. Sounds fragile also for upstream if the description text is not sanitized when it is entered into the database.

Ansuel commented 1 month ago

@hnyman well problem was something totally different and wasn't us adding new line char... More a too long entry getting corrupted.

If you want further reading :D https://gitlab.alpinelinux.org/alpine/apk-tools/-/merge_requests/236/diffs

Anyway with the 2 patch problem is fixed.

hnyman commented 1 month ago

Great that you found the real reason.

I read your explanation. And funnily enough, I had noticed from less/hexedit that the long descriptions ended in a non-printable char instead of the dot, but as none of occurences earlier in the file seemed to cause problems, I just thought that it may be an intended end marker for the longest descriptions... Sounds like that was a wrong guess. (Apparently other chars like 0x03 etc. are handled ok, but in this case the last wrong char was newline, breaking the database encoding.)

I will apply and test with your patches. Are you committing them now to OpenWrt repo as patches for our apk?

Ansuel commented 1 month ago

Are you committing them now to OpenWrt repo as patches for our apk?

They are in the gigantic APK wip pr.

pko76 commented 1 month ago

I have a similar problem: LuCI Master (24.298.52515~e34c268) / OpenWrt SNAPSHOT (r27931-945a335f66) root@OpenWrt:# apk verify packages.adb packages.adb: No such file or directory root@OpenWrt:# grep apk build.config grep: build.config: No such file or directory root@OpenWrt:# apk info ERROR: Unable to read database: No such file or directory ERROR: Failed to open apk database: No such file or directory root@OpenWrt:# apk --version apk-tools 3.0.0_pre20240806, compiled for armv7.

hnyman commented 1 month ago

@Ansuel 's fix for this has now been merged upstream at apk:

So, might it make sense to update our apk to that upstream HEAD and avoid taking the local patch separately to our marginally older apk version.

I close this bug, as we have the fix locally as patch in the apk WIP PR, and it has been fixed upstream.