search

koreader / koreader-base

Base framework offering a Lua scriptable environment for creating document readers
http://koreader.rocks/
GNU Affero General Public License v3.0
132 stars 105 forks source link

porting to a new device and UBUNTU_COMPAT_FLAGS #744

Closed pazos closed 5 years ago

pazos commented 5 years ago

Hi, I'm trying to port KOReader to a bq cervantes 4. The first build went well but can't run because luajit was complaining about glibc. So I lurked into the code and set UBUNTU_COMPAT_FLAGS for this platform.

Now luajit works but KOReader fails with the crash.log:

---------------------------------------------
                launching...
  _  _____  ____                _
 | |/ / _ \|  _ \ ___  __ _  __| | ___ _ __
 | ' / | | | |_) / _ \/ _` |/ _` |/ _ \ '__|
 | . \ |_| |  _ <  __/ (_| | (_| |  __/ |
 |_|\_\___/|_| \_\___|\__,_|\__,_|\___|_|

 It's a scroll... It's a codex... It's KOReader!

 [*] Current time: 10/12/18-21:46:27
 [*] Version: v2015.11-1878-g9b45a930_2018-10-12

ffi.load: blitbuffer
ffi.load (warning): libblitbuffer.so: cannot open shared object file: No such file or directory
ffi.load: SDL2
ffi.load (warning): libSDL2.so: cannot open shared object file: No such file or directory
ffi.load: SDL
ffi.load (warning): libSDL.so: cannot open shared object file: No such file or directory
10/12/18-21:46:27 INFO  SoftwareVersion:  openFW
10/12/18-21:46:27 INFO  initializing for device Cervantes
10/12/18-21:46:27 INFO  framebuffer resolution: {
    ["h"] = 1448,
    ["w"] = 1072
}
ffi.load: libs/libfreetype.so.6
ffi.load: libs/libmupdf.so
ffi.load (warning): /lib/arm-linux-gnueabi/libm.so.6: version `GLIBC_2.27' not found (required by libs/libmupdf.so)
./luajit: ./setupkoenv.lua:30: Not able to load dynamic library: libs/libmupdf.so
stack traceback:
    [C]: in function 'error'
    ./setupkoenv.lua:30: in function 'load'
    ./ffi/mupdf.lua:25: in main chunk
    [C]: in function 'require'
    frontend/ui/widget/htmlboxwidget.lua:10: in main chunk
    [C]: in function 'require'
    frontend/ui/widget/scrollhtmlwidget.lua:6: in main chunk
    [C]: in function 'require'
    frontend/ui/widget/dictquicklookup.lua:18: in main chunk
    [C]: in function 'require'
    frontend/apps/reader/modules/readerdictionary.lua:4: in main chunk
    [C]: in function 'require'
    frontend/apps/reader/readerui.lua:26: in main chunk
    [C]: in function 'require'
    ./reader.lua:212: in main chunk
    [C]: at 0x00013a05

Any hint on that?, thanks

pazos commented 5 years ago

For the record:

I'm using TC arm-linux-gnueabi-gcc (Ubuntu/Linaro 7.3.0-27ubuntu1~18.04) 7.3.0_x86

I'm going to try to build KOReader with vendor recommended TC

NiLuJe commented 5 years ago

Yeah, those flags date back to Ubuntu... 12.something, I think? So, a metric ton of time ago ;p.

They won't help much with today's stuff ;).

NiLuJe commented 5 years ago

You could potentially dig into which exact symbols MuPDF relies on (with readelf's help) and possibly manage to work that around, but it's basically a sand castle: everything might break again at the next update ;).

NiLuJe commented 5 years ago

So, yeah, the recommended way to handle that is either going with the vendor-recommended TC, or building one with only GCC updates, but matching kernel/libc , like we do on Kindle/Kobo.

Frenzie commented 5 years ago

As a quick hack (depending on what you want to focus on first) you could also try something like using the Kobo libs btw.

Is there some more technical info about this device? I did a quick search and just found some specs about size on the consumer oriented page.

NiLuJe commented 5 years ago

That's doable, but somewhat harder to pull off in the glibc case (especially in this case, because Kobo's handling of the dynamic loader is already iffy since they updated the glibc in the FW) ;).

There are a few threads on MR for doing that sort of things to use a hardfp libc on Kindle (mainly from twobob & knc1 IIRC, form... a good while ago :D).

pazos commented 5 years ago

Is there some more technical info about this device? I did a quick search and just found some specs about size on the consumer oriented page.

Most of the device is equal to kobos (except for soft float userspace): battery, lights, power, drivers for wifi and usbms/usbnet... Screen is based on neonode IR as well.

Software is based on Debian Wheezy (armel). 

cat /proc/cpuinfo 
Processor   : ARMv7 Processor rev 10 (v7l)
BogoMIPS    : 1988.29
Features    : swp half thumb fastmult vfp edsp vfpv3 
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x2
CPU part    : 0xc09
CPU revision    : 10

Hardware    : Freescale i.MX 6SoloLite NTX Board
dpkg -l | grep libc6
ii  libc6:armel                     2.13-38+deb7u10              armel
ii  libc6-dev:armel                 2.13-38+deb7u10              armel

So, yeah, the recommended way to handle that is either going with the vendor-recommended TC, or building one with only GCC updates, but matching kernel/libc , like we do on Kindle/Kobo.

recommended TC is unable to build KOReader or even FBInk. I will try other Ubuntu/Linaro TC.

pazos commented 5 years ago

Ok, building with arm-linux-gnueabi-gcc (Linaro GCC 4.9-2017.01) 4.9.4 I get

---------------------------------------------
                launching...
  _  _____  ____                _
 | |/ / _ \|  _ \ ___  __ _  __| | ___ _ __
 | ' / | | | |_) / _ \/ _` |/ _` |/ _ \ '__|
 | . \ |_| |  _ <  __/ (_| | (_| |  __/ |
 |_|\_\___/|_| \_\___|\__,_|\__,_|\___|_|

 It's a scroll... It's a codex... It's KOReader!

 [*] Current time: 10/13/18-22:11:11
 [*] Version: v2015.11-1878-g9b45a930_2018-10-12

ffi.load: blitbuffer
ffi.load (warning): libblitbuffer.so: cannot open shared object file: No such file or directory
ffi.load: SDL2
ffi.load (warning): libSDL2.so: cannot open shared object file: No such file or directory
ffi.load: SDL
ffi.load (warning): libSDL.so: cannot open shared object file: No such file or directory
10/13/18-22:11:12 INFO  SoftwareVersion:  openFW
10/13/18-22:11:12 INFO  initializing for device Cervantes
10/13/18-22:11:12 INFO  framebuffer resolution: {
    ["h"] = 1448,
    ["w"] = 1072
}
ffi.load: libs/libfreetype.so.6
Illegal instruction
NiLuJe commented 5 years ago

Possibly because it's hard float? As is Ubuntu's now that I think about it...

On Sat, Oct 13, 2018, 22:15 Martín Fernández notifications@github.com wrote:

Ok, building with arm-linux-gnueabi-gcc (Linaro GCC 4.9-2017.01) 4.9.4 I get

            launching...

| |/ / | \ _ _ | | | ' / | | | |) / \/ ` |/ ` |/ _ \ '| | . \ || | < / (| | (| | / | ||_\/|| __|\,|_,|__||

It's a scroll... It's a codex... It's KOReader!

[] Current time: 10/13/18-22:11:11 [] Version: v2015.11-1878-g9b45a930_2018-10-12

ffi.load: blitbuffer ffi.load (warning): libblitbuffer.so: cannot open shared object file: No such file or directory ffi.load: SDL2 ffi.load (warning): libSDL2.so: cannot open shared object file: No such file or directory ffi.load: SDL ffi.load (warning): libSDL.so: cannot open shared object file: No such file or directory 10/13/18-22:11:12 INFO SoftwareVersion: openFW 10/13/18-22:11:12 INFO initializing for device Cervantes 10/13/18-22:11:12 INFO framebuffer resolution: { ["h"] = 1448, ["w"] = 1072 } ffi.load: libs/libfreetype.so.6 Illegal instruction

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/koreader/koreader-base/issues/744#issuecomment-429571904, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG1ZjJb3mXIilc1tnEVe5YU1A_aUuxwks5ukkn0gaJpZM4XaGYq .

NiLuJe commented 5 years ago

What's the issue with the vendor TC?

On Sat, Oct 13, 2018, 23:24 NiLuJe ninuje@gmail.com wrote:

Possibly because it's hard float? As is Ubuntu's now that I think about it...

On Sat, Oct 13, 2018, 22:15 Martín Fernández notifications@github.com wrote:

Ok, building with arm-linux-gnueabi-gcc (Linaro GCC 4.9-2017.01) 4.9.4 I get

            launching...

| |/ / | \ _ _ | | | ' / | | | |) / \/ ` |/ ` |/ _ \ '| | . \ || | < / (| | (| | / | ||_\/|| __|\,|_,|__||

It's a scroll... It's a codex... It's KOReader!

[] Current time: 10/13/18-22:11:11 [] Version: v2015.11-1878-g9b45a930_2018-10-12

ffi.load: blitbuffer ffi.load (warning): libblitbuffer.so: cannot open shared object file: No such file or directory ffi.load: SDL2 ffi.load (warning): libSDL2.so: cannot open shared object file: No such file or directory ffi.load: SDL ffi.load (warning): libSDL.so: cannot open shared object file: No such file or directory 10/13/18-22:11:12 INFO SoftwareVersion: openFW 10/13/18-22:11:12 INFO initializing for device Cervantes 10/13/18-22:11:12 INFO framebuffer resolution: { ["h"] = 1448, ["w"] = 1072 } ffi.load: libs/libfreetype.so.6 Illegal instruction

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/koreader/koreader-base/issues/744#issuecomment-429571904, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG1ZjJb3mXIilc1tnEVe5YU1A_aUuxwks5ukkn0gaJpZM4XaGYq .

pazos commented 5 years ago

Possibly because it's hard float? As is Ubuntu's now that I think about it...

gnueabi is soft float (armel) and gnueabihf is hard float (armhf). This device supports hardfloat (it has a VFP enabled kernel).

I'll try to keep binary compatibility w/ stock os. If this is impossible I will switch to armhf.

What's the issue with the vendor TC?

For building FBInk need to set the flag -std=c99 and even then fails with

make strip CERVANTES=true
make[1]: se entra en el directorio '/home/pazos/Escritorio/FBInk-git'
make staticlib
make[2]: se entra en el directorio '/home/pazos/Escritorio/FBInk-git'
mkdir -p Release/shared/utf8 Release/static/utf8
arm-linux-gnueabi-gcc  -D_REENTRANT=1 -DFBINK_FOR_CERVANTES -std=c99 -DFBINK_WITH_FONTS -DFBINK_WITH_IMAGE -DFBINK_WITH_BUTTON_SCAN -O2 -fomit-frame-pointer -pipe -ftree-vectorize -funroll-loops  -DFBINK_VERSION='"v1.7.0-3-g3222cd7 for Cervantes"' -o Release/static/fbink.o -c fbink.c
fbink.c:57:10: warning: unknown option after ‘#pragma GCC diagnostic’ kind [-Wpragmas]
arm-linux-gnueabi-gcc  -D_REENTRANT=1 -DFBINK_FOR_CERVANTES -std=c99 -DFBINK_WITH_FONTS -DFBINK_WITH_IMAGE -DFBINK_WITH_BUTTON_SCAN -O2 -fomit-frame-pointer -pipe -ftree-vectorize -funroll-loops  -DFBINK_VERSION='"v1.7.0-3-g3222cd7 for Cervantes"' -o Release/static/utf8/utf8.o -c utf8/utf8.c
arm-linux-gnueabi-gcc-ar rc Release/libfbink.a Release/static/fbink.o Release/static/utf8/utf8.o
Makefile:233: recipe for target 'staticlib' failed
make[2]: *** [staticlib] Error 228
make[2]: se sale del directorio '/home/pazos/Escritorio/FBInk-git'
Makefile:271: recipe for target 'static' failed
make[1]: *** [static] Error 2
make[1]: se sale del directorio '/home/pazos/Escritorio/FBInk-git'
Makefile:297: recipe for target 'cervantes' failed
make: *** [cervantes] Error 2

KOReader fails to build some third party libs:

Makefile:103: recipe for target 'libtesseract.so' failed
make[5]: *** [libtesseract.so] Error 2
make[5]: se sale del directorio '/home/pazos/Escritorio/koreader/base/thirdparty/libk2pdfopt/build/arm-linux-gnueabi/libk2pdfopt-prefix/src/libk2pdfopt'
CMakeFiles/libk2pdfopt.dir/build.make:110: recipe for target 'libk2pdfopt-prefix/src/libk2pdfopt-stamp/libk2pdfopt-build' failed
make[4]: *** [libk2pdfopt-prefix/src/libk2pdfopt-stamp/libk2pdfopt-build] Error 2
make[4]: se sale del directorio '/home/pazos/Escritorio/koreader/base/thirdparty/libk2pdfopt/build/arm-linux-gnueabi'
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/libk2pdfopt.dir/all' failed
make[3]: *** [CMakeFiles/libk2pdfopt.dir/all] Error 2
make[3]: se sale del directorio '/home/pazos/Escritorio/koreader/base/thirdparty/libk2pdfopt/build/arm-linux-gnueabi'
Makefile:83: recipe for target 'all' failed
make[2]: *** [all] Error 2
make[2]: se sale del directorio '/home/pazos/Escritorio/koreader/base/thirdparty/libk2pdfopt/build/arm-linux-gnueabi'
Makefile.third:213: recipe for target 'build/arm-linux-gnueabi/libs/libk2pdfopt.so.2' failed
make[1]: *** [build/arm-linux-gnueabi/libs/libk2pdfopt.so.2] Error 2
make[1]: se sale del directorio '/home/pazos/Escritorio/koreader/base'
NiLuJe commented 5 years ago

@pazos : Yeah, I'm using C99 constructs, so that makes sense ;).

As for the build failure, the Makefile also currently expects GCC plugins to be enabled. You can try to pass/set AR=arm-linux-gnueabi-ar & RANLIB=arm-linux-gnueabi-ranlib, that should help for FBInk ;).

As for KOReader, the actual error is probably higher up in the backlog, I don't see anything besides the final make failures there ;).

pazos commented 5 years ago

@NiLuJe: yeah, for FBInk export CROSS_TC=arm-linux-gnueabi will set AR=arm-linux-gnueabi-gcc-ar instead of arm-linux-gnueabi-ar. The same happens with ranlib. This is incompatible with old toolchains, it seems.

About KOReader I can't see other errors. Is the build process logged to somewhere?

pazos commented 5 years ago

Yup, kindlepw2 TC produces working binaries for the device. After building KOReader with this toolchain:

---------------------------------------------
                launching...
  _  _____  ____                _
 | |/ / _ \|  _ \ ___  __ _  __| | ___ _ __
 | ' / | | | |_) / _ \/ _` |/ _` |/ _ \ '__|
 | . \ |_| |  _ <  __/ (_| | (_| |  __/ |
 |_|\_\___/|_| \_\___|\__,_|\__,_|\___|_|

 It's a scroll... It's a codex... It's KOReader!

 [*] Current time: 10/14/18-22:32:16
 [*] Version: v2015.11-1881-gf0d79e2a_2018-10-14

ffi.load: blitbuffer
ffi.load (warning): libblitbuffer.so: cannot open shared object file: No such file or directory
ffi.load: SDL2
ffi.load (warning): libSDL2.so: cannot open shared object file: No such file or directory
ffi.load: SDL
ffi.load (warning): libSDL.so: cannot open shared object file: No such file or directory
10/14/18-22:32:16 INFO  SoftwareVersion:  openFW
10/14/18-22:32:16 INFO  initializing for device Cervantes
10/14/18-22:32:16 INFO  framebuffer resolution: {
    ["h"] = 1448,
    ["w"] = 1072
}
ffi.load: libs/libfreetype.so.6
ffi.load: libs/libmupdf.so
ffi.load: libs/libwrap-mupdf.so
./luajit: frontend/apps/reader/modules/readerdictionary.lua:8: module 'json' not found:
    no field package.preload['json']
    no file 'common/json.lua'
    no file 'rocks/share/lua/5.1/json.lua'
    no file 'frontend/json.lua'
    no file './json.lua'
    no file '/usr/local/share/luajit-2.1.0-beta3/json.lua'
    no file '/usr/local/share/lua/5.1/json.lua'
    no file '/usr/local/share/lua/5.1/json/init.lua'
    no file 'common/json.so'
    no file 'common/json.dll'
    no file '/usr/lib/lua/json.so'
    no file 'rocks/lib/lua/5.1/json.so'
    no file './json.so'
    no file '/usr/local/lib/lua/5.1/json.so'
    no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
    [C]: in function 'require'
    frontend/apps/reader/modules/readerdictionary.lua:8: in main chunk
    [C]: in function 'require'
    frontend/apps/reader/readerui.lua:26: in main chunk
    [C]: in function 'require'
    ./reader.lua:212: in main chunk
    [C]: at 0x000139b1

So it seems that the problem isn't related to the compiled libraries, but related to the json module. Do I need to install some program/library on build device?

NiLuJe commented 5 years ago

That's actually a good question :D.

I have no idea where we're pulling the json module from. The docs mention a JSON.lua that's nowhere to be found, and yet this works just fine pretty much everywhere...

NiLuJe commented 5 years ago

Scratch that, we get it via luajson w/ luarocks ;).

NiLuJe commented 5 years ago

Was there anything about weird and/or missing luarocks calls on your build log?

(IIRC, it's annoying to handle: you need luarocks, and one that'll put rocks in a Lua 5.1 compatible path, which some distributions actually mess up, especially when they're no longer using Lua 5.1 or LuaJIT).

EDIT: And no, I don't think we log anywhere by default (unless I missed something in the kodev script), so you'd have to handle that yourself w/ I/O redirections.

NiLuJe commented 5 years ago

LuaRocks 3 appears to have made the situation easier to handle with new --lua-dir && --lua-version flags, but I have not tested this myself ;).

pazos commented 5 years ago

(IIRC, it's annoying to handle: you need luarocks, and one that'll put rocks in a Lua 5.1 compatible path, which some distributions actually mess up, especially when they're no longer using Lua 5.1 or LuaJIT).

Seems that Ubuntu 18.04 is on that boat:

Installing https://luarocks.org/luajson-1.3.4-1.src.rock
Missing dependencies for luajson 1.3.4-1:
   lpeg >= 0.8.1 (not installed)

luajson 1.3.4-1 depends on lpeg >= 0.8.1 (not installed)
Installing https://luarocks.org/lpeg-1.0.1-1.src.rock
gcc -O2 -fPIC -I/usr/include/lua5.2 -c lpcap.c -o lpcap.o
lpcap.c:6:10: fatal error: lua.h: No existe el archivo o el directorio
 #include "lua.h"
          ^~~~~~~
compilation terminated.

Error: Failed installing dependency: https://luarocks.org/lpeg-1.0.1-1.src.rock - Build error: Failed compiling object lpcap.o

Switching to Ubuntu 16.04 which plays nice w/ lua 5.1

pazos commented 5 years ago

The illegal instruction seems related to my (kernel) device not-compliant with cortex-a8 specs (ie: no simd support)

I think we need a new arm7-a without neon in Makefile.defs.

ARMV7_GENERIC_ARCH:=-march=armv7-a -mtune=generic-armv7-a -mfpu="vfpv3" -mthumb

I hope I will be able to sort out this quickly because I'm spending most of my free time on this:disappointed:

UPDATE: :smile: , issue solved.