Web UI and NickelMenu early feedback request

[shermp]

Over the past week or so, I've been working on a UI overhaul of KU. Namely, I've ripped out all the fbink "UI", and replaced it with a new web based UI that is actually interactive.

I'm also ditching kfmon in favor of NickelMenu (sorry @NiLuJe , but I don't think this whole thing would work with kfmon).

And I'm ditching the whole "do stuff behind Nickel's back" approach. I think it adds a lot of complexity for little gain. Yes that means using the DB while Nickel is also using it, but it seems looking at sqlite docs, with the right connection settings, it should be pretty safe, although one might want to backup their DB before testing, just in case. FWIW, I haven't seen any DB corruption with the new code.

I think I'm about ready for some early feedback if anyone is willing to give it a go. NickelMenu is required to launch it. I'm currently using the following NickelMenu config:

menu_item :main :Kobo UNCaGED :cmd_spawn :/mnt/onboard/.adds/kobo-uncaged/nm-start-ku.sh
chain  :nickel_extras :web_browser
menu_item :main :Book Scan :nickel_misc :rescan_books_full
chain :nickel_misc :rescan_books_full

Although this could use some refinement. @geek1011 do you know if the rescan_books actions are synchronous or not? I'm not really sure. Sometimes tapping "Book Scan" once works, other times I need to tap it a second time. Also, the non-full variant doesn't appear to populate series?

Once the browser is launched, you will need to navigate to localhost:8181. I would recommend setting this to your home page, or add it as a favorite.

Please be aware, there is still plenty of jank. Lots of jank. First, the web UI is a retrofit. Second, I'm not experienced with writing web apps. Third, the Kobo browser is truly awful. Fourth, I suck at web design, which makes point three even worse.

Pinging @tmjwid as well, as this might solve your Wifi issues when it's release worthy. Mainly because I'm letting Nickel handle all the wifi stuff via the web browser.

My main areas of concern are it's still possible for KU to get "stuck", requiring killing the process(es). Also, there's currently nothing stopping multiple launches. Probably need to deal with that in the startup script, any ideas on that @NiLuJe ? Other than that, I want to try and reduce the jank. I won't say no to any help if there's any interest.

The new code is in the webui-nm branch. I'm afraid there's no fancy CI/CD setup with build artifacts 😭 On a sidenote, anyone interested in dockerizing koxtoolchain? I could use NickelTC I suppose, but KU doesn't need such an old toolchain...

EDIT: A couple of features of the new Web UI.

• Change settings without needing to edit the config file.
• Enter password via browser UI, if required instead if hardcoding them in the config file. Passwords are also now associated with their respective Calibre library UUID, instead of using a dumb list.

[pgaskin]

On a sidenote, anyone interested in dockerizing koxtoolchain

I've been working on that on and off over the last week (the thing is, I rarely use the other TCs myself, so it's not a priority). For this specific scenario, you should be able to just use the existing Dockerfile, dump the sysroot stuff, and switch the ct-ng target. Something like this:

FROM debian:buster-slim AS base

USER root

RUN apt-get update -qqy && \
    DEBIAN_FRONTEND=noninteractive apt-get install -qqy \
        autoconf autoconf-archive automake bash bison build-essential \
        busybox-static bsdutils bzip2 coreutils curl diffutils file findutils \
        flex gawk git gperf grep gzip jq libtool make nano openssh-client perl \
        rsync sed unzip wget xz-utils zip && \
    rm -rf /var/lib/apt/lists

RUN mkdir -p /tc/x-tools

FROM base AS build

ENV HOME=/tc

# note: downloads to /tc/ctng-src
RUN git init /tc/ctng-src && \
    git -C /tc/ctng-src remote add origin https://github.com/NiLuJe/crosstool-ng.git && \
    git -C /tc/ctng-src fetch origin && \
    git -C /tc/ctng-src checkout --recurse-submodules 23ba174c7ebdefc09dc610286c92619c610e4d27 && \
    git -C /tc/ctng-src submodule update --init --recursive

# note: builds to /tc/ctng-out
RUN cd /tc/ctng-src && \
    ./bootstrap && \
    ./configure --prefix="/tc/ctng-out" && \
    make -j12 && \
    make install && \
    rm -rf /tc/ctng-src

# note: downloads to /tc/tc-cache, mount as a Docker volume for offline builds
#       later (files will only be downloaded if needed)
RUN mkdir /tc/tc-src /tc/tc-cache && \
    /tc/ctng-out/bin/ct-ng -C /tc/tc-src arm-kobo-linux-gnueabihf && \
    echo 'CT_SAVE_TARBALLS=y' >> /tc/tc-src/.config && \
    echo 'CT_LOCAL_TARBALLS_DIR="/tc/tc-cache"' >> /tc/tc-src/.config && \
    echo 'CT_EXPERIMENTAL=y' >> /tc/tc-src/.config && \
    echo 'CT_ALLOW_BUILD_AS_ROOT=y' >> /tc/tc-src/.config && \
    echo 'CT_ALLOW_BUILD_AS_ROOT_SURE=y' >> /tc/tc-src/.config && \
    echo 'CT_LOG_PROGRESS_BAR=n' >> /tc/tc-src/.config && \
    /tc/ctng-out/bin/ct-ng -C /tc/tc-src oldconfig && \
    /tc/ctng-out/bin/ct-ng -C /tc/tc-src updatetools && \
    /tc/ctng-out/bin/ct-ng -C /tc/tc-src build CT_ONLY_DOWNLOAD=y

# note: builds to /tc/x-tools/arm-kobo-linux-gnueabihf, should be relocatable
RUN /tc/ctng-out/bin/ct-ng -C /tc/tc-src build CT_FORBID_DOWNLOAD=y && \
    rm -rf /tc/tc-src

FROM base AS toolchain

COPY --from=build /tc/x-tools/arm-kobo-linux-gnueabihf/ /tc/arm-kobo-linux-gnueabihf/
ENV PATH="/tc/arm-kobo-linux-gnueabihf/bin:${PATH}"

CMD ["/bin/bash"]

[pgaskin]

do you know if the rescan_books actions are synchronous or not? I'm not really sure. Sometimes tapping "Book Scan" once works, other times I need to tap it a second time. Also, the non-full variant doesn't appear to populate series?

Mostly; that may be due to the UI updating (see if it works after opening and closing a book instead of doing another scan); and yes, that's correct.

[pgaskin]

Please be aware, there is still plenty of jank. Lots of jank. First, the web UI is a retrofit. Second, I'm not experienced with writing web apps. Third, the Kobo browser is truly awful. Fourth, I suck at web design, which makes point three even worse.

I'll take a look. I'm mainly a backend or systems person, but I sometimes do frontend web stuff if needed.

Edit: It's not too bad, but some of the JS is slightly more dated than it needs to be, some of the code needs to be formatted, and it really needs some more CSS. Do you want me to make it look similar to Nickel, or like something else entirely (I'll work on it sometime in the next week and a half)?

[pgaskin]

Oh, and I suggest you ship a NM config with KU (put it in .adds/nm/ku or something like that).

[NiLuJe]

Concurrent writes should be much less annoying since the switch to WAL.

As long as you yourself are not doing threading, even a simple single-threaded build of SQLite will do just fine.

(In fact, the threading modes & open flags make no differences as far as cross-process concurrency is involved, that's handled via Posix advisory locks, no matter what).

[NiLuJe]

As far as avoiding duplicate processes, you can check if a process by that name already exists via (p)kill -0, c.f., https://github.com/NiLuJe/kfmon/blob/a17f3060513748d8d9903afe979a118347bc9427/scripts/on-animator.sh#L5

(And kill(2) for why this works).

EDIT: Since this is a shell script, and we don't have exec -a to alter process names, you'll probably want to go w/ pkill -f ;).

Or, since we're on Linux, you can also check if /proc/<PID> exists, assuming you know the PID (or tried to get it via pidof/pgrep, and confirmed that you got an int out of it).

Or you can even create a pidfile if you want to go the extra mile ;). (IIRC, the stock busybox build might even support the start-stop-daemon applet).

[shermp]

Thanks for the feedback guys!

Edit: It's not too bad, but some of the JS is slightly more dated than it needs to be, some of the code needs to be formatted, and it really needs some more CSS. Do you want me to make it look similar to Nickel, or like something else entirely (I'll work on it sometime in the next week and a half)?

Yeah, I know it needs more CSS. The default form element styling in particular is horrible (especially the checkboxes!). It's difficult to figure out what exactly is supported, as old qtwebkit isn't exactly a common target, so it's difficult to consult compatibility matrices. I tend to err on the side of stuff that works on early safari, or maybe early chrome.

Also, the CSS for the password modal doesn't really work 😞 It looks fine when it opens, but as soon as one taps the password entry field to enter the password, all the text disappears. Maybe I need to add some JS to refresh the content on activating the input field?

Oh, and I suggest you ship a NM config with KU (put it in .adds/nm/ku or something like that).

I'm planning on it! Just need to nail down the exact config I'm going to use.

...

My main issue with killing KU is identifying the processes. There's three of them. The shell process which QProcess spawns. The startup script, then the KU binary. eg:

1336 root       0:00 /bin/sh -c /mnt/onboard/.adds/kobo-uncaged/nm-start-ku.sh
1337 root       0:00 {nm-start-ku.sh} /bin/sh /mnt/onboard/.adds/kobo-uncaged/nm-start-ku.sh
1341 root       0:00 /mnt/onboard/.adds/kobo-uncaged/bin/kobo-uncaged

[NiLuJe]

My main issue with killing KU is identifying the processes. There's three of them. The shell process which QProcess spawns. The startup script, then the KU binary. eg:

1336 root       0:00 /bin/sh -c /mnt/onboard/.adds/kobo-uncaged/nm-start-ku.sh
1337 root       0:00 {nm-start-ku.sh} /bin/sh /mnt/onboard/.adds/kobo-uncaged/nm-start-ku.sh
1341 root       0:00 /mnt/onboard/.adds/kobo-uncaged/bin/kobo-uncaged

If the only issue is preventing concurrent startups, discriminating between those is not really necessary, right?

If any of 'em are up, it's a trip to an early abort, and that's it.

If you did need to actually SIGTERM one of 'em, I'd try with the earliest one. pgrep has fancy filtering options, including by PPID. The QProcess one will be reparented to init, so pgrep -f nm-start-ku.sh -P 1 ought to do the trick ;).

[NiLuJe]

Also, I imagine using exec in the cmd_spawn would get rid of one of those ;).

[pgaskin]

You can also anchor the pkill -f to the start of the command with ^, as it's a regexp.

[shermp]

Another thought, any ideas where I should save the cached passwords? Currently just saving them to the KU directory for now, but I would like to make it at leas a little harder for "little johnny" to find.

I was thinking of saving them somewhere on rootfs, but where? Traditionally, one would use the home directory but kobo... /etc perhaps?

Probably not going to bother protecting it further with encoding/encryption. The password is saved in plaintext on calibres end anyway.

[shermp]

Arrrgh, I hate JavaScript sometimes.

Trying to generate a clickable list of instances (if there are multiple Calibre instances running on the network). I can generate and show the list fine. Problem is in firefox the onclick event never fires 😞

function getInstances() {
        var xhr = new XMLHttpRequest();
        xhr.open('GET', instancePath);
        xhr.onload = function() {
            if (xhr.status === 200) {
                instList = JSON.parse(xhr.responseText);
                for (var i = 0; i < instList.length; i++) {
                    var instNode = document.createElement('div');
                    instNode.dataset.instanceId = i;
                    var pAddr = document.createElement('p');
                    var pDesc = document.createElement('p');
                    pAddr.innerHTML = instList[i].Addr;
                    pDesc.innerHTML = instList[i].Description;
                    instNode.append(pAddr, pDesc);
                    instNode.id = 'cal_instance_' + i;
                    instNode.onclick = function() {setInstance(instNode.id);};
                    instanceContainer.append(instNode);
                }
                instanceModal.style.display = "block";
            }
            else {
                console.log('Request failed.  Returned status of ' + xhr.status);
            }
        };
        xhr.send();
    }
    function setInstance(id) {
        var el = document.getElementById(id)
        var xhr = new XMLHttpRequest();
        xhr.open('POST', instancePath);
        var instance = instList[el.dataset.instanceId];
        xhr.onload = function() {
            if (xhr.status === 205) {
                instanceModal.style.display = "none";
            } else {
                console.log('Unexpected status code. Expected 205, got ' + xhr.status);
            }
        }
        xhr.send(JSON.stringify(instance));
    }

Any ideas?

[pgaskin]

First, you don't need to set the ID. You can just pass the node itself. Second, you should be using addEventListener instead of onclick.

Overall, I actually suggest you use Vue + a es6 promise polyfill + a fetch polyfill instead of all this JavaScript, as it will make this kind of UI trivial to implement.

[shermp]

I guess at the end of the day, My aim was/is to minimize any external dependencies. Also, trying to do as little as possible in JS, because crappy qtwebkit. But I'm open to the idea of other alternatives.

[shermp]

I've been looking at Vue.js (and similar frameworks) and ugh, I was really hoping to avoid stuff like npm (or yarn), transpilers and the like.

No matter what I end up doing frontend wise though, I'm definitely thinking of doing most of the rendering client-side and making the Go side of things simpler. The Go code can then just serve JSON, rather than rendering HTML/JS templates.

Maybe I can boil down the API to something like:

/ # render the main application. Single Page App perhaps?
/config # to get and set the configuration using GET/POST
/exit # To exit the application at the config screen
/disconnect # to disconnect from Calibre
/messages # Subscribe to Server Sent Events
/auth # get and set Calibre auth info

[tmjwid]

Hey sorry I've been MIA, I'll have a play with this new build when I get the chance. I'm guessing I'll need to build the toolchain and compile for this version yeah? I'll spin up a VM just in case.

[shermp]

Hey sorry I've been MIA, I'll have a play with this new build when I get the chance. I'm guessing I'll need to build the toolchain and compile for this version yeah? I'll spin up a VM just in case.

Don't be in too much of a hurry. I'm currently in the middle of a major refactor of this particular functionality. There was a reason I haven't yet made it a PR :)

[shermp]

And don't worry about setting up a toolchain. I'll attach a build to this thread when I have one ready.

[shermp]

Right, WebUI, take two. Out goes the horrible mess of templates and page redirection. In comes the single page app (SPA).

I decided against using a UI framework like VueJS, because a) I didn't feel like learning another new framework, and b) I like the fact that one can jump in and tinker with the code (on device even!) without needing to install any javascript dev tools like node/npn/yarn etc.

There still needs to be more work done. Styling is still shite (the default Kobo styles are truly horrid). Now that config is done graphically, I'm thinking of ditching the toml config file, and just saving settings a JSON. Also, need to work on proper NickelMenu integration.

And speaking of NickelMenu, I managed to break Nickel when I went to rescan. For some reason, the database was still locked when Nickel tried to use it. Needless to say, Nickel was Not Happy. I had to reboot using telnet, because the power button was unresponsive.

So yeah, I need to REALLY make sure the database is not locked before doing a library scan.

New code lives in the webui-refactor branch.

[NiLuJe]

IIRC, the only way the database will be (write) locked on a WAL db is if someone (i.e., Nickel) is actively writing to it (possibly just committing, actually?).

IIRC, there's also no 100% proof way to check if the DB is locked, because it might get locked between the check and the action you wanted to make sure wouldn't happen on a locked DB.

(The sane approach is to have every db action sanely handle a locked db (i.e., via sqlite3busy*), which, err, might be tricky here ^^).

[NiLuJe]

https://sqlite.org/rescode.html#busy mentions using BEGIN IMMEDIATE to at least avoid getting a write lock in the middle of a transaction, which sounds neat/useful ;).

[NiLuJe]

Forcing WAL checkpointing (https://www.sqlite.org/c3ref/wal_checkpoint_v2.html) might also help?

[shermp]

Note, I'm using go-sqlite3 to access the database, and accessing it through Go's SQL interface, so I don't have direct control over the DB connection.

My main issue I think is the very rudimentary NM config I'm using doesn't do any checks. Probably the safest thing to do is to check all the KU processes has ended before triggering the book scan.

If @geek1011 ever manages to get series parsing for epubs working, I may end up skipping database writes altogether.

It does highlight what I think MIGHT be a NM weakness though. If a blocking process like rescan_books_full freezes, it might be possible for the Kobo to get into an unusable state. Note, I could not trigger a reboot by pressing the power button for over 30s. Thankfully, I had telnet access at the time...

I don't know if there's a way to mitigate this risk. Perhaps "don't play with Nickel's DB" might be the only option.

[NiLuJe]

sickel should be murdering a hung or busy-looping Nickel, AFAIK? (I don't recall the exact details of how the sickel -> nickel ping works :s).

What did things look like over telnet at the time?

[shermp]

Apart from Nickel using around 60-80% CPU usage (via top) and the DB errors in the syslog, it was otherwise normal.

Initially, I didn't think anything was wrong, but when the "Importing Content" screen never appeared, I started to get a bit worried. It was then I noticed the UI was completely frozen.

[NiLuJe]

Re: sickel

Possibly a dbus ping?

In any case, if I SIGSTOP nickel, sickel will reboot after ~1 minute:

01:13:45.975967 [2c3b2f44] pselect6(7, [3<anon_inode:[eventfd]> 6<socket:[1831]>], [], [], {tv_sec=9, tv_nsec=980000000}, {NULL, 8}) = 1 (in [6], left {tv_sec=4, tv_nsec=996720260})
01:13:50.961220 [2c15756c] recvmsg(6<socket:[1831]>, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\1\0\1\0\0\0\0\31\3\0\0}\0\0\0\1\1o\0\1\0\0\0/\0\0\0\0\0\0\0\6\1s\0\30\0\0\0com.kobo.watchdog.Sickel\0\0\0\0\0\0\0\0\2\1s\0\30\0\0\0com.kobo.watchdog.Sickel\0\0\0\0\0\0\0\0\3\1s\0\4\0\0\0Ping\0\0\0\0"..., iov_len=2048}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = 144
01:13:50.985238 [2c15756c] recvmsg(6<socket:[1831]>, {msg_namelen=0}, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
01:13:50.986004 [2c3aaebc] write(3<anon_inode:[eventfd]>, "\1\0\0\0\0\0\0\0", 8) = 8
01:13:50.986402 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3446, tv_nsec=31833426}) = 0
01:13:50.986718 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3446, tv_nsec=32143926}) = 0
01:13:50.987047 [2c1576cc] sendmsg(6<socket:[1831]>, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\2\1\1\0\0\0\0\372\2\0\0\30\0\0\0\6\1s\0\4\0\0\0:1.0\0\0\0\0\5\1u\0\31\3\0\0", iov_len=40}, {iov_base="", iov_len=0}], msg_iovlen=2, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 40
01:13:50.988767 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3446, tv_nsec=34228926}) = 0
01:13:50.988987 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3446, tv_nsec=34394926}) = 0
01:13:50.989125 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3446, tv_nsec=34524926}) = 0
01:13:50.989247 [2c3b2f44] pselect6(7, [3<anon_inode:[eventfd]> 6<socket:[1831]>], [], [], {tv_sec=9, tv_nsec=966000000}, {NULL, 8}) = 1 (in [3], left {tv_sec=9, tv_nsec=965986375})
01:13:50.989862 [2c3aae2c] read(3<anon_inode:[eventfd]>, "\1\0\0\0\0\0\0\0", 8) = 8
01:13:50.990069 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3446, tv_nsec=35478676}) = 0
01:13:50.990221 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3446, tv_nsec=35624051}) = 0
01:13:50.990348 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3446, tv_nsec=35743051}) = 0
01:13:50.990469 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3446, tv_nsec=35861801}) = 0
01:13:50.990597 [2c3b2f44] pselect6(7, [3<anon_inode:[eventfd]> 6<socket:[1831]>], [], [], {tv_sec=9, tv_nsec=965000000}, {NULL, 8}) = 0 (Timeout)
01:14:00.966806 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3456, tv_nsec=12660049}) = 0
01:14:00.967939 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3456, tv_nsec=13635896}) = 0
01:14:00.968709 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3456, tv_nsec=14331839}) = 0
01:14:00.969383 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3456, tv_nsec=15003530}) = 0
01:14:00.970037 [2c3b2f44] pselect6(7, [3<anon_inode:[eventfd]> 6<socket:[1831]>], [], [], {tv_sec=9, tv_nsec=987000000}, {NULL, 8}) = 0 (Timeout)
01:14:10.970130 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3466, tv_nsec=15977184}) = 0
01:14:10.971253 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3466, tv_nsec=16938867}) = 0
01:14:10.972006 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3466, tv_nsec=17632818}) = 0
01:14:10.972691 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3466, tv_nsec=18319769}) = 0
01:14:10.973347 [2c3b2f44] pselect6(7, [3<anon_inode:[eventfd]> 6<socket:[1831]>], [], [], {tv_sec=9, tv_nsec=984000000}, {NULL, 8}) = 0 (Timeout)
01:14:20.972931 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3476, tv_nsec=18772896}) = 0
01:14:20.974054 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3476, tv_nsec=19749701}) = 0
01:14:20.977097 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3476, tv_nsec=22909174}) = 0
01:14:20.978023 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3476, tv_nsec=24788058}) = 0
01:14:20.979895 [2c3b2f44] pselect6(7, [3<anon_inode:[eventfd]> 6<socket:[1831]>], [], [], {tv_sec=9, tv_nsec=981000000}, {NULL, 8}) = 0 (Timeout)
01:14:30.978288 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3486, tv_nsec=24150431}) = 0
01:14:30.979425 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3486, tv_nsec=25115459}) = 0
01:14:30.980184 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3486, tv_nsec=25811979}) = 0
01:14:30.980865 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3486, tv_nsec=26499125}) = 0
01:14:30.981539 [2c3b2f44] pselect6(7, [3<anon_inode:[eventfd]> 6<socket:[1831]>], [], [], {tv_sec=9, tv_nsec=975000000}, {NULL, 8}) = 0 (Timeout)
01:14:40.969587 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3496, tv_nsec=15438765}) = 0
01:14:40.970721 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3496, tv_nsec=16401139}) = 0
01:14:40.971494 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3496, tv_nsec=17120764}) = 0
01:14:40.972189 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3496, tv_nsec=17816138}) = 0
01:14:40.972855 [2c3b2f44] pselect6(7, [3<anon_inode:[eventfd]> 6<socket:[1831]>], [], [], {tv_sec=9, tv_nsec=984000000}, {NULL, 8}) = 0 (Timeout)
01:14:50.963426 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3506, tv_nsec=9265370}) = 0
01:14:50.964533 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3506, tv_nsec=12455270}) = 0
01:14:50.968532 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3506, tv_nsec=17051751}) = 0
01:14:50.972262 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3506, tv_nsec=18024739}) = 0
01:14:50.979396 [2c3b2f44] pselect6(7, [3<anon_inode:[eventfd]> 6<socket:[1831]>], [], [], {tv_sec=9, tv_nsec=988000000}, {NULL, 8}) = 0 (Timeout)
01:15:00.980738 [2c3c831c] clock_gettime(CLOCK_MONOTONIC, {tv_sec=3516, tv_nsec=26600410}) = 0
01:15:00.984481 [2c37544c] gettimeofday({tv_sec=1590621300, tv_usec=987055}, NULL) = 0
01:15:00.989173 [2c3531dc] open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 7</etc/zoneinfo/Europe/Paris>
01:15:00.991482 [2c3aa29c] fstat64(7</etc/zoneinfo/Europe/Paris>, {st_dev=makedev(0xb3, 0x1), st_ino=33452, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=1024, st_blocks=6, st_size=2945, st_atime=1590595171 /* 2020-05-27T17:59:31+0200 */, st_atime_nsec=0, st_mtime=1288665484 /* 2010-11-02T03:38:04+0100 */, st_mtime_nsec=0, st_ctime=1420991891 /* 2015-01-11T16:58:11+0100 */, st_ctime_nsec=0}) = 0
01:15:00.994378 [2c3aa29c] fstat64(7</etc/zoneinfo/Europe/Paris>, {st_dev=makedev(0xb3, 0x1), st_ino=33452, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=1024, st_blocks=6, st_size=2945, st_atime=1590595171 /* 2020-05-27T17:59:31+0200 */, st_atime_nsec=0, st_mtime=1288665484 /* 2010-11-02T03:38:04+0100 */, st_mtime_nsec=0, st_ctime=1420991891 /* 2015-01-11T16:58:11+0100 */, st_ctime_nsec=0}) = 0
01:15:00.999343 [2c3b6598] mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aad3000
01:15:01.000330 [2c352cd8] read(7</etc/zoneinfo/Europe/Paris>, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\f\0\0\0\f\0\0\0\0\0\0\0\267\0\0\0\f\0\0\0\33\221`P\213\233Gx\360\233\327,p\234\274\221p\235\300H\360\236\211\376p\237\240*\360\240`\245\360\241\200\f\360\242.\22\360\243zL\360\2445\201\360\245^#p\246%5\360\247'\233\360\250X&p\251\7}\360\251\3564p\252\347_\360\253\327P\360\254\307A\360"..., 1024) = 1024
01:15:01.004034 [2c352cd8] read(7</etc/zoneinfo/Europe/Paris>, "\21\0\0\16\20\0\rPMT\0WEST\0WET\0CET\0CEST\0WEMT\0\0\1\1\0\0\1\1\0\0\0\1\1\0\0\0\0\0\0\0\0\0\0\1\1TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\r\0\0\0\r\0\0\0\0\0\0\0\270\0\0\0\r\0\0\0\37\377\377\377\377k\310J\213\377\377\377\377\221`P\213\377\377\377\377\233Gx\360\377\377"..., 1024) = 1024
01:15:01.008933 [2c3b9fd4] _llseek(7</etc/zoneinfo/Europe/Paris>, 869, [2917], SEEK_CUR) = 0
01:15:01.011348 [2c352cd8] read(7</etc/zoneinfo/Europe/Paris>, "\nCET-1CEST,M3.5.0,M10.5.0/3\n", 1024) = 28
01:15:01.012597 [2c351acc] close(7</etc/zoneinfo/Europe/Paris>) = 0
01:15:01.013418 [2c3b662c] munmap(0x2aad3000, 4096) = 0
01:15:01.016852 [2c3baf2c] socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 7<socket:[9204]>
01:15:01.020543 [2c3baa4c] connect(7<socket:[9204]>, {sa_family=AF_UNIX, sun_path="/dev/log"}, 110) = 0
01:15:01.029627 [2c3bad3c] send(7<socket:[9204]>, "<15>May 28 01:15:00 sickel: void SickelService::reboot() rebooting ", 67, MSG_NOSIGNAL) = 67
01:15:01.037090 [2c3b31b0] reboot(LINUX_REBOOT_MAGIC1, LINUX_REBOOT_MAGIC2, LINUX_REBOOT_CMD_RESTART

[shermp]

I'll see if I can trigger it again, or if it was just a fluke.

[NiLuJe]

Also, random comment about something that came up at the beginning of this thread: I imagine the koreader images could effectively serve as a "dockerized" variant of every koxtoolchain TC (except nickel & kobomk7, which KOReader doesn't use).

I don't do docker so I have no clue how this stuff works, but looking at the KOReader CI build matrix should make this clear ;).

[shermp]

Yep. You can basically have one Dockerfile which uses build arguments to determine which toolchain to create an image for. Alternatively, one could have a Dockerfile for each toolchain version. But I think one would probably make more sense.

Once you have an image, it can then be pushed to Docker Hub. It also looks like you can get Docker Hub to automatically build images for you on each commit.

I guess one could use the example Dockerfile @geek1011 posted upthread as a starter. Alternatively, one could have a Dockerfile which clones the koxtoolchain repo, and runs gen-tc.sh.

EDIT: One is fond of using the term 'one'...

[pgaskin]

I have one suspicion (a guess, I haven't checked any part of it yet) as to the cause of the power button freeze. I'll check this myself later, but sickel may not be active while Nickel is in USBMS mode. Also, USBMS mode may have implications related to the power button.

[pgaskin]

Note that Docker Hub is too slow for building toolchains (see my comments in the kobo-plugin-experiments for more on the other available options).

[shermp]

I have one suspicion (a guess, I haven't checked any part of it yet) as to the cause of the power button freeze. I'll check this myself later, but sickel may not be active while Nickel is in USBMS mode. Also, USBMS mode may have implications related to the power button.

Note, this new version of KU no longer enters USBMS mode. I'm trying to see if I can use it safely without USBMS, because that adds its own hurdles (dealing with wifi, and filesystem mounts)

[shermp]

Note that Docker Hub is too slow for building toolchains (see my comments in the kobo-plugin-experiments for more on the other available options).

I wondered about that. Good to know!

[NiLuJe]

The KOReader images are definitely taking the "one image per TC" approach ^^.

c.f., https://hub.docker.com/u/koreader

[shermp]

First, I've identified the fact that I'm simply showing the "all done" message too early. It's being shown before the database is closed. 99 times out of a 100, it probably won't be an issue, but if there are any cover resize stragglers...

That's an issue I'm going to fix for sure!

I've also been thinking about switching to a less elegant, but probably safer solution. Make KU only perform read only access on the DB (and open the DB in ro mode), and save out writes to a SQL file, that can then be executed in a separate NickelMenu action using the sqlite CLI tool. That way I can hopefully do the DB update in a blocking fashion, which hopefully avoids any issues with Nickel.

[shermp]

The KOReader images are definitely taking the "one image per TC" approach ^^.

c.f., https://hub.docker.com/u/koreader

Yeah, but each image can (and probably is?) generated from one Dockerfile, using build arguments/environment variables to control which variant is built.

[NiLuJe]

IIRC, Kobo doesn't ship the SQLite CLI tool.

(It certainly didn't use to, as I deemed it safe to ship a symlink to mine in the Python packages. [As for why SQLite ships w/ Python and not the core shell tools, the short answer is: ICU ^^]).

[shermp]

IIRC, Kobo doesn't ship the SQLite CLI tool.

(It certainly didn't use to, as I deemed it safe to ship a symlink to mine in the Python packages. [As for why SQLite ships w/ Python and not the core shell tools, the short answer is: ICU ^^]).

I was planning on shipping my own, assuming it's small enough, and can be statically linked.

[NiLuJe]

Short of the (optional) ICU dependency, it can be built without readline (IIRC), and there's no longer any third-party deps at that point ;).

(Short of a build dep: tcl which might be needed for modular builds).

[shermp]

Yeah, looking at the docs, a minimal build looks to be as simple as

gcc -DSQLITE_THREADSAFE=0 shell.c sqlite3.c -ldl

[shermp]

Ok, KU now delays its "Done" user messages to the very last possible moment, and crucially, after the database is closed. Also added a basic NickelMenu config.

I need to add more safety to the startup script (by killing any previous KU instances). And only allow the rescan_books_full to occur if no KU process is running.

The only thing I'm worried about now is the possibility of a Nickel DB write during KU operation. I hope being in browser helps here, but who knows?

[NiLuJe]

You'll get thrown a SQLITE_BUSY if Nickel is actively writing somewhere for some reason, so, providing both sides have a sane busy handler, it should behave ;).

EDIT: And it's apparently handled via Open query parameters in go-sqlite3 (for the stock busy_timeout handler).

EDIT²: Which you're already using ;p. There's also the _txlock parameter for the IMMEDIATE stuff from earlier ;).

[shermp]

You'll get thrown a SQLITE_BUSY if Nickel is actively writing somewhere for some reason, so, providing both sides have a sane busy handler, it should behave ;).

We ARE talking about Nickel you know ^ ^

[NiLuJe]

Sidebar about SQLITE_BUSY: I find the human-readable variant from sqlite3_errstr/sqlite3_errmsg so incredibly confusing when compared to SQLITE_LOCKED, when the two actually mean very different things, that I kind of jumped the gun and went for the internal API in KFMon to get at the constant name instead: https://github.com/NiLuJe/kfmon/blob/1958fba1bec3ecf203742e242f66d4850ffca0d1/kfmon.c#L2785 -> https://github.com/sqlite/sqlite/blob/d63c76fb31a0e262fb12a93b171e95a4e30c1a2e/src/main.c#L1447-L1451

(That said, there's approximately 0% chance we'll ever hit SQLITE_LOCKED unless we've seriously screwed the pooch ^^. (Single thread, synchronous, single db connection. Which is also why a no-mutex, no shared cache, single-threaded SQLite build/connection works just fine).)

[shermp]

The more I think about it, the more I want to use the sqlite cli method of updating the DB. I'm kind of uncomfortable leaving a rw connection open for an arbitrarily length of time, especially if the user can navigate away from the browser at any time, and access any of the library views, or start a sync, or...

With CLI approach, KU can can open the DB read-only, get a current list of books, then close the DB, and never connect to it again. I can then run the DB update in a UI blocking manner, which should hopefully minimize any potential whoopsies.

Looking around, it's difficult to find resources on how to create a safe SQL query, but it seems for sqlite, escaping ' in strings should be enough for my purposes. Everyone shouts about using prepared statements/parameters, but that won't work for the CLI.

[shermp]

Awww, the rescan_books_full really doesn't appear to be blocking :( Testing with the following config:

menu_item :main :KU - Import Books :dbg_toast :Starting replacement SQL
chain :cmd_output :5000:/mnt/onboard/.adds/kobo-uncaged/scripts/sql-replacements.sh
chain  :dbg_toast :Starting Lib Scan 1
chain  :nickel_misc :rescan_books_full
chain  :dbg_toast :Starting Update SQL
chain  :cmd_output     :5000:/mnt/onboard/.adds/kobo-uncaged/scripts/sql-updates.sh
chain  :dbg_toast :Starting Lib Scan 2
chain  :nickel_misc :rescan_books_full

I get the Starting Update SQL and :Starting Lib Scan 2 toasts appearing while the first 'Importing Content' screen is still showing.

@geek1011 do you have any ideas on how I might be able to deal with this in NM?

[NiLuJe]

Crappy workaround ideas ahead:

Hmmm, something something SQL (inside a blocking shell script)?

Along the lines of periodically checking if a specific TRIGGER has fired based on whatever the full_scan is supposed to do with the new data?

Or waiting for the database not to be held in a write lock for more than n seconds?

(... or parsing the syslog output :D. I feel dirty.)

[shermp]

(... or parsing the syslog output :D. I feel dirty.)

Don't worry. I've had that particular thought as well. Doesn't make it any less dirty though...

[shermp]

I'm wondering the sync/scan process emits some sort of signal that one could wait for using QEventLoop?

[pgaskin]

I'm wondering the sync/scan process emits some sort of signal that one could wait for using QEventLoop?

Well, we are already inside of it, so I can't exactly block it. I could add a hidden option to run the remaining options in another thread, but I'd rather not if there's a better way.