Initial testing

[shermp]

Starting an issue thread to gather early testing results.

@NiLuJe if you have any issues or comments, leave 'em here. Probably better than continuing the discussions in PR's

[shermp]

I deleted a book. And transferred a book :D

I haven't really changed anything @NiLuJe , so if it doesn't work, make sure Calibre is listening, and the ports aren't being blocked.

Issues so far:

• FBInk printing is not behaving how I want it to. It SHOULD show up in a centred box, with the previous contents being cleared for each print. I've probably got some sums wrong in kuPrintEink.go
• Need to improve status messages to let users have a better idea of what KU is doing, especially when waiting for cover thumbnails to be generated, which leads in to the next issue...
• Cover/thumbnail generation is s l o w. Like really slow. I obviously need to find a faster library to do this. Probably needs to be based on a C/C++ lib with ARM optimizations.

[NiLuJe]

I'll do a round of testing tomorrow ;).

What are you currently using to generate the thumbnails? I don't know if that's doable in this context, but it might be worth letting Nickel handle all that, because otherwise, doing it right is going to be expensive, no matter what ;).

[shermp]

I'm using a Go library. Just noticed that I asked for it to use Lanczos algorithm. Perhaps (ok, almost certainly) this was a tad ambitious, and the quality is probably lost on such small images anyway. Will try switching to bilinear and see if that helps (it will, just not sure by how much...)

I really want to get thumbnail generation working at an acceptable performance level. Nothing more annoying (for me) to wait at the library screen while Nickel takes its sweet time generating thumbnails...

I'm using a go routine for image generation, so it doesn't block the rest of the code, at least not until we're ready to exit. So long as I can get performance good enough that it's all done by the time the user "ejects" from Calibre, that's good enough for me.

Note when you are having a go, the correct way to exit is to "eject" the device in Calibre, like you would with a USB connection.

[shermp]

So, I've got thumbnail generation to an acceptable performance level. Mainly by deciding not to bother with the full cover at all. KU now requests a (much) smaller cover size (the dimensions of the - N3_LIBRARY_FULL.parsed image), and downscales that where necessary. Nickel can generate the full cover from the ebook, when required.

Also fixed an issue around updating the Nickel DB with NULL values. I was sending the string 'NULL' instead of actually setting NULL values. Changed the variable types to pointers, and Go handled the rest :)

And I haven't had any troubles exiting and returning from Nickel. Thanks for making the shell scripts so much more robust @NiLuJe

[NiLuJe]

After a false start because I have no idea how Go works, and despite a go get, I ended up apparently building old sources, it now appears to be going better!

I'd forgotten that you had to enable a service in Calibre for the Wireless thingy, which probably explains my earlier failures (speaking of, that probably warrants a user-visible warning on-screen, something along the lines of: couldn't connect to Calibre, check that it's running and the Wireless connection service has been started).

• The Running Kobo-UNCaGED/Creating KU object step still takes a good long while over here, an on-screen hint that it might be the case could be helpful here, too ;).

• The KoboUNCaGED log entries are timestamped twice:

May  4 17:04:31 UNCaGED: USBMS mode entered . . .
May  4 17:04:31 UNCaGED: Running Kobo-UNCaGED
May  4 17:04:31 KoboUNCaGED[3376]: 2019/05/04 17:04:31 Started Kobo-UNCaGED
May  4 17:04:31 KoboUNCaGED[3376]: 2019/05/04 17:04:31 /mnt/newonboard
May  4 17:04:31 KoboUNCaGED[3376]: 2019/05/04 17:04:31 Creating KU object
May  4 17:04:53 dropbear[3380]: Child connection from 192.168.0.42:54120
May  4 17:04:53 dropbear[3380]: Pubkey auth succeeded for 'root' with key sha1!! 
[root@(none) ~]# logread -f
May  4 17:07:21 KoboUNCaGED[3376]: 2019/05/04 17:07:21 Preparing Kobo UNCaGED!
May  4 17:07:22 KoboUNCaGED[3376]: 2019/05/04 17:07:22 Starting Calibre Connection

You don't need explicit timestamps, the syslog handles those ;).

• Definitely see what you meant with the OT FBInk prints ;). Haven't looked at the code, though.

[NiLuJe]

Yeah, when Calibre is down, I see a May 4 17:13:40 KoboUNCaGED[3834]: 2019/05/04 17:13:40 calibre server not found in the logs, but I never saw it on-screen, only the previous Preparing Kobo UNCaGED! was visible ;).

[NiLuJe]

Aaaaand, semi-success: I apparently successfully downloaded a book, I can see it and everything, but I get a "Oops! This document couldn't be opened" warning when I actually try to read it ;).

I did get the processing pass, I did get the second KU run (it also took a few minutes to start). I couldn't track everything because I of course had to take a call right at this moment -_-". But the logs look sane:

May  4 17:19:43 KoboUNCaGED[4573]: 2019/05/04 17:19:43 Started Kobo-UNCaGED
May  4 17:19:43 KoboUNCaGED[4573]: 2019/05/04 17:19:43 /mnt/newonboard
May  4 17:19:43 KoboUNCaGED[4573]: 2019/05/04 17:19:43 Creating KU object
May  4 17:22:06 KoboUNCaGED[4573]: 2019/05/04 17:22:06 Updating Metadata
May  4 17:22:06 UNCaGED: Unmounting onboard
May  4 17:22:09 UNCaGED: Onboard unmounted (0) . . .
May  4 17:22:09 UNCaGED: Going back to Nickel

Side-note: this is a KePub. I have both an ePub and a KePub for this book in my Calibre library, and it preferred the KePub (which is fine by me). That may be because of my format preferences in Calibre, in which case, perfect ;).

[NiLuJe]

FWIW, I can open that same book on my Forma, where I transferred it the usual way.

[shermp]

I haven't actually tested kepub yet. it SHOULD work, but obviously there may be some kinks.

The reason it takes so long to before it connects to Calibre, is because before connecting, I gather all the metadata, which is a 2.5 pass process. First, if it exists, I read the metadata.calibre file, and put it into a map, checking with the Nickel DB that the book still exists while I do so. Then it checks the database for books that aren't in the metadata cache, and adds them by opening the (k)epub and extracting metadata.

As you can see, depending on how many books you have, and how in sync the Nickel DB and calibre cache file are, this step can take a while. I'm not sure how to speed this step up, and still present calibre with an accurate list of books on device (as seen by Nickel).

As far as preferring kepub, that's because that's the extension listed first in the code. I should probably make that configurable.

As far as the timestamps on the logs go, that's automatically added by Go's default logger which I'm using.

Agreed I need to better display the state to the user.

It's the book corruption I'm most worried about however. Will need to look into that.

Thanks so much for helping to test this.

[shermp]

Ok, sent a kepub to my H2O, and it opened fine. So not a kepub issue. If the kepub is a known good file that Nickel reads correctly, then there's probably some corruption going on in the book saving code, or networking code. Joys...

[NiLuJe]

Okay, take two:

Still no dice on that book. A few things about it: there's a bang in the title/filename, it's in French. It also inherited a completely bogus "Series" entry on the second send (it doesn't actually belong to a Series).

The main content DB entry looked like this the first time around:

file:///mnt/onboard/Gare a Lou ! - Jean Teule (6336).kepub.epub|6|application/x-kobo-epub+zip|||file____mnt_onboard_Gare_a_Lou_!_-_Jean_Teule_(6336)_kepub_epub|Gare à Lou !|Jean Teulé|<p class="description">Avec Gare à Lou ! Jean Teulé revient à la veine fantastique qui avait fait le succès du Magasin des suicides et laisse libre court à un imaginaire plus débridé que jamais. Comme le disaient Mozart et Shakespeare : " Il est très agréable de jouir d'un don exceptionnel, mais il ne faut pas oublier que c'est une source inépuisable d'embêtements ". A 12 ans, Lou partage absolument cette opinion. Au prétexte qu'elle est en mesure de faire tomber immédiatement les pires calamités sur la tête de tous ceux qui la contrarient, on l'enferme dans un endroit secret en compagnie de militaires haut gradés pour qu'elle devienne une arme absolue capable de mettre en échec les plans malveillants des ennemis du pays ou, pire, d'ourdir de méchantes et sournoises manoeuvres afin de causer des torts effroyables à d'autres nations. De telles occupations n'offrent pas à une adolescente les satisfactions que la vie aurait pu lui promettre. D'autant que son super pouvoir, aussi extraordinaire soit-il, ne fonctionne pas toujours comme prévu. Rien ne pouvait mieux inspirer Jean Teulé que d'imaginer les horreurs qu'un être humain bien disposé peut infliger à ses contemporains.</p>|||/mnt/onboard/Gare a Lou ! - Jean Teule (6336).kepub.epub|Julliard|true||true||0|0|890|0|-1|0|2019-05-04T15:19:24Z|kepub_user|||2104362|0|0|-1|-1||fr||true|0|0|0||FALSE|||0|true|-1|2|||||-1||0|0|0.0|0|||||||0||||false|0|||false|true|0|0|||||||||false|false||||||true|true

Here's what Nickel says when it fails to open it:

May  5 18:12:18 nickel: (  5061.154 @ 0x142af20) QList<int> KepubBookWordCounter::wordsInBook(const Volume&, const QVector<Shortcover>&) EPUB PATH NOT FOUND:  "/mnt/onboard/Gare a Lou " 
May  5 18:12:18 nickel: (  5061.386 @ 0x9db9f0 / ui.warning) ZipStreamDevice::ZipStreamDevice(zip*, const QString&) 0x0 
May  5 18:12:18 nickel: (  5061.387 @ 0x9db9f0 / ui.debug) virtual bool KepubBookReaderBase::contentMissing() "file:///mnt/onboard/Gare a Lou ! - Jean Teule (6336).kepub.epub" Trying to load first chapter false 
May  5 18:12:19 nickel: (  5061.549 @ 0x9db9f0 / ui.warning) ZipStreamDevice::ZipStreamDevice(zip*, const QString&) 0x0 
May  5 18:12:19 nickel: (  5061.550 @ 0x9db9f0 / ui.debug) virtual bool KepubBookReaderBase::contentMissing() "file:///mnt/onboard/Gare a Lou ! - Jean Teule (6336).kepub.epub" Trying to load first chapter true 
May  5 18:12:19 nickel: (  5061.551 @ 0x9db9f0 / ui.warning) virtual void KepubBookReaderBase::reloadContent() markup check failed "file:///mnt/onboard/Gare a Lou ! - Jean Teule (6336).kepub.epub" true false

And here's the entry from the second send (I did tweak the book a bit, because of a page-break issue I caught on the Forma), where it gets assigned to book 6 of The Witcher series.

(Not sure if relevant: that's probably the latest Series created/updated on that device, and the latest book sent to the device before that. If not the very latest, in the latest batch, at least).

file:///mnt/onboard/Teule, Jean - Gare a Lou !.kepub.epub|6|application/x-kobo-epub+zip|||file____mnt_onboard_Teule,_Jean_-_Gare_a_Lou_!_kepub_epub|Gare à Lou !|Jean Teulé|<p class="description">Avec Gare à Lou ! Jean Teulé revient à la veine fantastique qui avait fait le succès du Magasin des suicides et laisse libre court à un imaginaire plus débridé que jamais. Comme le disaient Mozart et Shakespeare : " Il est très agréable de jouir d'un don exceptionnel, mais il ne faut pas oublier que c'est une source inépuisable d'embêtements ". A 12 ans, Lou partage absolument cette opinion. Au prétexte qu'elle est en mesure de faire tomber immédiatement les pires calamités sur la tête de tous ceux qui la contrarient, on l'enferme dans un endroit secret en compagnie de militaires haut gradés pour qu'elle devienne une arme absolue capable de mettre en échec les plans malveillants des ennemis du pays ou, pire, d'ourdir de méchantes et sournoises manoeuvres afin de causer des torts effroyables à d'autres nations. De telles occupations n'offrent pas à une adolescente les satisfactions que la vie aurait pu lui promettre. D'autant que son super pouvoir, aussi extraordinaire soit-il, ne fonctionne pas toujours comme prévu. Rien ne pouvait mieux inspirer Jean Teulé que d'imaginer les horreurs qu'un être humain bien disposé peut infliger à ses contemporains.</p>|||/mnt/onboard/Teule, Jean - Gare a Lou !.kepub.epub|Julliard|true||true||0|0|57|0|-1|0|2019-05-05T16:17:54Z|kepub_user|||1501912|0|0|-1|-1||fr||true|0|0|0||FALSE|||0|true|-1|2||The Witcher|6||-1||0|0|0.0|0|||||||0||||false|0|||false|true|0|0|||||||6.0||false|false||||||true|true

---

No issues with another book, in English, part of a series. It opens and was assigned to the proper series :).

[NiLuJe]

Oh, and if I actually read the error messages instead of just copy/pasting 'em: the failure to open it definitely appears to be because of the bang ;p.

So I guess sanitizing filenames to be FAT-friendly should be enough?

No idea what happened with the Series metadata the second time around, though.

FWIW, I deleted the first copy from within Nickel, and not via Calibre.

[NiLuJe]

And about the startup delay: progress bar? ;p.

I get that keeping things in sync can be tricky, but maybe some stuff could be cached? And then naively checked for changes via timestamps, as that's probably going to be the fastest. Walking the full onboard tree will take its toll no matter what, though :/.

Also, no idea how feasible that would be, but if the metadata pass could avoid having to re-do the whole thing that would probably help, too ;).

[shermp]

Thanks for continuing your testing. I had hoped that Calibre would sanitize the filename before sending it. Seems I'll have to do it on device instead :( EDIT: Just realised, I'm running on Windows, so calibre is already sanitizing filenames for me. Am going to have to think of how to handle this, as Lpath is basically used by calibre as an ID.

As far as the wrong series info, that was a bug I fixed locally last evening, I noticed it myself. Simple silly error of forgetting to null variables in a loop.

And I specifically don't walk the onboard tree, merely check the DB. The only time it should open a book is if the book exists in Nickel, but not in the metadata cache. Will instrument this to make sure though.I'll definitely have a bit of a think on how to speed this up though, as currently it is definitely slower than I'd like.

Thanks for the continued testing!

[shermp]

Well, here's the relevant comment from the calibre driver that it doesn't sanitize filenames: https://github.com/kovidgoyal/calibre/blob/56036fd77ae3cd1ddbeb9c9841a228e19785cdc3/src/calibre/devices/smart_device_app/driver.py#L434

Going to have to have a bit of a rethink as to how to handle this. There looks to be two possible options. I can ask calibre to send UUID filenames. Or there's the option CacheUsesLpaths that I could experiment with setting false. I guess the third option is to add the functionality as an option to the Calibre driver...

[shermp]

Well, that wasn't too hard. The driver API accounts for this scenario, and allows us to change the 'lpath' if the Calibre generated one isn't suitable. Hopefully https://github.com/shermp/UNCaGED/commit/cb4fc322c8372703e6e8c0ca1ba4f56abc5b1022 and https://github.com/shermp/Kobo-UNCaGED/commit/41798b24ee2f1567459e48f1d0c23fc41598f77f fixes the filepath issue.

Since this fix updates UNCaGED as well, you will probably need to update that manually. A go get -u github.com/shermp/UNCaGED/uc should do that.

Next up, will have a think about how to make metadata collection/management more efficient...

[shermp]

Ok, https://github.com/shermp/Kobo-UNCaGED/commit/bff9e8a9a89d6f54507e8df74e4c8282939fe5e4 should dramatically speed up metadata collection.

In hindsight, performing individual single row DB queries for potentially hundreds or even thousands of books really was a rather silly idea wasn't it?

Now there's only one multi-row query out to the DB, and I create the final map there instead. Both simpler, and MUCH faster.

[NiLuJe]

Success! :)

May  6 18:44:09 nickel: (  6123.566 @ 0x1756098 / ui.debug) Pruning file "file:///mnt/onboard/Kobo-UNCaGED.png" 
May  6 18:45:11 UNCaGED: Entering USBMS mode...
May  6 18:45:11 UNCaGED: Inserting USB
May  6 18:45:12 UNCaGED: Scanning for Button
May  6 18:45:16 UNCaGED: (Re)mounting onboard
May  6 18:45:18 UNCaGED: Enabling WiFi
May  6 18:45:19 UNCaGED: Acquiring IP
May  6 18:45:31 UNCaGED: USBMS mode entered . . .
May  6 18:45:31 UNCaGED: Running Kobo-UNCaGED
May  6 18:45:31 KoboUNCaGED[6455]: 2019/05/06 18:45:31 Started Kobo-UNCaGED
May  6 18:45:31 KoboUNCaGED[6455]: 2019/05/06 18:45:31 /mnt/newonboard
May  6 18:45:31 KoboUNCaGED[6455]: 2019/05/06 18:45:31 Creating KU object
May  6 18:45:39 KoboUNCaGED[6455]: 2019/05/06 18:45:39 Preparing Kobo UNCaGED!
May  6 18:45:39 KoboUNCaGED[6455]: 2019/05/06 18:45:39 Calibre Address 192.168.0.42:9090
May  6 18:45:39 KoboUNCaGED[6455]: 2019/05/06 18:45:39 Starting Calibre Connection
May  6 18:45:59 KoboUNCaGED[6455]: 2019/05/06 18:45:59 Send Book detail is: {...}
May  6 18:45:59 KoboUNCaGED[6455]: 2019/05/06 18:45:59 Sending OK-to-send packet
May  6 18:46:01 KoboUNCaGED[6455]: 2019/05/06 18:46:01 Send Book detail is: {...}
May  6 18:46:01 KoboUNCaGED[6455]: 2019/05/06 18:46:01 Sending OK-to-send packet
May  6 18:46:01 KoboUNCaGED[6455]: 2019/05/06 18:46:01 Send Book detail is: {...}
May  6 18:46:08 KoboUNCaGED[6455]: 2019/05/06 18:46:08 Sending OK-to-send packet
May  6 18:46:20 UNCaGED: Leaving USBMS . . .
May  6 18:46:20 UNCaGED: Disabling WiFi
May  6 18:46:21 UNCaGED: WiFi disabled (0) . . .
May  6 18:46:21 UNCaGED: Unmounting onboard
May  6 18:46:25 UNCaGED: Onboard unmounted (0) . . .
May  6 18:46:25 UNCaGED: Waiting for content processing
May  6 18:46:41 nickel: (  6275.708 @ 0x1fd0490 / ui.debug) Pruning file "file:///mnt/onboard/Kobo-UNCaGED.png" 
May  6 18:46:44 UNCaGED: Updating metadata . . .
May  6 18:46:44 UNCaGED: Entering USBMS mode . . .
May  6 18:46:44 UNCaGED: Scanning for Button
May  6 18:46:48 UNCaGED: (Re)mounting onboard
May  6 18:46:53 UNCaGED: Running Kobo-UNCaGED
May  6 18:46:53 KoboUNCaGED[6741]: 2019/05/06 18:46:53 Started Kobo-UNCaGED
May  6 18:46:53 KoboUNCaGED[6741]: 2019/05/06 18:46:53 /mnt/newonboard
May  6 18:46:53 KoboUNCaGED[6741]: 2019/05/06 18:46:53 Creating KU object
May  6 18:47:01 KoboUNCaGED[6741]: 2019/05/06 18:47:01 Updating Metadata
May  6 18:47:01 UNCaGED: Unmounting onboard
May  6 18:47:05 UNCaGED: Onboard unmounted (0) . . .
May  6 18:47:05 UNCaGED: Going back to Nickel
May  6 18:47:21 nickel: (  6315.593 @ 0x1fc3450 / ui.debug) Pruning file "file:///mnt/onboard/Kobo-UNCaGED.png"

[shermp]

Finally have printing to screen working properly. It's probably a little (ok, more than just a little) fancier than is strictly required, but hey gotta use at least some of my FBInk feature requests :p

If you want to test it, I've updated go-fbink-v2 with a new function to aid printing a Go image.RGBA structure, and am using that new function.

I have to say, the valign option was giving me all sorts of grief, mainly because I kept forgetting that it applies to image/raw printing as well. So I would set it to center for printing OT text, then forgot to reset it in subsequent calls to print the image. And that in turn made button_scan barf. And I didn't notice the reason because my eink refresh calls were hiding the displacement.

Anyways, it's working now, and everything looks a lot nicer.

[shermp]

On a related note, how much faster is providing print_raw_data with an 8bpp grayscale image instead of an RGBA image? If the speed up significantly faster, it might be worth it for me to switch to using that.

[NiLuJe]

It won't be in that context, as the FB is 32bpp, so it would ultimately be expanded to RGBA again ;).

So starting with RGBA input is the right thing to do here ;).

Unless working with RGBA on your end is so much slower than working in Y8, and that despite the cost of a single conversion to RGBA at draw time, it would still be faster, but I doubt it ;).

On Tue, May 7, 2019, 14:01 Sherman Perry notifications@github.com wrote:

On a related note, how much faster is providing print_raw_data with an 8bpp grayscale image instead of an RGBA image? If the speed up significantly faster, it might be worth it for me to switch to using that.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/shermp/Kobo-UNCaGED/issues/3#issuecomment-490050223, or mute the thread https://github.com/notifications/unsubscribe-auth/AAA3KZQMMPAQ2432RTB3N3TPUFVSJANCNFSM4HKXERBA .

[NiLuJe]

Yeah, I've been caught unaware by *align stuff, mainly when I forget that I've set it earlier... >_<".

c.f., the dump/restore test for fun interactions between that and the H2O viewport ;p.

Besides not forgetting to reset the flag and/or bracket stuff that needs with a set/reset, one could also theoretically use a different FBInkConfig instance, but that's potentially trickier, unless you set it up at the same time you _init/_reinit, so it gets the same values that _init will enforce in there...

I usually temporarily switch to full-screen updates to make sure I don't do something stupid when I start playing with stuff like that ;).

[NiLuJe]

Gave it a go, looks neat :).

Ann here's a couple of nitpicky suggestions on how to make use of that space ;).

I'd probably clear the footer on Eject in the first pass, and possibly print something there in the metadata pass when it's over. In the first pass, right now you get a progression % that doesn't mean anything anymore mixed with the "we're going to restart" stuff; and on the second pass, the only new info comes from the wrapper, above the message box ;).

May  7 18:39:06 nickel: (  6761.066 @ 0x1ff0cf0 / ui.debug) Pruning file "file:///mnt/onboard/Kobo-UNCaGED.png" 
May  7 18:39:28 UNCaGED: Entering USBMS mode...
May  7 18:39:28 UNCaGED: Inserting USB
May  7 18:39:28 UNCaGED: Scanning for Button
May  7 18:39:33 UNCaGED: (Re)mounting onboard
May  7 18:39:34 UNCaGED: Enabling WiFi
May  7 18:39:36 UNCaGED: Acquiring IP
May  7 18:39:47 UNCaGED: USBMS mode entered . . .
May  7 18:39:47 UNCaGED: Running Kobo-UNCaGED
May  7 18:39:48 KoboUNCaGED[7387]: 2019/05/07 18:39:48 Started Kobo-UNCaGED
May  7 18:39:48 KoboUNCaGED[7387]: 2019/05/07 18:39:48 /mnt/newonboard
May  7 18:39:48 KoboUNCaGED[7387]: 2019/05/07 18:39:48 Creating KU object
May  7 18:39:48 KoboUNCaGED[7387]: 2019/05/07 18:39:48 MBox Width: 756   MBox Height: 756
May  7 18:39:48 KoboUNCaGED[7387]: 2019/05/07 18:39:48 Top Margin: 336, Bottom Margin: 942
May  7 18:39:48 KoboUNCaGED[7387]: 2019/05/07 18:39:48 Top Margin: 487, Bottom Margin: 489
May  7 18:39:48 KoboUNCaGED[7387]: 2019/05/07 18:39:48 Top Margin: 940, Bottom Margin: 338
May  7 18:39:48 KoboUNCaGED[7387]: 2019/05/07 18:39:48 Top Margin: 336, Bottom Margin: 942
May  7 18:39:48 KoboUNCaGED[7387]: 2019/05/07 18:39:48 Top Margin: 487, Bottom Margin: 489
May  7 18:39:48 KoboUNCaGED[7387]: 2019/05/07 18:39:48 Top Margin: 940, Bottom Margin: 338
May  7 18:39:48 KoboUNCaGED[7387]: 2019/05/07 18:39:48 Opening NickelDB
May  7 18:39:48 KoboUNCaGED[7387]: 2019/05/07 18:39:48 Getting Kobo Info
May  7 18:39:48 KoboUNCaGED[7387]: 2019/05/07 18:39:48 Getting Device Info
May  7 18:39:48 KoboUNCaGED[7387]: 2019/05/07 18:39:48 Reading Metadata
May  7 18:39:48 KoboUNCaGED[7387]: 2019/05/07 18:39:48 1 Reading metadata.calibre
May  7 18:39:50 KoboUNCaGED[7387]: 2019/05/07 18:39:50 1 Gathering metadata
May  7 18:39:56 KoboUNCaGED[7387]: 2019/05/07 18:39:56 Preparing Kobo UNCaGED!
May  7 18:39:56 KoboUNCaGED[7387]: 2019/05/07 18:39:56 Top Margin: 336, Bottom Margin: 942
May  7 18:39:56 KoboUNCaGED[7387]: 2019/05/07 18:39:56 Top Margin: 487, Bottom Margin: 489
May  7 18:39:56 KoboUNCaGED[7387]: 2019/05/07 18:39:56 Top Margin: 940, Bottom Margin: 338
May  7 18:39:56 KoboUNCaGED[7387]: 2019/05/07 18:39:56 Calibre Address 192.168.0.42:9090
May  7 18:39:56 KoboUNCaGED[7387]: 2019/05/07 18:39:56 Starting Calibre Connection
May  7 18:39:56 KoboUNCaGED[7387]: 2019/05/07 18:39:56 Top Margin: 336, Bottom Margin: 942
May  7 18:39:56 KoboUNCaGED[7387]: 2019/05/07 18:39:56 Top Margin: 487, Bottom Margin: 489
May  7 18:39:56 KoboUNCaGED[7387]: 2019/05/07 18:39:56 Top Margin: 940, Bottom Margin: 338
May  7 18:39:56 KoboUNCaGED[7387]: 2019/05/07 18:39:56 Top Margin: 336, Bottom Margin: 942
May  7 18:39:56 KoboUNCaGED[7387]: 2019/05/07 18:39:56 Top Margin: 487, Bottom Margin: 489
May  7 18:39:56 KoboUNCaGED[7387]: 2019/05/07 18:39:56 Top Margin: 940, Bottom Margin: 338
May  7 18:40:59 KoboUNCaGED[7387]: 2019/05/07 18:40:59 Send Book detail is: {...}
May  7 18:40:59 KoboUNCaGED[7387]: 2019/05/07 18:40:59 Top Margin: 336, Bottom Margin: 942
May  7 18:40:59 KoboUNCaGED[7387]: 2019/05/07 18:40:59 Top Margin: 487, Bottom Margin: 489
May  7 18:40:59 KoboUNCaGED[7387]: 2019/05/07 18:40:59 Top Margin: 940, Bottom Margin: 338
May  7 18:41:03 KoboUNCaGED[7387]: 2019/05/07 18:41:03 Sending OK-to-send packet
May  7 18:41:04 KoboUNCaGED[7387]: 2019/05/07 18:41:04 Top Margin: 336, Bottom Margin: 942
May  7 18:41:04 KoboUNCaGED[7387]: 2019/05/07 18:41:04 Top Margin: 487, Bottom Margin: 489
May  7 18:41:04 KoboUNCaGED[7387]: 2019/05/07 18:41:04 Top Margin: 940, Bottom Margin: 338
May  7 18:41:18 KoboUNCaGED[7387]: 2019/05/07 18:41:18 Top Margin: 336, Bottom Margin: 942
May  7 18:41:18 KoboUNCaGED[7387]: 2019/05/07 18:41:18 Top Margin: 487, Bottom Margin: 489
May  7 18:41:18 KoboUNCaGED[7387]: 2019/05/07 18:41:18 Top Margin: 940, Bottom Margin: 338
May  7 18:41:18 KoboUNCaGED[7387]: 2019/05/07 18:41:18 Top Margin: 336, Bottom Margin: 942
May  7 18:41:18 KoboUNCaGED[7387]: 2019/05/07 18:41:18 Top Margin: 487, Bottom Margin: 489
May  7 18:41:18 KoboUNCaGED[7387]: 2019/05/07 18:41:18 Top Margin: 940, Bottom Margin: 338
May  7 18:41:18 KoboUNCaGED[7387]: 2019/05/07 18:41:18 Top Margin: 336, Bottom Margin: 942
May  7 18:41:18 KoboUNCaGED[7387]: 2019/05/07 18:41:18 Top Margin: 487, Bottom Margin: 489
May  7 18:41:18 KoboUNCaGED[7387]: 2019/05/07 18:41:18 Top Margin: 940, Bottom Margin: 338
May  7 18:41:18 UNCaGED: Leaving USBMS . . .
May  7 18:41:18 UNCaGED: Disabling WiFi
May  7 18:41:19 UNCaGED: WiFi disabled (0) . . .
May  7 18:41:19 UNCaGED: Unmounting onboard
May  7 18:41:23 UNCaGED: Onboard unmounted (0) . . .
May  7 18:41:23 UNCaGED: Waiting for content processing
May  7 18:41:39 nickel: (  6914.237 @ 0xe812f0 / ui.debug) Pruning file "file:///mnt/onboard/Kobo-UNCaGED.png" 
May  7 18:41:40 UNCaGED: Updating metadata . . .
May  7 18:41:40 UNCaGED: Entering USBMS mode . . .
May  7 18:41:41 UNCaGED: Scanning for Button
May  7 18:41:43 UNCaGED: (Re)mounting onboard
May  7 18:41:46 UNCaGED: Running Kobo-UNCaGED
May  7 18:41:46 KoboUNCaGED[7658]: 2019/05/07 18:41:46 Started Kobo-UNCaGED
May  7 18:41:46 KoboUNCaGED[7658]: 2019/05/07 18:41:46 /mnt/newonboard
May  7 18:41:46 KoboUNCaGED[7658]: 2019/05/07 18:41:46 Creating KU object
May  7 18:41:47 KoboUNCaGED[7658]: 2019/05/07 18:41:47 MBox Width: 756   MBox Height: 756
May  7 18:41:47 KoboUNCaGED[7658]: 2019/05/07 18:41:47 Top Margin: 336, Bottom Margin: 942
May  7 18:41:47 KoboUNCaGED[7658]: 2019/05/07 18:41:47 Top Margin: 487, Bottom Margin: 489
May  7 18:41:47 KoboUNCaGED[7658]: 2019/05/07 18:41:47 Top Margin: 940, Bottom Margin: 338
May  7 18:41:47 KoboUNCaGED[7658]: 2019/05/07 18:41:47 Top Margin: 336, Bottom Margin: 942
May  7 18:41:47 KoboUNCaGED[7658]: 2019/05/07 18:41:47 Top Margin: 487, Bottom Margin: 489
May  7 18:41:47 KoboUNCaGED[7658]: 2019/05/07 18:41:47 Top Margin: 940, Bottom Margin: 338
May  7 18:41:47 KoboUNCaGED[7658]: 2019/05/07 18:41:47 Opening NickelDB
May  7 18:41:47 KoboUNCaGED[7658]: 2019/05/07 18:41:47 Getting Kobo Info
May  7 18:41:47 KoboUNCaGED[7658]: 2019/05/07 18:41:47 Getting Device Info
May  7 18:41:47 KoboUNCaGED[7658]: 2019/05/07 18:41:47 Reading Metadata
May  7 18:41:47 KoboUNCaGED[7658]: 2019/05/07 18:41:47 1 Reading metadata.calibre
May  7 18:41:49 KoboUNCaGED[7658]: 2019/05/07 18:41:49 1 Gathering metadata
May  7 18:41:54 KoboUNCaGED[7658]: 2019/05/07 18:41:54 Updating Metadata
May  7 18:41:54 KoboUNCaGED[7658]: 2019/05/07 18:41:54 Top Margin: 336, Bottom Margin: 942
May  7 18:41:54 KoboUNCaGED[7658]: 2019/05/07 18:41:54 Top Margin: 487, Bottom Margin: 489
May  7 18:41:54 KoboUNCaGED[7658]: 2019/05/07 18:41:54 Top Margin: 940, Bottom Margin: 338
May  7 18:41:55 UNCaGED: Unmounting onboard
May  7 18:42:03 UNCaGED: Onboard unmounted (0) . . .
May  7 18:42:03 UNCaGED: Going back to Nickel
May  7 18:42:20 nickel: (  6954.977 @ 0xd37b68 / ui.debug) Pruning file "file:///mnt/onboard/Kobo-UNCaGED.png"

[NiLuJe]

Sidebar: I have no idea how the go get/build stuff works, but the build script will probably need some tweaks to make sure it always pulls/builds the latest stuff, even on rebuilds ;).

(Also, the same kind of info pointed out in the go-fbink-v2 README about how to set up the env probably needs to be shown somewhere ;)).

[shermp]

Yeah, I'm aware some the content needs more work, Yesterday was about making sure the thing actually functions properly :)

Not sure about downloading updates from the build script. If people are doing any dev work, they may want to specifically use their current version on disk, and not update on every run of the build script. Will have a think about it.

[shermp]

Have you got any ideas on making the enable_wifi bit of the shell script even safer? It currently doesn't check whether wpa_supplicant or udhcpc actually succeed in giving us a network.

wpa_cli looks like it should give us the wpa_supplicant status, but my initial naive attempt to check the status failed miserably.

[NiLuJe]

I have no idea how those are supposed to work, but a generic "are we connected" check might be enough? (i.e., ping/check if there's a gateway/try to read something over HTTP). I think all of those are implemented one way or another in KOReader ;).

[shermp]

Determining if we have an IP address should be easy enough. Parsing ip addr show ${INTERFACE} should do the job there. KU doesn't need an internet connection, just a LAN connection.

But there's no point trying to get an IP address if wpa_supplicant doesn't connect to a Wifi network. That's what I'd like to try and determine if possible.

[NiLuJe]

IIRC, it logs what happened to the syslog. Granted, parsing that wouldn't be a terribly pretty solution ;).

On Wed, May 8, 2019, 08:16 Sherman Perry notifications@github.com wrote:

Determining if we have an IP address should be easy enough. Parsing ip addr show ${INTERFACE} should do the job there. KU doesn't need an internet connection, just a LAN connection.

But there's no point trying to get an IP address if wpa_supplicant doesn't connect to a Wifi network. That's what I'd like to try and determine if possible.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/shermp/Kobo-UNCaGED/issues/3#issuecomment-490362222, or mute the thread https://github.com/notifications/unsubscribe-auth/AAA3KZUPGONDOQYNWCEW4JTPUJV5FANCNFSM4HKXERBA .

[shermp]

Got a couple of more things ticked off the mental list.

The DB is now the base source of metadata for non cached books. The read-from-epub function is now used as an additional source. It also doesn't crash the program when a "bad" epub is encountered (the go XML parser doesn't handle XML 1.1)

I've now added support for all formats that the Kobo supports, including mobi, PDF, txt, rtf etc. Obviously no metadata readers for those formats (yet).

Implemented the same book replacement procedure as the Calibre kobo driver does (it updates the file size in the DB, so Nickel doesn't treat it as a new book).

And fixed a couple of nasty nil pointer dereferencing bugs that slipped in.

[shermp]

Am trying to decide how best to handle connecting to a password protected Calibre library. The easiest way to do this would be to allow the user to enter it into a config file that could then be ingested. This approach does make it awkward if the user connects to multiple password protected Calibre libraries though.

The proper approach would be to have an on-screen keyboard to allow the user to enter the password on device. I even have my https://github.com/shermp/go-osk and https://github.com/shermp/go-kobo-input experiments that I could polish up. Main problem with this idea, is I don't really understand the coordinate shenanigans that go on with the touch input on Kobo devices. Let alone what happens when rotation is added to the mix :(

[NiLuJe]

FWIW, the USB connection screen can only ever be shown in Portrait (I have a vague memory of double-checking that on the Forma by forcing it from a landscape search screen).

So that basically "only" leaves the 3 different protocols to handle ;p.

Which does involve rotation, since the touch coordinates are never in the same orientation than the FB, even in Portait, but is slightly less brain-twisty than having to handle actual FB rotation on top of it ;).

See KOReader for this simpler approach, and Plato for the cherry on top of it if you really want to handle FB rotation.

On Sun, May 12, 2019, 09:33 Sherman Perry notifications@github.com wrote:

Am trying to decide how best to handle connecting to a password protected Calibre library. The easiest way to do this would be to allow the user to enter it into a config file that could then be ingested. This approach does make it awkward if the user connects to multiple password protected Calibre libraries though.

The proper approach would be to have an on-screen keyboard to allow the user to enter the password on device. I even have my https://github.com/shermp/go-osk and https://github.com/shermp/go-kobo-input experiments that I could polish up. Main problem with this idea, is I don't really understand the coordinate shenanigans that go on with the touch input on Kobo devices. Let alone what happens when rotation is added to the mix :(

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/shermp/Kobo-UNCaGED/issues/3#issuecomment-491572545, or mute the thread https://github.com/notifications/unsubscribe-auth/AAA3KZWG37ZVYOQ5E2U4JE3PU7B4TANCNFSM4HKXERBA .

[shermp]

Ahem, if one has the orientation patch installed and is locked to landscape then...

[NiLuJe]

Huh. That's weird (and unfortunately annoying).

Just double-checked, the Forma does rotate back to Portrait (the "Connect/Cancel" popup honors the current rotation before that, though, but that's an issue for FBInk, which I think I fixed, which is why I originally tried this ;p).

[NiLuJe]

Oh. One thing I hadn't checked, and which as it turns out renders the whole point moot: it does honor Inverted Portrait, so, blergh, rotation it is :/.

Random fun fact: I also just realized the device logo depends on the device (it's forma shaped on the Forma ;p).

[shermp]

After reading sources for Plato, KOreader and FBInk, I think I'm more confused than when I started :p

All I can really say is: ouch, brain.

[NiLuJe]

Yeah, it's a known side-effect ^^.

---

Back to the config file approach:

I assume Calibre sends some kind of identification? Or is there just an IP or IP/port combo?

(i.e., how do we tell two different Calibre instances apart? How would you present that in a config file?)

[shermp]

Calibre sends a UUID (from memory) upon connection. I THINK, although I would need to double check, that it is the UUID of the current library, in which case that would be a PITA to get (it's stored in the metadata.db file)

[shermp]

Starts wireshark

Oh, that's right, it also sends the library name as well, so that could be used in a config file. What it does not appear to do is send an instance identifier.

And the password is valid for all libraries in a calibre instance. And I bet if one had multiple calibre instances serving the same library...

Ugh.

One suspects security was tacked on as an afterthought by the wireless driver developer. We won't even mention the "password stored as plaintext" bit. Mind you, with the way the password is hashed when sent over the network, I don't think it's possible for the password hash only to be saved on the host, so there's that.

[shermp]

I've had a disgusting idea, that makes me feel filthy just thinking about it...

How about if KU accepts a basic list of passwords, that it then iterates over until a password is accepted by calibre, or the list ends?

I was efficient. I had this thought while already showering, so a bit of extra time cleaned off most of the filth :)

[shermp]

Right, implemented afore-mentioned password handling. May come up with something more ... refined at a later date.

Also added a basic config file. Am open to adding more knobs to it if someone thinks of something.

Builds are now versioned automatically based on git describe :) Also made some shell script tweaks.

The upstream UNCaGED has been updated for the latest changes regarding passwords etc.

I think I'm nearing the point where I can release a public alpha over on Mobileread, unless there are any showstopper bugs come to light.

[pgaskin]

Nice work! I don't use Calibre myself, so I haven't go around to testing this yet, but I've skimmed over the code (and tested one or two bits of it like the connect/mount/button stuff).

@NiLuJe Sidebar: I have no idea how the go get/build stuff works, but the build script will probably need some tweaks to make sure it always pulls/builds the latest stuff, even on rebuilds ;).

@shermp, have you considered using go modules? It works pretty well now, and I haven't run into any major bugs with it recently. This would remove the need for GOPATH, while allowing for proper versioning of dependencies. The only disadvantage is this would require go 1.11+.

@shermp Not sure about downloading updates from the build script. If people are doing any dev work, they may want to specifically use their current version on disk, and not update on every run of the build script. Will have a think about it.

Using go modules would allow you to do this using the replace directive.

[shermp]

Nice work! I don't use Calibre myself, so I haven't go around to testing this yet, but I've skimmed over the code (and tested one or two bits of it like the connect/mount/button stuff).

@NiLuJe Sidebar: I have no idea how the go get/build stuff works, but the build script will probably need some tweaks to make sure it always pulls/builds the latest stuff, even on rebuilds ;).

@shermp, have you considered using go modules? It works pretty well now, and I haven't run into any major bugs with it recently. This would remove the need for GOPATH, while allowing for proper versioning of dependencies. The only disadvantage is this would require go 1.11+.

@shermp Not sure about downloading updates from the build script. If people are doing any dev work, they may want to specifically use their current version on disk, and not update on every run of the build script. Will have a think about it.

Using go modules would allow you to do this using the replace directive.

Yeah, thought about it, still need to get my head around how it all works...

[pgaskin]

It is as simple as:

$ export GO11MODULE=on # to allow use of go modules in the gopath
$ go mod init
$ go mod tidy

And then you should be able to move your source dir anywhere and use the ordinary go commands with it (including go get).

To replace a dependency, add a line like replace github.com/user/repo => /path/to/local/files or replace github.com/user/repo => github.com/forked/repo version.

[shermp]

Thanks. I THINK I have it sussed now. Needed to update the parent UNCaGED library and go-fbink-v2 with working tags as well.

[pgaskin]

Yep, looks good to me. Also note that you can use untagged versions if you want.

[shermp]

Yep, looks good to me. Also note that you can use untagged versions if you want.

Yeah, figured that, but still nice to have it all "proper", since I have control over those particular libraries. Thanks for the advice!

[shermp]

Siiigh... I hate setting up my Kobo from scratch.

Somewhere along the line, the ability of my Kobo to recognize an SD card went walkies. Thankfully, a factory reset solved the issue. Not sure what where, when, or why it happened, as I never actually use an SD card on my Kobo

So annoying :(

[shermp]

Right, have added preliminary SD card support. Haven't been able to properly test it because kfmon and wpa_supplicant are playing silly buggers on my fresh 4.14 FW install.

@NiLuJe can you please check my shell scripts again? I've made a fair few changes regarding mount/unmount to add SD support. I suspect there may be some issue, because kfmon is behaving suspiciously as though it's honoring the block_spawns = 1 setting in my ini... Knowing me, I've made some silly face-palm worthy mistake :p

With regards to wifi on 4.14, connecting seems a bit of a crapshoot. Sometimes it connects, other times it doesn't. I definitely need to look into the monitoring of wpa_supplicant status.

Siigh... and it was all going so well...

[NiLuJe]

I'll take a look later today, in the meantime, could you post your KFMon log? It might shed some light on what's happening ;).

(It's in /usr/local/kfmon/kfmon.log).

On Sun, May 19, 2019, 09:39 Sherman Perry notifications@github.com wrote:

Right, have added preliminary SD card support. Haven't been able to properly test it because kfmon and wpa_supplicant are playing silly buggers on my fresh 4.14 FW install.

@NiLuJe https://github.com/NiLuJe can you please check my shell scripts again? I've made a fair few changes regarding mount/unmount to add SD support. I suspect there may be some issue, because kfmon is behaving suspiciously as though it's honoring the block_spawns = 1 setting in my ini... Knowing me, I've made some silly face-palm worthy mistake :p

With regards to wifi on 4.14, connecting seems a bit of a crapshoot. Sometimes it connects, other times it doesn't. I definitely need to look into the monitoring of wpa_supplicant status.

Siigh... and it was all going so well...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/shermp/Kobo-UNCaGED/issues/3?email_source=notifications&email_token=AAA3KZS2AXZVKTJETQG5AMLPWD72PA5CNFSM4HKXERBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVW4UZA#issuecomment-493734500, or mute the thread https://github.com/notifications/unsubscribe-auth/AAA3KZQB3UDRTKYVR5WW3U3PWD72PANCNFSM4HKXERBA .