implement LD_PRELOAD shim for rM1 apps

[raisjn]

as per https://github.com/ddvk/remarkable2-framebuffer/issues/11#issuecomment-721461066, we can try making a shim that intercepts system calls for apps built for the rM1

if done right, this would mean the other integration (rmkit, libremarkable, koreader, qt's eprenderer) work is not necessary.

[pgaskin]

I'd recommend doing this by keeping a static array of bools (the size of the maximum FD) to record if each one is one which needs to be shimmed. Next, intercept open/close to compare the path then update this array and redirect the file if needed. Then, intercept ioctl and any other functions to transform the data (I'd recommend keeping statically allocated thread-local memory to avoid needing to keep reallocating memory for every call) and call msgsend instead. You could also intercept mmap for debugging.

[raisjn]

thanks for the recommendation!

i have a prototype semi-working, just shimming out each piece as necessary.

• static array of bools: haven't done yet
• intercept open/close: yes
• intercept ioctl: intercepting MXCFB_UPDATE_SEND and the get/put screen info so far
• statically allocated data: not yet

• msgsend: yes

  can share code when it's working better.

EDIT: HOLY &^#%, the shim works :-D

[raisjn]

https://imgur.com/a/c3Wr8dS <--- image of koreader https://imgur.com/a/QLcWghA <--- video of retris

the shim in ipc branch in src/client/ works for libremarkable and rmkit. the major changes the frameworks need are now related to input handling and battery location.

for koreader: it pretty much crashes immediately, but it can draw that initial screen and sometimes half a modal dialog. i can draw to screen with fbink successfully. it is likely that rm2fbserver needs to do some rectangle boundary handling and implement more ioctls.

qt apps did not work immediately with the shim, as they complained about an unrecognized device id (mxsfb-lcdif). this needs further investigation.

big thanks to @linusCDE for spending time this morning explaining libremarkable stuff

[pgaskin]

EDIT: HOLY &^#%, the shim works :-D

i can draw to screen with fbink successfully

video of retris

:tada:

And yes, LD_PRELOAD can be quite fun to work with!

static array of bools: haven't done yet

Have a look at pgaskin/kobo-mods/kobo-dotfile-hack-ng.

statically allocated data: not yet

Out of curiosity, if you've tested it, how's the current performance overhead without it?

for koreader: it pretty much crashes immediately

Do you have a strace?

@NiLuJe might also have some ideas on why this happens.

qt apps did not work immediately with the shim, as they complained about an unrecognized device id (mxsfb-lcdif). this needs further investigation.

I'm not familiar with this, but remapping the device IDs might work. I'm not sure what side-effects this will have, though.

battery location

That can be shimmed too, but I think it'd be better to properly implement it to avoid scope creep for the shim, since the shim is mainly intended to hide the differences in communication with the FB.

---

Edit: One other thing: I'd recommend avoiding c++ stdlib features (if you must, prefer the Qt implementations of the same thing) in the shim if possible, since that can cause issues in some cases.

Edit 2: I'd also recommend dlsyming the next symbols during library initialization to improve performance, reduce the possibility of issues in multi-threaded applications, and prevent some edge cases involving conflicting libraries.

Edit 3: For debugging (if you'll be including that in the final version), consider writing to a ring buffer in shared memory instead of to stderr for better performance.

Edit 4: Another random note about integration with other libraries/apps: note that it'll be possible to link this shim directly into an application, either during or after compilation.

[raisjn]

I'm not at the perf improvement stage, please don't review code as though its in final form - it's just to show progress

If you want to run a test on kobo or rM to see overhead, you can write high res timers (+ the full struct of mxcfb_update) through IPC and compare when they were sent and received.

[NiLuJe]

Random "add it to the pile for later" comment: is there an easy way for the "host" application to detect the shim?

(Case in point: I currently abort in FBInk on the rM2 ^^).

[NiLuJe]

Oh, and, yeah, as far as KOReader is concerned, if you can get me a strace (and/or verbose debug logs from KOReader itself, which maybe might be flushed before stuff blows up if we're lucky ^^), I can certainly try to help ;).

Since it crashes early, you'll want to tweak the settings manually, to set the debug and debug_verbose keys to true in the settings.reader.lua config file in KOReader's folder.

[NiLuJe]

But, yeah, might be as easy for starters as simply handling MXCFB_WAIT_FOR_UPDATE_COMPLETE ;).

(You can pass -w to fbink to make it do the same, if you want a quicker testcase).

[NiLuJe]

I don't think anyone actually makes use of anything else, but if you want to go the (really) extra mile, the only other ioctls one might really have cause to use are MXCFB_SET_UPDATE_SCHEME, MXCFB_GET_PWRDOWN_DELAY, MXCFB_SET_PWRDOWN_DELAY, MXCFB_SET_AUTO_UPDATE_MODE, MXCFB_SET_TEMPERATURE, MXCFB_SET_TEMP_AUTO_UPDATE_PERIOD, MXCFB_DISABLE_EPDC_ACCESS & MXCFB_ENABLE_EPDC_ACCESS.

[raisjn]

EDIT: koreader video

I found what was was causing crashes with KOReader (or at least one issue) and now KOReader works as expected and I can interact with it. Plato also works as expected (libremarkable app with some crazy drawing, supposedly). The touch input doesn't really work for either app (because the input is no longer inverted on x axis?), but the stylus does.

Random "add it to the pile for later" comment: is there an easy way for the "host" application to detect the shim?

there will be

I don't think anyone actually makes us of anything else, but if you want to go the (really) extra mile, the only other ioctls one might really have cause to use are MXCFB_SET_UPDATE_SCHEME, MXCFB_GET_PWRDOWN_DELAY, MXCFB_SET_PWRDOWN_DELAY, MXCFB_SET_AUTO_UPDATE_MODE, MXCFB_SET_TEMPERATURE, MXCFB_SET_TEMP_AUTO_UPDATE_PERIOD, MXCFB_DISABLE_EPDC_ACCESS & MXCFB_ENABLE_EPDC_ACCESS.

thanks, i'll look into these (should give us a nice todo list)

Edit 4: Another random note about integration with other libraries/apps: note that it'll be possible to link this shim directly into an application, either during or after compilation.

this would be a very nice tool to have where we can smack a binary from being rM1 compatible into being rM2 compatible

[ddvk]

on qt apps complaining about lcdif, i think it is in fixscreen struct, fbioget

[NiLuJe]

It's the id field, yes.

https://github.com/torvalds/linux/blob/521b619acdc8f1f5acdac15b84f81fd9515b2aff/include/uapi/linux/fb.h#L158

[raisjn]

the shim works with libremarkable, koreader, qsgepaper and rmkit now.

@pgaskin ok, now can code review (what's in master) and make PRs if you want (thank you!). list of stuff so far:

• use bool array for FD to know whether to intercept: not sure if still need this based on current code
• use ring buffer / remove cerr / debugging print statements: not planning on including debugging code that's not behind #ifdef
• avoid c++? :-( but it's doable
• static init: sounds reasonable if it will reduce problems down the line. i don't think the perf is important here but haven't measured.

i'll take a look tomorrow at perf and report some numbers. still no fast drawing yet, needs investigation.

[raisjn]

For perf, I put a millisecond counter into the struct being passed over IPC. It takes 0ms (i didn't bother measuring nano seconds) to pass the struct over IPC. depending on the update being used it can take 1 - 3ms (for small rects like pencil drawing) to 100+ms (for full screen (1404x1872) high quality updates)

The perf can be tested by enabling the DEBUG_TIMING define during build

[NiLuJe]

Kernel is out: https://github.com/reMarkable/linux/tree/zero-sugar

[ddvk]

yes,

https://github.com/reMarkable/linux/blob/zero-sugar/drivers/video/fbdev/mxsfb.c

data ist just shuffled out to the display (if i can interpret this right)

On Sun, 8 Nov 2020 at 17:01 NiLuJe notifications@github.com wrote:

Kernel is out: https://github.com/reMarkable/linux/tree/zero-sugar

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ddvk/remarkable2-framebuffer/issues/12#issuecomment-723602603, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIYZFLWFSN7CXAPLQUDNB5DSO26FBANCNFSM4TKUUYEA .

[pgaskin]

Regarding the bool array, the idea was to allow an application to open/close multiple instances instead of using a shared FD. Otherwise, it wouldn't be completely transparent since it would act slightly differently than native applications. And, since you currently open the shared FD when the lib is loaded and LD_PRELOAD is propagated to child processes, this may cause unnecessary instances from children.

[raisjn]

the shim has been implemented and we are considering this the "official" way of interacting with the rm2fb server, but using msgq + shm alone is also fine.