Refresh in Oxide after app is close doesn't work correctly

[Witos]

Environment setup: remarkable2-framebuffer: 706f4cb.. oxide: af869c6... (with adding LD_PRELOAD=/home/root/fb/librm2fb_client.so before application launch)

LD_PRELOAD=/home/root/fb/librm2fb_server.so /usr/bin/remarkable-shutdown & systemctl start tarnish

Steps:

1. Start "Process Manager"
2. Press "Back"

Actual: "Process Manager" elements are still visible.

Expected: Oxide should look like after the launch.

I'm attaching video.

The code responsible for the redraw: https://github.com/Eeems/oxide/blob/af869c69304a9d8debf21dc705783cc9f1c7fdef/applications/system-service/application.h#L162

IMG_0993.mov.zip

[raisjn]

it could be waveform mode auto, i checked and both remux and oxide are using it.

[Eeems]

it could be waveform mode auto, i checked and both remux and oxide are using it.

I did ~steal code~ ~borrow code~ take inspiration from remux.

[raisjn]

if you change it to WAVEFORM_MODE_GC16, then it should work

[Eeems]

@Witos would you be able to test that?

[Witos]

Tested this patch: https://github.com/Witos/oxide/commit/8ca3ea450050bb0eb5e9cea3783e228bb06c0ed0 It didn't change anything.

[Witos]

Log from server: reMarkable: ~/ cat server.log OPENED SHARED MEM: /dev/shm/swtfb.01 at 751a9000, errno: 0 BIN FILE: /usr/bin/remarkable-shutdown ADDR: 21f54 REPLACING THE IMAGE with shared memory SWTCON initialized \o/ STARTING RM2FB Reading waveforms from /usr/share/remarkable/320_R299_AFC421_ED103TC2U2_VB3300-KCD_TC.wbf Running INIT (135 phases) 1404 1872 16 reMarkable: ~

[timower]

Yeah, I've tried to debug this but the fact that tarnish stops the wifi just makes this unnecessarily difficult. You can get more info from the server by adding:

#define DEBUG
#define DEBUG_DIRTY

to https://github.com/ddvk/remarkable2-framebuffer/blob/master/src/server/main.cpp#L37

It would be nice to see if the ioctl sent by tarnish is actually received. I suspect the problem is with oxide or tarnish and the way LD_PRELOAD gets propagated.

[Witos]

reMarkable: ~/ cat server.log OPENED SHARED MEM: /dev/shm/swtfb.01 at 75272000, errno: 0 BIN FILE: /usr/bin/remarkable-shutdown ADDR: 21f54 REPLACING THE IMAGE with shared memory SWTCON initialized \o/ STARTING RM2FB Reading waveforms from /usr/share/remarkable/320_R299_AFC421_ED103TC2U2_VB3300-KCD_TC.wbf Running INIT (135 phases) 1404 1872 16 WAITING FOR SEND UPDATE ON MSG Q Dirty Region: 0 0 1404 1872 doUpdate mxc: waveform_mode 3 mxc: update mode 0 mxc: update marker 1 final: waveform 3 flags 0

Dirty Region: 1218 6 103 43 doUpdate mxc: waveform_mode 3 mxc: update mode 0 mxc: update marker 2 final: waveform 3 flags 0

Dirty Region: 1218 6 43 43 doUpdate mxc: waveform_mode 3 mxc: update mode 0 mxc: update marker 3 final: waveform 3 flags 0

Dirty Region: 23 75 187 222 doUpdate mxc: waveform_mode 3 mxc: update mode 0 mxc: update marker 4 final: waveform 3 flags 0

Dirty Region: 0 0 1404 1872 doUpdate mxc: waveform_mode 3 mxc: update mode 0 mxc: update marker 1 final: waveform 3 flags 0

Dirty Region: 0 0 1404 1872 doUpdate mxc: waveform_mode 2 mxc: update mode 1 mxc: update marker 0 final: waveform 2 flags 1

Dirty Region: 23 6 1238 291 doUpdate mxc: waveform_mode 3 mxc: update mode 0 mxc: update marker 5 final: waveform 3 flags 0

[Witos]

Tried to get a screenshot: /opt/bin/rot screen call screenshot and it's all black.

[Eeems]

I'm using the following code to create a screenshot: https://github.com/Eeems/oxide/blob/master/applications/system-service/fb2png.cpp

[timower]

Dirty Region: 0 0 1404 1872
doUpdate

So the update is received by the server, if that's what gets shown after the restore happens.

I think the screenshot does not work as we don't set the bits per channel in the get screen info ioctl correctly. So vinfo.red.length and vinfo.red.mask will be incorrect. Can you dump the raw framebuffer you save and check if it's correct after restoring, before calling the update ioctl?

[Eeems]

I don't have an rM2 so I can't test anything. @Witos ? @timower Feel free to also spin up a build and try things out yourself.

[Witos]

@timower those are the values: vinfo -----------> bits_per_pixel=16, xres=1404, yres=1872 red: offset=56, length=1982819968, msb_right=2128790624 green: offset=16, length=8, msb_right=31750184 blue: offset=2128790336, length=2128790360, msb_right=63 finfo line_length=2808 vinfo <-----------

[Witos]

The screenshot code is not used when the screen is saved for app switching....

[Witos]

Refreshing works like that:

qDebug() << "Saving screen...";
int frameBufferHandle = open("/dev/fb0", O_RDWR);
char* frameBuffer = (char*)mmap(0, DISPLAYSIZE, PROT_READ | PROT_WRITE, MAP_SHARED, frameBufferHandle, 0);
qDebug() << "Compressing data...";
auto compressedData = qCompress(QByteArray(frameBuffer, DISPLAYSIZE));
close(frameBufferHandle);
screenCapture = new QByteArray(compressedData);

and then:

int frameBufferHandle = open("/dev/fb0", O_RDWR);
auto frameBuffer = (char*)mmap(0, DISPLAYSIZE, PROT_READ | PROT_WRITE, MAP_SHARED, frameBufferHandle, 0);
memcpy(frameBuffer, uncompressedData, DISPLAYSIZE);
mxcfb_update_data update_data;
mxcfb_rect update_rect;
update_rect.top = 0;
update_rect.left = 0;
update_rect.width = DISPLAYWIDTH;
update_rect.height = DISPLAYHEIGHT;
update_data.update_marker = 0;
update_data.update_region = update_rect;
update_data.waveform_mode = WAVEFORM_MODE_GC16;
update_data.update_mode = UPDATE_MODE_FULL;
update_data.dither_mode = EPDC_FLAG_USE_DITHERING_MAX;
update_data.temp = TEMP_USE_REMARKABLE_DRAW;
update_data.flags = 0;
ioctl(frameBufferHandle, MXCFB_SEND_UPDATE, &update_data);

[Eeems]

You might want to edit and wrap the code in ```

[Witos]

Dumped the buffer (char* frameBuffer). I attaching. It's not empty, at least, but says nothing to me. debugfb.txt

[Witos]

Experiment - I commented out:

//auto frameBuffer = (char*)mmap(0, DISPLAYSIZE, PROT_READ | PROT_WRITE, MAP_SHARED, frameBufferHandle, 0);
//memcpy(frameBuffer, uncompressedData, DISPLAYSIZE);

and tested. It doesn't change anything at all. The screen looks exactly the same as when those above were on.

[Eeems]

@Witos I think we need to solve why copying the framebuffer isn't producing a proper image yet first before we attempt to solve why restoring the framebuffer is broken.

[Witos]

@Eeems , how do you know it's not producing a proper image? Did you check the attached debugfb.txt?

[Eeems]

https://github.com/ddvk/remarkable2-framebuffer/issues/30#issuecomment-742795033

[raisjn]

i wonder if it is related to the call to close(fbfd) in fb2png and other places. rm2fb does not intercept the close and so closing the FB file descriptor and then re-opening may not work properly. one way to check would be to remove the close() calls and re-compile oxide. if it is the problem, then we should override close() and do nothing if close is called on the swtfb FD.

[raisjn]

ok, I checked and I don't think the close() is the reason oxide isn't redrawing itself after closing the currently running process. I think oxide just needs to refresh after the current app closes.

[Witos]

@Eeems, I was curious about the numbers returned by ioctl FBIOGET_VCREENINFO. They seemed invalid. I added memset: struct fb_var_screeninfo vinfo; memset(&vinfo, 0, sizeof(struct fb_var_screeninfo)); ioctl(fbfd, FBIOGET_VSCREENINFO, &vinfo)

This results in: vinfo -----------> bits_per_pixel=16, xres=1404, yres=1872 red: offset=0, lenght=0 green: offset=0, lenght=0 blue: offset=0, lenght=0 vinfo <-----------

[ddvk]

yes, we need to fix those

On Mon, 14 Dec 2020 at 08:31 Witos notifications@github.com wrote:

@Eeems https://github.com/Eeems, I was curious about the numbers returned by ioctl FBIOGET_VCREENINFO. They seemed invalid. I added memset: struct fb_var_screeninfo vinfo; memset(&vinfo, 0, sizeof(struct fb_var_screeninfo)); ioctl(fbfd, FBIOGET_VSCREENINFO, &vinfo)

This results in: vinfo -----------> bits_per_pixel=16, xres=1404, yres=1872 red: offset=0, lenght=0 green: offset=0, lenght=0 blue: offset=0, lenght=0 vinfo <-----------

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ddvk/remarkable2-framebuffer/issues/30#issuecomment-744235637, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIYZFLU3TUPJXNA7EFTCZPLSUW5MDANCNFSM4UUBRVHQ .

[Witos]

       else if (request == FBIOGET_VSCREENINFO) {
    1
    2         fb_var_screeninfo *screeninfo = (fb_var_screeninfo *)ptr;
    3         screeninfo->xres = 1404;
    4         screeninfo->yres = 1872;
    5         screeninfo->grayscale = 0;
    6         screeninfo->bits_per_pixel = 16;
    7         return 0;
    8       }

[Witos]

This is still not the reason why the original refresh is not working, because it doesn't use vscreeninfo at all. It just copies the raw memory around. How to perform raw refresh, @raisjn, without this fb copying?

[raisjn]

there should be something in qt where you can redraw the app or main window

[ddvk]

can you try with the latest master? i see that the screenshot of oxide uses the rgb offsets

[Witos]

@ddvk , will try that this evening (UTC+1) @raisjn , will try to find something, but it seems it's not that obvious to force repaint in Qt

[ddvk]

you can force repaint by hiding the main window, then showing it with Qt.callLater

On Mon, 14 Dec 2020 at 09:05 Witos notifications@github.com wrote:

@ddvk https://github.com/ddvk , will try that this evening (UTC+1) @raisjn https://github.com/raisjn , will try to find something, but it seems it's not that obvious to force repaint in Qt

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ddvk/remarkable2-framebuffer/issues/30#issuecomment-744256778, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIYZFLXIZTN3RJENGXICRF3SUXBM5ANCNFSM4UUBRVHQ .

[Eeems]

ok, I checked and I don't think the close() is the reason oxide isn't redrawing itself after closing the currently running process. I think oxide just needs to refresh after the current app closes.

That defeats the entire point of recalling the stored screen. Not to mention there is no easy way to ensure that other apps redraw themselves on resume. Redrawing by hiding and showing the main window is what I use to do, but that was replaced with saving and recalling the screen so that when resuming any application it would work properly, instead of having to have every application specifically support being resumed.

[Witos]

Screenshot feature works fine after @ddvk patch. Attaching the screenshot captured as a proof.

[Eeems]

Does screen recalling when switching apps work now? Or is that still broken?

[Witos]

No, it's still broken. Attaching screenshot when the screen is malformed.

[Eeems]

And this is with WAVEFORM_MODE_GC16?

[raisjn]

oxide from toltec testing now works for me using master (ddvk's fix for screen info + close() impl)

[Witos]

Good job @raisjn , good job @ddvk - I confirm it works, after applying your 2 patches.