Does not work on reMarkable 2

[Utopiah]

Worked well with reMarkable 1. Tried with reMarkable and despite no error and a window appearing it remains black. I tried with the version I received the device with 2.2.1.88 then latest update 2.3.1.27 but to no avail.

Happy to provide logs or any other information that could help.

Here is the output I have without ffmpeg banner.

fabien@fabien-CORSAIR-ONE-i160:~/Prototypes/reStream$ ./reStream.sh 
Your remarkable does not have lz4.
Falling back to gzip, your experience may not be optimal.
Go to https://github.com/rien/reStream/#sub-second-latency for a better experience.
Input #0, rawvideo, from 'pipe:':  0KB vq=    0KB sq=    0B f=0/0   
  Duration: N/A, start: 0.000000, bitrate: 1054310 kb/s
    Stream #0:0: Video: rawvideo (Y1[0][16] / 0x10003159), gray16le, 1408x1872, 1054310 kb/s, 25 tbr, 25 tbn, 25 tbc

[harrylepotter]

@rien gist updated ( https://gist.github.com/harrylepotter/fd4f275892122217e9806394ba9fe89f) It's unfortunately a bit behind the work you've done and hasn't got the block size calculations in it

[raisjn]

I think it would be really cool if we can use LD_PRELOAD (or the systemwide LD_PRELOAD) to modify xochitl and swap out the pointer location from where it's pointing now to shared memory. this would then expose the framebuffer as /dev/shm/<name of shared mem> and reStream and others can just dump from there.

PS: unrelated (and sorry to interrupt) but we have a channel on discord (#rm2 on https://discord.gg/JSSGnFY) that we are going to try to use to coordinate rM2 framebuffer reversing, if you want to chat more there :-D

[tadfisher]

Personally, I feel that hooking into xochitl or depending on its process memory is brittle and will lead to a broken app ecosystem. We have the code to reverse-engineer it, so we should work on libreMarkable to add support for the rm2 framebuffer.

[Eeems]

Personally, I feel that hooking into xochitl or depending on its process memory is brittle and will lead to a broken app ecosystem. We have the code to reverse-engineer it, so we should work on libreMarkable to add support for the rm2 framebuffer.

Not just libremarkable I hope ;)

[harrylepotter]

@tadfisher totally agree. I'm actually wondering if RM is going to rectify the /dev/fb0 issue themselves so that we can continue to use existing stuff that uses it.. that failing, @raisjnn's idea is pretty awesome!

[benneti]

Looking at the line from @harrylepotter's comment

71432000-72bf0000 rw-s a8100000 00:06 248 /dev/fb0

I also looked at /proc/$pid/maps for the 2.4.0.27 version of xochitl to find:

713a4000-72b62000 rw-s a8100000 00:06 275        /dev/fb0
72b62000-732e8000 rw-p 00000000 00:00 0 

When using dd if=/proc/242/mem bs=1 count=2628288 skip=1924538376 > out.data (0x72b62008=1924538376) analogous to the above comment this results in the right image. Would it not be simpler to use the information in /proc/$pid/maps which is actually meant to list the different memory sections of the program than to search for a signature in the hexdump of the memory itself (of course this only holds if it is no coincidence that the right adress is right after the adress of fb0 but it seems that we have two data points that suggest this).

[rien]

Ooh that looks promising! I think this will be fairly stable, because the code will first map /dev/fb0 to place the waveforms and then allocate space for the actual framebuffer. I wonder why you need to add 8 bytes though.

[raisjn]

Would it not be simpler to use the information in /proc/$pid/maps which is actually meant to list the different memory sections of the program than to search for a signature in the hexdump of the memory itself (of course this only holds if it is no coincidence that the right adress is right after the adress of fb0 but it seems that we have two data points that suggest this).

this is best idea so far :-D

[rien]

I've implemented this in my latest commit in #31, is someone able to confirm this works? I'm available on the reMarkable discord for troubleshooting if it isn't.

[BeneCollyridam]

It is working for me now. However it is upside down (which can be fixed by adding hflip and vflip to video_filters It seems that by default it is in portrait mode, but adding -p makes it landscape It also takes ~50 sec to update the screen (I am using lz4) Let me know, if I can do some more testing.

[rien]

@BeneCollyridam can you confirm the video isn't scrambled? Which version of xochitl is on your reMarkable?

[BeneCollyridam]

@rien Yes, the video is not scrambled. I am on REMARKABLE_RELEASE_VERSION=2.3.1.27

[benneti]

Which commit are you on?

[Axenntio]

For me, the last committed version in #31, I have an error with head (so I don't have a window opening)

[...]
head: invalid option -- 'c' aq=    0KB vq=    0KB sq=    0B f=0/0
BusyBox v1.30.1 (2020-09-22 21:32:06 UTC) multi-call binary.

Usage: head [OPTIONS] [FILE]...
[...]

Edit: @thomsten has already mention this problem on the 2.4.1.30 software

[adamm]

I have the same error as @Axenntio saying "head: invalid option". It seems busybox 1.30.1's head doesn't support -c. Odd because the docs say it does.

Anyway, the fix described in PR https://github.com/rien/reStream/pull/31#discussion_r516919070 to grab debian's head(1) followed by the video_filter transpose=2 fixed it for

[ldecicco]

Thanks to @ddvk and @raisjn we have xofb to expose the framebuffer on /dev/shm/xofb and a way to stream the framebuffer in color (check this patch)!

This is a screenshot taken while streaming my rm2 with the above patch and library.

[vale981]

Thanks to @ddvk and @raisjn we have xofb to expose the framebuffer on /dev/shm/xofb and a way to stream the framebuffer in color (check this patch)!

This is a screenshot taken while streaming my rm2 with the above patch and library.

Any chance to work that into #31?

[stefanofornari]

If it may help, here is the output of trying reStream on my RM2:

[ste reStream] o(= git branch
* feature/support-reMarkable-2
  master
[ste reStream] =)o ./reStream.sh
[SSH] true
[SSH] cat /sys/devices/soc0/machine
[SSH] pidof xochitl
xochitl's PID: 248
[SSH] grep -C1 '/dev/fb0' /proc/248/maps | tail -n1 | sed 's/-.*$//'
framebuffer is at 0x72b1e000
[SSH] [ -f /opt/bin/lz4 ]
[SSH] [ -f ~/lz4 ]
[SSH] while dd if=/proc/248/mem bs=4096 skip=469790 count=642 2>/dev/null | tail -c+8 | head -c 2628288; do true; done | $HOME/lz4
ffplay version 4.2.4-1ubuntu0.1 Copyright (c) 2003-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
  configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
head: invalid option -- 'c' aq=    0KB vq=    0KB sq=    0B f=0/0   
BusyBox v1.30.1 (2020-09-22 21:32:06 UTC) multi-call binary.

Usage: head [OPTIONS] [FILE]...
Input #0, rawvideo, from 'pipe:':
  Duration: N/A, bitrate: 525657 kb/s
    Stream #0:0: Video: rawvideo (Y800 / 0x30303859), gray, 1872x1404, 525657 kb/s, 25 tbr, 25 tbn, 25 tbc

[xenuser]

I am not sure if feedback is wanted at this point of time and if it's helpful.

You guys are simply awesome. Managed to get my Remarkable 2 to work with your streaming script (feature branch) by downloading "head", scp-ing it over to the Remarkable2 and name it "shead" (there is no /usr/local/bin on my device). I then replaced "head -c" with "shead -c" in reStream.sh. Now runing reStreamsh gives me the image:

Image is still upside-down, but performance is great. Almost no delay while sharing the screen of the device.

Edit: I switched into landscape view, screen was still turned. I removed the added filter in reStream.sh so no "transpose=2" is added. The screen then shows up in the correct direction.

[Foxei]

@xenuser Have you used the xofb server to expose the framebuffer?

[owulveryck]

I give my 2 cents here.

thanks to all the information in this thread (thank you all!), I successfully grabbed a picture from my RM2 updated with firmware 2.5.

Here is the recap (please correct me if I something is wrong):

Remarkable version

reMarkable: ~/ cat /usr/share/remarkable/update.conf
[General]
#REMARKABLE_RELEASE_APPID={98DA7DF2-4E3E-4744-9DE6-EC931886ABAB}
#SERVER=https://get-updates.cloud.remarkable.engineering/service/update2
#GROUP=Prod
#PLATFORM=reMarkable2
REMARKABLE_RELEASE_VERSION=2.5.0.27

Locating the global framebuffer

reMarkable: ~/ strace xochitl
...
563 openat(AT_FDCWD, "/dev/fb0", O_RDWR)    = 5
564 ioctl(5, FBIOGET_FSCREENINFO, 0x7ee9d5f4) = 0
565 ioctl(5, FBIOGET_VSCREENINFO, 0x42f0ec) = 0
566 ioctl(5, FBIOPUT_VSCREENINFO, 0x42f0ec) = 0

Global framebuffer is located at 0x42f0ec-4 =0x42f0e8 (4387048 in decimal)

Now get a picture:

#!/bin/sh
pid=`pidof xochitl`
addr=`dd if=/proc/$pid/mem bs=1 count=4 skip=4387048  2>/dev/null | hexdump | awk '{print $3$2}'`
skipbytes=`printf "%d" $((16#$addr))`
dd if=/proc/$pid/mem bs=1 count=2628288 skip=$skipbytes > out.data

Note: 2628288 = 1404*1872 (the number of pixel in the screen of the remarkable)

Get the picture back on my laptop and convert it:

convert -depth 8 -size 1872x1404+0 gray:out.data out.png

which gives:

Edit: I've uploaded a gist in Go that takes a screenshot on the remarkable

[Foxei]

@xenuser Can you provide the compile „shead“ binary? This would make the installation process way easier as it does not require you to have a working toolchain. Similar to the way the lz4 library is provided as a binary.

[Eeems]

@xenuser Can you provide the compile „shead“ binary? This would make the installation process way easier as it does not require you to have a working toolchain. Similar to the way the lz4 library is provided as a binary.

You can install head with toltec/entware already.

[1dotd4]

$ cat /usr/share/remarkable/update.conf
[General]
#REMARKABLE_RELEASE_APPID={98DA7DF2-4E3E-4744-9DE6-EC931886ABAB}
#SERVER=https://get-updates.cloud.remarkable.engineering/service/update2
#GROUP=Prod
#PLATFORM=reMarkable2
REMARKABLE_RELEASE_VERSION=2.5.0.27

This was the best I was able to do. I took it from the feature/support-reMarkable-2 branch.

Thanks to this thread I was able to grab the grayscale live screen.

skip_bytes="$((0x$skip_bytes_hex))" # that +8 was getting everything funny
...
taking=$(($window_offset + $window_bytes))
head_fb0="dd if=/proc/$pid/mem bs=$page_size skip=$window_start_blocks count=$window_length_blocks of=/tmp/fb0_thing 2>/dev/null && dd if=/tmp/fb0_thing bs=$taking count=1 2>/d
ev/null | tail -c+$window_offset" # collect the chunk in /tmp/ then read the portion I need and I tail as head -c didn't work for me
...
video_filters="hflip,vflip"

This gives me a latency of 0.5s on avarage.

I'm looking forward for the range of the colored framebuffer — preferably without patching.

[Foxei]

@1dotd4 If you take the way with entware and install the head command, you will get around 2-3 FPS (the same you are getting). The current idea is to replace the dd, head and tail combination entirely. I have a working prototype that gives around 12-15 FPS but there ist some one else who is doing the same think. For more information take a look at the pull request in this Repo.

What is your idea vor the colored output? I took a close look at the memory In /proc/.../mem and for what I found, there is only the grayscale image. How would you realize this without patching?

[xenuser]

@xenuser Have you used the xofb server to expose the framebuffer?

Sorry for the late reply. @Foxei I didn't add xofb if that was your question. The "head" binary was downloaded from debian armhf, the link is located in the pull request here: https://github.com/rien/reStream/pull/31

With the "head" chain command from the pull request, performance is already okey for me.

[darthoctopus]

I was able to get a significant performance improvement by using dd from the latest busybox (let's say I place it at ~/bin/dd) and changing head_fb0 as

head_fb0="bin/dd if=/proc/$pid/mem bs=$window_bytes skip=$skip_bytes count=1 iflag=skip_bytes 2>/dev/null"

[Foxei]

Hi @darthoctopus, What do you mean with significant? How many FPS do you get? 🤔

[rien]

Hi all, I've added a binary to do all of the framebuffer reading on the remarkable. If someone wants to help finishing this, please try the installation instructions over at https://github.com/rien/reStream/tree/feature/support-reMarkable-2#installation and let me know if everything works on the rM2. Please also try the -p option and check if the view is in portrait mode then, by default it should be in landscape mode.

[darthoctopus]

There appears to be an offset error; the image scans a few pixels to the right each frame for me (I am running 2.5.0.27). The previous shell-only implementation worked fine.

[Foxei]

Hello @rien, I am glade to help you. But I have to inform you that your dumping script is not working as expected. The image is off and the rotation is not landscape.

Using your rust executable:

https://user-images.githubusercontent.com/25650006/103414876-0400f200-4b80-11eb-92d8-b22b4363e1f2.mov

[Foxei]

For the rM2. You offset should be 8 byte from the start of the framebuffers memory region as mapped in the /proc/pid/mmap file and you should dump 2628288 bytes. To ensure compatibility between rM1 and rM2 I suggest to dump the memory in blocks of 1872 bytes.

If you ensure that you will bet an image like this (I also fixed the rotation manually):

https://user-images.githubusercontent.com/25650006/103415126-19c2e700-4b81-11eb-8719-6d3ec8e989e4.mov

Unfortunately I cannot do any further testing today. Good Luck fixing the issue and a happy new year everyone.

[rien]

I have updated the binary (I've probably used an old build). Can someone test with the latest binary?

I should be dumping the right amount of bytes with the correct offset. But it's hard to test without a rM2 unfortunately.

[1dotd4]

@Foxei

How would you realize this without patching?

In the current version reMarkable 2 have stream is in beta (requires cloud), but I guess there could be a way to intercept it somewhere, just don't have the time right now to see that rabbit hole.

Maybe setting routes to the machine via usb and having a custom server that read the stream? Not sure, just thinking.

[rien]

I know the rM1 could handle color because the framebuffer was a the 'normal' one built into Linux. The rM2 uses special hardware and drives the display from user-space, so I don't think there is any color information present.

From what I remember from the cloud stream, it sends file updates (e.g. the lines you draw) instead of an actual "video stream" of the screen. If the file contains images in color, it will be rendered by your browser.

[BeneCollyridam]

@rien I just tried with the updated binary. By default it is in portrait mode, and when I add the -p flag, it is upside down.

[rien]

Thanks for trying out @BeneCollyridam, does that mean the video isn't broken anymore? How is the latency/fps?

I just tried with the updated binary. By default it is in portrait mode, and when I add the -p flag, it is upside down.

OK, I 've updated the script now. Can you try again?

[BeneCollyridam]

For me it is now in landscape when running with -p and when running without it is in portrait, but upside down.

[rien]

Ok, I've looked at @Foxei's PR and I've updated it again. I've forced pushed so you'll have to git fetch && git reset --hard feature/support-reMarkable-2 probably.

[BeneCollyridam]

It works perfectly now. I haven't measured, but it feels great (with regards to fps and latency).

[bencord0]

I've just pulled the latest commit on this branch (cf6b1050a7f2c46593c3f61977684ef8378626e3 in case you force push again).

Can confirm that it works on the rm2 too.

I've only stumbled onto this project recently, and wow, great work!

[Foxei]

@rien I have to be honest, I am A little bit proud that my work was not completely dropped. 😉 If reStream is normal working fine you should post it on Reddit. There are a bunch of people waiting for this. Just post a short Exclamation and a gif. I read tons of complains about the live view feature of remarkable.

If you have any problem in the future just drop me a line. I am down about any research about the rM2.

[Utopiah]

Best way to start 2021!

[1ykos]

I don't know what I did or what happened, It might be that the firmware was updated to 2.5.0.27 (it is at that version now anyways), but reStream was working as intended on my reMarkable2 just a few days ago and today it stopped working.

reMarkable: ~/ ./restream 
Error: No line containing /dev/fb0 in /proc/218/maps file
reMarkable: ~/ ps | grep xochitl
  218 root      265m S    /usr/bin/xochitl --system
  925 root      2496 S    grep xochitl
reMarkable: ~/ grep "/dev/fb0" /proc/218/maps
reMarkable: ~/ 

Any Ideas what I could try? Is there a reason we are not reading from /dev/fb0 directly?

[PhilippeDuchon]

I just checked my own, it is at 2.5.0.27 and reStream works fine - both in landscape and portrait modes - as far as I can tell.

[1ykos]

I also did install rm2fb ... https://github.com/ddvk/remarkable2-framebuffer maybe that has something to do with it?

[Foxei]

Hello @1ykos, the reason why we are not reading from /dev/fb0 directly is that the rM2 does not provide the image in this way anymore. 😢 We have to locate the frame buffer from the QtApplication itself and the dump the memory from there.

[Foxei]

If you redirected your frame buffer to /dev/fb0 you should use the old rM1 way of streaming. Hard code the rM verison to 1.0 and the image sizes to 1872px times 1404px with the corresponding number of bytes per pixel you are outputting. This should do the trick.

[1ykos]

@Foxei Ah that is what the checksum was about. Indeed /dev/fb0 does not change when the displayed image changes.

[Eeems]

I also did install rm2fb ... https://github.com/ddvk/remarkable2-framebuffer maybe that has something to do with it?

Yes, this is not compatible with rm2fb yet. See #41