Does not work on reMarkable 2

[Utopiah]

Worked well with reMarkable 1. Tried with reMarkable and despite no error and a window appearing it remains black. I tried with the version I received the device with 2.2.1.88 then latest update 2.3.1.27 but to no avail.

Happy to provide logs or any other information that could help.

Here is the output I have without ffmpeg banner.

fabien@fabien-CORSAIR-ONE-i160:~/Prototypes/reStream$ ./reStream.sh 
Your remarkable does not have lz4.
Falling back to gzip, your experience may not be optimal.
Go to https://github.com/rien/reStream/#sub-second-latency for a better experience.
Input #0, rawvideo, from 'pipe:':  0KB vq=    0KB sq=    0B f=0/0   
  Duration: N/A, start: 0.000000, bitrate: 1054310 kb/s
    Stream #0:0: Video: rawvideo (Y1[0][16] / 0x10003159), gray16le, 1408x1872, 1054310 kb/s, 25 tbr, 25 tbn, 25 tbc

[Eeems]

We don't yet know how to work with the rM2's framebuffer, so all older software that interacts with the screen will not work. You can find more information on the current research here https://github.com/Eeems/oxide/issues/48 and here https://github.com/koreader/koreader/issues/6792

[rien]

This script makes use of the standardized /dev/fbX to read the framebuffer, so it is weird it doesn't work. Can you try reading the framebuffer manually (cat /dev/fb0) and say if there is any output? It could be the case that they output less data or another encoding.

I don't have an rM2, so I won't be able to do troubleshooting myself unfortunately. Good luck with the research.

[Eeems]

This script makes use of the standardized /dev/fbX to read the framebuffer, so it is weird it doesn't work. Can you try reading the framebuffer manually (cat /dev/fb0) and say if there is any output? It could be the case that they output less data or another encoding.

I don't have an rM2, so I won't be able to do troubleshooting myself unfortunately. Good luck with the research.

It's not a standard framebuffer, @ddvk has been having quite the time trying to figure out how it even works.

[rien]

I've got my hands on a framebuffer capture and I was not able to figure out which pixel format is used. I do have the following conclusions of my analysis:

• While one framebuffer dump is 32MB in size, the data is all zeroes after 24893440 bytes.
• The data seems to be grouped by 4 bytes.
• Each of these groups range from 0000 0041 to 0000 0061 in the capture I have. The last byte only has the values 0x41, 0x51 0x53, 0x61, maybe this is dark - light grey - dark grey - white?

[Eeems]

@ddvk is this in line with your findings?

[rien]

I've got a second dump, where there are more values for the last byte. But they are in the same range (0x41 - 0x61).

[ddvk]

@Eeems it is complicated, but the waveforms are there somewhere

[raisjn]

@rien: from a trace that i saw, the FB reports as rgba32 when queried with fbioget_vscreeninfo ioctl during xochitl startup

(FBIOGET_VSCREENINFO)
xres        = 260
yres        = 1408
xres_virtual    = 260
yres_virtual    = 23936
xoffset     = 0
yoffset     = 0
bits_per_pixel  = 32    grayscale   = 0
red : offset = 16,  length =8,  msb_right = 0
green   : offset = 8,   length =8,  msb_right = 0
blue    : offset = 0,   length =8,  msb_right = 0
transp  : offset = 0,   length =0,  msb_right = 0
nonstd      = 0
activate    = 0
height      = 0x0
width       = 0x0
accel_flags(OBSOLETE) = 0
pixclock    = 28800
left_margin = 1
right_margin    = 1
upper_margin    = 1
lower_margin    = 143
hsync_len   = 1
vsync_len       = 1
sync        = 0
vmode       = 0
rotate      = 0
colorspace  = 0
reserved    = 4387708
-----
(FBIOPUT_VSCREENINFO)
xres        = 260
yres        = 1408
xres_virtual    = 260
yres_virtual    = 23936
xoffset     = 0
yoffset     = 22528
bits_per_pixel  = 32    grayscale   = 0
red : offset = 16,  length =8,  msb_right = 0
green   : offset = 8,   length =8,  msb_right = 0
blue    : offset = 0,   length =8,  msb_right = 0
transp  : offset = 0,   length =0,  msb_right = 0
nonstd      = 0
activate    = 0
height      = 0x0
width       = 0x0
accel_flags(OBSOLETE) = 0
pixclock    = 28800
left_margin = 1
right_margin    = 1
upper_margin    = 1
lower_margin    = 143
hsync_len   = 1
vsync_len       = 1
sync        = 0
vmode       = 0
rotate      = 0
colorspace  = 0
reserved    = 4387708

rgba32 is easier to deal with than rgb565, as you've noticed it's just groups of 4 bytes :-D i am surprised you have data in your framebuffer based on what other people have reported. if rgba32 works, i'll be pretty happy.

btw, what's the md5sum or sha256 of the buffer?

[raisjn]

regarding the bytes filled with zero:

on the original rM, the framebuffer is twice as large as it needs to be (the height is 2x what is actually shown on screen), so seeing all zeroes after a certain point is not surprising to me. i think the reason for 2x is usually for double buffering purposes (write to second screen, then swap it in all at once)

[thomsten]

I've got access to a reMarkable 2 as well if anyone needs some commands run.

btw, what's the md5sum or sha256 of the buffer?

You mean?

reMarkable: ~/ md5sum /dev/fb0 
480a37f069653d6a99aa603492eb7d66  /dev/fb0
reMarkable: ~/ sha256sum /dev/fb0 
3b5db951652767e4e1abaa964107d5e37cd1ea972297c0b273c1b1dcb8bb1d27  /dev/fb0

[ddvk]

yoi are running version 2.2.1.82?

On Fri, 23 Oct 2020 at 11:30 Thomas Stenersen notifications@github.com wrote:

I've got access to a reMarkable 2 as well if anyone needs some commands run.

btw, what's the md5sum or sha256 of the buffer?

You mean?

reMarkable: ~/ md5sum /dev/fb0 480a37f069653d6a99aa603492eb7d66 /dev/fb0 reMarkable: ~/ sha256sum /dev/fb0 3b5db951652767e4e1abaa964107d5e37cd1ea972297c0b273c1b1dcb8bb1d27 /dev/fb0

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rien/reStream/issues/28#issuecomment-715222749, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIYZFLQBYZDX4E6S3CFOOSLSMFEJXANCNFSM4ST2JWJA .

[thomsten]

2.3.1.27

[ddvk]

What does xochitl's log say, e.g: the line: Reading waveforms from /usr/share/remarkable/320_R292_AFBC21_ED103TC2M1_TC.wbf

[thomsten]

This part?

Oct 21 06:42:04 reMarkable systemd[1]: Started reMarkable main application.
Oct 21 06:42:04 reMarkable xochitl[240]: Registering exit handlers
Oct 21 06:42:04 reMarkable xochitl[240]: 42:04.816                          we're running on an epaper device (int main(int, char**) /usr/src/debug/xochitl/2.3+gitAUTOINC+3ccb7cc15b-r0/git/src/main.cpp:139)
Oct 21 06:42:06 reMarkable xochitl[240]: Reading waveforms from /usr/share/remarkable/320_R292_AFBC21_ED103TC2M1_TC.wbf
Oct 21 06:42:06 reMarkable xochitl[240]: Running INIT (111 phases)
Oct 21 06:42:06 reMarkable xochitl[240]: 42:06.719                          SWTCON initialized \o/
Oct 21 06:42:06 reMarkable xochitl[240]: 42:06.748                          EPD platform plugin loaded!

[ddvk]

ah, sorry,my mistake, didn't see that it is sha2 and not 1, mine is the same, the framebuffer does not change.

sha256sum /dev/fb0 3b5db951652767e4e1abaa964107d5e37cd1ea972297c0b273c1b1dcb8bb1d27 /dev/fb0

[rien]

The framebuffers I was sent were also the same, so I fear there is no useful information in there.

Has somebody tried strace xochitl to look which syscalls happen?

[ddvk]

yes, fbpan and sometimes fbblank

On Fri, 23 Oct 2020 at 12:50 Rien notifications@github.com wrote:

The framebuffers I was sent were also the same, so I fear there is no useful information in there.

Has somebody tried strace xochitl to look which syscalls happen?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rien/reStream/issues/28#issuecomment-715265697, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIYZFLWYAES6PMKVUIYEZ6DSMFNXFANCNFSM4ST2JWJA .

[thomsten]

Attached a strace: xochitl-strace.log

Suppose the interesting bits are here:

openat(AT_FDCWD, "/dev/fb0", O_RDWR)    = 5
ioctl(5, FBIOGET_FSCREENINFO, 0x7eac565c) = 0
ioctl(5, FBIOGET_VSCREENINFO, 0x3ee1c4) = 0
ioctl(5, FBIOPUT_VSCREENINFO, 0x3ee1c4) = 0
mmap2(NULL, 24893440, PROT_READ|PROT_WRITE, MAP_SHARED, 5, 0) = 0x7141e000
ioctl(5, FBIOPUT_VSCREENINFO, 0x3ee1c4) = 0
ioctl(5, FBIOBLANK, 0)                  = 0

[tadfisher]

Here's an strace dump using -ff, which collects separate traces for each child process. https://gist.github.com/tadfisher/a6ad91552e418077c2ed32d0a9e40a2e

trace.2415 is the main process, which does the FB setup before forking off the process in trace.2417, which appears to call FBIOPAN_DISPLAY for every screen refresh. I do see a couple of calls to FBIOPUT_VSCREENINFO which may point to actual framebuffer data.

I do not see any calls to MXCFB_* ioctls, and the driver in /sys/class/graphics/fb0 does appear to be mxs-lcdif, so they may have repurposed the mxsfb driver for framebuffer updates and are somehow setting EPDC parameters via userspace. I'm not experienced with this whatsoever, though, so I hope at least the traces can help.

[raisjn]

@tadfisher i have been examining the xochitl binary with bokluk and what you've said lines up with what we've seen. a few things:

• the fb0 seems to have waveforms placed in it when xochitl starts running (and thats why everyone has the same values in their framebuffer)

• there is a "vsync and flip thread"

• there is a "generator thread"

• the driver is mxsfb (according to the /sys/ graphics subdirectories) but i have not come across mention of mxsfb or mxcfb that treats fb0 like waveform storage (or RAM) for the display.

• there is no mxcfb_update (like in rM1)

it will be really nice when we can find the memory area that corresponds to what is being displayed on screen (for capturing purposes). it might be in xochitl's process memory (instead of system wide) though.

[PhilippeDuchon]

I don't have the technical skills to help, but I have access to both a rM1 and rM2, so if there are tests I can try and report here, let me know. I'd very much like to be able to use the script on the new device!

[tadfisher]

@raisjnn The pointer to global framebuffer data is located at address 0x003ee1c0, and the buffer itself is 2628288 bytes long. Dumping the buffer results in the raw bitmap data, which is interleaved a bit strangely, but should be trivial to reconstruct. You can open the dump directly in GIMP in "Raw image data" format and play with the sliders to get a picture. I used width=1404.

If you would like a ghidra export, I can provide one.

[raisjn]

awesome, nice work! it looks like the first half of that is something else? the image itself reminds me of when there is a bpp mismatch or resolution mismatch (it ends up printing the image a few times)

btw, are you able to find the same address in remarkable-shutdown binary?

[ddvk]

try converting it like this

convert -depth 8 -size 1872x1404+0 gray:raw out.png

[harrylepotter]

@tadfisher i'm curious how you managed to get the dump at address 0x003ee1c0. I tried using devmem and it threw me a bus error.

[tadfisher]

@harrylepotter This location is in xochitl's address space, so I'm not surprised. To grab the pointer, use dd on /dev/<pid>/mem, skipping 0x003ee1c0 bytes and with a count of 4. Translate the result from litlle-endian, which should get you an address in the heap toward the upper half of the address space. Find the memory map for that address in /dev/<pid>/maps and dump it with dd again.

[rien]

@tadfisher great you were able to dump the image!

If someone could write a script which is able to fetch the memory address and then dump that memory, I can fix it in the reStream script to work with the reMarkable 2. The following things should be done:

1. pid=$(pidof xochitl)
2. Interpret the 4 bytes in /dev/$pid/mem at 0x003ee1c0 (4121024 in decimal) as little-endian integer.
3. Use the address of the previous step to read 24893440 bytes from /dev/$pid/maps.

To check whether de data is correct, you can use @ddvk's convert command. I haven't tested it so I am not sure the pixel format is correct.

If I had a rM2 I would do it myself, but that's unfortunately not the case.

[raisjn]

i don't think this is portable across compiles of xochitl, so you'd want to make sure the memory address is correct before dumping

[rien]

We could make a mapping between the xochitl sha1 and the memory address of the framebuffer

[harrylepotter]

@tadfisher @rien hmmmmm. Just got a working image:

#!/bin/sh
$pid=`pidof xochitl`
dd if=/proc/$pid/mem bs=1 count=4 skip=4121024 2>/dev/null | hexdump | awk '{print $2$3}'

returns 000872bf

assuming there's a straightforward means of converting endianness in bash to 72bf0008, this would appear to resolve to:

71432000-72bf0000 rw-s a8100000 00:06 248        /dev/fb0
72bf0000-73376000 rw-p 00000000 00:00 0 

in /proc/$pid/maps

running dd if=/proc/$pid/mem bs=1 count=2628288 skip=1925120008 > out.data (where 1925120008=0x72bf0008) then produced a file which when run against convert -depth 8 -size 1872x1404+0 gray:out.data out.png produces the following result:

...for some reason running dd if=/proc/$pid/mem bs=2628288 count=1 skip=1925120008 doesn't produce the same result... Not sure why it doesn't like the bigger block size..

[harrylepotter]

@rien :

#!/bin/sh
pid=`pidof xochitl`
addr=`dd if=/proc/$pid/mem bs=1 count=4 skip=4121024 2>/dev/null | hexdump | awk '{print $3$2}'`
skipbytes=`printf "%d" $((16#$addr))`
dd if=/proc/$pid/mem bs=1 count=2628288 skip=$skipbytes > out.data

[thomsten]

Can confirm @harrylepotter's approach works for dumping image from reMarkable 2 👍 Weird that it doesn't work increasing the block size.., Even trying to read bs=4 count=1 gives Input/output error.

[tadfisher]

@harrylepotter @thomsten If you increase the block size, you have to divide skip and count by that size.

Also, increasing the block size past 4096 will not increase performance as that's the page size for i.MX7.

[harrylepotter]

so

dd if=/proc/237/mem bs=3647 count=721 skip=527864

works.. i'm guessing we want bs=the highest denominator of the address closest to 4096

[harrylepotter]

i've tried updating reStream.sh with the width & height swapped, and using pixel_format=gray8... I'm getting some results, but something appears to be off - is there an offset error or something? example.mp4.zip

Gist here: https://gist.github.com/harrylepotter/fd4f275892122217e9806394ba9fe89f

[harrylepotter]

@tadfisher also tried re-piping to dd with

dd if=/proc/237/mem bs=3647 count=721 skip=527864 2>/dev/null | dd bs=2628288 count=1 2>/dev/null

but no dice. Can't figure out if i'm doing something wrong on the DD side, or on the ffmpeg side at this stage...

[darthoctopus]

After some trial and error, I've gotten it to work with the following definition of head_fb0: dd if=/proc/$pid/mem bs=3060 count=920 skip=628906 2>/dev/null > temp && dd if=temp bs=2628288 count=1 2>/dev/null

Unfortunately, the latency in this case is pretty high. I'm also not sure if this will continue to work across e.g. restarts of xochitl (e.g. if the location of the framebuffer gets updated). The address supplied by @tadfisher appears not to work for me (possibly because I am on version 2.4 of the software).

edit: as expected; it did not work after restarting xochitl. it also seems unwieldy to me to have to recompute some integer factorisation when confronted with the updated address every time the script is run, in any case.

[harrylepotter]

@darthoctopus - i just tried something similar to what you put together but created a mountpoint in ram first:

mount -t tmpfs -o size=32m tmpfs ram/

..i'm not sure whats up with the pipe, but:

[rien]

Sweet! @harrylepotter what command did you end up using for $head_fb0? Is it necessary to change the -pixel_format, or does it also work with rgb565le?

[rien]

If you update the reStream gist you posted (or create a new one), then I will create a PR with the needed changes.

Can you also paste the output of sha1sum "$(which xochitl)", so I can make the mapping for the address. Or does someone know a reliable way to dump the xochitl version?

[rien]

Can someone with a reMarkable 2 tell me the following: What grep 'REMARKABLE_RELEASE_VERSION' /usr/share/remarkable/update.conf returns (here my rM 1 gives 2.3.0.16). Whether grep 'deviceid' /etc/remarkable.conf starts with RM2..-...-....? (my rM 1 gives RM102-002-33421).

[xiu]

here you go:

reMarkable: ~/ grep 'REMARKABLE_RELEASE_VERSION' /usr/share/remarkable/update.conf
REMARKABLE_RELEASE_VERSION=2.3.1.27
reMarkable: ~/ grep 'deviceid' /etc/remarkable.conf
deviceid=RM110-039-45166

[Stivanification]

On my reMarkable 2 I get

reMarkable: ~/ grep 'REMARKABLE_RELEASE_VERSION' /usr/share/remarkable/update.conf
REMARKABLE_RELEASE_VERSION=2.3.1.27

and

reMarkable: ~/ grep 'deviceid' /etc/remarkable.conf
deviceid=RM110-042-07857

So it doesn't start with RM2, if that was your idea. However, I get

reMarkable: ~/  cat /sys/devices/soc0/machine
reMarkable 2.0

[rien]

Thank you! If someone else can confirm the rM2's device id starts with RM110 that would be great!

Nvm, /sys/devices/soc0/machine is even better!

[benneti]

On my remarkable I get:

$ grep 'REMARKABLE_RELEASE_VERSION' /usr/share/remarkable/update.conf
REMARKABLE_RELEASE_VERSION=2.4.0.27

now the interesting part: there is no deviceid in /etc/remarkable.conf but a large devicetoken.

$ cat /sys/devices/soc0/machine
reMarkable 2.0

[rien]

I've started a PR over at #31, if someone with a reMarkable 2 can try it out and fix possible bugs that would be great!

[xiu]

Just tried it and managed to get it to start after two modifications. It takes a long time (1 minute or 2) to finally start ffplay and takes the same time to refresh. Here's a screenshot:

[benneti]

Im am trying to get the correct adress for 2.4 but there is no /dev/<pid of xochitl>/mem and /proc/<pid of xochitl>/mem gives me an input/output error when trying to use dd or cat. I do not usually access my memory manually so any pointers would be appreciated.

[harrylepotter]

couple of other things:

head_fb0="dd if=/proc/237/mem bs=3647 count=721 skip=527864 2>/dev/null > /mnt/ram/temp && sleep 0.06 && dd if=/mnt/ram/temp bs=2628288 count=1 2>/dev/null"

• sleep 0.04 reduces the framerate to ~15fps. This appears to make latency waaaaaaaaay better (because we're rate limiting whats coming back on the ssh buffer)
• using a tmpfs mountpoint seems to make things faster (...less io), but it would be better if we could figure out whats up with the pipes to avoid having commands to set this up.. Also... /tmp might be in ram already? Not sure

[darthoctopus]

/tmp is indeed already in RAM on tmpfs, and by default appears to be 500 MB (more than sufficient for the framebuffer).

On Thu, Oct 29, 2020 at 08:50, harrylepotter notifications@github.com wrote:

couple of other things:

head_fb0="dd if=/proc/237/mem bs=3647 count=721 skip=527864 2>/dev/null > /mnt/ram/temp && sleep 0.06 && dd if=/mnt/ram/temp bs=2628288 count=1 2>/dev/null" sleep 0.04 reduces the framerate to ~15fps. This appears to make latency waaaaaaaaay better (because we're rate limiting whats coming back on the ssh buffer) using a tmpfs mountpoint seems to make things faster (...less io), but it would be better if we could figure out whats up with the pipes to avoid having commands to set this up.. Also... /tmp might be in ram already? Not sure — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.