remarkable bricked, unclear why

[bprunevi]

I have, in that order :

1. Enabled wifi from xochitl for the first time since unboxing it back in november,
2. Connected to it using ssh (through usb, with ssh keys and after writing down the password)
3. Used that internet connection to install toltec, using the three commands in their readme
4. added opkg to my PATH ? not sure why I had to do that but I had to add /opt/bin to $PATH
5. I have installed rm2fb from toltec (opkg install rm2fb)
6. Have installed koreader manually using the latest release at the time.
7. started the koreader service
8. it runs well, I'm happy to see it works
9. I then disabled xochtil, enabled koreader, since I want koreader at startup not xochitl
10. After using koreader for about 10 minutes it crashed down. 10 . The remarkable does start but does not start xochitl or koreader at startup (not so surprised, could be a lot of things)
11. I cannot ssh into it, it looks like sshd does not start at all. (This i did not expect !)

If anyone knows why the remarkable won't let me ssh into it I am genuinely interested, I guess I knew I was playing with fire but even looking back now I have no idea how i could lose ssh access doing what I did.

Note : I am not expecting to get it to work again, or complaining, or asking for refund or whatever. I am just sharing my experience, proving myself my own stupidity once again and asking if by any chance anyone knows what could have happened there.

[Eeems]

11. I cannot ssh into it, it looks like sshd does not start at all. (This i did not expect !)

Over USB?

[bprunevi]

Over USB?

Yes, I have never used ssh over wifi (and wifi won't work now either since it's not connected)

[danshick]

It may be a bit early to point you in this direction, and others may have intermediate troubleshooting steps for you to try first, but there are ways to recover from this even if you can't connect via traditional means.

https://github.com/ddvk/remarkable2-recovery

[ddvk]

it is very strange the ssh is not starting...

[danshick]

it is very strange the ssh is not starting...

If @bprunevi does end up building pogo pins, I hope they are able to pull logs before restoring the factory image. Very curious what's going on.

[bprunevi]

If @bprunevi does end up building pogo pins, I hope they are able to pull logs before restoring the factory image. Very curious what's going on.

will do

[briankaemingk]

I had that issue too. I would try a hard reset, then try to ssh with a usb. That did the trick for me.

On Mon, Mar 29, 2021, 17:25 Aidan Musnitzky @.***> wrote:

same issue for me - just installed rm2fb and then white screen. cannot ssh.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ddvk/remarkable2-framebuffer/issues/39#issuecomment-809813875, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACXK2ZDGOFUFBXIVCNKVMF3TGEK63ANCNFSM4WXENPJQ .

[Eeems]

@bprunevi Any luck?

[Eeems]

@bprunevi poke?

[rmadhwal]

Hi, I seem to have run into the same issue as op after reinstalling toltec on a device without realising that I already had it

reMarkable: ~/ wget http://toltec-dev.org/bootstrap
Connecting to toltec-dev.org (172.104.246.107:80)
bootstrap            100% |**************************************************************************************************************************************************************************************************************|  8367  0:00:00 ETA
reMarkable: ~/ echo "aa808c87fae9a9ad8c5f264044f8d770f7cd8ac07f96d2c5df648f7a20856d3d  bootstrap" | sha256sum -c && bash bootstrap
bootstrap: OK
INFO:  Fetching secure wget
INFO:  Bootstrapping Opkg
2021-12-09 18:35:48 URL:https://bin.entware.net/armv7sf-k3.2/installer/opkg [752476/752476] -> "/home/root/.local/bin/opkg" [1]
2021-12-09 18:35:49 URL:https://toltec-dev.org/stable/rmall/toltec-bootstrap_0.2.1-1_rmall.ipk [6649/6649] -> "/tmp/toltec-bootstrap_0.2.1-1_rmall.ipk" [1]
INFO:  Installing Toltec and Entware
rm: can't remove '/opt': Device or resource busy
ERROR: Unexpected error on line bootstrap:179 in function main
ERROR: This script failed to install. If you can't solve the above
       issue yourself, please report it at:

           <https://github.com/toltec-dev/toltec/issues>

       (Please also include these error logs to help solving the
       problem faster. Thank you!)
Removing mount over '/opt'
Removed /etc/systemd/system/local-fs.target.wants/opt.mount.
reMarkable: ~/ ls
IJapD8h.png       bootstrap         chessmarkable     entware-reenable  log.txt           log.txt.old       plato-media       screenshots
reMarkable: ~/ opkg update
-sh: opkg: command not found
reMarkable: ~/ echo "aa808c87fae9a9ad8c5f264044f8d770f7cd8ac07f96d2c5df648f7a20856d3d  bootstrap" | sha256sum -c && bash bootstrap
bootstrap: OK
INFO:  Fetching secure wget
INFO:  Bootstrapping Opkg
2021-12-09 18:36:40 URL:https://bin.entware.net/armv7sf-k3.2/installer/opkg [752476/752476] -> "/home/root/.local/bin/opkg" [1]
2021-12-09 18:36:41 URL:https://toltec-dev.org/stable/rmall/toltec-bootstrap_0.2.1-1_rmall.ipk [6649/6649] -> "/tmp/toltec-bootstrap_0.2.1-1_rmall.ipk" [1]
INFO:  Installing Toltec and Entware
Mounting '/home/root/.entware' over '/opt'
Created symlink /etc/systemd/system/local-fs.target.wants/opt.mount → /lib/systemd/system/opt.mount.
Downloading https://bin.entware.net/armv7sf-k3.2/Packages.gz
Updated list of available packages in /opt/var/opkg-lists/entware
... (normal installation of toltec)

After this opkg installed fine without any issues and remarkable was still running fine, but running anything gave me this:

chessmarkable: /lib/libc.so.6: version `GLIBC_2.28' not found (required by /opt/lib/librm2fb_client.so.1)

After some time my ssh connection broke and I can't get back into my remarkable (even with a hard reset). On boot, the remarkable says "remarkable is starting" and loads into a blank screen. I can't seem to ssh over usb (ssh times out). Any suggestions ?

[matteodelabre]

@rmadhwal Hi! Are you using a reMarkable 1 or 2? Do you recall what your system version was?

[rmadhwal]

I'm on remarkable 2, was 2.5 if I'm not mistaken. I've had it since earlier this year and hadn't updated it so definitely wasn't a newer version. I suspect this is an issue with corrupted lib files, and will look at using https://github.com/ddvk/remarkable2-recovery when I have some time

[Eeems]

I don't believe toltec supports OS 2.5 anymore as it's too old.

[rmadhwal]

Hmm I see, any pointers on how I can go about recovering my device once I manage to mount it ?

[matteodelabre]

What puzzles me is that installing Toltec should not result in lib corruption since it installs its libs to /opt. So I can’t explain why you lost SSH access. Do you see anything in your host computer’s dmesg when you plug in the USB cable & start the device?

[rmadhwal]

Sorry for the quality, I don’t have a Linux machine on hand so I booted Ubuntu off a usb and have no means of getting the text output out. The remarkable’s output is near the bottom at 14:54

[matteodelabre]

This is encouraging, so the device seems to be running and you can see it on the bus. Is the USB network interface up, i.e., do you see anything in ip addr that looks like what I got below?

6: enp0s20f0u2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    link/ether XX:XX:XX:XX:XX:XX brd ff:ff:ff:ff:ff:ff
    inet 10.11.99.2/29 metric 10 brd 10.11.99.7 scope global dynamic enp0s20f0u2
       valid_lft 58sec preferred_lft 58sec
    inet6 fe80::XXXX:XXXX:XXXX:XXXX/64 scope link
       valid_lft forever preferred_lft forever

[rmadhwal]

The one at the bottom looks similar, but it doesn’t have an inet address

[rmadhwal]

[Eeems]

You could try setting it up manually. Your machine might just not be trying to connect automatically to it.

[rmadhwal]

Hmm okay I’ll give this a try later, though it’s a bit strange that the machine would randomly stop automatically connecting it

[rmadhwal]

Okay, I can see an inet address on it on osx, so you're probably right

en8: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=6407<RXCSUM,TXCSUM,VLAN_MTU,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
        ether ca:ca:60:81:1e:4a
        inet6 fe80::8d3:6b86:4b19:bf19%en8 prefixlen 64 secured scopeid 0xe
        inet 169.254.93.37 netmask 0xffff0000 broadcast 169.254.255.255
        nd6 options=201<PERFORMNUD,DAD>
        media: autoselect (100baseTX <full-duplex>)
        status: active

So the usb network interface is indeed up, but ssh still results in connection refused

[rmadhwal]

port scan confirms that only 6881, 19575, 19576 and 19577 are open on the host. though is it strange that the ip is 169.x.x.x instead of the normal 10.x.x.x format ?

[Eeems]

That is odd, are you use that's the usb device and not something else?

[rmadhwal]

yep, pretty sure, it goes away when I disconnect it and is back when it's connected

[rotech]

The 169.254.x.x IP address is the automatic private range. On Mac and Windows this is assigned when there is no other IP available, or there is some other issue with the network. Looks like the underlying interface is up, but the remarkable isn't actually communicating over it.

[rmadhwal]

Hi, so I managed to follow the recovery and take a dump of my remarkable’s filesystem. Drawing blanks on locating the log files (var/log doesn’t seem to have anything interesting), I stopped xochitl before my ssh session died so the log file in the root user directory doesn’t have anything interesting. Also, does anyone have any ideas on how can I easily tap into an init script to get xochitl running for now ? Thanks in advance.

Also I guess I should mention that the backup partition on my remarkable was somehow corrupted so I had to reformat that.

[Eeems]

The reMarkable uses systemd. Likely you'll just need to remove some of the unit files for toltec items, and then add a link for the xochtil unit if it's missing. Likely you'll want to delete /etc/systemd/system/xochitl.service.d/toltec-wrapper.conf and then look for link in one of the *.wants folders for rm2.service and remove that.

If that still isn't working, you'll probably want to grab one of the journald logs and see what it says is failing to start.

[rmadhwal]

Cool thanks for that pointer.

seems like the toltec-wrapper.conf is malformatted and has two ExecStart= under [Service], I wonder if that’s causing some issues. Also I don’t see any xochitl units, the only thing I have under xochitl.service.d is toltec-wrapper.conf. I see a rm2fb.service under xochitl.service.wants, is that what you’re referring to in the second bit ?

[Eeems]

seems like the toltec-wrapper.conf is malformatted and has two ExecStart= under [Service], I wonder if that’s causing some issues.

This is intentional, and not actually an issue: https://github.com/toltec-dev/toltec/blob/stable/package/xochitl/toltec-wrapper.conf#L1-L3

Also I don’t see any xochitl units, the only thing I have under xochitl.service.d is toltec-wrapper.conf.

The xochitl unit is under /usr somewhere, and likely linked under a *.wants under there somewhere.

I see a rm2fb.service under xochitl.service.wants, is that what you’re referring to in the second bit ?

Yes, you'll want to remove that.

[rmadhwal]

Is this the one ? See a lot of dangling links, wonder where all those files went (I can’t locate the file, this is not because it’s mounted)

[Eeems]

That looks correct. Odd that there are so many dangling links... I recommend that you message me on the community discord server. I might be able to get you sorted in another way.

[rmadhwal]

Sure, will do thanks!

[rmadhwal]

Update: managed to get it working again with @Eeems help. Steps I followed to fix it: 1) Downloaded the raw patch file for the version closest to mine off https://thelastzombie.github.io/remarkable-firmware/ then used https://github.com/ddvk/remarkable-update/tree/main/extractor to extract it 2) Rsyncd all the files from the previous step into the system (second) partition of my remarkable 3) Restarted the remarkable, xochitl booted up and I don't have a 500 dollar paperweight anymore :)

[Eeems]

@rmadhwal since this is resolved, could you close the issue?

[Eeems]

Never mind, I was given access to the repo, so I can do it myself now :)