Open qjoly opened 1 year ago
ddvk
commented
1 year ago the tablet doesn't trust: https://remarkable.redacted, if you are not using an official CA (e.g let's encrypt) and using some self signed CA you need to add it to the trusted CAs on the tablet.
qjoly
commented
1 year ago I'm using let's encrypt, the tablet trust https://remarkable.redacted since I can curl without adding -k to accept untrusted cert :(
Eeems
commented
1 year ago Apr 18 08:02:15 reMarkable rmfake-proxy[11273]: 2023/04/18 08:02:15 cert-file=/home/root/rmfakecloud/proxy.bundle.crt key-file=/home/root/rmfakecloud/proxy.key listen-addr=:443 upstream-url=https://remarkable.redacted
Apr 18 08:03:34 reMarkable rmfake-proxy[11273]: 2023/04/18 08:03:34 http: TLS handshake error from 192.168.1.84:40556: remote error: tls: unknown certificate authorityThese lines lead me to believe that it doesn't trust it though. Did you test the curl call from the device, or your computer?
qjoly
commented
1 year ago Apr 18 08:02:15 reMarkable rmfake-proxy[11273]: 2023/04/18 08:02:15 cert-file=/home/root/rmfakecloud/proxy.bundle.crt key-file=/home/root/rmfakecloud/proxy.key listen-addr=:443 upstream-url=https://remarkable.redacted Apr 18 08:03:34 reMarkable rmfake-proxy[11273]: 2023/04/18 08:03:34 http: TLS handshake error from 192.168.1.84:40556: remote error: tls: unknown certificate authorityThese lines lead me to believe that it doesn't trust it though. Did you test the curl call from the device, or your computer?
Sorry for late answer,
I can curl from the device without add untrusted cert:
root@reMarkable:~# ./installer.sh gencert
CA exists
Private key exists
Pub key exists
crt exists
root@reMarkable:~# curl https://remarkable.redacted
<!doctype html><html lang="en"><head><meta charset="utf-8"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="RM FakeApi"/><title>rmfakecloud</title><script defer="defer" src="/static/js/main.9c2de5b1.js"></script><link href="/static/css/main.d94d89ba.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>root@reMarkable:~# 
zeigerpuppy
commented
1 year ago Continuing error reporting of sync here instead of https://github.com/ddvk/rmfakecloud/issues/237
I tried:
/usr/local/share/ca-certificates/ca.crt and re-ran installer on tabletRM_TRUST_PROXY=true to server (docker-compose) configlog.txt shows successful pairing but then the following errors:
May 01 10:57:31.861 Debug: UserToken: setting a new userToken ("XXXXXXXXXX"...) (/usr/src/debug/xochitl/override+gitAUTOINC+6a003d604f-r0/git/src/network/src/usertoken.cpp:73, setUserToken)
May 01 10:57:31.954 Warning: Notifications socket is not OK: UnconnectedState (/usr/src/debug/xochitl/override+gitAUTOINC+6a003d604f-r0/git/src/notifications/src/notifications.cpp:187, checkIfShouldConnect)
May 01 10:57:32.626 Warning: Could not find hostname for service "notifications" (/usr/src/debug/xochitl/override+gitAUTOINC+6a003d604f-r0/git/src/network/src/servicehostname.cpp:44, parseNetworkReply)I checked the setting in etc/hosts and they are directing to the proxy properly
# rmfake_start
127.0.0.1 hwr-production-dot-remarkable-production.appspot.com
127.0.0.1 service-manager-production-dot-remarkable-production.appspot.com
127.0.0.1 local.appspot.com
127.0.0.1 my.remarkable.com
127.0.0.1 internal.cloud.remarkable.com
127.0.0.1 ping.remarkable.com
# rmfake_endOKecho Q | openssl s_client -connect localhost:443 -verify_hostname local.appspot.com -CAfile /etc/ssl/certs/ca-certificates.crt 2>&1 | grep Verify
p Verify
Verify return code: 0 (ok)
Verify return code: 0 (ok)proxy.service looks sane (has my server address) and systemctrl shows the service running with the correct address.Bit stumped as to what may be the cause of failed tablet-> server sync
Looks like the main error is Could not find hostname for service "notifications" (also showing in the xochitl log)
Any ideas?
ddvk
commented
1 year ago is there something in the rmfakecloud's logs with "notifications" ?
develop-Greenant
commented
1 year ago I think this may be caused by a mismatch between the SSL ciphers on the remarkable and the nginx reverse proxy:
When testing wget from client: wget -qO- https://myrmfakecloud.server.net:
wget: note: TLS certificate validation not implemented
wget: TLS error from peer (alert code 80): 80
wget: error getting response: Connection reset by peerOn the nginx server, it reports:
2023/07/15 15:04:03 [error] 21487#21487: *7 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.113, server: myrmfakecloud.server.net, request: "GET /notifications/ws/json/1 HTTP/1.1", upstream: "http://[::1]:3000/notifications/ws/json/1", host: "myrmfakecloud.server.net:443"
2023/07/15 15:04:15 [crit] 21487#21487: *11 SSL_do_handshake() failed (SSL: error:14201044:SSL routines:tls_choose_sigalg:internal error) while SSL handshaking, client: 10.0.0.113, server: 0.0.0.0:443A connection from other clients is fine.
So, I think the remarkable is trying to use an old cipher/SSL version and the nginx server rejects it.
Tried to loosen the default ciphers with the following in nginx reverse-proxy, but still getting the error:
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
ssl_ciphers ALL;Any ideas which cipher may be needed?
I guess this also explains why it stopped working (nginx server update woud have restricted old, insecure ciphers)
Eeems
commented
1 year ago wget on the rM doesn't support SSL at all out of box, but that isn't related to the proxy. You can grab a version of it that does work to use for testing here: http://toltec-dev.org/thirdparty/bin/wget-v1.21.1
develop-Greenant
commented
1 year ago Thanks for tip regarding wget
pgnhdcrt
commented
10 months ago Related issue?
Let's start with the known good, working configuration:
Server: Clean installation from source of rmfakecloud 0.0.15 on AlmaLinux 9.3. Have also tried the Docker image (I'm using a VM, so have snapped and reverted multiple times).
STORAGE_URL=https://<server>
PORT=443
Tablet: Automagic installation of rmfakecloud-proxy 0.0.3 on OS 3.7.0.1930.
ExecStart is calling https://<server>
In this configuration, sync works beautifully and there are no errors in any of the logs. I took packet captures in this config as a baseline.
Change to the problematic configuration. Note that these are the only changes to the above working config:
Server (rmfakecloud restarted post-change):
STORAGE_URL=https://<server>:3000
PORT=300
Tablet (changed via 'installer.sh setcloud', then restarted xochitl):
ExecStart is calling https://<server>:3000
In this configuration, sync is not working.
Tablet: xochitl log shows rm.network.notifications Notifications socket is not OK: UnconnectedState (checkIfShouldConnect /__w/xochitl/xochitl/src/notifications/src/notifications.cpp:190)
Server: Nothing in the logs, however, packet capture (at the server) shows port 443 traffic from the tablet which coincides with the xochitl errors. All other traffic is on port 3000, as expected.
Rebooted the tablet. Sync is still not working, xochitl log still shows the "notifications" error. The weird part? Packet capture no longer shows the port 443 traffic. Changed capture location to the router/AP (OpenWRT), and there is zero traffic (on any port) coming from the tablet when the xochitl errors occur.
My initial thought was a hardcoded sync call to 443, but post-reboot I'm stumped. I'm not familiar enough with the various bits to perform deeper inspection / logging on the tablet (are there debug settings for -proxy?). Happy to dig deeper, just point me in the right direction.
Update: Correction: #271 resolved the issue for me as well. #271 did not resolve the issue of being unable to change to a different (non 443) port. Apologies for the confusion.
mfussenegger
commented
10 months ago I'm also seeing rm.network.notifications Notifications socket is not OK errors.
The tablet is already on 3.8.2.1965 (is this supported?)
Built rmfakecloud from source, to include https://github.com/ddvk/rmfakecloud/commit/f1827cd952f1190de6d6f15c1824ec7b026609fc
rmfakecloud is running with plain HTTP on another host in a local network.
Pairing worked
rmapi cli seems to work
Connectivity is okay (ping, wget -qO-, etc. are working)
echo Q | openssl s_client -connect localhost:443 -verify_hostname local.appspot.com -CAfile /etc/ssl/certs/ca-certificates.crt 2>&1 | grep Verify shows ok
I see various GET calls to /sync/v3/... and v1/reports endpoints in the logs
I'm not seeing any connections to /notifications/ws/json/1 in the server logs. (Should this be the case?)
Trying to connect manually with websocat to that endpoint works if using the STORAGE_URL
Trying websocat -v wss://local.appspot.com/notifications/ws/json/1 --header "Authorization. Bearer <token>" also works and I see connecting websocket from: ... on the server when doing this manually.
Tried running strace xochitl, but for some reason I'm not seeing any connect/socket related information
To isolate if it's the local proxy, or the server component I then tried a different variant by disabling the local proxy on the tablet, and instead editing /etc/hosts to point to a public server. Then I put rmfakecloud behind an nginx proxy with something like the following, and ssh reverse tunneling:
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name hwr-production-dot-remarkable-production.appspot.com;
server_name service-manager-production-dot-remarkable-production.appspot.com;
server_name local.appspot.com;
server_name my.remarkable.com;
server_name internal.cloud.remarkable.com;
server_name ping.remarkable.com;
ssl_certificate_key /etc/nginx/proxy.key;
ssl_certificate /etc/nginx/proxy.bundle.crt;
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
location / {
proxy_pass http://tunnel;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
}
}(proxy.key, and proxy.bundle.crt are generated with the rmfakecloud-proxy install script)
The result is pretty much the same: Pairing works, sync doesn't. Same Notifications socket is not OK in the log.
I'm not sure what else I could try. Unfortunately I haven't been able to get termshark working on the tablet. Does anyone know of a static build?
Update: https://github.com/ddvk/rmfakecloud/pull/271 fixed the issue for me
nemunaire
commented
10 months ago The error Tablet is not syncing (Notifications Socket is Not OK?) is due to nginx closing inactives connections.
The notifications service creates a websocket, and after 1 minute without any exchange between rmfakecloud and the tablet, nginx takes the initiative to close the connection. This error occurs, and on the server side, we can see a warning: msg="Can't read from ws websocket: close 1006 (abnormal closure): unexpected EOF", for the same reason: the tablet doesn't send a proper close. This is true, as it's nginx that closes the connection.
This is not a problem as the tablet reconnects a few seconds later.
It can be delayed with nginx by adding proxy_read_timeout 10800;. With this option the socket will have a maximal duration of 3h. Eg.:
location /notifications/ws/json/1 {
proxy_pass http://_YOURPROXYADDRESS_;
proxy_http_version 1.1;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_redirect off;
proxy_read_timeout 10800;
}@mfussenegger I just create a pull request for the 3.8 release.
Hi,
Since a few months, my tablet does not synchronize with RMFakecloud. I use the most recent Docker image (tag: always) and my remarkable is up to date (3.2.3.1595).
In log.txt, I have this message :
When I install rmfakecloud with 'magic script', I have a sed error:
but the reverse proxy is runned and fonctionnal:
(192.168.1.84 is my laptop. I tested the reverse proxy with the IP of the tablet)
Do you have any solution ? Thanks in advance