search

ddvk / rmfakecloud

host your own cloud for the remarkable
GNU Affero General Public License v3.0
789 stars 68 forks source link

Is it possible to connect macOS or iOS apps to rmfakecloud? #300

Closed aoaim closed 2 months ago

aoaim commented 7 months ago

I apologize for what may seem like a stupid question. I hope someone can help me.

I have properly installed rmfakecloud using docker on hetzner's vps, and my remarkable2 can connect to my rmfakecloud properly, and also can upload documents and thus synchronize them to my remarkable2 through rmfakecloud webpage properly. i feel very much comfortable using it! I find it very comfortable to use.

However, I have both a macbook pro and an iphone, and I would like to synchronize them with my remarkable2. I noticed that rmfakecloud works on the principle of self-signed certificates for the official RMarkable sync server, and uses a reverse proxy to forward the sync requests to rmfakecloud, which in turn provides a great alternative to the official RMarkable sync server - very important for me in mainland China. As you know, in mainland China's network environment, remarkable's synchronization service is completely unavailable.

I use Surge as my proxy/VPN software on my macbook pro and iphone. Surge is essentially a network debugger with MitM and Rewrite. This made me think of rmfakecloud, the principle of rmfakecloud seems to use MitM and Rewrite. so I tried to transfer the requests to the remarkable sync server to my rmfakecloud in Surge, here is my config file:

#!name= rM2rmfakecloud
#!desc=My rmfakecloud

[URL Rewrite]
^https?://hwr-production-dot-remarkable-production.appspot.com https://rmfakecloud.domain.com 302
^https?://service-manager-production-dot-remarkable-production.appspot.com https://rmfakecloud.domain.com 302
^https?://local.appspot.com https://rmfakecloud.domain.com 302
^https?://my.remarkable.com https://rmfakecloud.domain.com 302
^https?://ping.remarkable.com https://rmfakecloud.domain.com 302
^https?://internal.cloud.remarkable.com https://rmfakecloud.domain.com 302
^https?://webapp-prod.cloud.remarkable.engineering https://rmfakecloud.domain.com 302

[MITM]
hostname = %APPEND% hwr-production-dot-remarkable-production.appspot.com, service-manager-production-dot-remarkable-production.appspot.com, local.appspot.com, my.remarkable.com, ping.remarkable.com, internal.cloud.remarkable.com, webapp-prod.cloud.remarkable.engineering

After deployment, I opened the remarkable app on both ios and macOS. I entered the one-time code I got from rmfakeable. Luckily, I passed the verification and went to the normal interface of the app. But, strangely, the file is empty here, and obviously not synced up with my rmfakecloud -- even after having logged in successfully.

Am I missing some important domain name related to synchronization?

aoaim commented 7 months ago

I realized I was wrong and I didn't really log in. I just realized that typing in 8 random letters takes me to the app's file screen,, well I'm even more confused.

aoaim commented 7 months ago

I can see the rmfakecloud docker logs

time="2024-04-14T08:06:03Z" level=info msg="[ui-authmiddleware] User from token: a@a.a"
time="2024-04-14T08:06:03Z" level=info msg="[GIN] 2024/04/14 - 08:06:03 | 200 |     421.222µs |   154.17.12.197 | GET      \"/ui/api/newcode\""
time="2024-04-14T08:06:07Z" level=info msg=/token/json/3/device/delete
time="2024-04-14T08:06:07Z" level=info msg="[GIN] 2024/04/14 - 08:06:07 | 200 |     103.053µs |   154.17.12.197 | GET      \"/token/json/3/device/delete\""
time="2024-04-14T08:06:14Z" level=info msg=/token/json/2/device/new
time="2024-04-14T08:06:14Z" level=info msg="[GIN] 2024/04/14 - 08:06:14 | 200 |     124.032µs |   154.17.12.197 | GET      \"/token/json/2/device/new\""
time="2024-04-14T08:06:14Z" level=info msg=/token/json/2/user/new
time="2024-04-14T08:06:14Z" level=info msg="[GIN] 2024/04/14 - 08:06:14 | 200 |      85.681µs |   154.17.12.197 | GET      \"/token/json/2/user/new\""
y0grt commented 7 months ago

in order to make the mac/ios app work, you will need a reverse proxy with ssl enabled and a self-signed certificate in front of rmfakecloud (nginx e.g.), then make the device trust the ca used to sign that certificate, then make your dns point the domains below to your instance

hwr-production-dot-remarkable-production.appspot.com    
internal.cloud.remarkable.com
local.appspot.com
my.remarkable.com
ping.remarkable.com
service-manager-production-dot-remarkable-production.appspot.com
webapp-prod.cloud.remarkable.engineering
TheToto318 commented 6 months ago

in order to make the mac/ios app work, you will need a reverse proxy with ssl enabled and a self-signed certificate in front of rmfakecloud (nginx e.g.), then make the device trust the ca used to sign that certificate, then make your dns point the domains below to your instance

hwr-production-dot-remarkable-production.appspot.com  
internal.cloud.remarkable.com
local.appspot.com
my.remarkable.com
ping.remarkable.com
service-manager-production-dot-remarkable-production.appspot.com
webapp-prod.cloud.remarkable.engineering

Hey, Did you have additional information about how to make the windows/mac app work with rmfakecloud ? I tried to set up my reverse proxy to serve these domains with an certificates signed with my own CA (trusted by my devices, systemwide). With a DNS override, I can access to the rmfakecloud API within my browser with no warning and by using the official endpoint (internal.cloud.remarkable.com). But when I try to use the RM windows app, the sync does not work. Some thing strange is that when I enable my fiddler proxy (HTTPS flow analyser, use a user system trusted CA to sign all certificates for HTTPS trafic going through the proxy), the sync is working !

I suspect an issue with my certificate, but since I couldn't find any logs for the remarkable app on windows I'm not sure. If you have an idea...

Have a great day

y0grt commented 6 months ago

have you tried re-paring the app after the proxy is enabled? I have no experience with windows systems tho

On Sun, May 12, 2024 at 20:42, Thomas Roux @.***(mailto:On Sun, May 12, 2024 at 20:42, Thomas Roux < wrote:

in order to make the mac/ios app work, you will need a reverse proxy with ssl enabled and a self-signed certificate in front of rmfakecloud (nginx e.g.), then make the device trust the ca used to sign that certificate, then make your dns point the domains below to your instance

hwr-production-dot-remarkable-production.appspot.com internal.cloud.remarkable.com local.appspot.com my.remarkable.com ping.remarkable.com service-manager-production-dot-remarkable-production.appspot.com webapp-prod.cloud.remarkable.engineering

Hey, Did you have additional information about how to make the windows/mac app work with rmfakecloud ? I tried to set up my reverse proxy to serve these domains with an certificates signed with my own CA (trusted by my devices, systemwide). With a DNS override, I can access to the rmfakecloud API within my browser with no warning and by using the official endpoint (internal.cloud.remarkable.com). But when I try to use the RM windows app, the sync does not work. Some thing strange is that when I enable my fiddler proxy (HTTPS flow analyser, use a user system trusted CA to sign all certificates for HTTPS trafic going through the proxy), the sync is working !

I suspect an issue with my certificate, but since I couldn't find any logs for the remarkable app on windows I'm not sure. If you have an idea...

Have a great day

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

TheToto318 commented 6 months ago

have you tried re-paring the app after the proxy is enabled? I have no experience with windows systems tho

On Sun, May 12, 2024 at 20:42, Thomas Roux @.***(mailto:On Sun, May 12, 2024 at 20:42, Thomas Roux < wrote:

in order to make the mac/ios app work, you will need a reverse proxy with ssl enabled and a self-signed certificate in front of rmfakecloud (nginx e.g.), then make the device trust the ca used to sign that certificate, then make your dns point the domains below to your instance

hwr-production-dot-remarkable-production.appspot.com internal.cloud.remarkable.com local.appspot.com my.remarkable.com ping.remarkable.com service-manager-production-dot-remarkable-production.appspot.com webapp-prod.cloud.remarkable.engineering

Hey, Did you have additional information about how to make the windows/mac app work with rmfakecloud ? I tried to set up my reverse proxy to serve these domains with an certificates signed with my own CA (trusted by my devices, systemwide). With a DNS override, I can access to the rmfakecloud API within my browser with no warning and by using the official endpoint (internal.cloud.remarkable.com). But when I try to use the RM windows app, the sync does not work. Some thing strange is that when I enable my fiddler proxy (HTTPS flow analyser, use a user system trusted CA to sign all certificates for HTTPS trafic going through the proxy), the sync is working !

I suspect an issue with my certificate, but since I couldn't find any logs for the remarkable app on windows I'm not sure. If you have an idea...

Have a great day

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

Pairing and syncing are working great with the proxy enabled but doesn't when it's disabled.

y0grt commented 6 months ago

what is acting as a proxy when your fiddler proxy is disabled? when you open the windows app do you see any output on the rmfakecloud logs? like when you hit the api from the browser.

TheToto318 commented 6 months ago

what is acting as a proxy when your fiddler proxy is disabled? when you open the windows app do you see any output on the rmfakecloud logs? like when you hit the api from the browser.

It's my nginx reverse proxy that redirect all requests on remarkable domain to rmfakecloud (with a custom certificate from my CA). On the rmfakecloud side, I only see two discovery requests : 

internal.cloud.remarkable.com 10.10.10.2 192.168.1.1 - - [13/May/2024:10:25:05 +0200] "GET /discovery/v1/endpoints HTTP/2.0" 200 90 "-" "desktop/3.10.0.845 (windows build 22631 11)" 90 0.001 "0.001" 192.168.1.119:1711 90 0.001 200"-" "-"
internal.cloud.remarkable.com 10.10.10.2 192.168.1.1 - - [13/May/2024:10:25:06 +0200] "GET /discovery/v1/endpoints HTTP/2.0" 200 90 "-" "desktop/3.10.0.845 (windows build 22631 11)" 7 0.001 "0.001" 192.168.1.119:1711 90 0.001 200"-" "-"

But the sync is not starting...

This is what a nomal remarkable app requests look like when the sync is working (fiddler proxy enabled) : image

I tried to edit the "STORAGE_URL" env variable to "internal.cloud.remarkable.com" instead of my the domain name using my own public dns so that the discovery request return the "official" endpoints...

{"notifications":"internal.cloud.remarkable.com","webapp":"internal.cloud.remarkable.com"}

My feeling was that the remarkable app could detect a certificate mismatch between the domain used withing the "GET /notifications/ws/json/1" and the domain used to proprely sync the files "GET /sync..."

But it seems that the problem is the same... It's a shame that the app show no logs during sync.

y0grt commented 6 months ago

no you don't need to adjust the STORAGE_URL env variable. check the answer of the discovery endpoint using the official url against your rmfakecloud instance, and share your nginx config file.

TheToto318 commented 6 months ago

no you don't need to adjust the STORAGE_URL env variable. check the answer of the discovery endpoint using the official url against your rmfakecloud instance, and share your nginx config file.

Here is my reverse proxy config (fake official API): 

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    client_max_body_size 200M;

    server_name *.appspot.com;

    # Certificates
    ssl_certificate /config/keys/MY_CUSTOM_CA/appspot.com.pem;
    ssl_certificate_key /config/keys/MY_CUSTOM_CA/appspot.com-key.pem;

    # enable for ldap auth, fill in ldap details in ldap.conf
    #include /config/nginx/ldap.conf;

    include /config/nginx/local-services-rv.conf;

    location / {
        # enable the next two lines for http auth
        #auth_basic "Restricted";
        #auth_basic_user_file /config/nginx/.htpasswd;

        # enable the next two lines for ldap auth
        #auth_request /auth;
        #error_page 401 =200 /login;

        include /config/nginx/proxy.conf;
        resolver 127.0.0.11 valid=30s;
        set $upstream_app 192.168.1.119;
        set $upstream_port 1711;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

    }
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    client_max_body_size 200M;

    server_name *.remarkable.com;

    # Certificates
    ssl_certificate /config/keys/MY_CUSTOM_CA/appspot.com.pem;
    ssl_certificate_key /config/keys/MY_CUSTOM_CA/appspot.com-key.pem;

    # enable for ldap auth, fill in ldap details in ldap.conf
    #include /config/nginx/ldap.conf;

    include /config/nginx/local-services-rv.conf;

    location / {
        # enable the next two lines for http auth
        #auth_basic "Restricted";
        #auth_basic_user_file /config/nginx/.htpasswd;

        # enable the next two lines for ldap auth
        #auth_request /auth;
        #error_page 401 =200 /login;

        include /config/nginx/proxy.conf;
        resolver 127.0.0.11 valid=30s;
        set $upstream_app 192.168.1.119;
        set $upstream_port 1711;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

    }
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    client_max_body_size 200M;

    server_name *.remarkable.engineering;

    # Certificates
    ssl_certificate /config/keys/MY_CUSTOM_CA/appspot.com.pem;
    ssl_certificate_key /config/keys/MY_CUSTOM_CA/appspot.com-key.pem;

    # enable for ldap auth, fill in ldap details in ldap.conf
    #include /config/nginx/ldap.conf;

    include /config/nginx/local-services-rv.conf;

    location / {
        # enable the next two lines for http auth
        #auth_basic "Restricted";
        #auth_basic_user_file /config/nginx/.htpasswd;

        # enable the next two lines for ldap auth
        #auth_request /auth;
        #error_page 401 =200 /login;

        include /config/nginx/proxy.conf;
        resolver 127.0.0.11 valid=30s;
        set $upstream_app 192.168.1.119;
        set $upstream_port 1711;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

    }
}

This is the reverse proxy config for the public access (lets encrypt certificate): 

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    client_max_body_size 200M;

    server_name rmcloud.*; (my public domain....)

    include /config/nginx/ssl.conf;

    # enable for ldap auth, fill in ldap details in ldap.conf
    #include /config/nginx/ldap.conf;

    location / {
        # enable the next two lines for http auth
        #auth_basic "Restricted";
        #auth_basic_user_file /config/nginx/.htpasswd;

        # enable the next two lines for ldap auth
        #auth_request /auth;
        #error_page 401 =200 /login;

        include /config/nginx/proxy.conf;
        resolver 127.0.0.11 valid=30s;
        set $upstream_app 192.168.1.119;
        set $upstream_port 1711;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

    }
}

The discovery endpoint return the URL specified in the STORAGE_URL so I assume that I need to put my public domain, it is working with my tablet because rmfakecloud-proxy is installed on it and redirect all the trafic to my public domain.

But for the official app, I don't know what I need to set...

y0grt commented 6 months ago

no need to change STORAGE_URL, leave it on your public domain. a simpler configuration will do it

worker_processes auto;

events {
    worker_connections 1024;
}

http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    server {
        listen 80;
        server_name *.appspot.com *.remarkable.com *.remarkable.engineering;

        location / {
            return 301 https://$host$request_uri;
        }
    }

    server {
        listen 443 ssl;
        server_name *.appspot.com *.remarkable.com *.remarkable.engineering;

        ssl_certificate /etc/nginx/certs/server.crt;
        ssl_certificate_key /etc/nginx/certs/server.key;

        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers 'TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384';

        ssl_prefer_server_ciphers off;
        ssl_session_cache shared:SSL:10m;
        ssl_session_timeout 5m;

        location / {
            proxy_pass http://rmfakecloud:3000;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
}
TheToto318 commented 6 months ago

no need to change STORAGE_URL, leave it on your public domain. a simpler configuration will do it

worker_processes auto;

events {
    worker_connections 1024;
}

http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    server {
        listen 80;
        server_name *.appspot.com *.remarkable.com *.remarkable.engineering;

        location / {
            return 301 https://$host$request_uri;
        }
    }

    server {
        listen 443 ssl;
        server_name *.appspot.com *.remarkable.com *.remarkable.engineering;

        ssl_certificate /etc/nginx/certs/server.crt;
        ssl_certificate_key /etc/nginx/certs/server.key;

        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers 'TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384';

        ssl_prefer_server_ciphers off;
        ssl_session_cache shared:SSL:10m;
        ssl_session_timeout 5m;

        location / {
            proxy_pass http://rmfakecloud:3000;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
}

Hey, Thanks for your answer, I tried your nginx config (close to what I've made) but still the same problem, the app doesn't sync. I only got these hits on rmfakecloud :

internal.cloud.remarkable.com 10.10.10.2 192.168.1.1 - - [15/May/2024:22:34:53 +0200] "GET /discovery/v1/endpoints HTTP/2.0" 200 74 "-" "desktop/3.10.0.845 (windows build 22631 11)" 90 0.003 "0.000" 192.168.1.119:1711 74 0.002 200"-" "-"
internal.cloud.remarkable.com 10.10.10.2 192.168.1.1 - - [15/May/2024:22:34:53 +0200] "GET /discovery/v1/endpoints HTTP/2.0" 200 74 "-" "desktop/3.10.0.845 (windows build 22631 11)" 7 0.002 "0.001" 192.168.1.119:1711 74 0.002 200"-" "-"
y0grt commented 6 months ago

what is showing the nginx logs?

y0grt commented 6 months ago

and what version of rmfakecloud are you using?

TheToto318 commented 6 months ago

what is showing the nginx logs?

Specified in the last message :

internal.cloud.remarkable.com 10.10.10.2 192.168.1.1 - - [15/May/2024:22:34:53 +0200] "GET /discovery/v1/endpoints HTTP/2.0" 200 74 "-" "desktop/3.10.0.845 (windows build 22631 11)" 90 0.003 "0.000" 192.168.1.119:1711 74 0.002 200"-" "-"
internal.cloud.remarkable.com 10.10.10.2 192.168.1.1 - - [15/May/2024:22:34:53 +0200] "GET /discovery/v1/endpoints HTTP/2.0" 200 74 "-" "desktop/3.10.0.845 (windows build 22631 11)" 7 0.002 "0.001" 192.168.1.119:1711 74 0.002 200"-" "-"

and what version of rmfakecloud are you using?

Last version

y0grt commented 6 months ago

what do you get when you execute curl https://internal.cloud.remarkable.com/discovery/v1/endpoints and curl https://internal.cloud.remarkable.com/discovery/v1/webapp and what's showing nginx logs?

TheToto318 commented 6 months ago

Found the problem, I tried to curl the API on Linux and it showed me an certificate error. The problem was that I didn't exported the full chain for my certificate used in NGINX. It create no problem with windows but linux/the remarkable app does not trust it.

Problem solved ! Thanks you for your help

TheToto318 commented 6 months ago

To follow on integrating remarkable app with rmfakecloud, I managed to patch the android app to trust my own CA, with the correct DNS redirect the app is working flowlessly with rmfakecloud.

Maybe it could be relevant to create a guide for user who want to use rmfakecloud with official remarkable apps... @y0grt

y0grt commented 6 months ago

it's quite dependant on the setup of each user, but the instruction here are relevant https://ddvk.github.io/rmfakecloud/remarkable/setup/ didn't investigate but probably there is an easy way to run a proxy in android that could do the job of rmfakecloud-proxy.