Cannot connect to the selfhosted instace via full openvpn redirect

bytersproblem commented 1 week ago

Have you read a contributing guide?

[X] I have read CONTRIBUTING.md
[X] I have searched the existing issues and didn't find any that were similar
[X] I have considered creating a pull request with fixes instead of a bug report and want to proceed

Current Behavior

I am trying to connect to my selfhosted anytype server via OpenVPN from work. I got the config file for the clients like this (some data removed). The OpenVPN config uses redirecy gateway which means that I am getting the home network address when I check my exteranal IP. I am connecting to OpenVPN via UDP 1194. All resources on the home network are reachable.

id: 
networkId:
nodes:
  - addresses:
      - any-sync-node-1:1001
      - quic://any-sync-node-1:1011
      - 192.168.100.199:1001
      - quic://192.168.100.199:1011
    peerId: 
      - tree
  - addresses:
      - any-sync-node-2:1002
      - quic://any-sync-node-2:1012
      - 192.168.100.199:1002
      - quic://192.168.100.199:1012
    peerId: 
    types:
      - tree
  - addresses:
      - any-sync-node-3:1003
      - quic://any-sync-node-3:1013
      - 192.168.100.199:1003
      - quic://192.168.100.199:1013
    peerId: 
    types:
      - tree
  - addresses:
      - any-sync-coordinator:1004
      - quic://any-sync-coordinator:1014
      - 192.168.100.199:1004
      - quic://192.168.100.199:1014
    peerId: 
    types:
      - coordinator
  - addresses:
      - any-sync-filenode:1005
      - quic://any-sync-filenode:1015
      - 192.168.100.199:1005
      - quic://192.168.100.199:1015
    peerId: 
    types:
      - file
  - addresses:
      - any-sync-consensusnode:1006
      - quic://any-sync-consensusnode:1016
      - 192.168.100.199:1006
      - quic://192.168.100.199:1016
    peerId: 
    types:
      - consensus

Here is output of sudo docker ps

CONTAINER ID   IMAGE                                            COMMAND                  CREATED       STATUS                 PORTS                                                                                                                                                                                 NAMES
564bd38c50f8   ghcr.io/anyproto/any-sync-node:v0.3.34           "/bin/sh -c '/bin/an…"   6 weeks ago   Up 4 weeks             443/tcp, 0.0.0.0:1003->1003/tcp, :::1003->1003/tcp, 0.0.0.0:1013->1013/udp, :::1013->1013/udp, 0.0.0.0:8003->8000/tcp, :::8003->8000/tcp, 0.0.0.0:8083->8080/tcp, :::8083->8080/tcp   any-sync-dockercompose-any-sync-node-3-1
6f35d795b085   ghcr.io/anyproto/any-sync-filenode:v0.6.5        "/bin/sh -c '/bin/an…"   6 weeks ago   Up 4 weeks             443/tcp, 8080/tcp, 0.0.0.0:1005->1005/tcp, :::1005->1005/tcp, 0.0.0.0:1015->1015/udp, :::1015->1015/udp, 0.0.0.0:8004->8000/tcp, :::8004->8000/tcp                                    any-sync-dockercompose-any-sync-filenode-1
e237e1962596   ghcr.io/anyproto/any-sync-node:v0.3.34           "/bin/sh -c '/bin/an…"   6 weeks ago   Up 4 weeks             443/tcp, 0.0.0.0:1001->1001/tcp, :::1001->1001/tcp, 0.0.0.0:1011->1011/udp, :::1011->1011/udp, 0.0.0.0:8001->8000/tcp, :::8001->8000/tcp, 0.0.0.0:8081->8080/tcp, :::8081->8080/tcp   any-sync-dockercompose-any-sync-node-1-1
1cf48b275ad3   ghcr.io/anyproto/any-sync-node:v0.3.34           "/bin/sh -c '/bin/an…"   6 weeks ago   Up 4 weeks             443/tcp, 0.0.0.0:1002->1002/tcp, :::1002->1002/tcp, 0.0.0.0:1012->1012/udp, :::1012->1012/udp, 0.0.0.0:8002->8000/tcp, :::8002->8000/tcp, 0.0.0.0:8082->8080/tcp, :::8082->8080/tcp   any-sync-dockercompose-any-sync-node-2-1
090e5da52b01   ghcr.io/anyproto/any-sync-consensusnode:v0.1.7   "/bin/sh -c '/bin/an…"   6 weeks ago   Up 14 seconds          443/tcp, 8080/tcp, 0.0.0.0:1006->1006/tcp, :::1006->1006/tcp, 0.0.0.0:1016->1016/udp, :::1016->1016/udp, 0.0.0.0:8006->8000/tcp, :::8006->8000/tcp                                    any-sync-dockercompose-any-sync-consensusnode-1
3ac26928affc   ghcr.io/anyproto/any-sync-coordinator:v0.3.25    "/bin/sh -c '/bin/an…"   6 weeks ago   Up 4 weeks             443/tcp, 8080/tcp, 0.0.0.0:1004->1004/tcp, :::1004->1004/tcp, 0.0.0.0:1014->1014/udp, :::1014->1014/udp, 0.0.0.0:8005->8000/tcp, :::8005->8000/tcp                                    any-sync-dockercompose-any-sync-coordinator-1
ead4cfdf14a9   redis/redis-stack-server:7.2.0-v6                "redis-server --port…"   6 weeks ago   Up 4 weeks (healthy)   0.0.0.0:6379->6379/tcp, :::6379->6379/tcp                                                                                                                                             any-sync-dockercompose-redis-1
3fa9864b8fc4   minio/minio:RELEASE.2024-07-04T14-25-45Z         "/usr/bin/docker-ent…"   6 weeks ago   Up 4 weeks (healthy)   0.0.0.0:9000-9001->9000-9001/tcp, :::9000-9001->9000-9001/tcp                                                                                                                         any-sync-dockercompose-minio-1

Expected Behavior

To connect to my selfhosted vault

Steps To Reproduce

Install containers as descibed on https://github.com/anyproto/any-sync-dockercompose Forward ports mentioned Upload conf file created on initial docker raise Enter secret phrase Error:

10:39:10.754 › {"level":"ERROR","ts":"2024-09-04T10:39:10.754+0200","logger":"service-metrics","msg":"failed to send messages","unsent messages":93}

{"level":"ERROR","ts":"2024-09-04T10:39:10.754+0200","logger":"service-metrics","msg":"failed to send messages","unsent messages":93}

Environment

- OS: Ubuntu 22.04 Server
- Version:

Anything else?

No response

fb929 commented 1 week ago

Hi! I would like to clarify if you used the EXTERNAL_LISTEN_HOSTS setting? (wiki)

bytersproblem commented 1 week ago

yes. Initially, I pointed it to the domain that provides IP address to my selfhosted setup. it is the form anysync.domain.com

fb929 commented 1 week ago

I don't see Mongo in the list of running containers. Could you please show the output of the following command?

cat .env.override
cat docker-compose.override.yml
docker-compose logs mongo-1

bytersproblem commented 1 week ago

CONTAINER ID   IMAGE                                              COMMAND                  CREATED       STATUS                        PORTS                                                                                                                                                                                 NAMES
9c01c9c85d14   ghcr.io/anyproto/any-sync-tools:latest             "tail -f /dev/null"      6 weeks ago   Up About a minute (healthy)   443/tcp, 8000/tcp, 8080/tcp                                                                                                                                                           any-sync-dockercompose-netcheck-1
564bd38c50f8   ghcr.io/anyproto/any-sync-node:v0.3.34             "/bin/sh -c '/bin/an…"   6 weeks ago   Up About a minute             443/tcp, 0.0.0.0:1003->1003/tcp, :::1003->1003/tcp, 0.0.0.0:1013->1013/udp, :::1013->1013/udp, 0.0.0.0:8003->8000/tcp, :::8003->8000/tcp, 0.0.0.0:8083->8080/tcp, :::8083->8080/tcp   any-sync-dockercompose-any-sync-node-3-1
6f35d795b085   ghcr.io/anyproto/any-sync-filenode:v0.6.5          "/bin/sh -c '/bin/an…"   6 weeks ago   Up About a minute             443/tcp, 8080/tcp, 0.0.0.0:1005->1005/tcp, :::1005->1005/tcp, 0.0.0.0:1015->1015/udp, :::1015->1015/udp, 0.0.0.0:8004->8000/tcp, :::8004->8000/tcp                                    any-sync-dockercompose-any-sync-filenode-1
e237e1962596   ghcr.io/anyproto/any-sync-node:v0.3.34             "/bin/sh -c '/bin/an…"   6 weeks ago   Up About a minute             443/tcp, 0.0.0.0:1001->1001/tcp, :::1001->1001/tcp, 0.0.0.0:1011->1011/udp, :::1011->1011/udp, 0.0.0.0:8001->8000/tcp, :::8001->8000/tcp, 0.0.0.0:8081->8080/tcp, :::8081->8080/tcp   any-sync-dockercompose-any-sync-node-1-1
1cf48b275ad3   ghcr.io/anyproto/any-sync-node:v0.3.34             "/bin/sh -c '/bin/an…"   6 weeks ago   Up About a minute             443/tcp, 0.0.0.0:1002->1002/tcp, :::1002->1002/tcp, 0.0.0.0:1012->1012/udp, :::1012->1012/udp, 0.0.0.0:8002->8000/tcp, :::8002->8000/tcp, 0.0.0.0:8082->8080/tcp, :::8082->8080/tcp   any-sync-dockercompose-any-sync-node-2-1
090e5da52b01   ghcr.io/anyproto/any-sync-consensusnode:v0.1.7     "/bin/sh -c '/bin/an…"   6 weeks ago   Up About a minute             443/tcp, 8080/tcp, 0.0.0.0:1006->1006/tcp, :::1006->1006/tcp, 0.0.0.0:1016->1016/udp, :::1016->1016/udp, 0.0.0.0:8006->8000/tcp, :::8006->8000/tcp                                    any-sync-dockercompose-any-sync-consensusnode-1
3ac26928affc   ghcr.io/anyproto/any-sync-coordinator:v0.3.25      "/bin/sh -c '/bin/an…"   6 weeks ago   Up About a minute             443/tcp, 8080/tcp, 0.0.0.0:1004->1004/tcp, :::1004->1004/tcp, 0.0.0.0:1014->1014/udp, :::1014->1014/udp, 0.0.0.0:8005->8000/tcp, :::8005->8000/tcp                                    any-sync-dockercompose-any-sync-coordinator-1
ead4cfdf14a9   redis/redis-stack-server:7.2.0-v6                  "redis-server --port…"   6 weeks ago   Up About a minute (healthy)   0.0.0.0:6379->6379/tcp, :::6379->6379/tcp                                                                                                                                             any-sync-dockercompose-redis-1
49a6256d3fd5   mongo:7.0.2                                        "docker-entrypoint.s…"   6 weeks ago   Up About a minute (healthy)   0.0.0.0:27001->27001/tcp, :::27001->27001/tcp, 27017/tcp                                                                                                                              any-sync-dockercompose-mongo-1-1
ae9040eafa25   any-sync-dockercompose-generateconfig-processing   "/code/docker-genera…"   6 weeks ago   Up About a minute (healthy)                                                                                                                                                                                         any-sync-dockercompose-generateconfig-processing-1
3fa9864b8fc4   minio/minio:RELEASE.2024-07-04T14-25-45Z           "/usr/bin/docker-ent…"   6 weeks ago   Up About a minute (healthy)   0.0.0.0:9000-9001->9000-9001/tcp, :::9000-9001->9000-9001/tcp                                                                                                                         any-sync-dockercompose-minio-1
31bf5325bf3b   any-sync-dockercompose-generateconfig-anyconf      "/code/docker-genera…"   6 weeks ago   Up About a minute (healthy)                                                                                                                                                                                         any-sync-dockercompose-generateconfig-anyconf-1

It seems that mongo in some time exited. Now I have all the listed containers runnind. the log file of the monogo container seems to bee to long, it just can't stop listing. For the env.override and docker-compose-override.yml, I do not see them in my cloned dir. The reason why I raised this issue is not that my setup is not working, I just can't connect from my work, where we have an academic network, and therefore a litlle restricted, but I don't see why the OpenvPN coonection to my home can't login to my vault.

bytersproblem commented 1 week ago

I brought up docker images with the bash script provided.

bytersproblem commented 1 week ago

OK. I found the EXTERNAL_LISTEN_HOSTS which was commented out in .env.common. I uncommented it and did make restart. I now have a new client.yml and will try again tommorow.

fb929 commented 1 week ago

Please use the .env.override file for configuration. It is in the .gitignore and does not interfere with repository updates.

Also You need to check that there is network access between the client and the server. For example, by using the telnet command, which should be run on the client:

telnet 192.168.100.199 1001

bytersproblem commented 1 week ago

I tried everything as you proposed. When I connect to telnet i get

telnet 192.168.100.199 1001
Trying 192.168.100.199...
Connected to 192.168.100.199.
Escape character is '^]'.
Connection closed by foreign host.

fb929 commented 1 week ago

It looks like your network access is blocked. Here’s what an allowed connection should look like:

telnet 127.0.0.1 1001
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
^]
telnet> Connection closed.

You can also check accessibility by running the [netcheck](https://github.com/anyproto/any-sync-tools/blob/main/any-sync-netcheck/README.md) utility (although it will only check the availability of coordinators). Example of running netcheck here

There could be several reasons:

Firewall restriction on the client
Firewall restriction on the self-hosted server
The configuration error is related to the EXTERNAL_LISTEN_HOSTS option (which we discussed earlier)
A ban on establishing connections between clients within the VPN network (For OpenVPN, this is the client-to-client option)

bytersproblem commented 1 week ago

Is it possible, by any means, to make subdomains on my home server and to proxy_pass all the required components to a domain that would be connected to https, or http port? I could do that but am not sure whether the udp connection is requrired for anytype functioning or is it possible to use tcp ports on the docker containers. Thanks!

bytersproblem commented 1 week ago

Something funny occured. I tried to create a new vault with the config that I have (the previous proposal didn't work). I set up the vault and I was "online" since I was able to share one space to others. I tried to login to that vault from my android, it worked. Then I tried to sync to my existing (main) vault again, no success. Isn't it the problem with my old vault where I stored for more than 2 months now, I don't know!

fb929 commented 1 week ago

Is it possible, by any means, to make subdomains on my home server and to proxy_pass all the required components to a domain that would be connected to https, or http port? I could do that but am not sure whether the udp connection is requrired for anytype functioning or is it possible to use tcp ports on the docker containers. Thanks!

It is impossible to use http_proxy because we are using the gRPC protocol, not HTTP. From similar solutions, you can use haproxy, nginx with the stream, or iptables port forwarding. However, using a VPN seems like a much simpler solution.

UDP is optional; it is enough to forward TCP ports. UDP is needed for faster implementation using the QUIC protocol.

fb929 commented 1 week ago

Something funny occured. I tried to create a new vault with the config that I have (the previous proposal didn't work). I set up the vault and I was "online" since I was able to share one space to others. I tried to login to that vault from my android, it worked. Then I tried to sync to my existing (main) vault again, no success. Isn't it the problem with my old vault where I stored for more than 2 months now, I don't know!

I haven't encountered any issues with the synchronization of specific accounts. I think you still have an issue with the availability of the self-hosted server

bytersproblem commented 5 days ago

I sshed to the virtual machine that hosts the anysync server. I tried to connect with telnet to port 1001. Telnet cannot connect (returns the same as from VPN) either to the localhost 1001 or 192.168.100.199 1001. It seems that the problem is in the vitual macnihe system itself. But I can ssh to that machine (via VPN), and sync works fine as explained earlier when I'm not at the office. I think that I will suffice to using anytype at home and non-restricted networks, although many odd things confuse me.

fb929 commented 5 days ago

You should absolutely be able to connect to the 127.0.0.1:1001 address if you're on the host where any-sync-dockercompose is running. If you can't establish a connection, it's likely that some settings have been applied to the docker-compose network, and I doubt that you're able to connect to the self-hosted server from anywhere at all.

bytersproblem commented 4 days ago

telnet 127.0.0.1 1001
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
Connection closed by foreign host.

After that I tried to connect from my wifi hotspot on the phone. It worked. It was the first time I saw my default vault on the work laptop.

bytersproblem commented 4 days ago

I wanted to give Envoy a shot. Before I begin, I wanted to ask does the anytype clients read nginx's location directives in the config file. And is the port (which will be 443) obligatory in configuration? Thanks!

fb929 commented 3 days ago

After that I tried to connect from my wifi hotspot on the phone. It worked. It was the first time I saw my default vault on the work laptop.

If your phone and laptop are on the same Wi-Fi network, they will work with each other without a server. Unfortunately, there is no direct way to check the connection status to the server on the client, but there is an indirect sign: you need to make some changes to the document (for example, upload an image) and check the sync status. If the sync is in an infinite progress state, then there is no connection to the server.

UPD: I misled you earlier; there is indeed a connection status to the server on the desktop client. You can see it in the screenshot above, represented by a grey smiley face. If the connection is established, the smiley will change to ":)" and turn green.

bytersproblem commented 3 days ago

So there is peer-to-peet interchage. I get it. I didn't have much time to test, but once, in the store, I opened android anytype to get my checklist. And it synced with my home network. This experiment with the mobile data turn off wireless, and uses the network of the mobile provider. So I was not on the same network with my phone, since it used its wireless adapter in AP mode. I will test more tomorrow and will try Envoy as I said in the previous comment. I only need to know whether the conig for selfhosted server may contain urls that are not only domains but also paths like anysync.domain.com/node-1. Thank you very much for your patience.

fb929 commented 3 days ago

I wanted to give Envoy a shot. Before I begin, I wanted to ask does the anytype clients read nginx's location directives in the config file. And is the port (which will be 443) obligatory in configuration? Thanks!

You can use ports at your discretion, as long as they match what's in your client.yml. An important point is that the network config for clients and servers is shared, meaning you can't just manually edit client.yml and provide it to the client. That means you need to achieve the desired configuration by changing the settings in the .env.override file.

bytersproblem commented 3 days ago

I think I get it. WIll try tommorow an for the weekend. Thanks again!

fb929 commented 3 days ago

I only need to know whether the conig for selfhosted server may contain urls that are not only domains but also paths like anysync.domain.com/node-1

No, as I mentioned earlier, the server operates using the gRPC protocol. In the gRPC protocol, the concept of "location" like /node-1 does not exist

A URL like anysync.domain.com/node-1 resembles the HTTP protocol, which we do not use.

anyproto / any-sync-dockercompose