Cant share space on selfhosted network

pmffromspace commented 1 month ago

Have you read a contributing guide?

[X] I have read CONTRIBUTING.md
[X] I have searched the existing issues and didn't find any that were similar
[X] I have considered creating a pull request with fixes instead of a bug report and want to proceed

Current Behavior

Today I tried to convince my colleagues to use anytype with our selfhosted network, but disappointingly it ended up being a disaster. (selfhosted network with official docker-compose git repo and this works fine per person)

BUT

When I tried to create a collaborative space on the desktop, I first ran into this problem So I installed the latest alpha, but that didn't help either. Ok, so it is not fixed yet. I get it.

BUT

I tried to share a space from my Android phone - nothing happened, no link generated. Then I tried sharing from an iOS device. This is the first time I got an error message:

Error on iOS when trying to create a shared space invitation link:

X Error: store invite in ipfs: add file to sync queue: check blocks availability: walk DAG: process batch: check availability: no connection to any file client, generate invite (unknownError)

Uploading, syncing files, videos, everything else works fine with my self-hosted network, I can just share space/collaborate. Is this kind of pointless to use any type for teams at all?

What am I doing wrong? How can I help solve this problem?

Expected Behavior

I should be able to share spaces in self hosted network

Steps To Reproduce

selfhost network create vault create space try to share space fail

Environment

- OS: macOS 14.5, Android, iOS, iPadOS
- Version:
macOS: 0.40.18-alpha

Anything else?

No response

fuksman commented 1 month ago

Hey @pmffromspace,

We rechecked each client app and middleware library, and self-hosted collaborations work fine. The only setup we've found out which returns the error message you quoted is if any-sync-filenode container is manually shut down or unavailable for the apps.

Please check that filenode is running.

pmffromspace commented 1 month ago

Thanks for this informative and helpful reply @fuksman !

I double checked, ports are opened (udp&tcp) everything else regarding sync between devices works great and fast.

On Mac trying to share returns make shareable: unable to connect, internal error

How could I verify the proper functioning of the filenode from gui? Maybe upload a video from one device, download on another? -> I just tried, works, but interestingly the video file is NOT listed in the space synced files.

How could I further debug?

The composer logs are plenty, can you recommend a value to grep like docker compose logs | grep share ?

I checked the node and the filenode is running with what seems like normal logs: (docker logs any-sync-dockercompose-any-sync-filenode-1)

2024-05-27T17:22:16.935Z    INFO    common.net.pool GC: removed 0; cache size: 0
2024-05-27T17:22:16.935Z    INFO    common.net.pool GC: removed 0; cache size: 2
2024-05-27T17:22:17.216Z    INFO    filenode.index  persist {"dur": "24.125033ms", "handled": 0, "deleted": 0, "missed": 0, "errors": 0, "moved": 0, "moved kbs": 0}
2024-05-27T17:22:21.510Z    INFO    filenode.deletionLog    processing deletion log {"records": 0, "handled": 0, "deleted": 0, "dur": "3.016987ms"}

fb929 commented 1 month ago

make shareable: unable to connect, internal error

This happens when the client is "offline," meaning it cannot reach the server. Is the macOS client running on the same computer where any-sync-dockercompose is running? Please show the output of docker ps and the contents of the .env.override and etc/client.yml files.

pmffromspace commented 1 month ago

The any-sync-dockercompose is running on a ubuntu server in the local network.

cat .env.override.*

ANY_SYNC_NODE_VERSION=latest
ANY_SYNC_FILENODE_VERSION=latest
ANY_SYNC_COORDINATOR_VERSION=latest
ANY_SYNC_CONSENSUSNODE_VERSION=latest
ANY_SYNC_NODE_VERSION=prod
ANY_SYNC_FILENODE_VERSION=prod
ANY_SYNC_COORDINATOR_VERSION=prod
ANY_SYNC_CONSENSUSNODE_VERSION=prod
ANY_SYNC_NODE_VERSION=stage1
ANY_SYNC_FILENODE_VERSION=stage1
ANY_SYNC_COORDINATOR_VERSION=stage1
ANY_SYNC_CONSENSUSNODE_VERSION=stage1

id: REMOVED
networkId: REMOVED
nodes:
  - addresses:
      - any-sync-node-1:1001
      - quic://any-sync-node-1:1011
      - 192.168.42.51:1001
      - quic://192.168.42.51:1011
    peerId: REMOVED
    types:
      - tree
  - addresses:
      - any-sync-node-2:1002
      - quic://any-sync-node-2:1012
      - 192.168.42.51:1002
      - quic://192.168.42.51:1012
    peerId: REMOVED
    types:
      - tree
  - addresses:
      - any-sync-node-3:1003
      - quic://any-sync-node-3:1013
      - 192.168.42.51:1003
      - quic://192.168.42.51:1013
    peerId: REMOVED
    types:
      - tree
  - addresses:
      - any-sync-coordinator:1004
      - quic://any-sync-coordinator:1014
      - 192.168.42.51:1004
      - quic://192.168.42.51:1014
    peerId: REMOVED
    types:
      - coordinator
  - addresses:
      - any-sync-filenode:1005
      - quic://any-sync-filenode:1015
      - 192.168.42.51:1005
      - quic://192.168.42.51:1015
    peerId: REMOVED
    types:
      - file
  - addresses:
      - any-sync-consensusnode:1006
      - quic://any-sync-consensusnode:1016
      - 192.168.42.51:1006
      - quic://192.168.42.51:1016
    peerId: REMOVED
    types:
      - consensus

And I added firewall exceptions in ufw with

sudo ufw allow 1001 comment anysync
sudo ufw allow 1002 comment anysync
.
.
.

cat .env.common

EXTERNAL_LISTEN_HOSTS="192.168.42.51"

ANY_SYNC_NODE_1_HOST=any-sync-node-1
ANY_SYNC_NODE_1_PORT=1001
ANY_SYNC_NODE_1_ADDRESSES=${ANY_SYNC_NODE_1_HOST}:${ANY_SYNC_NODE_1_PORT}
ANY_SYNC_NODE_1_QUIC_PORT=1011
ANY_SYNC_NODE_1_QUIC_ADDRESSES=${ANY_SYNC_NODE_1_HOST}:${ANY_SYNC_NODE_1_QUIC_PORT}

ANY_SYNC_NODE_2_HOST=any-sync-node-2
ANY_SYNC_NODE_2_PORT=1002
ANY_SYNC_NODE_2_ADDRESSES=${ANY_SYNC_NODE_2_HOST}:${ANY_SYNC_NODE_2_PORT}
ANY_SYNC_NODE_2_QUIC_PORT=1012
ANY_SYNC_NODE_2_QUIC_ADDRESSES=${ANY_SYNC_NODE_2_HOST}:${ANY_SYNC_NODE_2_QUIC_PORT}

ANY_SYNC_NODE_3_HOST=any-sync-node-3
ANY_SYNC_NODE_3_PORT=1003
ANY_SYNC_NODE_3_ADDRESSES=${ANY_SYNC_NODE_3_HOST}:${ANY_SYNC_NODE_3_PORT}
ANY_SYNC_NODE_3_QUIC_PORT=1013
ANY_SYNC_NODE_3_QUIC_ADDRESSES=${ANY_SYNC_NODE_3_HOST}:${ANY_SYNC_NODE_3_QUIC_PORT}

ANY_SYNC_COORDINATOR_HOST=any-sync-coordinator
ANY_SYNC_COORDINATOR_PORT=1004
ANY_SYNC_COORDINATOR_ADDRESSES=${ANY_SYNC_COORDINATOR_HOST}:${ANY_SYNC_COORDINATOR_PORT}
ANY_SYNC_COORDINATOR_QUIC_PORT=1014
ANY_SYNC_COORDINATOR_QUIC_ADDRESSES=${ANY_SYNC_COORDINATOR_HOST}:${ANY_SYNC_COORDINATOR_QUIC_PORT}
ANY_SYNC_COORDINATOR_DEFAULT_LIMITS_SPACE_MEMBERS_READ=1000
ANY_SYNC_COORDINATOR_DEFAULT_LIMITS_SPACE_MEMBERS_WRITE=1000
ANY_SYNC_COORDINATOR_DEFAULT_LIMITS_SHARED_SPACES_LIMIT=1000

ANY_SYNC_FILENODE_HOST=any-sync-filenode
ANY_SYNC_FILENODE_PORT=1005
ANY_SYNC_FILENODE_ADDRESSES=${ANY_SYNC_FILENODE_HOST}:${ANY_SYNC_FILENODE_PORT}
ANY_SYNC_FILENODE_QUIC_PORT=1015
ANY_SYNC_FILENODE_QUIC_ADDRESSES=${ANY_SYNC_FILENODE_HOST}:${ANY_SYNC_FILENODE_QUIC_PORT}
ANY_SYNC_FILENODE_DEFAULT_LIMIT=1099511627776

ANY_SYNC_CONSENSUSNODE_HOST=any-sync-consensusnode
ANY_SYNC_CONSENSUSNODE_PORT=1006
ANY_SYNC_CONSENSUSNODE_ADDRESSES=${ANY_SYNC_CONSENSUSNODE_HOST}:${ANY_SYNC_CONSENSUSNODE_PORT}
ANY_SYNC_CONSENSUSNODE_QUIC_PORT=1016
ANY_SYNC_CONSENSUSNODE_QUIC_ADDRESSES=${ANY_SYNC_CONSENSUSNODE_HOST}:${ANY_SYNC_CONSENSUSNODE_QUIC_PORT}

MONGO_1_PORT=27001
MONGO_CONNECT=mongodb://mongo-1:${MONGO_1_PORT}
MONGO_REPLICA_SET=rs0
MONGO_URL=${MONGO_CONNECT}/?replicaSet=${MONGO_REPLICA_SET}

REDIS_HOST=redis
REDIS_PORT=6379
REDIS_URL="redis://${REDIS_HOST}:${REDIS_PORT}?dial_timeout=3&read_timeout=6s"

ANY_SYNC_ADMIN_HOST=0.0.0.0
ANY_SYNC_ADMIN_PORT=5000
EXTERNAL_ANY_SYNC_ADMIN_PORT=4040

pmffromspace commented 1 month ago

So I got the supposition that the problem lies somewhere with the EXTERNAL_LISTEN_HOSTS="192.168.42.51" The setup is a server in a subnet with portforwarding (NAT like, its because off DMZ / Network seperation) I can vpn to the server so Im "in the 192.168.42.* subnet" and then generating a sharing link worked, once.

I am now trying to figure out why its not repeatable.

As you can see, I configured multiple addresses in the client config .yml, plan is to sync in LAN, over VPN and via nebula mesh vpn network.

So here is where it gets interesting, it seems like the EXTERNAL_LISTEN_HOST is somehow the prefered ip address for the sharing process - can someone more familiar with the protocoll confirm?

fb929 commented 1 month ago

You need to ensure that the addresses from EXTERNAL_LISTEN_HOSTS are indeed present on your "docker host" and that the firewall/NAT/routing rules allow docker to forward traffic into the docker network without any issues.
So here is where it gets interesting, it seems like the EXTERNAL_LISTEN_HOST is somehow the prefered ip address for the sharing process - can someone more familiar with the protocoll confirm?

For the client, all addresses are equivalent. But internal addresses like "any-sync-consensusnode" are not accessible to the client because in 99% of cases, clients do not have access to the internal docker network. These internal addresses are present in the configuration so that the any-sync-* daemons can establish connections with each other within the Docker network.

fb929 commented 1 month ago

I just tried, works, but interestingly the video file is NOT listed in the space synced files.

If the file does NOT appear in the "space files", it means it HASN'T been synchronized with the server. It appears on the second client because the clients synchronized it directly through P2P.

pmffromspace commented 1 month ago

I tried alot of thinks, checked twice if ports are really reachable (using telnet and netcat) and the results of my investigation is following:

sharing spaces works on selfhosted networks if the server is in the same subnet / lan (closing this issue)
sharing spaces works via vpn (wireguard, docker)
sharing spaces does not work, when the anysync server is behind a firewall, even with port forwarding and both adresses configured
sharing spaces does not work via nebula mesh vpn

->This is probably a networking issue in my setup so I will close this issue now. Im not experienced enough to debug the NAT network setup stuff and frankly spend enough time figuring this out.

I want to thank @fuksman and @fb929 for their quick and very friendly and helpful support! I will stick to the project and spread the word about it, I like it alot. Thanks guys.

anyproto / any-sync-dockercompose