search

vrtmrz / livesync-bridge

44 stars 6 forks source link

Repeatedly seeing `WATCH: CONNECTION HAS BEEN CLOSED, RECONNECTING...` #1

Closed Teajey closed 11 months ago

Teajey commented 11 months ago

Hey, thanks for this, it's really hard to find a solution to syncing my obsidian vault with my blog, so I appreciate you sharing this code.

When I run this on my server all I see is this; any idea what I'm doing wrong?

...
meout=100000&heartbeat=5000
4/12/2023, 11:09:30 am  10  WATCH: CONNECTION HAS BEEN CLOSED, RECONNECTING...
4/12/2023, 11:09:40 am  10  WATCH: START: (since:now)
4/12/2023, 11:09:40 am  -1  Requesting ... get http://127.0.0.1:5984/brightscroll_posts/_changes?style=all_docs&filter=replicate%2Fpull&include_docs=true&since=now&feed=continuous&timeout=100000&heartbeat=5000
4/12/2023, 11:09:40 am  10  WATCH: CONNECTION HAS BEEN CLOSED, RECONNECTING...
4/12/2023, 11:09:50 am  10  WATCH: START: (since:now)
4/12/2023, 11:09:50 am  -1  Requesting ... get http://127.0.0.1:5984/brightscroll_posts/_changes?style=all_docs&filter=replicate%2Fpull&include_docs=true&since=now&feed=continuous&timeout=100000&heartbeat=5000
4/12/2023, 11:09:50 am  10  WATCH: CONNECTION HAS BEEN CLOSED, RECONNECTING...
4/12/2023, 11:10:00 am  10  WATCH: START: (since:now)
4/12/2023, 11:10:00 am  -1  Requesting ... get http://127.0.0.1:5984/brightscroll_posts/_changes?style=all_docs&filter=replicate%2Fpull&include_docs=true&since=now&feed=continuous&timeout=100000&heartbeat=5000
4/12/2023, 11:10:00 am  10  WATCH: CONNECTION HAS BEEN CLOSED, RECONNECTING...

Here's my config:

{
    "peers": [
        {
            "type": "couchdb",
            "name": "brightscroll",
            "database": "brightscroll_posts",
            "username": "brightscroll",
            "password": <omitted>,
            "url": "http://127.0.0.1:5984",
            "customChunkSize": 100,
            "minimumChunkSize": 20,
            "baseDir": "/home/thomas/posts"
        }
    ]
}
vrtmrz commented 11 months ago

Thank you for using this and the first question! It looks bridge could not connected to the remote database. 127.0.0.1 appears to be a local loopback address, which is what you intended? Does your CouchDB is on the same machine?

If this was intended, then we can check the log of CouchDB in next. If the authentication has been failed, CouchDB would have logged it.

Teajey commented 11 months ago

Yes, it's on the same machine. I should probably also add the detail that I'm using Docker, similar to the suggestion here: https://github.com/vrtmrz/obsidian-livesync/blob/main/docs/setup_own_server.md#docker-compose

I haven't got logging set up, but I see a lot of this in the container's STDOUT:

obsidian-livesync  | [notice] 2023-12-04T21:56:22.726746Z nonode@nohost <0.32740.33> fa3d691bb8 127.0.0.1:5984 172.22.0.1 undefined GET /brightscroll_posts/_changes?style=all_docs&filter=replicate%2Fpull&include_docs=true&since=now&feed=continuous&timeout=100000&heartbeat=5000 401 ok 1
obsidian-livesync  | [notice] 2023-12-04T21:56:27.497762Z nonode@nohost <0.354.0> -------- chttpd_auth_cache changes listener died because the _users database does not exist. Create the database to silence this notice.
obsidian-livesync  | [error] 2023-12-04T21:56:27.498209Z nonode@nohost emulator -------- Error in process <0.274.34> with exit value:
obsidian-livesync  | {database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,430}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,405}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,434}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,100}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,39}]},{chttpd_auth_cache,ensure_auth_ddoc_exists,2,[{file,"src/chttpd_auth_cache.erl"},{line,214}]},{chttpd_auth_cache,listen_for_changes,1,[{file,"src/chttpd_auth_cache.erl"},{line,160}]}]}

Notably

So perhaps there's a bit of setup for CouchDB that's missing?

vrtmrz commented 11 months ago

As you mentioned, possibly this is the reason of failure of authentication.

This database could be created by Single node setup from the UI, or following REST API. Would you please try this one?

couchHost, couchUser and couchPwd are http://127.0.0.1:5984, brightscroll, and the password you decided on.

$ curl -X POST "${couchHost}/_cluster_setup" -H "Content-Type: application/json" -d "{\"action\":\"enable_single_node\",\"username\":\"${couchUser}\",\"password\":\"${couchPwd}\",\"bind_address\":\"0.0.0.0\",\"port\":5984,\"singlenode\":true}"  --user "${couchUser}:${couchPwd}"
Teajey commented 11 months ago

Okay, I've done that, but I started seeing this 404 response

obsidian-livesync  | [notice] 2023-12-05T22:52:51.362279Z nonode@nohost <0.1675.0> cd6ba5c8d7 127.0.0.1:5984 172.22.0.1 brightscroll GET /brightscroll_posts/_changes?style=all_docs&filter=replicate%2Fpull&include_docs=true&since=now&feed=continuous&timeout=100000&heartbeat=5000 404 ok 2

So I sent a PUT as per https://docs.couchdb.org/en/stable/intro/tour.html But I'm still seeing 404s

obsidian-livesync  | [notice] 2023-12-05T22:55:38.288300Z nonode@nohost <0.3448.0> 0d3b71ca3a 127.0.0.1:5984 172.22.0.1 brightscroll PUT /brightscroll_posts 201 ok 34
obsidian-livesync  | [notice] 2023-12-05T22:56:04.270583Z nonode@nohost <0.3693.0> 1519d2b273 127.0.0.1:5984 172.22.0.1 brightscroll GET /brightscroll_posts/_local%2Fobsydian_livesync_milestone 404 ok 2
Teajey commented 11 months ago

These are the results if I manually run these requests with cURL:

GET /brightscroll_posts/_changes?style=all_docs&filter=replicate%2Fpull&include_docs=true&since=now&feed=continuous&timeout=100000&heartbeat=5000

results in

{"error":"not_found","reason":"Database does not exist."}

However

GET /brightscroll_posts

results in what looks like a successful response

{"instance_start_time":"1701816938","db_name":"brightscroll_posts","purge_seq":"0-g1AAAABPeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCeexAEmGBiD1HwiyEhlwqEtkSKqHKMgCAIT2GV4","update_seq":"0-g1AAAACLeJzLYWBgYMpgTmHgzcvPy09JdcjLz8gvLskBCeexAEmGBiD1HwiyMpgTGXKBAuxmyYYWJomW6HpwmJLIkFSPot3I3MLCwBxDexYA89sqAw","sizes":{"file":16692,"external":0,"active":0},"props":{},"doc_del_count":0,"doc_count":0,"disk_format_version":8,"compact_running":false,"cluster":{"q":2,"n":1,"w":1,"r":1}}
GET /_local%2Fobsydian_livesync_milestone

Also results in

{"error":"not_found","reason":"Database does not exist."}

(I'm noticing that I can't find /_local in the CouchDB docs, so I wonder if I'm using the wrong version? Also, I wonder if %2F and "obsydian" are mistakes in livesync-bridge?)

The database list looks ok

GET /_all_dbs
["_replicator","_users","brightscroll_posts"]
vrtmrz commented 11 months ago

Thank you for the details! Have you synchronised from Self-hosted LiveSync to a remote database at least once? Some design documents will be made then, but they seem not in your database. (A little oddly, the reason for the error should simply be missing). And possibly the Check database configuration and Fix buttons on the Self-hosted LiveSync might help. If you want to create manually, the following document should be put on your database.

{
  "_id": "_design/replicate",
  "ver": 2,
  "filters": {
    "default": "function (s, r) {\n        return !(\"remote\" in s && s.remote);\n      }",
    "push": "function (s, r) {\n        return true;\n      }",
    "pull": "function (s, r) {\n        return !(s.type && \"leaf\" == s.type);\n      }"
  }
}

_local is under the database path(Local documents). It would be brightscroll_posts/_local. And, indeed misspelt since first and kept for compatibility.

Teajey commented 11 months ago

I have a directory full of markdown files on my server that I want to synchronized to the database, and then I'll download them into Obsidian, instead of the other direction

Teajey commented 11 months ago

livesync-bridge is running happily now! I'm at the point of setting up Obsidian LiveSync on my phone, but unfortunately it's failing to contact the server when I test or check the database connection. It should work because I can reach the couchdb server via my webbrowser.

There's a toast popup that tells me to check the inspector, but I'm not sure where to find that I've found it, but I'm not seeing any network requests in the network tab when I press 'Test'

Edit 1: Okay, I got it to connect! But the files aren't being replicated starting from livesync-bridge to Obsidian, and I'm guessing it's just not designed to work like that?

Edit 2: I tried copying the markdown files from the server directly into my vault with scp. obsidian-livesync picked up on the files, and seems to be sending them back to the server, but I don't see any new documents appearing in CouchDB.

Edit 3: Nevermind, I've found one local non-replicating document:

{
  "_id": "_local/z_w0zD0Nqa1XuFzLrpnzyg==",
  "_rev": "0-2",
  "session_id": "9089d3c0-0e0c-44d3-b6b5-7e8d376fddf6",
  "history": [
    {
      "last_seq": 3,
      "session_id": "9089d3c0-0e0c-44d3-b6b5-7e8d376fddf6"
    },
    {
      "last_seq": 1,
      "session_id": "a03624a1-2389-4192-9330-2a7573ea64dd"
    }
  ],
  "replicator": "pouchdb",
  "version": 1,
  "last_seq": 3
}
Teajey commented 11 months ago

I think I'm very close to getting this working. All I'm noticing now is that for every file obsidian-livesync attempts to upload livesync-bridge logs WATCH: SKIP <some file name>.md: OUT OF TARGET FOLDER.

Since my livesync-bridge/dat/config.json's baseDir is set to an absolute path, I'm not sure what to make of it. Even if I point it to an empty directory I get the same message

haidahaha commented 11 months ago

@Teajey out of curiosity, do you have end-to-end encryption enabled? I can never make the shared livesync work with the encryption.

Teajey commented 11 months ago

No... For now I'm just trying to get it to work without that.

I'm still struggling to understand what this "OUT OF TARGET FOLDER" error message means

Teajey commented 11 months ago

Hi @vrtmrz. Could you please explain what "OUT OF TARGET FOLDER" means? I think it's the one problem standing between me and getting my system working

vrtmrz commented 11 months ago

Sorry for being away for a while! As @Teajey mentioned, baseDir is the reason. I missed this This has a different meaning for each type of Peer. In the case of couchdb, it is the path prefix which is synchronised in the Obsidian. (Yes, target folder).

If we want to synchronise the whole thing, we can leave it empty. Sorry for missing this, even though you had given me this information from the beginning.

Could you please try with an empty string?

Teajey commented 11 months ago

My config now looks like this but I'm still seeing the same error.

{
    "peers": [
         {
            "type": "couchdb",
            "name": "brightscroll",
            "database": "brightscroll_posts",
            "username": "brightscroll",
            "password": <omitted>,
            "url": "http://127.0.0.1:5984",
            "customChunkSize": 100,
            "minimumChunkSize": 20,
            "baseDir": ""
        }
    ]
}

This time when I launched livesync-bridge I saw the message display for every file again, as if I was starting it for the first time. If I edit a file in Obsidian and press 'Replicate,' the "OUT OF TARGET FOLDER" message prints for that file I edited.

Am I correct in understanding that baseDir is relative to the Obsidian vault's directory? So it's for choosing if you want only a certain subpath of the vault to be synchronised? I'm confused about how livesync-bridge knows which filesystem directory to replicate files to and from in that case.

I'm starting to wonder if I'm misunderstanding the use-case for livesync-bridge, as I don't want to synchronise part of an obsidian vault into another obsidian vault; I just want to replicate my Obsidian vault in a file directory on my server.

Or perhaps I just need to also add a "type": "storage" peer to my config, that points to the directory that I want to populate?

vrtmrz commented 11 months ago

Oh, I apologise to you that I have told you the erroneous explanation... Any files could not be accepted with the empty path of couchdb. It has been fixed at the latest commit.

Or perhaps I just need to also add a "type": "storage" peer to my config, that points to the directory that I want to populate?

I am really sorry also for this, I have completely misunderstood that you have just snipped the configuration. Yes, you should add a storage-type peer! LiveSync-Bridge would transfer all one's updates to others among the peers.

Teajey commented 11 months ago

After pulling the latest changes and adding the storage peer everything appears to be working as I'd hoped! Thank you so much for your help @vrtmrz!!