Closed flancian closed 3 months ago
That's not good. Can you see anything in the server logs? That would be more helpful in solving this.
Thank you for your response and for working on Silverbullet! It is amazing :) These are the docker logs:
flancian@paramita:~/bin$ docker run --restart unless-stopped --name silverbullet -p 5019:3000 -v /home/flancian/garden:/space zefhemel/silverbullet:latest
Will run SilverBullet with UID 1000, inferred from the owner of /space (set PUID environment variable to override)
Running SilverBullet as silverbullet (configured as PUID 1000 and PGID 1000)
Using KV as a database backend (/space/.silverbullet.db).
Going to start SilverBullet binding to 0.0.0.0:3000
Listening on http://localhost:3000/
SilverBullet is now running: http://localhost:3000
Using local disk as a storage backend: /space
Activated plug tasks
Activated plug editor
Activated plug query
Activated plug sync
Activated plug share
Activated plug template
Activated plug emoji
Activated plug federation
Activated plug search
Activated plug index
Activated plug markdown
Activated plug plug-manager
Booting up worker for index
Loaded 0 functions and 0 commands from space-script
Booting up worker for federation
Current space index version 5
Loading builtins attributes into index
Booted server with hostname 0.0.0.0
Requested file SETTINGS.md
alreadyFetching is on, skipping even triggering for fetchFileList.
TypeError: value too large (max 65536 bytes)
at doAtomicWriteInPlace (ext:deno_kv/01_db.ts:628:16)
at AtomicOperation.commit (ext:deno_kv/01_db.ts:407:32)
at zV.batchSet (file:///silverbullet.js:2376:13847842)
at Q0.batchSet (file:///silverbullet.js:2375:10458)
at Oc.batchSet (file:///silverbullet.js:73:11329)
at Oc.set (file:///silverbullet.js:73:11061)
at file:///silverbullet.js:2366:191876
at file:///silverbullet.js:73:4963
at CL.dispatchEvent (file:///silverbullet.js:73:4986)
at OX.dispatchEvent (file:///silverbullet.js:74:2280)
TypeError: value too large (max 65536 bytes)
at doAtomicWriteInPlace (ext:deno_kv/01_db.ts:628:16)
at AtomicOperation.commit (ext:deno_kv/01_db.ts:407:32)
at zV.batchSet (file:///silverbullet.js:2376:13847842)
at Q0.batchSet (file:///silverbullet.js:2375:10458)
at Oc.batchSet (file:///silverbullet.js:73:11329)
at Oc.set (file:///silverbullet.js:73:11061)
at file:///silverbullet.js:2366:191876
at file:///silverbullet.js:73:4963
at CL.dispatchEvent (file:///silverbullet.js:73:4986)
at OX.dispatchEvent (file:///silverbullet.js:74:2280)
Requested file prime.md
alreadyFetching is on, skipping even triggering for fetchFileList.
Requested file index.md
Requested file 2024-03-09.md
Requested file daily.md
Requested file Flancia pattern language.md
Requested file meditate.md
alreadyFetching is on, skipping even triggering for fetchFileList.
TypeError: value too large (max 65536 bytes)
at doAtomicWriteInPlace (ext:deno_kv/01_db.ts:628:16)
at AtomicOperation.commit (ext:deno_kv/01_db.ts:407:32)
at zV.batchSet (file:///silverbullet.js:2376:13847842)
at Q0.batchSet (file:///silverbullet.js:2375:10458)
at Oc.batchSet (file:///silverbullet.js:73:11329)
at Oc.set (file:///silverbullet.js:73:11061)
at file:///silverbullet.js:2366:191876
at file:///silverbullet.js:73:4963
at CL.dispatchEvent (file:///silverbullet.js:73:4986)
at OX.dispatchEvent (file:///silverbullet.js:74:2280)
Requested file Eugnosia.md
TypeError: value too large (max 65536 bytes)
at doAtomicWriteInPlace (ext:deno_kv/01_db.ts:628:16)
at AtomicOperation.commit (ext:deno_kv/01_db.ts:407:32)
at zV.batchSet (file:///silverbullet.js:2376:13847842)
at Q0.batchSet (file:///silverbullet.js:2375:10458)
at Oc.batchSet (file:///silverbullet.js:73:11329)
at Oc.set (file:///silverbullet.js:73:11061)
at file:///silverbullet.js:2366:191876
at file:///silverbullet.js:73:4963
at CL.dispatchEvent (file:///silverbullet.js:73:4986)
at OX.dispatchEvent (file:///silverbullet.js:74:2280)
TypeError: value too large (max 65536 bytes)
at doAtomicWriteInPlace (ext:deno_kv/01_db.ts:628:16)
at AtomicOperation.commit (ext:deno_kv/01_db.ts:407:32)
at zV.batchSet (file:///silverbullet.js:2376:13847842)
at Q0.batchSet (file:///silverbullet.js:2375:10458)
at Oc.batchSet (file:///silverbullet.js:73:11329)
at Oc.set (file:///silverbullet.js:73:11061)
at file:///silverbullet.js:2366:191876
at file:///silverbullet.js:73:4963
at CL.dispatchEvent (file:///silverbullet.js:73:4986)
at OX.dispatchEvent (file:///silverbullet.js:74:2280)
TypeError: value too large (max 65536 bytes)
at doAtomicWriteInPlace (ext:deno_kv/01_db.ts:628:16)
at AtomicOperation.commit (ext:deno_kv/01_db.ts:407:32)
at zV.batchSet (file:///silverbullet.js:2376:13847842)
at Q0.batchSet (file:///silverbullet.js:2375:10458)
at Oc.batchSet (file:///silverbullet.js:73:11329)
at Oc.set (file:///silverbullet.js:73:11061)
at file:///silverbullet.js:2366:191876
at file:///silverbullet.js:73:4963
at CL.dispatchEvent (file:///silverbullet.js:73:4986)
at OX.dispatchEvent (file:///silverbullet.js:74:2280)
TypeError: value too large (max 65536 bytes)
at doAtomicWriteInPlace (ext:deno_kv/01_db.ts:628:16)
at AtomicOperation.commit (ext:deno_kv/01_db.ts:407:32)
at zV.batchSet (file:///silverbullet.js:2376:13847842)
at Q0.batchSet (file:///silverbullet.js:2375:10458)
at Oc.batchSet (file:///silverbullet.js:73:11329)
at Oc.set (file:///silverbullet.js:73:11061)
at file:///silverbullet.js:2366:191876
at file:///silverbullet.js:73:4963
at CL.dispatchEvent (file:///silverbullet.js:73:4986)
at OX.dispatchEvent (file:///silverbullet.js:74:2280)
TypeError: value too large (max 65536 bytes)
at doAtomicWriteInPlace (ext:deno_kv/01_db.ts:628:16)
at AtomicOperation.commit (ext:deno_kv/01_db.ts:407:32)
at zV.batchSet (file:///silverbullet.js:2376:13847842)
at Q0.batchSet (file:///silverbullet.js:2375:10458)
at Oc.batchSet (file:///silverbullet.js:73:11329)
at Oc.set (file:///silverbullet.js:73:11061)
at file:///silverbullet.js:2366:191876
at file:///silverbullet.js:73:4963
at CL.dispatchEvent (file:///silverbullet.js:73:4986)
at OX.dispatchEvent (file:///silverbullet.js:74:2280)
TypeError: value too large (max 65536 bytes)
at doAtomicWriteInPlace (ext:deno_kv/01_db.ts:628:16)
at AtomicOperation.commit (ext:deno_kv/01_db.ts:407:32)
at zV.batchSet (file:///silverbullet.js:2376:13847842)
at Q0.batchSet (file:///silverbullet.js:2375:10458)
at Oc.batchSet (file:///silverbullet.js:73:11329)
at Oc.set (file:///silverbullet.js:73:11061)
at file:///silverbullet.js:2366:191876
at file:///silverbullet.js:73:4963
at CL.dispatchEvent (file:///silverbullet.js:73:4986)
at OX.dispatchEvent (file:///silverbullet.js:74:2280)
TypeError: value too large (max 65536 bytes)
at doAtomicWriteInPlace (ext:deno_kv/01_db.ts:628:16)
at AtomicOperation.commit (ext:deno_kv/01_db.ts:407:32)
at zV.batchSet (file:///silverbullet.js:2376:13847842)
at Q0.batchSet (file:///silverbullet.js:2375:10458)
at Oc.batchSet (file:///silverbullet.js:73:11329)
at Oc.set (file:///silverbullet.js:73:11061)
at file:///silverbullet.js:2366:191876
at file:///silverbullet.js:73:4963
at CL.dispatchEvent (file:///silverbullet.js:73:4986)
at OX.dispatchEvent (file:///silverbullet.js:74:2280)
TypeError: value too large (max 65536 bytes)
at doAtomicWriteInPlace (ext:deno_kv/01_db.ts:628:16)
at AtomicOperation.commit (ext:deno_kv/01_db.ts:407:32)
at zV.batchSet (file:///silverbullet.js:2376:13847842)
at Q0.batchSet (file:///silverbullet.js:2375:10458)
at Oc.batchSet (file:///silverbullet.js:73:11329)
at Oc.set (file:///silverbullet.js:73:11061)
at file:///silverbullet.js:2366:191876
at file:///silverbullet.js:73:4963
at CL.dispatchEvent (file:///silverbullet.js:73:4986)
at OX.dispatchEvent (file:///silverbullet.js:74:2280)
The developer console:
Hope this helps, please let me know if I should run in some other mode/get logs some other way! Thanks again.
I tried taking a look at silverbullet.js lines 2366 and 73 but I'm unsure how to map that back to the .ts yet. Not sure which max size check is being triggered.
Edit: I cloned the repo and built using deno, will try to reproduce from head against my garden.
I was able to reproduce on head against my garden, these are the server logs there.
flancian@paramita:~/silverbullet$ deno task watch-server ~/garden/
Task watch-server deno run -A --unstable-kv --unstable-worker-options --check --watch silverbullet.ts "/home/flancian/garden/"
Watcher Process started.
Using KV as a database backend (/home/flancian/garden/.silverbullet.db).
Going to start SilverBullet binding to 127.0.0.1:3000
SilverBullet will only be available locally, to allow outside connections, pass -L0.0.0.0 as a flag, and put a TLS terminator on top.
Listening on http://127.0.0.1:3000/
SilverBullet is now running: http://127.0.0.1:3000
Using local disk as a storage backend: /home/flancian/garden
Booting up worker for tasks
Activated plug tasks
Booting up worker for editor
Activated plug editor
Booting up worker for federation
Activated plug federation
Booting up worker for share
Activated plug share
Booting up worker for search
Activated plug search
Booting up worker for query
Activated plug query
Booting up worker for template
Activated plug template
Booting up worker for emoji
Activated plug emoji
Booting up worker for markdown
Activated plug markdown
Booting up worker for index
Activated plug index
Booting up worker for sync
Activated plug sync
Booting up worker for plug-manager
Activated plug plug-manager
Loaded 0 functions and 0 commands from space-script
Current space index version 5
Loading builtins attributes into index
Booted server with hostname 127.0.0.1
Requested file index.md
TypeError: value too large (max 65536 bytes)
at doAtomicWriteInPlace (ext:deno_kv/01_db.ts:628:16)
at AtomicOperation.commit (ext:deno_kv/01_db.ts:407:32)
at DenoKvPrimitives.batchSet (file:///home/flancian/silverbullet/lib/data/deno_kv_primitives.ts:35:33)
at PrefixedKvPrimitives.batchSet (file:///home/flancian/silverbullet/lib/data/prefixed_kv_primitives.ts:16:25)
at DataStore.batchSet (file:///home/flancian/silverbullet/lib/data/datastore.ts:47:20)
at DataStore.set (file:///home/flancian/silverbullet/lib/data/datastore.ts:29:17)
at file:///home/flancian/silverbullet/server/server_system.ts:121:24
at file:///home/flancian/silverbullet/common/hooks/event.ts:91:48
at EventHook.dispatchEvent (file:///home/flancian/silverbullet/common/hooks/event.ts:95:10)
at EventedSpacePrimitives.dispatchEvent (file:///home/flancian/silverbullet/common/spaces/evented_space_primitives.ts:47:27)
TypeError: value too large (max 65536 bytes)
at doAtomicWriteInPlace (ext:deno_kv/01_db.ts:628:16)
at AtomicOperation.commit (ext:deno_kv/01_db.ts:407:32)
at DenoKvPrimitives.batchSet (file:///home/flancian/silverbullet/lib/data/deno_kv_primitives.ts:35:33)
at PrefixedKvPrimitives.batchSet (file:///home/flancian/silverbullet/lib/data/prefixed_kv_primitives.ts:16:25)
at DataStore.batchSet (file:///home/flancian/silverbullet/lib/data/datastore.ts:47:20)
at DataStore.set (file:///home/flancian/silverbullet/lib/data/datastore.ts:29:17)
at file:///home/flancian/silverbullet/server/server_system.ts:121:24
at file:///home/flancian/silverbullet/common/hooks/event.ts:91:48
at EventHook.dispatchEvent (file:///home/flancian/silverbullet/common/hooks/event.ts:95:10)
at EventedSpacePrimitives.dispatchEvent (file:///home/flancian/silverbullet/common/spaces/evented_space_primitives.ts:47:27)
Requested file index.md
Requested file index.md
Requested file Library/Core/Widget/Table of Contents.md
Requested file index.md
Requested file index.md
TypeError: value too large (max 65536 bytes)
at doAtomicWriteInPlace (ext:deno_kv/01_db.ts:628:16)
at AtomicOperation.commit (ext:deno_kv/01_db.ts:407:32)
at DenoKvPrimitives.batchSet (file:///home/flancian/silverbullet/lib/data/deno_kv_primitives.ts:35:33)
at PrefixedKvPrimitives.batchSet (file:///home/flancian/silverbullet/lib/data/prefixed_kv_primitives.ts:16:25)
at DataStore.batchSet (file:///home/flancian/silverbullet/lib/data/datastore.ts:47:20)
at DataStore.set (file:///home/flancian/silverbullet/lib/data/datastore.ts:29:17)
at file:///home/flancian/silverbullet/server/server_system.ts:121:24
at file:///home/flancian/silverbullet/common/hooks/event.ts:91:48
at EventHook.dispatchEvent (file:///home/flancian/silverbullet/common/hooks/event.ts:95:10)
at EventedSpacePrimitives.dispatchEvent (file:///home/flancian/silverbullet/common/spaces/evented_space_primitives.ts:47:27)
Requested file index.md
Requested file index.md
Requested file index.md
TypeError: value too large (max 65536 bytes)
at doAtomicWriteInPlace (ext:deno_kv/01_db.ts:628:16)
at AtomicOperation.commit (ext:deno_kv/01_db.ts:407:32)
at DenoKvPrimitives.batchSet (file:///home/flancian/silverbullet/lib/data/deno_kv_primitives.ts:35:33)
at PrefixedKvPrimitives.batchSet (file:///home/flancian/silverbullet/lib/data/prefixed_kv_primitives.ts:16:25)
at DataStore.batchSet (file:///home/flancian/silverbullet/lib/data/datastore.ts:47:20)
at DataStore.set (file:///home/flancian/silverbullet/lib/data/datastore.ts:29:17)
at file:///home/flancian/silverbullet/server/server_system.ts:121:24
at file:///home/flancian/silverbullet/common/hooks/event.ts:91:48
at EventHook.dispatchEvent (file:///home/flancian/silverbullet/common/hooks/event.ts:95:10)
at EventedSpacePrimitives.dispatchEvent (file:///home/flancian/silverbullet/common/spaces/evented_space_primitives.ts:47:27)
Requested file _plug/emoji.plug.js
Requested file SETTINGS.md
TypeError: value too large (max 65536 bytes)
at doAtomicWriteInPlace (ext:deno_kv/01_db.ts:628:16)
at AtomicOperation.commit (ext:deno_kv/01_db.ts:407:32)
at DenoKvPrimitives.batchSet (file:///home/flancian/silverbullet/lib/data/deno_kv_primitives.ts:35:33)
at PrefixedKvPrimitives.batchSet (file:///home/flancian/silverbullet/lib/data/prefixed_kv_primitives.ts:16:25)
at DataStore.batchSet (file:///home/flancian/silverbullet/lib/data/datastore.ts:47:20)
at DataStore.set (file:///home/flancian/silverbullet/lib/data/datastore.ts:29:17)
at file:///home/flancian/silverbullet/server/server_system.ts:121:24
at file:///home/flancian/silverbullet/common/hooks/event.ts:91:48
at EventHook.dispatchEvent (file:///home/flancian/silverbullet/common/hooks/event.ts:95:10)
at EventedSpacePrimitives.dispatchEvent (file:///home/flancian/silverbullet/common/spaces/evented_space_primitives.ts:47:27)
Requested file website/Library/Journal/New Page/Daily Note.md
Error GETting file website/Library/Journal/New Page/Daily Note.md Not found
Requested file index.md
Requested file index.md
Requested file index.md
Not handling {
"type": "HTMLTag",
"from": 345,
"to": 353,
"children": [
{
"from": 345,
"to": 353,
"text": "<strong>"
}
]
}
Not handling {
"type": "HTMLTag",
"from": 364,
"to": 373,
"children": [
{
"from": 364,
"to": 373,
"text": "</strong>"
}
]
}
Requested file SETTINGS.md
TypeError: value too large (max 65536 bytes)
at doAtomicWriteInPlace (ext:deno_kv/01_db.ts:628:16)
at AtomicOperation.commit (ext:deno_kv/01_db.ts:407:32)
at DenoKvPrimitives.batchSet (file:///home/flancian/silverbullet/lib/data/deno_kv_primitives.ts:35:33)
at PrefixedKvPrimitives.batchSet (file:///home/flancian/silverbullet/lib/data/prefixed_kv_primitives.ts:16:25)
at DataStore.batchSet (file:///home/flancian/silverbullet/lib/data/datastore.ts:47:20)
at DataStore.set (file:///home/flancian/silverbullet/lib/data/datastore.ts:29:17)
at file:///home/flancian/silverbullet/server/server_system.ts:121:24
at file:///home/flancian/silverbullet/common/hooks/event.ts:91:48
at EventHook.dispatchEvent (file:///home/flancian/silverbullet/common/hooks/event.ts:95:10)
at EventedSpacePrimitives.dispatchEvent (file:///home/flancian/silverbullet/common/spaces/evented_space_primitives.ts:47:27)
TypeError: value too large (max 65536 bytes)
at doAtomicWriteInPlace (ext:deno_kv/01_db.ts:628:16)
at AtomicOperation.commit (ext:deno_kv/01_db.ts:407:32)
at DenoKvPrimitives.batchSet (file:///home/flancian/silverbullet/lib/data/deno_kv_primitives.ts:35:33)
at PrefixedKvPrimitives.batchSet (file:///home/flancian/silverbullet/lib/data/prefixed_kv_primitives.ts:16:25)
at DataStore.batchSet (file:///home/flancian/silverbullet/lib/data/datastore.ts:47:20)
at DataStore.set (file:///home/flancian/silverbullet/lib/data/datastore.ts:29:17)
at file:///home/flancian/silverbullet/server/server_system.ts:121:24
at file:///home/flancian/silverbullet/common/hooks/event.ts:91:48
at EventHook.dispatchEvent (file:///home/flancian/silverbullet/common/hooks/event.ts:95:10)
at EventedSpacePrimitives.dispatchEvent (file:///home/flancian/silverbullet/common/spaces/evented_space_primitives.ts:47:27)
TypeError: value too large (max 65536 bytes)
at doAtomicWriteInPlace (ext:deno_kv/01_db.ts:628:16)
at AtomicOperation.commit (ext:deno_kv/01_db.ts:407:32)
at DenoKvPrimitives.batchSet (file:///home/flancian/silverbullet/lib/data/deno_kv_primitives.ts:35:33)
at PrefixedKvPrimitives.batchSet (file:///home/flancian/silverbullet/lib/data/prefixed_kv_primitives.ts:16:25)
at DataStore.batchSet (file:///home/flancian/silverbullet/lib/data/datastore.ts:47:20)
at DataStore.set (file:///home/flancian/silverbullet/lib/data/datastore.ts:29:17)
at file:///home/flancian/silverbullet/server/server_system.ts:121:24
at file:///home/flancian/silverbullet/common/hooks/event.ts:91:48
at EventHook.dispatchEvent (file:///home/flancian/silverbullet/common/hooks/event.ts:95:10)
at EventedSpacePrimitives.dispatchEvent (file:///home/flancian/silverbullet/common/spaces/evented_space_primitives.ts:47:27)
TypeError: value too large (max 65536 bytes)
at doAtomicWriteInPlace (ext:deno_kv/01_db.ts:628:16)
at AtomicOperation.commit (ext:deno_kv/01_db.ts:407:32)
at DenoKvPrimitives.batchSet (file:///home/flancian/silverbullet/lib/data/deno_kv_primitives.ts:35:33)
at PrefixedKvPrimitives.batchSet (file:///home/flancian/silverbullet/lib/data/prefixed_kv_primitives.ts:16:25)
at DataStore.batchSet (file:///home/flancian/silverbullet/lib/data/datastore.ts:47:20)
at DataStore.set (file:///home/flancian/silverbullet/lib/data/datastore.ts:29:17)
at file:///home/flancian/silverbullet/server/server_system.ts:121:24
at file:///home/flancian/silverbullet/common/hooks/event.ts:91:48
at EventHook.dispatchEvent (file:///home/flancian/silverbullet/common/hooks/event.ts:95:10)
at EventedSpacePrimitives.dispatchEvent (file:///home/flancian/silverbullet/common/spaces/evented_space_primitives.ts:47:27)
My garden is freely available at https://github.com/flancian/garden.git FWIW.
I took a look at server_system.ts:121 and it's got to do with updating a space snapshot: https://github.com/flancian/silverbullet/blob/be8c3303a417d30c88b645a145756a14722e1ff4/server/server_system.ts#L121
I'm unsure which value is too large here -- it sure would be useful if the TypeError exception said which variable triggered it but if it does I'm not seeing it? :)
Bisecting, syncing to c7f564507512867febeab7dcea1d859760aeabc7 fixes the issue.
3e94389c1ea3b41880f888449e0e58e54ff129d9 reproduces. It added the line that is causing the exception as expected, but I don't have a concrete hypothesis for why it's causing it when combined with my garden. FWIW my garden might be an outlier file-count-wise (unsure) with ~7k?
Ok. I understand the issue. Let me look into a fix.
Ok, if you can switch to the edge build and see if I fixed it? https://community.silverbullet.md/t/living-on-the-edge-builds/27/5
The fix is not great (it disables snapshot persisting for very large spaces, like yours apparently), but I think that's acceptable. These snapshots are only useful in environments where changes to the space's are made on disk while SilverBullet is not running, which is a rare scenario.
Thank you Zef, after pulling your fix I was able to confirm it :) Running off edge with docker also works.
On updates that take place while Silverbullet is not running: I am afraid these can happen with my workflow, as I also use [[wiki vim]] to edit my garden, and occasionally my Silverbullet instance can be down while I do so. Are there any side effects I should look out for while living on the edge? :)
Thanks again and have a great weekend!
The only thing that can happen is that those changes are not picked up by the indexer. If you create a new page, it may not appear in the page picker. If a task was created SB will not know about it until the next change you make.
This has always been the case (also pre 0.8) and this snapshot is supposed to fix it.
If you see issues like this a run of the "Space: Reindex" command will always fix it.
Thank you! It's been working fine usually until recently, although I've noticed a few issues coming up more often.
Please let me know if I should open a different issue, I added this here for now as it seems relatively likely this other issue also has to do with the size of my garden?
Also:
Yes, could you create a separate issue? I think the core issue here is scalability with your 7k of pages. I think nobody operates at that scale. Please give as many details in that issue that could be relevant: how many pages, do you have a lot of folders, a lot of attachments or all .md files? How big is the entire space altogether? etc.
I think nobody operates at that scale.
Not yet.
FWIW I was on the latest docker image (updated six hours ago) and I started getting crashes on startup. On inspecting the console, I saw that loading e.g. index.json results in an internal server error.
Reverting to 0.7.7 works around the issue.
Hope this helps :)