Crash

[ludos1978]

I get a crash when trying silverbullet on my large md-based filestructure on mac. It does incorporate lots of other files, linked structures etc.

Version: 3175b3c94870b01339aa6fee2b41c865bf0d5a79

Going to start Silver Bullet on 0.0.0.0:3000 serving pages from /Users/rspoerri/_SYNC/Hochschulen/_Notes_Hochschulen with db file data.db

<--- Last few GCs --->

[54317:0x7fe4b8008000]   330073 ms: Mark-Compact 1246.7 (1435.1) -> 1234.4 (1435.6) MB, 2188.3 / 0.0 ms  (average mu = 0.138, current mu = 0.080) task; scavenge might not succeed
[54317:0x7fe4b8008000]   331352 ms: Mark-Compact 1247.4 (1435.6) -> 1234.9 (1436.4) MB, 1116.3 / 0.0 ms  (average mu = 0.134, current mu = 0.127) task; scavenge might not succeed

<--- JS stacktrace --->

#
# Fatal javascript OOM in Ineffective mark-compacts near heap limit
#

[1]    54317 trace trap  silverbullet .

[zefhemel]

What is "lots"? How many files? And what do you mean with linked structures?

[ludos1978]

The size is approximately 305GBytes in 217'000 Files of all types of files among them approx 2000 Markdown files.

The documents are often linked with Markdown Tags such as also other Media are Linked that way such as Movies, CSV-Tables and Images.

[zefhemel]

Ok, but what you're seeing here is the full console output? It doesn't attempt to start indexing things even it seems? So maybe it blows up even trying to get the file listing. I've tested spaces with around 4k markdown files, but never 217k...

[ludos1978]

I just run it once more and now it runs an indexing which it did not do previously. However it aborts after 361/2122.

Going to start Silver Bullet on 0.0.0.0:3000 serving pages from /Users/rspoerri/_SYNC/Hochschulen/_Notes_Hochschulen with db file data.db
Going to load 8 plugs...
Loading emoji
Loading tasks
Loading markdown
Loading collab
Loading share
Loading directive
Loading plugmd
Loading core
Now reindexing space...
Now starting sandbox for core
[Sandbox log] Clearing page index...
[Sandbox log] Listing all pages
[Sandbox log] Indexing page 1/2122: ZHdK/_ADMIN/Anstellung-ZHdK
Now starting sandbox for tasks
Now starting sandbox for directive
[Sandbox log] Indexing page 2/2122: ZHdK/20210621103224
[Sandbox log] Indexing page 3/2122: ZHdK/20210630104300
...
lots more Sandbox log entries
...
[Sandbox log] Indexing page 361/2122: ZHdK/_KURSE/VGD_PROCEDURAL_DESIGN/_90_ABGABEN/Game Visuals 5 - 2020/MorenoVogel_SeverinBrodmann/GameVis5/Library/PackageCache/com.unity.shadergraph@7.3.1/Documentation~/Cosine-Node

<--- Last few GCs --->

[77509:0x7fca58008000]   325031 ms: Mark-Compact 1231.2 (1434.1) -> 1219.0 (1434.9) MB, 770.4 / 0.0 ms  (average mu = 0.183, current mu = 0.115) task; scavenge might not succeed
[77509:0x7fca58008000]   326123 ms: Mark-Compact 1232.1 (1434.9) -> 1219.6 (1435.4) MB, 966.5 / 0.0 ms  (average mu = 0.149, current mu = 0.115) task; scavenge might not succeed

<--- JS stacktrace --->

#
# Fatal javascript OOM in Ineffective mark-compacts near heap limit
#

[1]    77509 trace trap  silverbullet .

The folder it stops running in is a Unity Project (Unity3d.com) which apparently made it into my hierarchy :-)

I also have syncthing running on this directory (dropbox like folder syncing). Also i have opened several files in vscode while running it.

OS is: macOS Monterey 12.6, 64GB Ram App usage seems to stay at 1Gbyte Ram

---

...
[Sandbox log] Indexing page 961/2122: Zettelkasten/Website-Projekte-MD/2018-Cosmic_Watch

<--- Last few GCs --->

[77628:0x7fada0048000]   321112 ms: Mark-Compact 1233.9 (1434.8) -> 1221.6 (1435.6) MB, 909.0 / 0.0 ms  (average mu = 0.215, current mu = 0.137) task; scavenge might not succeed
[77628:0x7fada0048000]   322175 ms: Mark-Compact 1234.5 (1435.6) -> 1222.1 (1436.3) MB, 921.9 / 0.0 ms  (average mu = 0.176, current mu = 0.133) task; scavenge might not succeed

<--- JS stacktrace --->

#
# Fatal javascript OOM in Ineffective mark-compacts near heap limit
#

[1]    77628 trace trap  silverbullet .

a third run makes it even further to 961/2122, i cannot see anything specific causing it to abort. It's definitely not correlated with files opened in Visual Studio Code. Syncthing could open them, but they havent been changed in a long time, so it shouldnt access them as well.

---

Is it possible to make it more verbose? My folders do contain huge amounts of other file-types. Is it possible that these interfere?

[ludos1978]

a 4th run has it abort earlier then the 3th run (at 685/2122). It seems to be totally random when it fails.

[zefhemel]

I suspect that it's trying to index large or perhaps binary files. Since indexing right now is fully sequential, the last file name should be the place where things actually fail. Can you say anything about e.g. Zettelkasten/Website-Projekte-MD/2018-Cosmic_Watch is there anything weird in that file? It large, strange characters?

Since it's indexing files like ZHdK/_KURSE/VGD_PROCEDURAL_DESIGN/_90_ABGABEN/Game Visuals 5 - 2020/MorenoVogel_SeverinBrodmann/GameVis5/Library/PackageCache/com.unity.shadergraph@7.3.1/Documentation~/Cosine-Node I'm wondering if we somehow need to be more aggressive about excluding things, because I would imagine this is not a file you want to actually index or even navigate to. Do you "blacklist" such files anywhere, a .gitignore file perhaps? Theoretically SB could start interpreting a .gitignore file and apply the same filters.

[ludos1978]

2018-Cosmic_Watch is a very short md file, the only speciality possibly being a md-link to another file. Also it contains german Umlaute (äöü).

The second file is actually a markdown file which i wasnt aware of (documentations of a project internal file), which could be excluded. I removed it anyways because i dont need it anymore. Some way of excluding and/or including different folders would be nice.

This is actually the same file (content) as mine: https://github.com/needle-mirror/com.unity.shadergraph/blob/master/Documentation~/Cosine-Node.md

In my case i have multiple folder hierarchies i use (private/business/universities) where i store different topics. They cant be put together in a folder because they are synchronized with different parties.

[ludos1978]

It might not even have to do with the indexing. I zipped all project folders which reduced the number of markdown files quite a lot. But not the overall folder size.

this is my last run:

Going to start Silver Bullet on 0.0.0.0:3000 serving pages from /Users/rspoerri/_SYNC/Hochschulen with db file data.db
Going to load 8 plugs...
Loading emoji
Loading tasks
Loading markdown
Loading collab
Loading share
Loading directive
Loading plugmd
Loading core
Silver Bullet is now running: http://localhost:3000

<--- Last few GCs --->

[84185:0x7f9778048000]   768920 ms: Mark-Compact 1211.2 (1434.4) -> 1198.7 (1434.9) MB, 1076.1 / 0.0 ms  (average mu = 0.175, current mu = 0.103) task; scavenge might not succeed
[84185:0x7f9778048000]   769912 ms: Mark-Compact 1211.6 (1434.9) -> 1199.2 (1435.6) MB, 874.7 / 0.0 ms  (average mu = 0.149, current mu = 0.118) task; scavenge might not succeed

<--- JS stacktrace --->

#
# Fatal javascript OOM in Ineffective mark-compacts near heap limit
#

[1]    84185 trace trap  silverbullet .

It seems it crashed after the indexing the same way as it did previously.

---

another try resulted also in a crash after the indexing!