Obsidian import corrupts users.db and greatly increases world size

[4everGM]

Hi all,

I'm on V11 using Forge as hosting and using Chrome as default browser. I've used this tool in the past with no issue, but recently I've noticed that when I import (roughly 350 records, some with images, but nothing large) that it will import with no issues, but the world size goes up dramatically (from 8mg to 50+) and when I log out and then log back in, it tells me the user.db is corrupted. I realize Forge can be a little wonky because but had no issues with this in the past. Any thoughts on how to rectify. (I love the backlinks and ease of use of importing. (much better than Realm Works importer (although I do love that I can have secrets and reveals). Please keep up the good work!

[Praxxian]

Sorry to hear you're encountering this.

1. What's the total size of your vault in MB?
2. When you import, do you receive the confirmation that the import finished before logging out?
3. Any errors in the console during the import?

[Praxxian]

Haha, on mobile. Hit the wrong button.

[4everGM]

haha, I know the feeling…

1. I’ve tried this a couple of times; it starts at about 8mb and after import ballooned to about 44mb or higher. I have tried this on a couple of game systems (T2K4e and PF2e for example) to ensure it wasn’t system related (since systems like to set their own journal “skins”)
2. Yup, it confirms. Technically it works as intended, the files are there, but once I log exit back to main, I can’t access again. I tried a few different things, and only is an issue on importing from Obsidian
3. Not really, I see one “viewbox” type error which looks to me like something to do with the window prompt, but during import there are really no errors.

Regards, AJ

[joetheory]

I can confirm this experience and I have tested it on a few v11 worlds. One of them now has a user.db of around 125mb and causes the server to crawl. This plugin has become an essential part of my workflow and I cant wait to have it back. Here's to hoping @Praxxian has some time soon.

[joetheory]

I should note that I self host via docker so the issue is not platform specific.

[rikmarais]

Hey there 👋 Encountered this today as well (~500mb of data from image files stored in the markdown stored in buffer in the vault in the users db). When packed into leveldb, the size is substantially smaller, but storing the data in the users db seems like it would greatly increase load times under any network conditions.

I realize the problem is not trivial to solve, but I have some thoughts.

• Process the markdown and upload any embedded images with the FilePicker, then reference those uploaded files.
• upload and reference a file in a world-specific location which contains the vault data rather than storing it in users db. This is less ideal since it could lead to large single files if not minified/compressed but would likely work better than in the users which is accessed when the world loads

[joetheory]

FilePicker is being used to upload. I had this problem without using the function that uploads images so I think something else is the cause. Here is the function though if you see anything that stands out:

https://github.com/Praxxian/lava-flow/blob/6ac80d3bea6e6f36b706329c93b09b5e73c1e466/src/lava-flow.ts#L129C1-L129C1

[rexmortus]

I have had this same issue, with it smashing two of my games that depend on Obsidian vaults.

[Merlyn42]

Is this always an issue with Lava-Flow or is there something I can do to prevent it?

[matt-duffy]

Just to confirm that I am also facing this issue, users DB goes from 5KB to 25MB if I do an obsidian import (only 140 odd items) which kills mem usage of the Foundry server and makes it unusable

[matt-duffy]

I think this issue is line 92 in lava-flow.ts:

await (game as Game).user?.setFlag(LavaFlow.FLAGS.SCOPE, LavaFlow.FLAGS.LASTSETTINGS, savedSettings);

I have commented that line out and built a local copy and the problem with blowing up the users db file doesn't appear to happen anymore, I guess maybe that savedSettings object can get really big and therefore cause the issue?

[Praxxian]

Hey everyone I am so sorry for the lack of updates. Work has been kicking my butt. I took time off to work on this issue, and I spent the whole time sick.

Thank you @matt-duffy for troubleshooting. As soon as I can, I will check this out.

[rexmortus]

@Praxxian no apologies necessary. 💓 This is an open-source project motivated by the goodness of your heart. I hope I speak for all us Obsidian / FoundryVTT users ... we are grateful this plugin exists.

[4everGM]

Agreed, no apologies necessary. We appreciate you eyeballing it, and of course, work comes first!!!! Thanks dude!

[Praxxian]

Hello all! Version 3.0.1 is released and should resolve the issue. I had some trouble reproducing, but thanks to @matt-duffy I can definitely see there would be a problem for large vaults.

Hopefully someone can upgrade and try this out to confirm they are no longer impacted. Thanks!

[rexmortus]

Thanks @Praxxian, I updated to 3.0.1 and imported both my vaults into Foundry. Load times normal, no server crawl. Seems fixed to me!

[matt-duffy]

Yep I've tested with 3.0.1 as well and all working fine, thanks very much @Praxxian, fantastic module!