search

earlSt1 / vtt-compendium-folders

Collapsable folders in the compendium directory and folder structures inside compendiums for FoundryVTT
21 stars 18 forks source link

Migration fails due to exhausting system memory #215

Closed FrankIsAtank closed 11 months ago

FrankIsAtank commented 11 months ago

The migration to v11 caps a CPU and eats up memory until it reaches 100% after about 5 minutes, then the tab crashes. My system has 32 GB. Hard to imagine why a migration needs that much memory.

thejoester commented 11 months ago

Yeah I have 64GB and its the same thing

earlSt1 commented 11 months ago

Hi, is this a sidebar migration or migrating a compendium?

FrankIsAtank commented 11 months ago

Hi, is this a sidebar migration or migrating a compendium?

Unsure. Whichever migration occurs when you click the button on the new Professor Farnsworth popup that appears when you login.

earlSt1 commented 11 months ago

Can you both send me the contents of the Import/Export dialog in the module settings? I'm mainly interested in the Export side

gludington commented 11 months ago

Different person, but similar issue. Attaching my
compendiumfolders.json.txt

I have quite a few compendiums, both from modules and my own, and I will see many hundreds of thousands of 10250foundry.js:616 Foundry VTT | Registered callback for renderCompendiumDirectory hook

before the tab finally gives up. I have been reloading when this happens, and it appears to pick up where it left off, though I am not sure if there is any damage yet to whatever compendium was in progress at the time of the tab crash.

earlSt1 commented 11 months ago

Thanks, will check it out in a bit. In the console there may be more clues as to where its failing if you show verbose logging. In chrome there should be a dropdown button where you can select log levels. It says Default by default i think.

On Sun, 16 Jul 2023, 05:07 gludington, @.***> wrote:

Different person, but similar issue. Attaching my compendiumfolders.json.txt https://github.com/earlSt1/vtt-compendium-folders/files/12063272/compendiumfolders.json.txt

I have quite a few compendiums, both from modules and my own, and I will see many hundreds of thousands of 10250foundry.js:616 Foundry VTT | Registered callback for renderCompendiumDirectory hook

before the tab finally gives up. I have been reloading when this happens, and it appears to pick up where it left off, though I am not sure if there is any damage yet to whatever compendium was in progress at the time of the tab crash.

— Reply to this email directly, view it on GitHub https://github.com/earlSt1/vtt-compendium-folders/issues/215#issuecomment-1636975385, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADI2CKASGJZT5E4LJQIN2YTXQNSGFANCNFSM6AAAAAA2C5FMHI . You are receiving this because you commented.Message ID: @.***>

gludington commented 11 months ago

Those lines were from the verbose section of the logs. If I skip migration and go check out the "Configure Settings" section, it only shows two of my compediums as present in as migration options, though it starts with the (admittedly) large Baileywiki modules each time, and I seem unable to get past that.

earlSt1 commented 11 months ago

Ok, i'm a bit confused, so the Sidebar migration button (when you click on the popup window when you first launch the module), is the one generating those log messages. That migration doesnt actually touch the contents of the compendiums, its just the sidebar folders. The section in the module settings is for migrating folders inside each compendium. Also compendiums will only show in that settings dialog if they are unlocked, as I'm not able to modify them if they are locked.

I just tested your sidebar config in my test world, and it seemed to migrate and create the folders fine. Do you have any other modules enabled that could be interfering with the migration?

gludington commented 11 months ago

Interesting. It may be a false positive then, in that the last element I see in the UI is that folder migration has started before I get hundreds of thousands of those hook messages in console, and the browser eventually halts. If I disable EVERY module, migration appears to succeed. After re-enabling the modules, the world loads again, though the sidebar has a large number of duplicates (e.g. I have 15 "Actor" folders, 12 "Item" folders, and so on, each with different (or sometimes no) contents, presumably from the browser dying mid-migration, for whatever cause.

ref2335 commented 11 months ago

Same issue here, encountered similar results with Edge and Firefox browsers.

Nedrapter commented 11 months ago

Also experiencing the same, upgraded today to v11 306 and migration crashes the browser tab, even with 64gb RAM. I do have other modules enabled but, many of them might be having folders that need migration too, so I'm not sure whether makes sense disabling randomly (I have lots of modules, compendium and folders).

thejoester commented 11 months ago

Honestly I just gave up, uninstalled the module, and manually rebuilt all my compendiums.

Nedrapter commented 11 months ago

Honestly I just gave up, uninstalled the module, and manually rebuilt all my compendiums.

Wish I could do the same, but I've been building folders for like 2 years...and some 3rd part content creators have compendiums with 250+ folders....

Update: Checking out the console, even after page crashes, these counters keep going image

And looking at the Task Manager the RAM doesn't reach 100% or even close, but still the page crashes, saying that it ran out of memory.

earlSt1 commented 11 months ago

Honestly I just gave up, uninstalled the module, and manually rebuilt all my compendiums.

Wish I could do the same, but I've been building folders for like 2 years...and some 3rd part content creators have compendiums with 250+ folders....

Update: Checking out the console, even after page crashes, these counters keep going image

And looking at the Task Manager the RAM doesn't reach 100% or even close, but still the page crashes, saying that it ran out of memory.

Hi, just to clarify this is after clicking the sidebar migration? (The button on the popup window with professor farnsworth when you first boot up in v11)

earlSt1 commented 11 months ago

Another thing to try is not having the compendium sidebar visible when doing the migration (so stay on the actors tab for example). It may skip trying to render the directory which looks to be the problem

Nedrapter commented 11 months ago

Just tried that, but it had no effect.

earlSt1 commented 11 months ago

Just tried that, but it had no effect.

Ok, will try and find some time to check this out today.

earlSt1 commented 11 months ago

Alrighty, I've managed to replicate this issue by enabling the module Compendium Browser, disabling this module allows me to do the sidebar migration successfully. Can you try with disabling this module if you are using it? If you're not can you send your module list and system?

Either way, if you disable all modules other than content modules (which just have compendiums in them), and CF + libWrapper, then it should work fine.

Nedrapter commented 11 months ago

It is working now, indeed! Is there a way to migrate all unlocked compendiums' folder data in one go?

earlSt1 commented 11 months ago

It is working now, indeed! Is there a way to migrate all unlocked compendiums' folder data in one go?

Great! And no unfortunately, i had a feeling that if i added that option people would run into memory issues again. You can call the migration code using a macro, so you could try batching them in groups of 5 or 10, but its more reliable to do them one by one.

See this comment for an example macro: https://github.com/earlSt1/vtt-compendium-folders/issues/208#issuecomment-1568141857

earlSt1 commented 11 months ago

I'll be closing this issue for now, as it's likely that a lot of these issues have stemmed from the Compendium Browser module being enabled.

For anyone else encountering this issue, disable Compendium Browser before running any migrations. Once the migrations have been done you can safely reenable it.