search

sveltia / sveltia-cms

Alternative to Netlify/Decap CMS. Fast, lightweight, Git-based headless CMS. Modern UX, first-class i18n support, open source & free. Made with Svelte.
MIT License
955 stars 49 forks source link

Seemingly arbitrary maximum limit of entries using local git repository #224

Open xelivous opened 1 month ago

xelivous commented 1 month ago

I'm noticing some strange behaviour with the local git repository. I currently have 6 collections with the following quantities (161, 35, 1, 2, 2, 157) for a total of 358 items.

If I create a new item and bring the total up to 359, I can browse through the site and edit any information as normal. However if I refresh the page (or simply add in another collection with more than 1 entry), I will get an "unexpected error". Looking at the console log provides the following stack trace:

image

As soon as I remove any item from any of my collections (or comment out a collection) to bring the total down to 358 or lower I am able to enter the CMS once more.

kyoshino commented 1 month ago

Thanks for the heads-up. The CMS itself doesn’t set any limitation, but the error message comes from Chrome. I’ll figure out what’s going on.

kyoshino commented 1 month ago

I have duplicated a collection on my client instance, bringing the total number of entries to 1,000+, but it still works 🤔

Do you have many images, or some large files?

Perhaps it’s a Linux-specific issue. I’m on macOS. Will create a Linux VM to test.

xelivous commented 1 month ago

The 161 collection have 1x image, average around 250kb size. The 35 collection all have 1x small image each (<30kb). The other collections don't have any images. "All assets" reports 224 images.

I'll try to create some random test configs of my own to see if i can replicate it some other way.

After doing some testing, it might be some kind of random memory limit? I had 4GB of memory left on my computer but maybe chrome was running into some limit before that based off of the testing below.

  1. My indexed DB was fine at 1.1MB
  2. I browsed some thumbnails in the asset library and the indexedDB went up to 1.7MB
  3. refreshed the page and it failed to load
  4. Cleared the indexedDB, disabled cache
  5. Refreshed, Loaded fine again, refreshed, loaded fine
  6. Browsed some thumbnails to raise it above the limit, and it failed to load again.
  7. Fully closed chrome and everything else, to free up as much ram as possible on my system
  8. Reloaded up sveltia, browse through thumbnails, indexedDB up to 7MB, refresh, works fine
  9. Wait a few minutes, refresh, fails once (same tracelog as original error), refresh again, works fine
  10. The more that I refresh now, the more likely it is to fail. After about 20 refresh cycles it's now failing 100% of the time. If I restart chrome it will likely start working again.

Could be a weird bug with the local storage api.

kyoshino commented 1 month ago

Thanks for your investigation! I’ll create a bunch of images and try again.

kyoshino commented 1 month ago

I have generated 500 images in the asset library but these are still loaded (though It takes a sec; it’s a separate issue)

I think URL.createObjectURL() used for displaying thumbnails is causing a memory leak because Sveltia CMS is not revoking these URLs properly. I’ll fix the issue.

kyoshino commented 1 month ago

I’ve made the changes in v0.40.2. Do you see any difference?

xelivous commented 1 month ago

Yes, still the same issue after ensuring to pin it to 0.40.2

kyoshino commented 1 month ago

Sorry to hear that. I’ll set up a Linux VM with Parallels.

kyoshino commented 1 month ago

I have tested with Linux Mint but still can’t reproduce the problem. Will do more research.

kyoshino commented 1 week ago

Sorry it took so long. Not sure if you’re still seeing the problem but I’ve realized that the problem is probably not with the thumbnails, but with the entry file reading part, so have added some try...catch to prevent crashes. Let me know if the latest version solves your problem.

xelivous commented 1 week ago

Haven't had too many issues with my "medium"-sized repository for a while now since i've constantly restarted the OS/browser relatively often enough. However I am still having issues with my "massive" repo (1084 pages reported by hugo + 824 images) with v0.46.5 unfortunately.

I tried digging through with the debugger to see where exactly it's failing but i just can't figure it out from stepping through.

It does fail within this line though, and seems to be in the areas that you've changed; if i put a breakpoint on the line below that it will never reach it. More specifically it manages to _iterate() properly in that function and then fails when resolving the final promises here. You've already changed the hash function to be async so i'm not sure what else would be failing in there.

I'll look into making a local dev copy of sveltia this weekend to see if i can see where/how exactly it's failing instead of trying to work around with local overrides in the debugger.

Thanks

kyoshino commented 1 week ago

I will retry to reproduce the issue locally.