Performance degradation based on library size

[DrazorV]

Hello! I will try to document my real world experience to better help understanding the performance impact especially as libraries grow. Please fill free to correct me if something is not accurate or should be mentioned in a different issue.

Setup

• Synology Nas 723+ (AMD R1600 and 18GB ram)
• Manyfold on a docker-compose stack with redis7 and pg-16

Library mostly stls and 3mf but has images and lys files too

Loading Times

• Home Page: 500ms
• Settings: 115ms
• Creators: 6.39s
• Collections: 6.50s
• Library Page: 31.71s

CPU/Ram Usage Everything below is documented while I perform a scan (not the initial scan)

Manyfold Conainer

Postgres Container

Redis Container

Additional Info

• Manyfold container has 12gb ram limit (never reached)
• Redis container had 256mb limit that I had to remove because it kept shutting down
• Noticed some ram usage increase when an additional user logged in (from 2.1GB to 3.5GB)
• Pagination (number of elements per page) does not affect loading time for the Library Page tried with 2, 12 and 50 elements.
• "Create tags from model directory name" setting is enabled
• Activity page doesn't load at all (it did work for couple of minutes but stopped loading after the scan progressed)

Since pagination doesn't affect loading times, it safe to assume that the performance hit doesn't rely on the number of models per page nor the type or size of models and their previews. After some discussions we came to the conclusion that it could be affected from the number of tags created which is close to 16000 for me and/or the number of database queries.

If there is anything else I can provide please ask!

[Floppy]

This improved about 33% with the N+1 fixes in #2085 , which is a start. After more investigation there's definitely some slowdown in the building of the tag list in ModelFilters, which ends up doing a lot of queries, and in this case with 16000 tags, I mean a LOT of queries.

[Floppy]

I've release v0.74.0 with a bunch of tag performance improvements; hopefully this will help! I'd also recommend changing your tag visibility threshold to at least 2 (the new default) in settings.

[DrazorV]

Hello, I deleted my db and I just started a new scan on my library. I will comeback with feedback about performance improvements soon.

[Floppy]

Thanks, it'll still take a long time to scan I expect, but the UI performance should be better...

[Floppy]

@DrazorV just checking in to see how the performance is at the moment - has anything improved over the last few months?

[DrazorV]

Performance has certainly improved. I feel there is space for optimization. Initial scan takes time. Model preview images could be loaded in a lower resolution to improve site performance, and models without image previews could use an image of the 3d model instead of loading the whole model. Loading 20 models in a single page takes a lot of time.

[Floppy]

Yeah, those aspects definitely can be improved; I've checked to make sure they're covered in other issues, and of course generally I'll keep working on performance. The next release will include a performance dashboard and some memory stats, so I'm keeping performance in mind and hopefully making it more visible!

Regarding general performance of the site though, rendering pages, etc, would you say it's in a usable state for you, so this general issue can be closed? Obviously we can open issues for specific slow parts that stand out.

Existing issues:

• 1706 (generating images to preview 3d models)

• 3111

• 2970 (upcoming work to send 3d models more efficiently)

[DrazorV]

@Floppy yes I think it's fair to say that it is usable, we can close the issue. I don't know if the slow loading of images is totally on manyfold because I am hosting it in network with a 20mbps upload speed.

[Floppy]

it's definitely something we can try to at least not make worse :D