Making PiGallery2 better for unraid and in general for massive libaries of files?

[kardo22]

I like PiGallery2, its a lightweight easy solution to gallerize your images, but when you start getting many images inside one folder of many folders inside a folder with sub albums it can get very slow to render the page.

My first though was that it had to do with sqlite so i switched to SQL but no change. It seems it actually does not have anything to do with the DB but for me its a raw disk read speed issue. Unraid has never been known for being speedy especially when you read multiple datastreams from multiple disks in the array.

My problem is if i open a folder with 10 000+ pictures and those pictures are spread out over the unraid array (in this case 4 different disks) the read speed on some of those threads are incredible slow and makes it take almost 3 minutes + to open the folder for the first time. Once the folder has been opened once, it opens almost instantly the next time.

This image below shows whath appens when folder in pigallery2 is opened.

It makes me think that if the folder was indexed and reindexed NEVER got triggered by a user opening the folder and only have a cronjob that can run on its own schedule and can do the heavy lifting instead and have users only be servered thumbnails from the cache.

As it is now you cannot set "Folder reindexing sensitivity" to "Never" and soley rely on a cronjob to do this process for you which is bad. I would want to 100% rely on cronjobs to do this process and never ever serve users data directly from the array but only serve them thumbnails (unless they click a image ofcourse then the data would be read from the array).

I do understand the downsides of this, the gallery would only show an update if the scheduled cronjob has ran etc, but its actually far much more a usuable solution if you have many pictures and a big gallery, especially on unraid that is known for being very slow when it comes to accessing data on the array especially if you are doing multiple smaller reads which images are. Unraid has ssd for cache so you would not have to wait for any disk of software logic when accessing the thumbnails because they are on the cache.

Would love to see this feature in a future version of PiGallery2.

• OS: Unraid: 6.10.3

[rhatguy]

I'm running pigallery2 on unraid with 42k pictures and I don't experience this issue. My pictures are well sorted into yearly folders and then subfolders based on event, so no single folder has more than a few hundred pictures directly in the folder. The yearly folders do have thousands of pictures in a hierarchy. I can also "flatten" an entire yearly folder which effectively displays every picture on that page as if they were all in a single folder. That does take a few seconds to load, but nothing like the 3 minutes you mentioned. I do have a very beefy unraid server, so that may be masking any performance issues somewhat.

Curious if you're using the default unraid pigallery2 install? I recently had to change the "repsoitory" for my docker container to bpatrik/pigallery2:nightly-debian-bullseye as the default unraid install was pulling a VERY old version of PiGallery2. There have been SIGNIFICANT performance improvements since 1.9.4 which the default unraid install was pulling relatively recently for me.

[rhatguy]

I should mention that all of my photos live on an unraid BTRFS pool on SSD and not on the "array" devices, which may also have some impact to this. Are you saying that PiGallery2 is checking and reading if needed all of the full size pictures which you have living on the array devices every time you open a folder?

[kardo22]

I should mention that all of my photos live on an unraid BTRFS pool on SSD and not on the "array" devices, which may also have some impact to this. Are you saying that PiGallery2 is checking and reading if needed all of the full size pictures which you have living on the array devices every time you open a folder?

yes this is the problem, if you are running your files from a ssd there wont be any issues, but my picture library is massive beyond ssd capacity. The problematic folders are the one with 10 000k ultra high res images in one folder, these span over multiple regular spin disks on the array and it becomes sluggish, i know my usecase is extreme and no body else probably has this problem but it could be easily resolved by having a setting that makes it impossible for a user browsing a share to trigger reindexing and only rely on reindexing to be done via cronjobs

[bpatrik]

Hi,

Disclaimer: I'm not familiar with unraid and as the readme says that app can only handle a few photos (less than 2k or so) per folder/search result. (I.e.: I can't provide support above that)

That being said I think it's fine to add 'never' option to the indexing severity. Should be an easy addition to the config enum and and to the indexing manager. If you can sand a PR that would be appreciated or I can come along like in a month and do myself.

Also some background: On every folder load the app fetches the folder modification date. If the date is newer than the last indexed date, it will do a reindexing. (It also does a reindexing if a timeout passed). If I recall correctly this is how low severity works.

Other words the app reads the folder metadata on every folder listing. That can be also cause of the slowness. (See code in the indexingmanager)

Also: the app generates thumbnails and scaled down photos. The original photo will be only loaded if someone zoom-s in the photo. So apart from reading folder metadata/reindexing, the app should not touch the original folder/image.

[bpatrik]

In my previous comment I meant GalleryManager.ts at this function: '  public async listDirectory'

[bpatrik]

Added a never index option. That should help with this one.

[bpatrik]

Although for security reasons the app also checks if the original photo exists when the app shows any thumbnail. To remove that it would take more consideration, but at least setting the indexing to never, would make the first load faster.

The problem of removing the check for the original source when serving an image is: If you delete a photo (like one that is compromising), not checking the original source, the app could still server the thumbnails until the tmp folder is cleaned up. So I think that check should stay there.

[bpatrik]

Closing this due to inactivity. Feel free to reopen if you think otherwise.

[boxleits]

Hi @bpatrik, you wrote that "the app also checks if the original photo exists when the app shows any thumbnail.". I would like to hack the code and switch off that check. At first glace i could not find the spot, where this check is implemented. Could you maybe give me a hint?

[bpatrik]

Here are the endpoints where you should start digging: https://github.com/bpatrik/pigallery2/blob/7b7e980ddc6631a66efe08c9cdc068ed52c31949/src/backend/routes/GalleryRouter.ts#L65-L160

Bat actually you probably need this (same file): https://github.com/bpatrik/pigallery2/blob/7b7e980ddc6631a66efe08c9cdc068ed52c31949/src/backend/routes/GalleryRouter.ts#L183-L200

Here is the code that check if the file exists: https://github.com/bpatrik/pigallery2/blob/7b7e980ddc6631a66efe08c9cdc068ed52c31949/src/backend/middlewares/GalleryMWs.ts#L191-L194