search

loris-imageserver / loris

Loris IIIF Image Server
Other
209 stars 87 forks source link

Break up Image and Info cache into more directories #431

Closed martinlovell closed 4 years ago

martinlovell commented 6 years ago

Currently the image and info cache create a directory per ID in the root of the cache (after the protocol) in ImageRequest.cache_path and InfoCache._get_info_fp/_get_color_profile_fp. This could cause performance problems with some file systems because of many entries in one directory.

The implementation could optionally split the entries into more directories by doing something like SimpleHTTPResolver._ident_file_structure for those who server up many different IDs. It could be an option so those who don't need it, don't have to use the resources to build the file structure for each request.

jpstroop commented 6 years ago

FWIW (and if I recall correctly), the cache used to be a pairtree-like structure, but that was changed for some reason. Might be worth poking through old issues/commits to see why.

bcail commented 5 years ago

We are running into this issue, and it seems like this suggested change would be helpful. @alexwlchan do you have any thoughts?

bcail commented 5 years ago

Looks like issue #143 and PR #146 are related. @jpstroop, @alexwlchan seems like we could do something like _ident_file_structure, but still use identifier name. Something like this: cache/ab/cd/ident/...

How does that sound?

bcail commented 5 years ago

PR #479 would implement this change for the InfoCache (using the same directory format that the SimpleHTTPResolver uses).

I would like to do the same thing for the derivative ImageCache. Anyone opposed? Anyone need the derivatives cache to stay the way it currently is?

jrhoads commented 4 years ago

PR #479 Was merged. Does that close this issue?

bcail commented 4 years ago

Yes, with #483. Closing.