search

cantaloupe-project / cantaloupe

High-performance dynamic image server in Java
https://cantaloupe-project.github.io/
Other
267 stars 107 forks source link

Tif file performance #455

Open qudratullah1061 opened 3 years ago

qudratullah1061 commented 3 years ago

Hi,

I hope you are doing great. I have a question regarding cantaloupe, I have images which are tif but cantaloupe is taking around 45sec, 50sec to make each tile from my source image and I am not able to see it in any viewer because it is a lot of time. I read in your documentation to use pyramidal tiff or tiled tiff for best performance. Is my attached file does not meet requirements? How can i convert these files? Can you send me pyramidal tiff file as sample if you have any so i can test things on cantaloupe server by uploading sample file if performance issue fixed so i can change my files format or structure. As it takes huge time, Also i did not found any pyramidal tif file on internet. Is my attached file format is correct or should i change it? Your advice will be very helpful to me. My max file size is around 600mb from my collection.

http://54.85.241.249:8182/iiif/2/baf__d288c1c83b2ed88089882a959a54a1e64fd27399e8a8177523a91e29505b5c17/full/full/0/default.tif

You can download sample tif file from here.

Thank you.

demonshreder commented 3 years ago

I have the same issue for serving TIF images, is there any specific ways to improve performance for the Java2D processor, or is there a way to cache all the tiles forcibly instead of processing then caching on-demand?

The performance is quite well once the derivative cache is hit but as mentioned above it takes more than 30s for first hit.

DiegoPino commented 3 years ago

@demonshreder we also saw some TIFF performance issue, not as big as 30s(more in the 5s time for large TIFFS) but sometimes, specially when using S3 cached derivatives we end with some strange partially broken Tiles with gray pieces...(which can be of running out of memory). What we are thinking of right now (as we love Cantaloupe) is to test if we can, by issuing ASYNC HEAD requests force the cache to be "pre-warmed" and if space/disk is not an issue for you can make Caches last for ever if needed (or even have an external cache removal strategy). The challenge is to make sure the tiles requested (crop/size) for the pre-warming match the ones your Viewers would expect. Also we have not tested if a HEAD request will trigger the caching/generation of derivative caching or we need to make a GET and close the connection quickly afterwards.

But we are confident that caching is the key here and memory too.

demonshreder commented 3 years ago

Hi @DiegoPino, thanks for explaining the situation at hand.

I am using a FileSystemCache (HDD) which is indeed quite slow, however I am seeing responses for cached tiles within a second. The problem arises when the uncached tiles keep getting requested and the response time increases in O(n) manner. For now I have added another cantaloupe instance to nginx to load balance, with shared cache.

Both cantaloupe instances are running 4.1.8 on JVM 11, with max JVM heap size of 4GB RAM each. Best I can do now is to take a list of all URLs and send requests to everything I need to be warmed up. The server has 8 cores of Intel Xeon, no GPU.

Edit: Typo, add hardware details.

adolski commented 3 years ago

Ordinary non-tiled, non-pyramidal TIFF files are going to be slow, unless they are small. Larger TIFFs must be pyramidal and tiled if you want to have any hope of good performance. Please refer to the TIFF section on this page and also see this excellent page for info on creating them.

jbarth-ubhd commented 3 years ago

Pyramidal tiffs still require ~2s per request: https://github.com/cantaloupe-project/cantaloupe/issues/296

Withdrawn. 90 % of the problem was our delegate.rb without caching.