search

Helioviewer-Project / JHelioviewer-SWHV

Space Weather JHelioviewer ☉
https://swhv.oma.be
Mozilla Public License 2.0
49 stars 16 forks source link

A new option to set image cache size? #283

Open GitatHub opened 3 months ago

GitatHub commented 3 months ago

The current image cache size is set to 8GB on disk. Given that modern computers often have hard drives of 1TB or more, could an option be added to customize the image cache size, as long as it doesn’t risk crashing the application?

bogdanni commented 3 months ago

8GB is about 8000 AIA images (there is an overhead in the cache store). Is there an actual need to cache many more than that, given that new images are constantly being acquired?

GitatHub commented 3 months ago

8GB is about 8000 AIA images (there is an overhead in the cache store). Is there an actual need to cache many more than that, given that new images are constantly being acquired?

Thanks. Yes, it's rare to need more than 8GB of cache, but I'm currently working on multiple events and have maxed out the fixed 8GB. Each time I reuse an image layer, I have to wait for the application to reload the images. If increasing the cache size doesn’t cause any issues with the application, it would be beneficial to have more customization options. This would also reduce server load and bandwidth usage.

bogdanni commented 3 months ago

For this use case, I guess it would be OK to restart the program for the new size to take effect. In my testing, increasing the cache size preserves the data already there, while decreasing the size leads to clearing of the cache.

GitatHub commented 3 months ago

For this use case, I guess it would be OK to restart the program for the new size to take effect. In my testing, increasing the cache size preserves the data already there, while decreasing the size leads to clearing of the cache.

What do you mean by 'restart for new size'? Are you saying that if I restart the program, the cache will increase by 8GB each time? Does this mean JHV doesn’t track the cache used but just requests a new cache size with each restart? If so, wouldn’t this extend the cache size with every restart? However, the cache directory on my disk is always smaller than 8GB. Additionally, even if restarting increases cache size, there is another issue that needs to manually set the time ranges after each restart :-) (see issue #281).

Cache size:

~/JHelioviewer-SWHV/Cache$ du -sh ./ 5.4G ./

Screenshot from 2024-08-08 20-00-44

bogdanni commented 3 months ago

After changing the size in Settings, you will have to restart the program for the new value for the size to be used.

The JPIP image cache is held into the JPIPStream-* directory for image data and in JPIPLevel-* directory for some metadata about the resolution of the downloaded data. I cannot reproduce that size disparity from your computer: the calculation which is done to display the size in the Settings window recurses those two directories and adds up the sizes of the files, i.e., what du also does.

GitatHub commented 3 months ago

I understand that if there were an option to set the cache size in the Settings, the change would only take effect after a restart. Currently, I’m using version 4.7.0.11268, which doesn’t include this option. As for the cache size shown in my screenshot, please disregard it. On my computer, I’ve also seen different outputs, such as Nautilus showing: 3,195 items, totaling 9.0 GB.

dmueller3000 commented 3 months ago

Yes, it's rare to need more than 8GB of cache, but I'm currently working on multiple events and have maxed out the fixed 8GB. Each time I reuse an image layer, I have to wait for the application to reload the images.

Hello, For your use case, you might also consider downloading the image layers as .jpx files by selecting the respective layer in the layer manager and clicking on the download icon. Personally, that's my preferred approach in this case because it gives you full control over the data on your computer and ensures that JHV always has access to the highest resolution, even if the server connection might be slow.

GitatHub commented 3 months ago

Yes, it's rare to need more than 8GB of cache, but I'm currently working on multiple events and have maxed out the fixed 8GB. Each time I reuse an image layer, I have to wait for the application to reload the images.

Hello, For your use case, you might also consider downloading the image layers as .jpx files by selecting the respective layer in the layer manager and clicking on the download icon. Personally, that's my preferred approach in this case because it gives you full control over the data on your computer and ensures that JHV always has access to the highest resolution, even if the server connection might be slow.

So will JHV automatically check and read the downloaded jp2 images in ~/JHelioviewer-SWHV/Downloads before fetching data from the server? That sounds good. Thanks.

bogdanni commented 3 months ago

Not automatically, they have to be loaded manually.

For example, data referred to by state files is requested only from the server, thus indeed this JPIP (streaming) cache reduces a lot the network traffic and waiting times.

GitatHub commented 3 months ago

Not automatically, they have to be loaded manually.

For example, data referred to by state files is requested only from the server, thus indeed this JPIP (streaming) cache reduces a lot the network traffic and waiting times.

I see. A user can load these images from "Open Image Layer" in "File" menu. Thanks.