Closed larsrinn closed 1 year ago
Also, it might be of interest how I noticed that: My MacOS complained about my disk being full. When looking into it, it turned out the wetterdienst-cache-folder had a size of more than 500 GB.
I don't want to use the cache. But if you want to, I think a configurable maximum cache size with a reasonable default would be useful.
Dear @larsrinn ,
I added a first fix for the caching configuration although we will probably apply further changes regarding the current way we handle concurrency with settings.
As for the maximum cache size we are relying on the features implemented in fsspec and there it is currently not possible to define a maximum cache size unfortunately. I will however try to see if we can make our own implementation for this.
Dear Lars,
apologies that you experienced troubles with Wetterdienst cache storage.
Wetterdienst cache is enabled
At https://github.com/earthobservations/wetterdienst/issues/827#issuecomment-1355835985:
That log message was misleading, it was fixed with GH-828.
At https://github.com/earthobservations/wetterdienst/issues/827#issuecomment-1355822602:
Using both
Settings.cache_disable = True
, orenviron['WD_CACHE_DISABLE'] = 'True'
, to turn off caching, worked perfectly well for me. I was able to confirm that no directory has been created at either/Users/amo/Library/Caches/wetterdienst
(macOS) or/root/.cache/wetterdienst
(Linux/Docker), after running the program.
My MacOS complained about my disk being full. When looking into it, it turned out the wetterdienst-cache-folder had a size of more than 500 GB.
I am very sorry about that. As @gutzbenj outlined, there is apparently no option to limit the size of the cache. Regarding the WD_CACHE_DISABLE
setting, there may have been hiccups in the past. Can you confirm it works with recent versions of Wetterdienst?
With kind regards, Andreas.
After the latest settings refactoring, this now works as expected. Closing for now but we may come back if cache size can be limited in a future release of fsspec.
Is your feature request related to a problem? Please describe.
Running this
logs
Wetterdienst cache is enabled
Describe the solution you'd like
Log:
Wetterdienst cache is disabled
And also: I don't think logging a warning in both directions is appropriate. For us, this clutters our Sentry account with issues that aren't really issues. Wouldn't a log-level of info be sufficient?
Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.
Additional context
Output of
wetterdienst.info()