search

nginx-modules / ngx_cache_purge

nginx module which support to purge ngx_http_(fastcgi|proxy|scgi|uwsgi)_module cache backend
Other
208 stars 33 forks source link

/purge/* will cause 100% of disk io usage #32

Open cdliaozhi opened 3 years ago

cdliaozhi commented 3 years ago

I use Azure VM with nginx as the reverse proxy. And add a 64g Premium SSD as cache disk. When I use /purge/* to clear the cache, the disk IO exceeds 100%. image

denji commented 3 years ago

Configuration file you have may be affected by other nginx settings like thread async i/o, cache lock. What is the number of objects to be deleted (cache folder)? This module has no barrier or throttling.

cdliaozhi commented 3 years ago

My OS is FreeBSD 12.0. No async i/o. No matter I set proxy_cache_lock on or off, the same failure will occur. Even if I purge a new site, just 28 pages.

cdliaozhi commented 3 years ago

I found that when I purge with wildcard, it will traverse all my cache files. My cache files is very huge, so it cause 100% of disk io usage.

Denis Denisov notifications@github.com 于2020年9月23日周三 上午12:52写道：

Configuration file you have may be affected by other nginx settings like async i/o. What is the number of objects to be deleted (cache folder)?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/nginx-modules/ngx_cache_purge/issues/32#issuecomment-696846912, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQ4GSITOPFU4HADPQPKSAZTSHDI6VANCNFSM4RVN4WBA .

midorinet commented 3 years ago

I'm having the same issue here.. any workaround for big cache size like > 300GB ?

cdliaozhi commented 3 years ago

Just about 10GB.

midorinet notifications@github.com 于2020年9月29日周二 上午11:08写道：

I'm having the same issue here.. any workaround for big cache size like > 300GB ?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/nginx-modules/ngx_cache_purge/issues/32#issuecomment-700398564, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQ4GSIVTEZTEBVPTDQGCWW3SIFFSNANCNFSM4RVN4WBA .

ennorehling commented 3 years ago

Earlier today, I explained why this happens, here: https://github.com/FRiCKLE/ngx_cache_purge/issues/55#issuecomment-777272264 It's a fundamental problem with the way the plugin decides what is matched by a partial cache_key. There's apparently no easy way to find the cache_key for a given cache node, so instead it opens every file in the cache and reads the cache_key from that (it's stored as a string directly after the header). This is obviously going to hurt the disk. IMHO, this plugin is fundamentally flawed, and this isn't even the only problem with it.

pei-jikui commented 3 years ago

delete the file from disk in a worker process is REALLY NOT a good idea.