thephpleague / flysystem

Abstraction for local and remote filesystems
https://flysystem.thephpleague.com
MIT License
13.36k stars 827 forks source link

Memory exhausted because of the flysystem cache #1539

Closed tristanbes closed 2 years ago

tristanbes commented 2 years ago

Bug Report

Q A
BC Break no
Version v1

Summary

Hello, We had an incident where the application ended up in error 500 because of memory exhausted.

We narrowed down the issue to the flysystem cache using Redis: flysystem pushed so many items in this single redis key that it took a lot of memory for PHP to retrieve this object and json_decode it resulting in a memory exhaustion

We have a 196mo memory limit on the prod. Querying that redis key to check for values made our Redis GUI client crash (TablePlus)

Workaround

I'm not looking for a flysystem v1 fix. The immediate fix was to empty this redis entry.

Question

I'm looking to know if this is a behaviour you're aware of and if this has been fixed in v2 or v3 by implementing a cache eviction / cache limit

Example of a value stored inside the cache:

[{"article\/main-picture\/62e2a0f310b0e179632735.png":false,"article\/main-picture":{"path":"article\/main-picture","dirname":"article","basename":"main-picture","filename":"main-picture","type":"dir"},"article":{"path":"article","dirname":"","basename":"article","filename":"article","type":"dir"},"article\/main-picture\/6305f58d4cefa539118766.png":{"path":"article\/main-picture\/6305f58d4cefa539118766.png","dirname":"article\/main-picture","basename":"6305f58d4cefa539118766.png","extension":"png","filename":"6305f58d4cefa539118766","size":216102,"mimetype":"image\/png","type":"file"},"article\/main-picture\/5d526e14bf820589871494.jpg":false,"article\/main-picture\/6305f5a52c514266851611.jpg":{"path":"article\/main-picture\/6305f5a52c514266851611.jpg","dirname":"article\/main-picture","basename":"6305f5a52c514266851611.jpg","extension":"jpg","filename":"6305f5a52c514266851611","size":63520,"mimetype":"image\/jpeg","type":"file"},"sitemap\/en\/articles.cat.news":{"path":"sitemap\/en\/articles.cat.news","dirname":"sitemap\/en","basename":"articles.cat.news","extension":"news","filename":"articles.cat","timestamp":1661334431,"size":1291,"mimetype":"text\/xml","type":"file"},"sitemap\/en":{"path":"sitemap\/en","dirname":"sitemap","basename":"en","filename":"en","type":"dir"},"sitemap":{"path":"sitemap","dirname":"","basename":"sitemap","filename":"sitemap","type":"dir"},"sitemap\/es\/articles.cat.news":{"path":"sitemap\/es\/articles.cat.news","dirname":"sitemap\/es","basename":"articles.cat.news","extension":"news","filename":"articles.cat","timestamp":1661299350,"size":2333,"mimetype":"text\/xml","type":"file"},"sitemap\/es":{"path":"sitemap\/es","dirname":"sitemap","basename":"es","filename":"es","type":"dir"},"sitemap\/it\/articles.dog-cat.news":{"path":"sitemap\/it\/articles.dog-cat.news","dirname":"sitemap\/it","basename":"articles.dog-cat.news","extension":"news","filename":"articles.dog-cat","timestamp":1661299355,"size":2348,"mimetype":"text\/xml","type":"file"},"sitemap\/it":{"path":"sitemap\/it","dirname":"sitemap","basename":"it","filename":"it","type":"dir"},"sitemap\/es\/articles.dog-cat.news":{"path":"sitemap\/es\/articles.dog-cat.news","dirname":"sitemap\/es","basename":"articles.dog-cat.news","extension":"news","filename":"articles.dog-cat","timestamp":1661212944,"size":2473,"mimetype":"text\/xml","type":"file"},"sitemap\/pl\/googleNews":{"path":"sitemap\/pl\/googleNews","dirname":"sitemap\/pl","basename":"googleNews","filename":"googleNews","timestamp":1661334466,"size":10319,"mimetype":"text\/xml","type":"file"},"sitemap\/pl":{"path":"sitemap\/pl","dirname":"sitemap","basename":"pl","filename":"pl","type":"dir"}},[]]
frankdejonge commented 2 years ago

V2 and V3 do not have this cache anymore and V1 is in security fix only mode.