search

Krukov / cashews

Cache with async power
MIT License
421 stars 28 forks source link

_empty and _empty_in_redis marker objects leak #235

Closed robert-schmidtke closed 5 months ago

robert-schmidtke commented 5 months ago

Hi,

I'm heavily using the Redis client-side backend, and every now and then I see that get returns an object or _EmptyCopy. I cannot reproduce this locally, it only occurs in my service (a FastAPI application), which is handling thousands of requests per second. I will try and build a minimal example to reproduce, but wanted to flag this up early, just in case anything comes to mind where that could occur.

Looking at the code, the only possible scenario I could imagine is where any of the value is _empty_in_redis check fails for any reason. Maybe because the value got pickled/unpickled somehow and then the is check fails? (Even though the _local_cache is a Memory which uses the NonPickler by default).

I don't think it's possible that within the same process, the _empty_in_redis marker object could get created twice (e.g. in different threads) and then treated as different.

A workaround would be to make the marker objects slightly smarter when checking __eq__ between them, but for this would be good to understand the underlying issue first.

I'll be adding to this ticket as I gather more information.

robert-schmidtke commented 5 months ago

I believe this is an issue on our side where we have been messing with with registered picklers, causing the _empty and _empty_in_redis objects to be pickled and unpickled, making the identity checks fail.

We will revisit and reopen if needed, but pretty sure this is on us -- sorry!