search

MagicStack / immutables

A high-performance immutable mapping type for Python.
Other
1.14k stars 57 forks source link

immutables._map.MapMutation is not iterable / does not implement items() #55

Open atzannes opened 3 years ago

atzannes commented 3 years ago

I am wondering if this is by design or because it hasn't been useful up till now.

My use-case is implementing a middleware type of cache. The cache subscribes to some external data that gets updated and tracks it, and it allows clients to get an immutable view of the cache. The cache can be a nested immutables.Map dictionary. What works is to call mutate and finish for each new piece of data that comes in (N times each if the updated key is at depth N), but this can be too slow when the cache receives a large batch of updates. Using a profiler, I see my code spending ~35% of the time processing the incoming data to be added to the cache and ~65% of the time in the mutate/finish calls.

What I would like to implement is a lazy-finish protocol that calls mutate on updates as needed but not finish. Instead, we traverse the cache and make any needed finish calls when the client requests a view into the cache. What is preventing me from realizing this implementation (I think) is that the immutables._map.MapMutation objects that are returned by the un-finished mutate calls, are not iterable and they don't implement items(), but I need to finish the inner MapMutation objects before I finish the outer ones.

As a nasty hack, I can finish() an outer MapMutation to get its keys and use them to find and recurse into any inner MapMutation values to finish those, before I call finish() a second time on the outer MapMutation to really finish it this time.

1st1 commented 3 years ago

I am wondering if this is by design or because it hasn't been useful up till now.

It's by design to keep everything simple. Mutation objects have the same internal structure as immutable objects, and enjoy the same iterator implementation. And that implementation assumes that the underlying tree is static and cannot change while iterating. That said, it's certainly possible to add the necessary code to error out the iteration if there was a mutation during it -- feel free to open a PR for that.

atzannes commented 3 years ago

I'll try to take a cut at this next month when I have a bit more time. Just to make sure I understood correctly, you are referring to the RuntimeError: dictionary changed size during iteration error, right? (As opposed to mutations that only modify values.)

atzannes commented 3 years ago

Great, thank you for the feedback. I will certainly keep this as a future weekend project. Unfortunately, the need is not pressing to justify spending work time on it, and life just got busy, so it won't be soon, but I am not dropping the ball (just yet)