Closed GodTamIt closed 7 months ago
I think it's better to provide a default limited chunk size rather than using None as default--doesn't seem like a default behaviour.
Iter_chunk still seems significantly slower, although the chunk size is pretty large (25000).
Iter_chunk still seems significantly slower, although the chunk size is pretty large (25000).
That's because it's doing 4x more work for less than 4x time
I think it's better to provide a default limited chunk size rather than using None as default--doesn't seem like a default behaviour.
This is now done.
Iter_chunk still seems significantly slower, although the chunk size is pretty large (25000).
@Congyuwang iter chunk is faster. It's probably more helpful to look at the it/s
metric than the total seconds because the number of items used to be different between single-thread and multi-thread. I've updated the benchmark to not do this.
I'm comparing iter vs. iter_chunk. Not multithreaded vs. single threaded. Seems that based on the previous benchmark, multithreaded iter chunk is slower than multithreaded iter where GIL is not released.
Looks like we have some kind of deadlock.
This allows callers to get chunks at a time from the iterator and in such scenarios, it makes sense to drop the GIL.
Dropping the GIL is not only useful for scenarios when there are multiple iterators but when code may be calling other GIL-dropping code.