codspeed-hq[bot]
commented
2 weeks ago CodSpeed Performance Report
Merging #3180 will improve performances by ×2.2
Comparing colin/swordfish-mono-id (509e645) with main (84db665)
Summary
⚡ 2 improvements
✅ 15 untouched benchmarks
Benchmarks breakdown
| Benchmark | main |
colin/swordfish-mono-id |
Change | |
|---|---|---|---|---|
| ⚡ | test_iter_rows_first_row[100 Small Files] |
421.4 ms | 375 ms | +12.36% |
| ⚡ | test_show[100 Small Files] |
32.7 ms | 14.9 ms | ×2.2 |
codecov[bot]
Implements monotonically increasing id as a streaming sink with
max_concurrency = 1.I tested multithreaded and single threaded implementations and found that there was no performance gain in multithreaded. This is because monotonically increasing id is a memory bound operator, all it does is allocate an array of ints for the id. Multiple threads trying to do this in parallel are bottlenecked by memory bandwidth.
It is actually also much simpler to implement this as a single threaded operation, as we just need to keep a running count of the lengths of morsels seen so far. This is effectively just
row_number.Note:
pyfunc_into_table_iterfunction, which consumes python iterators in scan tasks (used in read_lance and read_generator), where the consumer only callsnext()on the iterator once. This PR fixes that.