Open teh-cmc opened 1 year ago
Thanks for the PR. This hits a lot of code so I want to tune in @jorgecarleitao as well.
In Rerun, we've seen up to 50% reduced memory requirements in some use cases with this PR.
Could you elaborate a bit on this? What did you benchmark? How was the data type such a huge bottleneck?
What is the new data type size and what was the old one?
Somewhat related, In polars we use smallstring
which might also be an interesting route.
Attention: 16 lines
in your changes are missing coverage. Please review.
Comparison is base (
3ddc6a1
) 83.39% compared to head (3c3c6ed
) 83.96%.:exclamation: Current head 3c3c6ed differs from pull request most recent head 40541b4. Consider uploading reports for the commit 40541b4 to get more accurate results
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
Any chance to get this merged? We do a lot of cloning of datatypes, and the memory use adds up extremely quickly. Using Arc
reduces the memory footprint tremendously.
Fixes #439
The entire PR pretty much comes down to this diff:
everything else is just a lot of grunt work and pain to accommodate for these new types.
As mentioned in https://github.com/jorgecarleitao/arrow2/issues/439#issuecomment-1506984699: I went for the path of least resistance, so this isn't optimal, but it is already quite the improvement.
I have branches ready for
arrow2_convert
,polars
andrerun
. In Rerun, we've seen up to 50% reduced memory requirements in some use cases with this PR.