search

owid / etl

A compute graph for loading and transforming OWID's data
https://docs.owid.io/projects/etl
MIT License
71 stars 20 forks source link

Update frame repacking to use new pyarrow data types #2470

Open larsyencken opened 5 months ago

larsyencken commented 5 months ago

Context

We do frame repacking to make our data frames smaller on disk and faster to work with. It has some slight annoyances, such as making more variables categorical, which complicates group-bys, for example.

What

With Pandas 2.2, we have the option to change our frame repacking to use new pyarrow data types, which are supposed to be much more efficient.

That would also bring our data catalog into compatibility with more of the data ecosystem (e.g. Polars, Nushell and friends).

larsyencken commented 5 months ago

Blocked on:

larsyencken commented 4 months ago

From discussion: we could also consider a gradual migration, e.g. enabling pyarrow types for new datasets rather than applying it over everything.

We could also bench the performance gains to see if they're worth it.

larsyencken commented 4 months ago

Pablo noted that adding commitments to our future selves can mean that "small" data updates can expand. We should try to make more things as automatic as possible.

stale[bot] commented 2 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Marigold commented 2 months ago

Don't close it. We should at least compare the performance (CPU & mem) of current repacking vs new pyarrow dtypes.

Marigold commented 1 month ago

(While looking into this and trying to optimize reading the table, I discovered that loading is actually really fast (<1s), but setting primary key as an index is slow.)