search

elixir-explorer / explorer

Series (one-dimensional) and dataframes (two-dimensional) for fast and elegant data exploration in Elixir
https://hexdocs.pm/explorer
MIT License
1.12k stars 123 forks source link

Improve performance of "concat_columns" #938

Closed philss closed 4 months ago

philss commented 4 months ago

This PR changes the implementation of "concat_columns" for lazy and eager dataframes.

Before this we were doing a join of a temporary column which consumes a lot of memory and is slower than concatenating each column of the DFs.

There is a small change in behaviour for lazy frames, causing frames of different sizes to be merged with nil values for the smaller columns.

Closes #937