elixir-explorer / explorer

Series (one-dimensional) and dataframes (two-dimensional) for fast and elegant data exploration in Elixir
https://hexdocs.pm/explorer
MIT License
1.12k stars 123 forks source link

Add native expression for Series.split/2 #875

Closed H12 closed 8 months ago

H12 commented 8 months ago

Fixes #874

Let me know if I missed anything!

H12 commented 8 months ago

Here are the results of testing this in iex:

Mix.install([{:explorer, github: "H12/explorer"}])
# => :ok

require Explorer.DataFrame, as: DF
# => Explorer.DataFrame

df = DF.new(a: ["foo,bar", "bar,baz"])
# => Explorer.DataFrame<
#      Polars[2 x 1]
#      a string ["foo,bar", "bar,baz"]
#    >

DF.mutate(df, b: split(a, ","))
# => Explorer.DataFrame<
#      Polars[2 x 2]
#      a string ["foo,bar", "bar,baz"]
#      b list[string] [["foo", "bar"], ["bar", "baz"]]
#    >

Curiously, this does not work in Livebook -- something to do with mismatched rustler versions between explorer and kino_explorer. Bumping the rustler & explorer versions in kino_explorer then results in a compiler failure w/ this error:

== Compilation error in file lib/explorer/polars_backend/native.ex ==
** (ErlangError) Erlang error: :enoent
    (elixir 1.16.0) lib/system.ex:1101: System.cmd("cargo", ["metadata", "--format-version=1"], [cd: "native/explorer"])
    (rustler 0.31.0) lib/rustler/compiler/config.ex:91: Rustler.Compiler.Config.metadata!/1
    (rustler 0.31.0) lib/rustler/compiler/config.ex:78: Rustler.Compiler.Config.build/1
    (rustler 0.31.0) lib/rustler/compiler.ex:8: Rustler.Compiler.compile_crate/3
    lib/explorer/polars_backend/native.ex:32: (module)
could not compile dependency :explorer, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile explorer --force", update it with "mix deps.update explorer" or clean it with "mix deps.clean explorer"

For the record, that output was on the latest nightly build of the native MacOS distribution & the following dependencies:

Mix.install([
  {:kino, "~> 0.12.3"},
  {:kino_explorer, github: "H12/kino_explorer"}
])

I presume this is an issue w/ outdated/missing dependencies in Livebook/Kino that will get resolved eventually, but I'd also be happy to help figure that out too if it's not already being worked on. Wouldn't quite know where to start tho 😅

josevalim commented 8 months ago

:green_heart: :blue_heart: :purple_heart: :yellow_heart: :heart: