search

elixir-explorer / explorer

Series (one-dimensional) and dataframes (two-dimensional) for fast and elegant data exploration in Elixir
https://hexdocs.pm/explorer
MIT License
1.12k stars 121 forks source link

caching rust deps locally #67

Closed dustinfarris closed 2 years ago

dustinfarris commented 3 years ago

Not sure if this is possible, but could you suggest a way to keep from having to recompile polars everytime we update to the latest commit here?

philss commented 3 years ago

@dustinfarris this is not supported right now, but you can disable the compilation of the Polars by adding the option skip_compilation?: true to the Explorer.PolarsBackend.Native module, where Rustler is configured. Reference for these options can be found in Rustler's doc.

Just make sure that the file priv/native/libexplorer.* exists. This file is generated when Polars is compiled with Rustler and the extension can changed based on your OS.

We are researching a way to use precompiled Polars. Stay tuned :)

dustinfarris commented 2 years ago

@cigrainger was this resolved?

cigrainger commented 2 years ago

@dustinfarris precompiled Polars will be available soon. I was closing as I thought @philss answer solved the specific issue -- if that's not right I'm happy to reopen.

philss commented 2 years ago

Just to give an update: we are waiting for this PR to get merged (or we publish that work as a separated package): https://github.com/rusterlium/rustler/pull/419

I can ping you here when we have the feature :)

dustinfarris commented 2 years ago

@dustinfarris precompiled Polars will be available soon. I was closing as I thought @philss answer solved the specific issue -- if that's not right I'm happy to reopen.

ok got it thanks for clarifying — i'll keep watching the repo for an update on precompiled.

fyi for anyone else watching this, we went with a slightly different workaround:

config :explorer, Explorer.PolarsBackend.Native,
  # By default, Polars compiles inside a build directry in deps/explorer.
  # Fully compiling Polars can take well over 15 minutes. This config moves the
  # location outside of deps, so that Polars does not have to fully re-compile
  # whenever deps is cleaned or re-created.  To force Polars to recompile, just
  # remove _explorer from your project root.
  target_dir: Path.expand("_explorer/build")

which has been working fairly well in CI (except when GitHub Actions randomly busts our cache for no reason)