ENH: Replace pandas dependence/use with narwhals

cluhmann commented 1 month ago

Before

No response

After

No response

Context for the issue:

With the rise in popularity of packages such as polars, arrow, and others PyMC's dependence on pandas is looking less and less universally useful. Narwhals was developed for developers of python libraries that consumes dataframes who wish wishing to make their libraries completely dataframe-agnostic. So maybe we consider using that instead of pandas per se? Narwhals has no dependencies and has negligible overhead, so it seems relatively lightweight.

It will require some refactoring as it relies on (as subset of) the polars API.

ricardoV94 commented 1 month ago

Last time I checked we don't really depend on pandas, it's arviz that does

cluhmann commented 1 month ago

Does that mean that we can use polars dataframes when using PyMC?

ricardoV94 commented 1 month ago

Does that mean that we can use polars dataframes when using PyMC

As data? Not sure, there's some special logic for handling pandas.

But PyMC does not depend on pandas, so maybe you are requesting a new feature, not a change of dependency

ricardoV94 commented 1 month ago

Btw pandas special logic is:

Dispatch so pt.as_tensor accepts pandas series/matrices

I don't think we can replace that by narwhals since dispatch works on types at runtime. We would need to dispatch on polars as well.

Special logic in pm.data? and observed for detecting nans and triggering automatic imputation. This could possibly be backend agnostic although that code is pretty spaghetti, so someone would need to check.

2.1 It may actually already work because IIRC it's all based on duck typing

fonnesbeck commented 4 weeks ago

Yeah, was taking a peek at this today -- if we were using pandas-specific functionality (merging, etc.) then it would make sense to use narwhals. For the post part we are taking DataFrames (and Series) and turning them into ndarrays. The only exception may be in deriving dims from indexes, since polars does not use indexes.

MarcoGorelli commented 4 weeks ago

Hey, thanks for looking into Narwhals 🙏

First, thanks for opening https://github.com/pymc-devs/pymc/pull/7463, it's great to see Polars support come along - facilitating that was one of my goals with Narwhals, and if it can happen even without it, even better 💪

I think #7463 is already a net-positive, I just wanted to leave some comments, in case they're of interest:

pandas is currently a required a dependency of PyMC, and so is ArviZ. Would you consider making them both optional dependencies?
regarding the dispatch, I think Narwhals could simplify it: you call nw.from_native(..., eager_only=True) on the input, pass that to dataframe_to_tensor_variable, and then just do:
```
@_as_tensor_variable.register(nw.DataFrame)
@_as_tensor_variable.register(nw.Series)
def dataframe_to_tensor_variable(df: , *args, **kwargs) -> TensorVariable:
    return pt.as_tensor_variable(df.to_numpy(), *args, **kwargs)
```
to_numpy is part of the Narwhals API, and nw.from_native is a practically-free operation. But it can of course also be done without Narwhals, as you've done 🙌
if you need to do something pandas-specific, you can still do that without pandas being a required dependency:
```
if (pd := sys.modules.get('pandas', None)) is not None and isinstance(df, pd.DataFrame):
    # pandas-specific logic goes here
```
In fact, that's what Altair do, using is_pandas_dataframe

No hard feeling of course if you keep the current approach, I was wanted to point out the possibilities ♾️ All the best, and it was really fun meeting some of you at PyData London!

EDIT: upon further inspection, I was wrong about the pytensor part, Narwhals wouldn't help there (unless you used it in PyTensor too), I think it would only potentially help in convert_data?

pymc-devs / pymc