pandas nested data infeasible operations

[mroeschke]

pandas generally discourages having nested data in a DataFrame or Series. For nested data in pandas, I tend to group the types of nested data as:

1. Array-like (N-D)

   
   In [4]: nested_array_like = pd.Series([[1, 2], [2, 3]])

In [5]: nested_array_like Out[5]: 0 [1, 2] 1 [2, 3] dtype: object

The only behavior somewhat defined and tested for array-like (python `list` specifically) is addition which acts like an `append`

In [14]: nested_array_like + nested_array_like Out[14]: 0 [1, 2, 1, 2] 1 [2, 3, 2, 3] dtype: object

And there is `explode` which encourages users to `unnest` their data

In [15]: nested_array_like.explode() Out[15]: 0 1 0 2 1 2 1 3 dtype: object

Some operations I have seen tried by users with array-like data is:

* `groupby` the array-like values
* element-wise operations (e.g. add 2 to each element in the array)
* reduction-wise operations per array-like value (e.g. `sum` each array)
* indexing/selecting/slicing the array-like values
* containment operations (e.g. 2 in each array -> True/False)

2. Key-Value-like

In [6]: nested_kv_like = pd.Series([{1:2}, {2:3}])

In [7]: nested_kv_like Out[7]: 0 {1: 2} 1 {2: 3} dtype: object

The only behavior supported and tested for dict-like is `dict.get` via the `str` accessor (which is somewhat strange IMO)

In [13]: nested_kv_like.str.get(1) Out[13]: 0 2.0 1 NaN dtype: float64

A lot of the same operations described above I've seen users try with key-value-like data except specifically treating the `keys` or `values` as "arrays"

[martindurant]

These are certainly worthwhile processing models that we will chase. However, I was wondering if you knew of specific datasets or workflows that people were choosing not to process with python/pandas because it was too awkward or slow?

We are finding that nested/ragged data just doesn't show up a lot in python exactly because no one knows what to do with them - even though they are ubiquitous in the real world. We could probably do something interesting with the likes of https://www.kaggle.com/datasets/eliasdabbas/web-server-access-logs , for instance. We have the following specific cases in mind for examples:

• chicago taxis, which have details of the exact ride path as a sequence of lat/lon pairs
• million-songs, which has various analyses of the bars of various songs
• NYC building outline polygons
• scientific telemetry from floating buoys

Any other suggestions?

[martindurant]

https://pythonspeed.com/articles/json-memory-streaming/ a smallish example we can directly compare to; takes 23MB for ak in memory, but a very complicated typestring.

[mroeschke]

However, I was wondering if you knew of specific datasets or workflows that people were choosing not to process with python/pandas because it was too awkward or slow?

Ah I see. Sorry I am not too familiar of public-ish datasets/workflows for this case.