search

skrub-data / skrub

Prepping tables for machine learning
https://skrub-data.org/
BSD 3-Clause "New" or "Revised" License
1.21k stars 97 forks source link

Adding a transformer to sessionize a table #750

Open jeromedockes opened 1 year ago

jeromedockes commented 1 year ago

Problem Description

In a table representing events with timestamps, it is often useful to detect sessions -- groups of events that occurred close to each other.

For example, a few visits to the hospital a few days apart are likely to be exams and follow-ups related to the same medical event. Or we may want to detect when a user walks away from an app, website, service etc and later comes back to it. Then features like number of sessions, session length etc can be extracted.

Examples from Ibis, polars and SODA using logs from a video game:

https://ibis-project.org/how-to/timeseries/sessionize https://www.pola.rs/posts/the-expressions-api-in-polars-is-amazing/ https://github.com/soda-inria/survival-analysis-benchmark/blob/main/notebooks/tutorial_part_2.py

Would it be in the scope of skrub to have a transformer that adds a "session id" column to a dataframe, and maybe extracts some features such as session length?

Feature Description

transform a table by detecting sessions and adding related information

Alternative Solutions

No response

Additional Context

No response

koaning commented 8 months ago

I wouldn't mind working on this if folks agree. This session stuff is the the prime villain in any story about data leakage in ML pipelines.

GaelVaroquaux commented 8 months ago

I wouldn't mind working on this if folks agree. This session stuff is the the prime villain in any story about data leakage in ML pipelines.

Awesome.

However, I would delay working on this, as with @JeromeDockes we are in the middle of a huge refactor of skrub.

We need to put notes of this discussion on the skrub repo

jeromedockes commented 8 months ago

@koaning part of the changes Gaël mentions is being worked on in #877 Feel free to provide feedback and advice!

koaning commented 5 months ago

One observation since responding to this all those months ago ...

Do we generally want to add sessions to the dataset before calling train/test split? I think we might! The session may be what we need in order to split events into groups which define the train/test splits. If that's the case ... do we want to have a pipeline that runs before we split train/test?

GaelVaroquaux commented 5 months ago

do we want to have a pipeline that runs before we split train/test?

At some point we will have to address this, and I think that it means that at some point the "Recipe" should know how to split data.

In the short term, I would rather not move forward with showing explicitly a pipeline before the split, as I worry about leaks.