docs: Create and Populate How-To Guides

[p-a-a-a-trick]

Create+Populate How-To Guides

There is a lack of how to guides on the website. Let's make a list of them and start populating a useful section for code schematics.

I hope to keep this issue up as a rolling list of How to Guides, so if you stumble on this and your need isn't included in the docs or this list, then comment your request here!

v3.2

• [x] How to compute the top k records
• [x] How to register tables in a DuckDB backend #4386
• [x] How to ffill and bfill using window functions (source) #4402

v4.0

• [x] How to join a table expression to a pandas dataframe (merged) (docs PR)
• [x] (WIP) How to chain ibis expressions (_ API) (#4914)
• [x] How to sessionize a log of events based on the elapsed time between successive transactions for a give user identifier (#4843).
• [ ] How to convert column structures into columns (unpack)

v4.1+

• [ ] How to create a table with a user-defined schema and efficiently load a large CSV file into it (handling common annoying things like the choice of the separator and custom date / time formats).
• [ ] How to efficiently sample random rows from a large table (or expr) to quickly compute approximate descriptive statistics such as quantiles.
• [ ] How to perform an asof_join (source)
• [ ] How to interpolate dates (and/or other sequences)
• [ ] [maybe?] How to use Ibis to generate and save Substrait/SQL
• [ ] How to create a backend (incl. a section on SQLAlchemy backends specifically, but should focus on min requirements to get one up and running w/o SQLAlch)
• [ ] How to chain joins (unknown if this should even be a how-to guide, but if #4309 is a fact of life then we should add this)

Let me know if you have any ideas and I'll add them to the list.

[cpcloud]

I think the things in the wiki are useful bits of information to put in the how-tos.

[ogrisel]

Here are some more ideas:

• How to create a table with a user-defined schema and efficiently load a large CSV file into it (handling common annoying things like the choice of the separator (and maybe quotes) and custom date / time formats).

• How to sessionize a log of events based on the elapsed time between successive transactions for a give user identifier.

• How to efficiently sample random rows from a large table (or expr) to quickly compute approximate descriptive statistics such as quantiles.

For the first item about CSV loading, I don't know if there is a standard way to use the COPY statements of popular databases from ibis. Personally I used a raw_sql command with duckdb to be able to specify a custom date format in a recent experiment.

The second is about event-based data and should probably cross-reference #4402 (about ffil).

[ogrisel]

@p-a-a-a-trick I made a typo in the word "sessionize".

[p-a-a-a-trick]

@p-a-a-a-trick I made a typo in the word "sessionize".

Got it; thanks @ogrisel! I'll probably knock a few more of these out in the next couple of months. Let me know if you have any more ideas—I can't guarantee they'll get in before the next release but I hope to have most (if not all) of these in before then.

I hope to keep this issue as a rolling list of requested How To guides, not sure if it'll ever be closed.

[ogrisel]

For reference the 3 howto ideas above stem for my first-time experience with ibis and duckdb which is quite pleasant so far:

https://github.com/soda-inria/survival-analysis-benchmark/tree/main/datasets/kkbox_churn

[jcmkk3]

Below is a list of some resources that I think are good examples to better learn tools in the data ecosystem. Maybe some parts of these could be translated to use Ibis. Sometimes one of the hardest parts is to just figure out a good narrative flow or interesting example use case when writing tutorials.

• Tidy Tuesday Project Contains lots of interesting datasets. Both raw and "clean" data, along with the cleaning scripts that produced the clean versions.
• Tidy Tuesday Screencasts - David Robinson (video) Good examples of EDA workflows.
• Tidy Tuesday Screencasts - David Robinson (code)
• Intro to VisiData Good example of an interesting, but well paced introduction to the tool.
• Calm Code - Pandas Pipe
• Computational Notebooks - palewire Lots of good examples of real world data wrangling and analysis.
• First Python Notebook Another interesting, but well paced introduction to data tooling.
• R for Data Science A classic for a reason. Lots of good examples as well as ways to think about data analysis tasks.
• Reproducible Data Analysis in Jupyter A good example of a workflow for exploring data in python.
• Arquero notebook collection Some nice examples of interleaving a bit of a story and interesting tidbits while teaching a tool.

Many of these examples have other elements outside of just data manipulation like visualization or maybe some stats/machine learning. It could get a little bit tricky to choose which other libraries to feature, but I do think that it is useful to see how different tools interact.

[ogrisel]

Another good related post on how to sessionize data (using polars):

• https://www.pola.rs/posts/the-expressions-api-in-polars-is-amazing/

The dataset is good (not too big, not too small and realistic) and the multistep task is a good opportunity to show how to factorize code with functions chained via the .pipe method.

[NickCrews]

I'll throw in data import/export. I find guidance on this topic currently pretty limited in the docs, but this is a step that every single user is going to do. Here is a nice example from vaex. There are other good examples of How-tos there as well.

[lostmygithubaccount]

closing this per recent docs updates, how-tos available on the site: https://ibis-project.org/how-to/configure/basics