Dataframes

[johari]

CSV and the relational model are examples of dataframes.

Spreadshetes

Dataframes in spreadsheets facilitate

• New kinds of operations on tabular data (ones you see in relational algebra)
• A convenient way to consolidate data from multiple sources (an important and time consuming task)
• Might facilitate relational programming (#71)

Dataframes and spreadsheets

Is there any paper or article that hints to first-class support for dataframes inside cells?

Related work

Streaming dataframes

Some data never stops. It arrives continuously in a constant, never-ending stream. This happens in financial time series, web server logs, scientific instruments, IoT telemetry, and more. Algorithms to handle this data are slightly different from what you find in libraries like NumPy and Pandas, which assume that they know all of the data up-front. It’s still possible to use NumPy and Pandas, but you need to combine them with some cleverness and keep enough intermediate data around to compute marginal updates when new data comes in. (https://matthewrocklin.com/blog/work/2017/10/16/streaming-dataframes-1)

Apache Spark

• https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#creating-streaming-dataframes-and-streaming-datasets

DataSpread

Dataspread combines the intuitiveness and flexibility of spreadsheets and the scalability and power of databases (http://dataspread.github.io/)

Apache arrow

• https://arrow.apache.org/

cuDF

• https://github.com/rapidsai/cudf

External links

• https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html

[johari]

Closing this issue for now as dataframes are not the main focus for now. I'm still curious about what would be a set of good core primitives to manipulate data frames. Perhaps it'll be a set of formulas that will mimic basic SQL queries and/or pandas API.