Pandata: The scalable open-source analysis stack

Pandata is a fully open source, high-performance, modern Python data-analytics stack usable in any scientific, engineering, or analytical domain.

Are you limited by your domain’s software stack?

Every scientific or engineering discipline has its own computing needs. Many such disciplines have developed entirely separate sets of tools for:

• Storing data
• Reading data
• Processing data
• Plotting data
• Analyzing data
• Modeling data
• Exploring data

These stacks are largely tied to outdated architectures and assumptions:

• Not cloud or remote friendly: Tied to a local desktop GUI or OS
• Not scalable: Tied to a single processor (CPU), whether for technical or licensing reasons
• Not general purpose: Narrow audience, maintained by few, and unknown outside the field

It’s all just data – time for a better way!

Pandata: the scalable open-source analysis stack

Instead of your outdated stack, use modern Python data-science tools that are:

• Domain independent: Maintained, used, and tested by many people all across the world
• Efficient: Run at machine-code speeds using vectorized data or JIT compilation
• Scalable: Run on anything from a single-core laptop to a thousand-node cluster
• Cloud friendly: Fully usable for local or remote compute using data on any file storage system
• Multi-architecture: Runs on your desktop and on Mac/Windows/Linux CPUs and GPUs
• Scriptable: Run in batch mode for parameter searches and unattended operation
• Compositional: Select which tools you need and put them together to solve your problem
• Visualizable: Support rendering even the largest datasets without conversion or approximation
• Interactive: Support fully interactive exploration, not just rendering static images or text files
• Shareable: Deployable as web apps for use by anyone anywhere
• OSS: Free, open, and ready for research or commercial use, without restrictive licensing

But I don’t do data science, you say? You do! Data science is what’s shared across lots of disciplines; it’s not just for AI and ML (though it supports those well too!)

What is Pandata and why do I need it?

Pandata is just a name for a specific collection of Python libraries maintained separately by different people. Pandata libraries are designed to work well with each other to achieve the goals listed above (being scalable, interactive, etc.). You don't need Pandata for anything other than to know which libraries are designed to work well together in this way. Just use any library from Pandata and be happy, knowing that if you need something covered by one of the other libraries, you can use them together without jeopardizing scalability, interactivity, and so on.

Who runs Pandata?

Pandata is just this informational website set up by the authors of some of the Pandata tools; there's no management or policies or software development specifically associated with Pandata. But if you have questions or ideas about what to do with Pandata, feel free to open an issue on our Github repo for discussion!

Examples

There are lots of examples online of applying Pandata libraries to solve problems, including:

• Pangeo: JupyterHub, Dask, Xarray for climate science; Pandata is Pangeo but for any field!
• Project Pythia: Geoscience-specific examples, many of which use Pandata tools
• Attractors: Processing huge datasets with Numba, rendering with Datashader
• Census: Reading chunked data from Parquet, rendering with Dask + Datashader
• Ship traffic: Rendering spatially indexed data with interactive lookup
• Landsat: Intake data catalog, xarray n-D data, hvPlot+Bokeh plotting, Dask ML
• Minian: Jupyter, Dask, and HoloViews for neuroscience

See the Pandata paper from SciPy 2023 for all the details, and then just download and use any of the packages in Pandata in any combination and enjoy having all this power at your fingertips!