The Open Data Platform for your community Open Data
Datadex is a fully open-source, serverless, and local-first Data Platform that improves how communities collaborate on Open Data. Datadex is not a new tool, it is a pattern showing an opinionated bridge between existing ones.
[!TIP] Datasets generated by this project are ready to explore and consume at HuggingFace.
Learn more about the approach in this post or check other real-world production implementations of the Datadex pattern working in the following repositories:
git
. Improves data access and empowers data scientists to conduct research and helps to guide community-driven analysis and decisions. Version your data as code! Publish and share your reusable models for others to build on top. Datasets should be both reproducible and accessible!Python
and SQL
.There are two ways to get started with Datadex or your own Open Data Portal/Platform based on the Datadex pattern, Python Virtual Environment or Docker / Dev Containers.
If you hit any issue, please open an issue!
You can install all the dependencies inside a Python virtual environment by running make setup
. To do that, clone the repository and run the following commands from the root folder.
make setup
Alternatively, you can rely on your system's Python installation to create a virtual environment and install the dependencies.
# Create a virtual environment
python3 -m venv .venv
source .venv/bin/activate
# Install the package and dependencies
pip install -e ".[dev]"
Now, you should be able to spin up Dagster UI (make dev
or dagster dev
) and access it locally.
Using VSCode Remote Containers is the fastest and smoother way to start using Datadex, but requires you to have Docker running. Open the project in VSCode and click on the bottom right corner to open the project in a container.
Once inside the develpment environment, you'll only need to run make dev
to spin up the Dagster UI locally. You'll also have a few extra extensions installed and configured to work with the project.
The development environment can also run in your browser thanks to GitHub Codespaces!
This project started after thinking how an Open Data Protocol could look like!
data
projects.