prefect-kv

Welcome!

A simple key-value store for use with Prefect.

Getting Started

Python setup

Requires an installation of Python 3.7+.

We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.

These tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to the Prefect documentation.

Installation

Install prefect-kv with pip:

pip install prefect-kv

Using a key-value store

The entire store is saved to Prefect's API each time a key is updated. Each time a key is read, the entire store is loaded from Prefect's API. This exposes a simple way to persist and retrieve values across flow runs and tasks, independently of your dataflow.

from prefect_kv import KVStore

# Create a new KV store named 'demo-store'
store = KVStore(name="demo-store")

# Set a value in the store
store.set("foo", "test")

# Get a value from the store
print(store.get("foo"))

# Get a value from the store with a default
print(store.get("does-not-exist, "that's okay!"))

# Set a value in the store using item syntax
store["bar"] = "another test"

# Get a value from the store using item syntax
print(store["bar"])

# View the whole store
print(f"Here's the whole thing: {store.dict()}")

If you exit the process and create a new store object with same name, you will still be able to access your values:

from prefect_kv import KVStore

# Get the existing KV store named 'demo-store'
store = KVStore(name="demo-store")

print(store["foo"])
print(store["bar"])

Resources

If you encounter any bugs while using prefect-kv, feel free to open an issue in the prefect-kv repository.

If you have any questions or issues while using prefect-kv, you can find help in either the Prefect Discourse forum or the Prefect Slack community.

Feel free to ⭐️ or watch prefect-kv for updates too!

Development

If you'd like to install a version of prefect-kv for development, clone the repository and perform an editable install with pip:

git clone https://github.com/madkinsz/prefect-kv.git

cd prefect-kv/

pip install -e ".[dev]"

# Install linting pre-commit hooks
pre-commit install