Citi/parfun

Lightweight parallelisation library for Python.

Parfun is a lightweight library providing helpers to make it easy to write and run a Python function in parallel and distributed systems.

The main feature of the library is its @parfun decorator that transparently executes standard Python functions following the map-reduce pattern:

from parfun import parfun
from parfun.combine.collection import list_concat
from parfun.partition.api import per_argument
from parfun.partition.collection import list_by_chunk

@parfun(
    split=per_argument(
        values=list_by_chunk
    ),
    combine_with=list_concat,
)
def list_pow(values: List[float], factor: float) -> List[float]:
    return [v**factor for v in values]

Features

Provides significant speedups to existing Python functions
Does not require any deep knowledge of parallel or distributed computing systems
Automatically estimates the optimal sub-task splitting (the partition size)
Automatically handles data transmission, caching and synchronization.
Supports various distributed computing backends, including Python's multiprocessing, Scaler or Dask.

Benchmarks

Parfun efficiently parallelizes short-duration functions.

When running a short 0.28-second ML function on an AMD Epyc 7313 16-Cores Processor, Parfun provides an impressive 7.4x speedup. Source code for this experiment here.

Benchmark Results

Quick Start

The built-in Sphinx documentation contains detailed usage instructions, implementation details, and an exhaustive API reference.

Use the doc Make target to build the HTML documentation from the source code:

make doc

The documentation's main page can then ben found at docs/build/html/index.html.

Take a look at our documentation's quickstart tutorial to get more examples and a deeper overview of the library.

Contributing

Your contributions are at the core of making this a true open source project. Any contributions you make are greatly appreciated.

We welcome you to:

Fix typos or touch up documentation
Share your opinions on existing issues
Help expand and improve our library by opening a new issue

Please review our community contribution guidelines and functional contribution guidelines to get started 👍.

Code of Conduct

We are committed to making open source an enjoyable and respectful experience for our community. See CODE_OF_CONDUCT for more information.

License

This project is distributed under the Apache-2.0 License. See LICENSE for more information.

Contact

If you have a query or require support with this project, raise an issue. Otherwise, reach out to opensource@citi.com.

Citi / parfun

readme