Nicer API for writing a `Backend` - Githubissues

utdemir / distributed-dataset

A distributed data processing framework in Haskell.

BSD 3-Clause "New" or "Revised" License

114 stars 5 forks source link

Nicer API for writing a `Backend` #6

Open utdemir opened 5 years ago

utdemir commented 5 years ago

Currently the API to write a Backend is quite limited, it forces us to create a new process, and does not look pretty. With a nicer API:

We can use #5 to get rid of the Python wrapper we're currently using
We can create a backend which uses threads and an in memory shuffle store, it should be considerably faster than localProcessBackend and localTmpShuffleStore.