substantic / rain

Framework for large distributed pipelines
https://substantic.github.io/rain/docs/
MIT License
748 stars 54 forks source link

Rust subworker #54

Closed gavento closed 6 years ago

gavento commented 6 years ago

Locally tested rust subworker with many tests, some finished APIs, docs.

Must be tested with: cargo test -- --test-threads=1 (parallel tests clash their working dir settings)

Missing/WIP pieces include:

gavento commented 6 years ago

@spirali Would you please take a look and test it against the worker? I am on a weak connection for the next few days and that complicates things. Also, feedback is welcome! (along with integration tests ;)

spirali commented 6 years ago

I have put basis for integration testing of rust subworker here: https://github.com/spirali/rain/commit/bc4cea8a44c9d1333a9fad898af51baa8410f773

I will do some more review later.

spirali commented 6 years ago

I suggest that we could standardize some operations that should by implemented by tester of subworker library (e.g. hello world, a kind of concat, return object without change, etc). This allows us to simply reuse one set of basic python tests for all subworkers.

gavento commented 6 years ago

Almost everything not dealing with attributes is finished and somewhat polished.

New and shiny docs and examples, many tests, flexible registration with register_task!, simplified writing to Output (no more built-in locks). There are still TODOs. Feedback welcome.

gavento commented 6 years ago

@spirali That sounds reasonable, do you want to open an issue/PR with a summary of the requirements, or make a writeup doc?