search

so-dang-cool / dt

dt - duct tape for your unix pipes
https://dt.plumbing/
BSD 3-Clause "New" or "Revised" License
430 stars 12 forks source link

Parallel execution #46

Open booniepepper opened 1 year ago

booniepepper commented 1 year ago

Probably this should wait for async but if I'm feeling cheeky I could write something roughly compatible with it. See https://ziglang.org/news/0.11.0-postponed-again/

Start with a form of each that parallelizes whatever it's doing.

Move on to map and filter and others if it's simple to extend. In anything parallel, ordering of operations, or the resulting data, is not guaranteed. In these cases, there will still need to be locking on the state for any writes.

booniepepper commented 1 year ago

This really absolutely requires that at least everything within dt's state is following immutable semantics.

This may require reference counting and/or copy-on-write. For example, reversing/sorting a quote/string should have the effect that it only reverses/sorts that single string, and can conceptually be considered as creating a new string. If it's a reverse/sort on the only remaining copy of a quote/string then it's fine if the implementation involves reusing the allocated memory (for example: in-place reverse or sort with no extra allocation)

There can't be a compromise on this from my perspective, it's key to making sure a dt program can be read and remain simple enough to follow exactly what it will perform. (And factor it and refactor it etc)

For side effects... You're on your own! Behavior once you start doing parallel writes to files or sockets or anything else outside the world of dt values will be completely nondeterministic. As an example, a flurry of activity all trying to write or append to the same file will be delegated to whatever filesystem locking behavior exists (if any)