Discussion: RxJs based fork of @start

[sparebytes]

The Start project inspired me to make a fork based on RxJs. It's called Hark. I thought I'd share it here in case anyone found it interesting. Documentation is sorely lacking but check out the harkfile.ts to get an idea of how it works.

https://github.com/sparebytes/hark

The big changes are:

1. It uses RxJs under the hood in to help facilitate complex asynchronous interactions. It also helps ensure that certain tasks are only executed once. EG, only build the models project once even if both the api and ui projects are requesting the models project to be built at the same time.
2. You build your own CLI using clipanion because you might want to provide special params, eg --watch, for certain tasks.
3. Base classes for handling monorepos are provided. These classes are extended to describe tasks that can be run on the monorepo as a whole and tasks which are specific to each package.

Here is an example build --watch scenario with Hark:

As you can see, the API server doesn't start until the first time all of the dependencies have finished (Babel API sources, TTypescript of models). In this example, we might be using Typesmith to transform the models directly into validation functions. Whenever a change occurs, dependents are notified. In this case, the API will restart if either the API or the Models complete a compilation. It also has a debouncer to ensure it won't needlessly restart too many times in a short period of time.

[deepsweet]

Hi!

It looks very interesting, especially if you really enjoy RxJs. I personally find it quite complicated even though I understand it :)

I see that it's a hard fork, but I'd appreciate some mention of the original Start. Every plugin is licensed with MIT, and it'd be nice to reflect that somehow and somewhere.

So far our idea is to make next version of Start to use async iterables all the way, which is a bit opposite to the observables:

active push producer (observable) -> passive push consumer (observer)
passive pull producer (iterable) -> active pull consumer (iterator)

It depends on perspective of how one looks at FS: it's either a static data (files) that you need to lazily pull from disk (iterable), or it's an emitter to where you push files (observable, conceptually shines here in FS watch mode, but fake'ish otherwise).

The main "issue" with Start as for today is that each file doesn't flow down the pipe right away:

• file1, file2, fil3
• read1, read2, read3
• babel1, babel2, babel3
• write1, write2, write3

Async iterables idea will allow us to make:

• file1, read1, babel1, write1
• file2, read2, babel2, write2
• file3, read3, babel3, write3

using at least concurrency if not even a real parallelism with Worker Threads.

As a bonus we'd finally be able to extract logger from plugins, just like in your harkfile, and put it outside as a part of a pipe ("sequence" in current terms).

Some ground work we've done so far:

• iterama
• piall
• workerama
• iva (WIP)

[sparebytes]

Sorry, I fixed the license to include NexTools. For the record all the package.json retained their original license and author. Hark is just an experiment and I don't expect to put much more work into it.

I was just searching for a good iterable library and one is!

Yes RxJS is a blessing and curse but I went with it for it's power and and mature API. Mostly I was looking for parallelism and the ability to cache the results of previous tasks in a monrepo settings.

I'm sure you already have this use case covered but I thought I'd ask about it. Regarding (file1, read1, babel1, write1) pipe. At the end the pipe you might want a buffer of sorts that waits until every file has been compiled before proceeding on to another step. EG

👎 No Good
Wacher -> file1 -> read1 -> babel1 -> write1 -> Run API
   |   -> file2 -> read2 -> babel2 -> write2 -> Restart API
   |   -> file3 -> read3 -> babel3 -> write3 -> Restart API

👍 Okay
Wacher -> file1 -> read1 -> babel1 -> write1 -> Counter
   |   -> file2 -> read2 -> babel2 -> write2 ->    | 
   |   -> file3 -> read3 -> babel3 -> write3 ->    | 
   |   ->               3 Total              -> Counter -> Run API
   |          (programmer edits a file)            |
   |   -> file4 -> read4 -> babel4 -> write4 ->    | 
   |   ->               4 Total              -> Counter -> Run API

It's exciting to get a peek at the next evolution of Start. Looking forward to trying it once a prototype is out.