Proposal: A protocol watch processes can implement

[VanCoding]

Hi guys

Because I was unhappy about the situation with watch-processes, or mainly their inability to coordinate with other watch-processes, I recently started my hobby project task-graph-protocol.

The main idea I had was a simple protocol that watch-processes could implement to communicate with the task-runner, so that the task runner knows when a task is completed even if the process does not exit, and the task knows when to re-start.

I haven't heard of wireit before and because this whole protocol doesn't make sense without a task-runner implementing it, I also implemented a simple task runner for it. But wireit of course is much more mature, and it would be much simpler to integrate the protocol into it than to extend my task runner to be as powerful as wireit.

So my proposal is to integrate the task-graph-protocol or the idea of it into wireit. And if you're really going to do it, we could try to bring all the tools like typescript, webpack, etc on board. That's also much easier if the protocol is backed by a major task runner.

What do you think? :)

Btw, I have already implemented the protocol for TypeScript, it's included in the repository.

[aomarks]

Hi @VanCoding. Awesome, I'm very interested in this topic!

We have casually discussed something along these lines before -- I filed https://github.com/google/wireit/issues/32 but there isn't much detail there so let's discuss here.

My thought was that we could implement the Bazel persistent worker protocol, which is described at https://bazel.build/remote/persistent. From a high level, I think it has similar goals to your protocol.

Are you familiar with the Bazel persistent worker protocol? I would be curious to get your take on it, and how it compares to your design. An advantage of piggy-backing on the Bazel worker protocol for this use-case would be that any tools that add support for it would get persistent worker support in both Wireit and Bazel.

[VanCoding]

@aomarks Wow, that response was quick! :sweat_smile:

No, I'm not familiar with the Bazel worker protocol, but I'm going to give it a read tomorrow.

But in the meantime as a sidenote: The design I came up with basically had 2 goals in mind. 1) It should be as simple to implement as possible, so that build tools can implement it very easily. 2) the task runner should not have to know if a process is implementing the protocol or not and it should be easily detectable at runtime. So a process can at any point "upgrade" or "opt-in" to the protocol.

My protocol also has a functionality to signal to the task runner that the task's files have changed. This could be complementary to the file watching logic wireit already has. If the build tool supports the feature, it would no longer be needed to specify which files to watch in the task again.

With that said, I'll come back to you after reading about the Bazel worker protocol :smiley:

[VanCoding]

@aomarks I've read the page about Persisent Workers now. It looks like a really powerful protocol! I think it could definitely be used to achieve what we want, but I'm not sure if it really is intended for that. It seems to focus more on multiplex workers and sandboxing, which are things that are also interesting, but could also be solved separately.

Maybe I misunderstood something, but I didn't quite get why the protocol even needs to be aware of multiplexing and sandboxing. Multiplexing could be the tasks job to do, and sandboxing could be entirely the task runners job, without the task even noticing it.

My gut feeling is that it's probably easier to make tool creators implement a protocol that's stripped down to solve only the one thing that cannot really be done without them implementing it, and hence is simpler for them to understand and implement.

I mean the only thing that we really need is a way to tell a running process to start the task again, and maybe a way to tell it to only restart the task when we tell it to. Everything else could be solved without a protocol at all. For example, getting notified when the task completed, and if it failed could also be done by parsing the stdout & stderr (which you're currently even doing in wireit).

But maybe there doesn't even have to be the one and only protocol. It's also possible to support multiple. Tool creators could then pick the one that fits their needs best. task-graph-protocol could then be the zero-config option :stuck_out_tongue_winking_eye:

[VanCoding]

@aomarks Were you able to think about it a bit more?

I placed some comments on issues of other task runners in the mean time to get some more feedback on the protocol. See: https://github.com/VanCoding/task-graph-protocol/issues/3