Could Shake use the Remote Execution API?

[joneshf]

I was looking to see the progress of Cloud Shake recently, and it looks like it's still in development. Since it's still in development, have you considered leaning on the Remote Execution API?

The idea being that Shake could be another client like Bazel, BuildStream, Pants, or Recc, and the server stuff would be taken care of by something else.

To be honest, I wonder if the client side of things couldn't be implemented by an oracle in Shake proper. It seems like it would subsume what Cloud Shake does, but maybe I'm missing some other pieces of what Cloud Shake does. The idea sounds possible in my head, but I've tried to look too deeply at whether that would work in practice or not. Might be an idea to try out in an external package.

What are your thoughts?

[ndmitchell]

Even when complete, Cloud Shake wouldn't offer remote execution facilities, so it's not an either/or situation - they could both be useful.

I'm sure it's possible, but it's not as direct as other build systems. For example, in Ninja you run commands specifying the dependencies for a command. However, in Shake you run rules, specifying dependencies for rules - which means there isn't a 1:1 mapping between dependencies and commands. Shake also allows you to declare dependencies afterwards (needed) and for certain local artifacts that are correlated with other artifacts, Shake lets you avoid mentioning them at all. Neither of those would work with a remote execution API.

That said, if you want to give it a go, I think you just need to edit Command.hs with a Remote flag, and then run the command, with all dependencies already declared up to that point in the rule. I'd be keen to see the results!

[joneshf]

Oh sorry, I should have been clearer. The Remote Execution API also defines caching. My understanding of Cloud Shake is that the thing it adds to Shake proper is remote caching. That's what I was thinking could be subsumed by the Remote Execution API. Is there more that Cloud Shake is intended to do?

I don't fully get the difference between a dependency and a command as you laid out. I'll try reading some more documentation and seeing if that clears it up.

For transparency, it would probably be quite a while before I did anything here. I don't currently use Shake for anything, so I don't really have the drive to explore this too deeply.

[ndmitchell]

A rule defines an action. An action can execute multiple commands. RE (remote execution) caches and runs at the level of commands. Shake naturally works at the level of rules/actions, and requires consistency at that level. Cloud Shake caches at the level of rules. That means you can have an expensive Haskell computation cached in Cloud Shake, but if its run as an internal apart of the build, it wouldn't be suitable for RE. It could well be that if we had RE we wouldn't care about Cloud Shake, but they do somewhat different things.

[joneshf]

I think I'm starting to get it. Thanks for explaining that!