Keeping the make process running as a daemon (not --keep-going, not 'watch make')

[daviddeutsch]

I've asked this on the make mailing list but didn't have any luck. Apologies if this is out of scope for remake.

I use make to build a document tree, again and again, when prerequisites change. I observe outside change events and use inotify on some files and when a change has occurred, I re-run make. So my problem is not detecting the change - I already know when a change has occurred and where it happened.

My problem is that the more complex a document tree gets, the more makefiles I put on the build graph and the more files make has to parse on each startup. In some of my projects, I have thousands of makefiles amounting to multiple MBs of rules that make has to read and reread. It handles that remarkably well, but it can lead to long startup times, which makes me wonder: Would it be possible to keep the make process itself running as a daemon with the state that it has generated?

Put differently: There are some applications of make where booting up make and building the dependency graph is the most expensive part, by far. Would it be possible to keep the make process running and that graph in memory?

Related to that: It further seems like a waste to generate so many .mk and .d files only to restart make and read them in. It would be great if I there was an option to tell make to not create files and read them in, but directly pipe their output back into memory.

[rocky]

All of this sounds perfectly reasonable.

One of the aspects about remake was to allow a platform whereby people could make wilder, more aggressive kinds of changes. In contrast, since GNU Make is relied on heavily, maintainers have to be cautious and conservative about changes.

So please feel free to make changes to the code base in order to do these sorts of things.

[daviddeutsch]

Yes, I figured core make would be too conservative - as they should be!

I will definitely try to make something happen, but I'm in no way proficient enough in C to get anywhere soon.

Would you be able to make an estimate on how big of a task my proposal would be? I've tried to get a quote from another developer, but just getting to understand the make codebase is a task in and of itself…

I suspect from your reply that the answer is no, but - if I put together a budget for this, would you be interested in a collaboration here? For now, the best I can offer is web development work*, but in time, I could look into putting together something… "proper".

*The other thing I'm looking into is putting together a browser-based debugger to help understand the make graph. I'd be writing this for my own projects, eventually, anyways.

[rocky]

The problem I have in coming up with an estimate is that it kind of depends on how much you want to do, how flaky it can be, and in more detail what the specification is.

For how much you want to do, I believe there are OS mechanisms for getting notified of filesystem changes. These are likely to be OS specific and not cover all OS's. Alternatively you could poll but then you would be doing massive statd's.

So that suggests that some work might be needed just to profile the system for feasibility.

On the spec side, there's what is it you want. Are you always looking for a specific target only when you deamonize is the target somehow indicated afterwards? Some targets are "phony" so that means they always get rebuilt whenever you want them. If you specify a target at the outset, then clearly it can't be a PHONY target which is always "out of date". But I suppose phony in a chain of dependencies are okay and are likely to happen.

The above is to give you some idea of the challenges in coming up with an estimate.

As for the problem of estimating cost, one way to do this is just start breaking this up into bounties on one of the various bounty networks. If the ask is too great, you'll get no takers, and then you can adjust the bounty accordingly.

[daviddeutsch]

My preference would actually be to not do any sort of polling or file watching at all and leave that up to discrete outside triggers. Just offer something like a socket (or websocket) that you could send… triggers into. As I mentioned in my first post - knowing when and what has changed is already a solved problem on my end.

Ideally, the command socket would allow for the same arguments as a make call, so: Tell make a goal to build and perhaps set a few variables. The only thing that this would skip is the necessity of reading in all of the makefiles - except for when it creates new makefiles (which, again, would be incredibly cool if remake wrote those straight into its own memory instead of an external file… that would save a LOT of disk thrashing on my end… could call it something like --prefer-virtual-includes).

It would probably still need to check through all of the files in the dependency tree. I guess you could then further improve the performance by supplying --assume-old= when you call on the remake daemon and thus tell it to not check any files but work off of a discrete list of known outdated files. But yeah - it would redo phony targets and everything.

Thinking some more about it, maybe it'd be possible to have remake write some kind of remake.lock file when you call on it with an argument like --daemon. Then, further calls to remake would see that there is an instance of remake running and just hand the commands over to that. So:

> remake my-goal -j4 var1=test var2=test2 --daemon

# running in the background now, creating a remake.lock file...

> remake my-other-goal -j1 var1=false var2=test5 --daemon

# this now forwards its command to the running daemon

> remake my-special-goal --assume-old=src/theOnlyFileThatChanged --daemon

# this does the same, but does not check any files on whether they are outdated or not except the one provided

> remake --stop-daemon

# kill the daemon and remove the .lock file

That being said, I can totally see how it might make sense for an application called "remake" to be able to do its own file watching. It could be quite an asset because I've seen a lot of people ask for a make --watch (instead of a watch make). I just see that as a separate step.

I occasionally stream my work on twitch. If you'd be interested, I could do a stream where I show off how I use make and you could watch (or, rather: skip through) at your leisure.