Provide an easier way to programmatically start the daemon

[mvdan]

This is a continuation of https://github.com/ipfs/go-ipfs/issues/5983#issuecomment-463708585, since that's an old thread that was closed.

If one wants to execute ipfs daemon and wait for it to start, the only way right now is to run it and wait for Daemon is ready on stdout:

$ ipfs daemon
Initializing daemon...
go-ipfs version: 0.7.0-ea77213e3
Repo version: 10
System version: amd64/linux
Golang version: go1.15.2
Swarm listening on /ip4/127.0.0.1/tcp/4001
[...]
WebUI: http://127.0.0.1:5001/webui
Gateway (readonly) server listening on /ip4/127.0.0.1/tcp/8080
Daemon is ready

This is what js-ipfsd-ctl does: https://github.com/ipfs/js-ipfsd-ctl/blob/master/src/ipfsd-daemon.js#L220

Note that I'm not interested in the Go APIs to run an IPFS node directly; I'm interested in executing the already-installed ipfs daemon from a separate program and any language.

I propose we add ipfs daemon --fork, which starts the daemon in a sub-process and returns when it's ready.

It would also be very useful to get information about the daemon that's been started, in a way that's easier for machines to parse. For example, the app starting the daemon is probably interested in fields above like go-ipfs version and WebUI, but the human-friendly output is not particularly consistent.

It would be great if, instead, you'd get something like:

$ ipfs daemon --fork
# we can print progress and errors here via stderr; json below is stdout
{
  "version": "0.7.0-ea77213e3",
  "repoVersion": "10",
  "webUI": "http://127.0.0.1:5001/webui",
  ...
}
$ # daemon is now running

It's fine if the JSON output replacing the plaintext were behind another flag, like --json. I know that feature is somewhat separate from --fork, but I think the use case is extremely similar - gathering information from the started daemon without parsing the human-readable stdout lines.

cc @andrew @daviddias @olizilla @Stebalien

[mvdan]

One more thought - the forking does complicate controlling the process in a programmatic way. Right now, I can simply use https://golang.org/pkg/os/exec/#Cmd.Wait to notice when the daemon exits or crashes. If this new flag forks, then Wait will instead tell me when the daemon is ready, not when it's exited. I'd have to notice when it exits some other way.

One possible way would be for the JSON to provide a PID as well, and then it's up to the caller to wait on that PID if they wish.

Another possible way would be to not fork. The caller would wait for a JSON object to be printed to stdout, meaning that the daemon has started, and then wait for the process to finish to notice when the daemon exits. This is sort of similar to what we have right now, but the JSON would be properly documented and easier to parse, as opposed to the plaintext stdout lines we have right now.

I think I prefer the latter, since it feels more powerful overall. But the former would be easier to use directly from the command line.

[olizilla]

I'm a big +1 on having a command that lets us start the daemon and exits when it's ready... people shouldn't need to write stuff like this https://github.com/alexanderschau/ipfs-pinning-action/blob/main/waitForIpfs.go

If forking creates complications, we could add a seperate command like ipfs daemon wait (similar to lotus sync wait)... scripts would then background the ipfs daemon command, and run ipfs daemon wait to pause until it's ready. Having this feature as a flag to ipfs daemon seems preferable though.

[djdv]

Another possible way would be to not fork. The caller would wait for a JSON object to be printed to stdout, meaning that the daemon has started, and then wait for the process to finish to notice when the daemon exits. This is sort of similar to what we have right now, but the JSON would be properly documented and easier to parse, as opposed to the plaintext stdout lines we have right now.

For what it's worth I needed something similar recently. And quickly hacked together this https://github.com/djdv/go-filesystem-utils/blob/439fcbb37fa7fa546cabf9fc279916e900d672d6/cmd/executor/executor.go#L31

[SomajitDey]

For the time-being, I am using this one-liner to start the daemon in background and wait until it's ready.

ipfs daemon 2>/dev/null | grep -i -o -m1 'Daemon is ready' & tail -f --pid=$! /dev/null

Outputs something when Daemon is ready, nothing otherwise.

[schomatis]

Seems like a big chunk of this use case could be satisfied with a Lotus-like tool to tell when the API is receiving requests (independent of which process is servicing it, maybe the current ipfs daemon or maybe a previous instance already running).

As an extra we can better structure daemon output as JSON provided the user clearly understand that the "Daemon is ready" flag does not guarantee its request will be serviced since we do start the HTTP server in another goroutine and do not monitor when it's up.

(Alternatively we could merge the two above and monitor the server within the daemon command itself to print/emit/run callback when the server is accepting requests.)

Way down on my priority list would be baking some forking procedure into go-ipfs logic. I think the devop scripting and automating this could easily monitor the launched ipfs daemon for errors or early termination.

cc @aschmahmann for feedback on this to get a scoped useful first step toward making this pain point better (won't move on this until confirmation)