Adding handlePanic method to capture panic

Tulsishah commented 1 year ago

Before, when an application crashed in the background, it was not getting logs anywhere. I made handlePanic method which will capture the panic and print panic messages in logfile

stapelberg commented 1 year ago

This is unusual.

Panics go to STDERR by default. Why is STDERR not captured in your environment?

Even if the default panic behavior is not sufficient, why is the panic not recovered and logged at a higher level (in the program that uses jacobsa/fuse)?

Tulsishah commented 1 year ago

This is unusual.

Panics go to STDERR by default. Why is STDERR not captured in your environment?

Even if the default panic behavior is not sufficient, why is the panic not recovered and logged at a higher level (in the program that uses jacobsa/fuse)?

So in gcsfuse, we are using fuse library, and whenever we run gcsfuse in foreground mode and panic occurs, it is printing stderr on the console, which is not handled by gcsfuse. But when we ran gcsfuse in background mode, the stderr got vanished. And this stderr is not available anywhere in the system also. In gcsfuse all the operations are calling by this handleops method. That's why I made defer handlepanic call to print the panic message. Do you have any lead on achieving the same goal, or do you have any alternate ideas?

stapelberg commented 1 year ago

But when we ran gcsfuse in background mode, the stderr got vanished. And this stderr is not available anywhere in the system also

Can you describe how I can run gcsfuse in background mode? It’s surprising to me that STDERR is lost in this situation.

How can I reproduce the panic that you’re trying to diagnose here?

Tulsishah commented 1 year ago

But when we ran gcsfuse in background mode, the stderr got vanished. And this stderr is not available anywhere in the system also

Can you describe how I can run gcsfuse in background mode? It’s surprising to me that STDERR is lost in this situation.

How can I reproduce the panic that you’re trying to diagnose here?

Clone gcsfuse code
Put panic("message") in createObject method
cd gcsfuse
Run go run . --implicit-dirs --max-conns-per-host 100 --log-file=log.txt --log-format="text" your-bucket-name your-mounting-dir
cd your-mounting-dir
run mkdir test

gcsfuse will crash and panic message got vanished.

same thing when you will do through

Run go run . --foreground --implicit-dirs --max-conns-per-host 100 --log-file=log.txt --log-format="text" your-bucket-name your-mounting-dir
cd your-mounting-dir
run mkdir test

gcsfuse will crash and crash logs will print on console.

stapelberg commented 1 year ago

Okay. The reason why the STDERR vanishes is because the daemonize code doesn’t set cmd.Stderr. When adding it like so, the panic is printed:

--- i/vendor/github.com/jacobsa/daemonize/daemonize.go
+++ w/vendor/github.com/jacobsa/daemonize/daemonize.go
@@ -211,6 +211,7 @@ func startProcess(
        cmd.Args = append(cmd.Args, args...)
        cmd.Env = append(cmd.Env, env...)
        cmd.ExtraFiles = []*os.File{pipeW}
+       cmd.Stderr = os.Stderr

        // Change working directories so that we don't prevent unmounting of the
        // volume of our current working directory.

Looking at the stack trace of a panic, I see that you already have an error-handling related wrapper:

github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers.(*errorMapping).CreateFile(0xc000920180, {0x1153da8?, 0xc0009236b0?}, 0xc00074d710?)
    /home/michael/go/src/github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers/error_mapping.go:164 +0x33

If you still need to recover panics for some reason, maybe you could recover panics in there?

Tulsishah commented 1 year ago

Okay. The reason why the STDERR vanishes is because the daemonize code doesn’t set cmd.Stderr. When adding it like so, the panic is printed:
--- i/vendor/github.com/jacobsa/daemonize/daemonize.go
+++ w/vendor/github.com/jacobsa/daemonize/daemonize.go
@@ -211,6 +211,7 @@ func startProcess(
        cmd.Args = append(cmd.Args, args...)
        cmd.Env = append(cmd.Env, env...)
        cmd.ExtraFiles = []*os.File{pipeW}
+       cmd.Stderr = os.Stderr

        // Change working directories so that we don't prevent unmounting of the
        // volume of our current working directory.
Looking at the stack trace of a panic, I see that you already have an error-handling related wrapper:
github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers.(*errorMapping).CreateFile(0xc000920180, {0x1153da8?, 0xc0009236b0?}, 0xc00074d710?)
  /home/michael/go/src/github.com/googlecloudplatform/gcsfuse/internal/fs/wrappers/error_mapping.go:164 +0x33
If you still need to recover panics for some reason, maybe you could recover panics in there?

In Background mode, where does it printed?

stapelberg commented 1 year ago

In Background mode, where does it printed?

To the terminal from which you started it in background mode.

If you start background mode from a systemd service, I would expect the output to make it into the system journal by default.

vitalif commented 1 year ago

Capturing panics is bad, in fact I also have it in https://github.com/yandex-cloud/geesefs left from Goofys but I plan to remove it. It only hides bugs. So I wouldn't add panic handlers if you asked me

stapelberg commented 1 year ago

Yeah, I think @Tulsishah found a better solution for this. I’ll close the PR.

jacobsa / fuse

Adding handlePanic method to capture panic #139