Pocketbase Crashed in our system with error

[abhilashsajeev]

Our pocket base crashes some times with log

fatal error: concurrent map iteration and map write
created by [github.com/pocketbase/pocketbase/tools/routine.FireAndForget](http://github.com/pocketbase/pocketbase/tools/routine.FireAndForget)
        /home/runner/work/pocketbase/pocketbase/tools/routine/routine.go:18 +0xac

goroutine 37076 [IO wait, 11 minutes]:
internal/poll.runtime_pollWait(0x7fdffa184c68, 0x72)
        /opt/hostedtoolcache/go/1.19.4/x64/src/runtime/netpoll.go:305 +0x89
internal/poll.(*pollDesc).wait(0xc000c3b100?, 0xc0009336f1?, 0x0)
        /opt/hostedtoolcache/go/1.19.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /opt/hostedtoolcache/go/1.19.4/x64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000c3b100, {0xc0009336f1, 0x1, 0x1})
        /opt/hostedtoolcache/go/1.19.4/x64/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000c3b100, {0xc0009336f1?, 0xc0008400c0?, 0xc000656f68?})
        /opt/hostedtoolcache/go/1.19.4/x64/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc0000120e0, {0xc0009336f1?, 0xc000a33d60?, 0x1?})
        /opt/hostedtoolcache/go/1.19.4/x64/src/net/net.go:183 +0x45
net/http.(*connReader).backgroundRead(0xc0009336e0)
        /opt/hostedtoolcache/go/1.19.4/x64/src/net/http/server.go:678 +0x3f
created by net/http.(*connReader).startBackgroundRead
        /opt/hostedtoolcache/go/1.19.4/x64/src/net/http/server.go:674 +0xca

How to prevent this and enable graceful restarts

[ganigeorgiev]

We already have panic recovering and graceful shutdown handling. For autorestart you can setup a systemd service as shown in https://pocketbase.io/docs/going-to-production/#systemd-service.

Could you provide more information when you are experiencing this error? What is your PocketBase version? Do you use PocketBase as framework or the prebuilt executable?

[abhilashsajeev]

@ganigeorgiev My pocketbase version is 0.10.4. Specific situation of crash is not yet identified. but the last logs before crash is given above. it shows fatal error: concurrent map iteration and map write

[ganigeorgiev]

I'm not sure how to reproduce it.

The only way I can think this error could occur is if there is a log write in the process and the server was abruptly stopped/killed for some reason (eg. in case of OOM).

FireAndForget is generally called for 2 reasons:

• background log writes
• public sending emails (eg. verification/password-reset) when the result of the call shouldn't be awaited to prevent enumerations (this eventually will be replaced with a mailqueue in the future)

If you manage to narrow down the cause of the error, please let me know and I'll try to investigate it in more details.

But in any case, it is recommended to create a systemd service to keep the process up and restart on its own if needed.

You can also temporary enable the debug mode with --debug.

[ganigeorgiev]

Another possibility for the error could be also if you have custom Go code that modifies a models.Record concurrently or in a OnRecordAfterRequestVerificationRequest/OnRecordAfterRequestPasswordResetRequest hook since it runs concurrently with the FireAndForget (I may eventually consider adding a mutex to the models.Record in the next release).

[ganigeorgiev]

Just for info if anyone is following here, the models.Record data and expand operations should be now concurrent safe in the latest v0.12.0 release.