task_mgr::spawn's shutdown_process_on_error` doesnt work reliably [P:3] [S:0]

[LizardWizzard]

Steps to reproduce

Was discovered via https://github.com/neondatabase/neon/issues/3387

Apparently synthetic size calculation task had an error that triggered shut down. Synthetic size calculation shouldnt lead to pageserver shutdown, and this is fixed in #3392. But shutdown on error should still work even if its triggered erroneously. This is what this issue all about.

Expected result

pageserver restart.

Actual result

pageserver was stuck in semi-alive state when some of the tasks were stopped and some continue running. Postgres protocol listener was shut down so this resulted in connection refused errors during basebackups.

Environment

prod.

Logs, links

• https://neondb.slack.com/archives/C033RQ5SPDH/p1674231060776979
• Captured stacktraces (many of threads are idle tokio workers)

[shanyp]

consider having a timeout for this one

[jcsp]

Fixed in https://github.com/neondatabase/neon/pull/6105 -- we now exit(1) if a shutdown_process=true case.