protect from panics in goroutines

[smira]

Inspired by #369, follow-up for #372.

There are two major sources of goroutines launched by Omni which might not have panic protection:

1. errgroup.Go
2. go func()

I think we need to ensure all these runs have a panic handling mechanism.

I propose to add the following helper function:

func recoverPanic(f func() error, logger zap.Logger) (err error) {
   defer func() {
       if p := recover(); p != nil {
             ....
             err =  
             logger.
       }
     }()

     return f()

And the introduce sub-versions of this function - the one which accepts func() and returns no error (for go func()) case.

And then ensure that every invocation of a goroutine is wrapped accordingly

[smira]

let's also make sure it's unit-tested :)