setJSExceptionHandler only firing once?

[JeremyBradshaw7]

I have this in my App.jsx:

// CAPTURE UNHANDLED EVENTS TO PREVENT APP CRASH
const errorHandler = (e, isFatal) => {
  ErrorService.logError('Unhandled JS Exception', e, {}, {name: e.name, isFatal});
  if (isFatal) {
    Alert.alert(
      'Unexpected error occurred',
      `${isFatal ? 'Fatal ' : ''} ${e.name} ${e.message}

This error has been logged. We recommend you close the app and restart it.
`,
      [{
        text: 'Close'
      }]
    );
  } else {
    console.log(e); // So that we can see it in the ADB logs in case of Android if needed
  }
};
setJSExceptionHandler(errorHandler, true);

And have a test button on a screen to test this:

                <RoundButton iconName='bolt' label='Unhandled JS Error' onPress={() => {
                  throw new Error('This is an unhandled javascript error causing an app crash!');
                }} />

I press the button and see the alert (at least on a test build I do, I don't see it in dev mode though I think I should as I set the 2nd param to setJSExceptionHandler to true). I press it again and nothing happens, I'd expect the alert to show again - it only shows again after I restart the app - this doesn't suggest to me that it has gracefully handled the error and that in fact users would HAVE to restart the app to get things working again.