Improve observability of JWT decoding errors

[robrap]

This ticket is for improving observability of errors during JWT decoding in our shared library. See https://github.com/openedx/edx-drf-extensions/blob/master/edx_rest_framework_extensions/auth/jwt/decoder.py#L67-L74

Acceptance Criteria:

• [ ] Add simpler observability of JWT decoding errors via New Relic (through logs and/or custom attributes).
• [ ] Update New Relic dashboard(s) as appropriate. See notes.

Questions and notes:

• Possible improvements include:
  • wrapping all exceptions in a custom exception that is easier to monitor, or
  • setting a custom attribute with the error class information and then re-raising the existing error.
  • both or other?
• Would we want to remove this custom attribute that may end up reporting redundantly? https://github.com/openedx/edx-drf-extensions/blob/7179ec861aec6d1df1e3d633ca03ce13962bc32d/edx_rest_framework_extensions/auth/jwt/decoder.py#L243-L245
• The description for the custom attribute annotation (if we go that route) should mention to use the exceptions in the log to find where the exception is happening. This could go into text on the monitoring dashboard as well.
• See the relevant New Relic dashboard(s) documented here: https://openedx.atlassian.net/wiki/spaces/AC/pages/3739975683/So+you+want+to+touch+JWTs#Observability. There is a TODO in the doc for potentially consolidating the most important data on to a single dashboard, which could be done on a separate ticket or some progress could be made here.

[robrap]

Related, we have a New Relic dashboard with the chart "Token verification failed count (by app)":

SELECT count(*) FROM Log WHERE message = 'Token verification failed.' FACET entity.name SINCE 1 day ago TIMESERIES MAX

This is showing failures against discovery that seem to be from a bad actor, based on other research. We think someone might be calling twice with 2 tokens, one good and one bad, for each request. See https://onenr.io/0qwy8MrgGQn. Note that the failures are "unauthenticated", so we can see that these were JWT requests.

You can also see the log details, and see that there isn't enough to filter out known bad callers: https://onenr.io/0gR76X2L9wo

SELECT * FROM Log WHERE message = 'Token verification failed.' SINCE 1 day ago

Ideally we'd be able to filter out this data from the bad actor, but I'm not sure how. This may be a separate, but related ticket.