search

edx / edx-arch-experiments

A plugin to include applications under development by the architecture team at edx
GNU Affero General Public License v3.0
0 stars 3 forks source link

"None cannot be handled by custom serializers" when producing event to Kafka #788

Open timmc-edx opened 1 week ago

timmc-edx commented 1 week ago

We have intermittently seen errors involving "None cannot be handled by custom serializers" when producing to the event bus.

Example log message that has been pulled apart for readability, and redacted:

Error delivering message to Kafka event bus.
error=None cannot be handled by custom serializers (and default=None was not set)
full_topic='prod-course-catalog-info-changed'
event_key=None
signal=<OpenEdxPublicSignal: org.openedx.content_authoring.course.catalog_info.changed.v1>
initial_topic='course-catalog-info-changed'
event_key_field='catalog_info.course_key'
event_data={'catalog_info': CourseCatalogData(course_key=CourseLocator('...', '...', '...', None, None), name=None, schedule_data=CourseScheduleData(start=datetime.datetime(...), pacing='self', end=datetime.datetime(...), enrollment_start=None, enrollment_end=None), hidden=False, invitation_only=False)}
event_metadata=EventsMetadata(event_type='org.openedx.content_authoring.course.catalog_info.changed.v1', id=UUID('843664a9-6a0a-11ef-a2ab-8fe8a1efeee6'), minorversion=0, source='openedx/cms/web', sourcehost='ip-...', time=datetime.datetime(...), sourcelib=(9, 11, 0))
event_data_as_json=None
event_metadata_as_json=None

Another event on a different topic and with a different signal:

Error delivering message to Kafka event bus.
error=None cannot be handled by custom serializers (and default=None was not set)
full_topic='prod-learning-course-access-role-lifecycle'
event_key=None
signal=<OpenEdxPublicSignal: org.openedx.learning.user.course_access_role.added.v1>
initial_topic='learning-course-access-role-lifecycle'
event_key_field='course_access_role_data.course_key'
event_data={'course_access_role_data': CourseAccessRoleData(user=UserData(id=..., is_active=True, pii=UserPersonalData(username='...', email='...', name='')), org_key='...', course_key=None, role='data_researcher')}
event_metadata=EventsMetadata(event_type='org.openedx.learning.user.course_access_role.added.v1', id=UUID('6ebe1a20-6a06-11ef-b971-59131e46b91e'), minorversion=0, source='openedx/lms/web', sourcehost='ip-...', time=datetime.datetime(...), sourcelib=(9, 11, 0))
event_data_as_json=None
event_metadata_as_json=None

The traceback is the same:

Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.11/site-packages/edx_event_bus_kafka/internal/producer.py", line 296, in send
    context.event_data_as_json = json.dumps(get_signal_serializer(signal).to_dict(event_data))
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.11/site-packages/openedx_events/event_bus/avro/serializer.py", line 131, in to_dict
    return _event_data_to_avro_record_dict(event_data, serializers=self.serializers)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.11/site-packages/openedx_events/event_bus/avro/serializer.py", line 76, in _event_data_to_avro_record_dict
    json.dumps(
  File "/usr/lib/python3.11/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
          ^^^^^^^^^^^
  File "/usr/lib/python3.11/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/encoder.py", line 258, in iterencode
    return _iterencode(o, 0)
           ^^^^^^^^^^^^^^^^^
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.11/site-packages/openedx_events/event_bus/avro/serializer.py", line 72, in value_to_dict
    return attr.asdict(value, value_serializer=_get_non_attrs_serializer(serializers))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.11/site-packages/attr/_funcs.py", line 71, in asdict
    v = value_serializer(inst, a, v)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.11/site-packages/openedx_events/event_bus/avro/serializer.py", line 44, in _serialize_non_attrs_values
    raise Exception("None cannot be handled by custom serializers (and default=None was not set)")
Exception: None cannot be handled by custom serializers (and default=None was not set)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/edx/app/edxapp/venvs/edxapp/lib/python3.11/site-packages/edx_event_bus_kafka/internal/producer.py", line 56, in record_producing_error
    raise EventProductionException(error)
edx_event_bus_kafka.internal.producer.EventProductionException: None cannot be handled by custom serializers (and default=None was not set)
timmc-edx commented 1 week ago

I suspect this is actually a bug in the code that creates the event data in the first place. For example, that course_key=None is very suspicious; a spot-check of successfully sent messages on that topic shows a non-None course key, and the code that sends the event does not indicate that the course key is optional.