search

endlessm / azafea

Service to track device activations and usage metrics
Mozilla Public License 2.0
10 stars 2 forks source link

Error replaying unknown metrics #102

Open adarnimrod opened 4 years ago

adarnimrod commented 4 years ago

In the dev environment we got the following error running metrics-2 replay-unknown:

[ERROR] azafea.event_processors.endless.metrics.events._base: Metric event c75af67f-cf2f-433d-a060-a670087d93a1 takes no payload, but got <<@a{sv} {}>>
[ERROR] azafea.event_processors.endless.metrics.events._base: An error occured while processing the event:
Traceback (most recent call last):
  File "/azafea/azafea/event_processors/endless/metrics/events/_base.py", line 520, in replay_unknown_singular_events
    payload=payload)
  File "<string>", line 4, in __init__
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 441, in _initialize_instance
    manager.dispatch.init_failure(self, args, kwargs)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 438, in _initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/azafea/azafea/event_processors/endless/metrics/events/_base.py", line 122, in __init__
    payload_fields = self._parse_payload(payload)
  File "/azafea/azafea/event_processors/endless/metrics/events/_base.py", line 145, in _parse_payload
    raise WrongPayloadError(f'Metric event {self.__event_uuid__} needs a '
azafea.event_processors.endless.metrics.events._base.WrongPayloadError: Metric event 449ec188-cb7b-45d3-a0ed-291d943b9aa6 needs a a{sv} payload, but got [{'AllowSystemInstallation': <true>, 'AllowUserInstallation': <true>, 'IsAdministrator': <true>, 'OarsFilter': <('oars-1.1', @a{ss} {})>, 'AppFilter': <(false, @as [])>}, {'AllowSystemInstallation': <false>, 'AllowUserInstallation': <true>, 'IsAdministrator': <false>, 'OarsFilter': <('oars-1.1', @a{ss} {})>, 'AppFilter': <(false, @as [])>}, {'AllowSystemInstallation': <true>, 'AllowUserInstallation': <true>, 'IsAdministrator': <true>, 'OarsFilter': <('oars-1.1', @a{ss} {})>, 'AppFilter': <(false, @as [])>}] (aa{sv})
[ERROR] azafea.event_processors.endless.metrics.events._base: An error occured while processing the event:
Traceback (most recent call last):
  File "/azafea/azafea/event_processors/endless/metrics/events/_base.py", line 520, in replay_unknown_singular_events
    payload=payload)
  File "<string>", line 4, in __init__
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 441, in _initialize_instance
    manager.dispatch.init_failure(self, args, kwargs)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 438, in _initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/azafea/azafea/event_processors/endless/metrics/events/_base.py", line 122, in __init__
    payload_fields = self._parse_payload(payload)
  File "/azafea/azafea/event_processors/endless/metrics/events/_base.py", line 145, in _parse_payload
    raise WrongPayloadError(f'Metric event {self.__event_uuid__} needs a '
azafea.event_processors.endless.metrics.events._base.WrongPayloadError: Metric event 449ec188-cb7b-45d3-a0ed-291d943b9aa6 needs a a{sv} payload, but got [{'AllowSystemInstallation': <true>, 'AllowUserInstallation': <true>, 'IsAdministrator': <true>, 'OarsFilter': <('oars-1.1', @a{ss} {})>, 'AppFilter': <(false, @as [])>}, {'AllowSystemInstallation': <false>, 'AllowUserInstallation': <true>, 'IsAdministrator': <false>, 'OarsFilter': <('oars-1.1', @a{ss} {})>, 'AppFilter': <(false, @as [])>}] (aa{sv})
[ERROR] azafea.event_processors.endless.metrics.events._base: An error occured while processing the event:
Traceback (most recent call last):
  File "/azafea/azafea/event_processors/endless/metrics/events/_base.py", line 520, in replay_unknown_singular_events
    payload=payload)
  File "<string>", line 4, in __init__
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 441, in _initialize_instance
    manager.dispatch.init_failure(self, args, kwargs)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 438, in _initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/azafea/azafea/event_processors/endless/metrics/events/_base.py", line 122, in __init__
    payload_fields = self._parse_payload(payload)
  File "/azafea/azafea/event_processors/endless/metrics/events/_base.py", line 145, in _parse_payload
    raise WrongPayloadError(f'Metric event {self.__event_uuid__} needs a '
azafea.event_processors.endless.metrics.events._base.WrongPayloadError: Metric event 449ec188-cb7b-45d3-a0ed-291d943b9aa6 needs a a{sv} payload, but got [{'AllowSystemInstallation': <true>, 'AllowUserInstallation': <true>, 'IsAdministrator': <true>, 'OarsFilter': <('oars-1.1', @a{ss} {})>, 'AppFilter': <(false, @as [])>}, {'AllowSystemInstallation': <false>, 'AllowUserInstallation': <true>, 'IsAdministrator': <false>, 'OarsFilter': <('oars-1.1', @a{ss} {})>, 'AppFilter': <(false, @as [])>}] (aa{sv})
[ERROR] azafea.event_processors.endless.metrics.events._base: An error occured while processing the event:
Traceback (most recent call last):
  File "/azafea/azafea/event_processors/endless/metrics/events/_base.py", line 520, in replay_unknown_singular_events
    payload=payload)
  File "<string>", line 4, in __init__
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 441, in _initialize_instance
    manager.dispatch.init_failure(self, args, kwargs)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 438, in _initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/azafea/azafea/event_processors/endless/metrics/events/_base.py", line 122, in __init__
    payload_fields = self._parse_payload(payload)
  File "/azafea/azafea/event_processors/endless/metrics/events/_base.py", line 145, in _parse_payload
    raise WrongPayloadError(f'Metric event {self.__event_uuid__} needs a '
azafea.event_processors.endless.metrics.events._base.WrongPayloadError: Metric event 449ec188-cb7b-45d3-a0ed-291d943b9aa6 needs a a{sv} payload, but got [{'AllowSystemInstallation': <true>, 'AllowUserInstallation': <true>, 'IsAdministrator': <true>, 'OarsFilter': <('oars-1.1', @a{ss} {})>, 'AppFilter': <(false, @as [])>}, {'AllowSystemInstallation': <false>, 'AllowUserInstallation': <true>, 'IsAdministrator': <false>, 'OarsFilter': <('oars-1.1', @a{ss} {})>, 'AppFilter': <(false, @as [])>}, {'AllowSystemInstallation': <true>, 'AllowUserInstallation': <true>, 'IsAdministrator': <true>, 'OarsFilter': <('oars-1.1', @a{ss} {})>, 'AppFilter': <(false, @as [])>}] (aa{sv})
[ERROR] azafea.event_processors.endless.metrics.events._base: An error occured while processing the event:
Traceback (most recent call last):
  File "/azafea/azafea/event_processors/endless/metrics/events/_base.py", line 520, in replay_unknown_singular_events
    payload=payload)
  File "<string>", line 4, in __init__
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 441, in _initialize_instance
    manager.dispatch.init_failure(self, args, kwargs)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 438, in _initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/azafea/azafea/event_processors/endless/metrics/events/_base.py", line 122, in __init__
    payload_fields = self._parse_payload(payload)
  File "/azafea/azafea/event_processors/endless/metrics/events/_base.py", line 145, in _parse_payload
    raise WrongPayloadError(f'Metric event {self.__event_uuid__} needs a '
azafea.event_processors.endless.metrics.events._base.WrongPayloadError: Metric event 449ec188-cb7b-45d3-a0ed-291d943b9aa6 needs a a{sv} payload, but got [{'AllowSystemInstallation': <true>, 'AllowUserInstallation': <true>, 'IsAdministrator': <true>, 'OarsFilter': <('oars-1.1', @a{ss} {})>, 'AppFilter': <(false, @as [])>}, {'AllowSystemInstallation': <false>, 'AllowUserInstallation': <true>, 'IsAdministrator': <false>, 'OarsFilter': <('oars-1.1', @a{ss} {})>, 'AppFilter': <(false, @as [])>}, {'AllowSystemInstallation': <true>, 'AllowUserInstallation': <true>, 'IsAdministrator': <true>, 'OarsFilter': <('oars-1.1', @a{ss} {})>, 'AppFilter': <(false, @as [])>}] (aa{sv})
[ERROR] azafea.event_processors.endless.metrics.events._base: An error occured while processing the event:
Traceback (most recent call last):
  File "/azafea/azafea/event_processors/endless/metrics/events/_base.py", line 520, in replay_unknown_singular_events
    payload=payload)
  File "<string>", line 4, in __init__
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 441, in _initialize_instance
    manager.dispatch.init_failure(self, args, kwargs)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 438, in _initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/azafea/azafea/event_processors/endless/metrics/events/_base.py", line 122, in __init__
    payload_fields = self._parse_payload(payload)
  File "/azafea/azafea/event_processors/endless/metrics/events/_base.py", line 145, in _parse_payload
    raise WrongPayloadError(f'Metric event {self.__event_uuid__} needs a '
azafea.event_processors.endless.metrics.events._base.WrongPayloadError: Metric event 449ec188-cb7b-45d3-a0ed-291d943b9aa6 needs a a{sv} payload, but got [{'AllowSystemInstallation': <true>, 'AllowUserInstallation': <true>, 'IsAdministrator': <true>, 'OarsFilter': <('oars-1.1', @a{ss} {})>, 'AppFilter': <(false, @as [])>}, {'AllowSystemInstallation': <false>, 'AllowUserInstallation': <true>, 'IsAdministrator': <false>, 'OarsFilter': <('oars-1.1', @a{ss} {})>, 'AppFilter': <(false, @as [])>}, {'AllowSystemInstallation': <true>, 'AllowUserInstallation': <true>, 'IsAdministrator': <true>, 'OarsFilter': <('oars-1.1', @a{ss} {})>, 'AppFilter': <(false, @as [])>}] (aa{sv})
[ERROR] azafea.event_processors.endless.metrics.events._base: An error occured while processing the event:
Traceback (most recent call last):
  File "/azafea/azafea/event_processors/endless/metrics/events/_base.py", line 520, in replay_unknown_singular_events
    payload=payload)
  File "<string>", line 4, in __init__
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 441, in _initialize_instance
    manager.dispatch.init_failure(self, args, kwargs)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 438, in _initialize_instance
    return manager.original_init(*mixed[1:], **kwargs)
  File "/azafea/azafea/event_processors/endless/metrics/events/_base.py", line 122, in __init__
    payload_fields = self._parse_payload(payload)
  File "/azafea/azafea/event_processors/endless/metrics/events/_base.py", line 145, in _parse_payload
    raise WrongPayloadError(f'Metric event {self.__event_uuid__} needs a '
azafea.event_processors.endless.metrics.events._base.WrongPayloadError: Metric event 449ec188-cb7b-45d3-a0ed-291d943b9aa6 needs a a{sv} payload, but got [{'AllowSystemInstallation': <true>, 'AllowUserInstallation': <true>, 'IsAdministrator': <true>, 'OarsFilter': <('oars-1.1', @a{ss} {})>, 'AppFilter': <(false, @as [])>}, {'AllowSystemInstallation': <false>, 'AllowUserInstallation': <true>, 'IsAdministrator': <false>, 'OarsFilter': <('oars-1.1', @a{ss} {})>, 'AppFilter': <(false, @as [])>}, {'AllowSystemInstallation': <true>, 'AllowUserInstallation': <true>, 'IsAdministrator': <true>, 'OarsFilter': <('oars-1.1', @a{ss} {})>, 'AppFilter': <(false, @as [])>}] (aa{sv})
**
GLib:ERROR:../glib/gvariant-serialiser.c:1367:g_variant_serialised_n_children: code should not be reached
pwithnall commented 4 years ago 
GLib:ERROR:../glib/gvariant-serialiser.c:1367:g_variant_serialised_n_children: code should not be reached

certainly that last error looks like my bug with parental controls, which should be fixed by #107

dbnicholson commented 4 years ago

Oh, good catch. Even though we want to fix this particular case, azafea shouldn't crash when sent an unexpected variant. The exception should be caught, logged and the event rejected. I've never looked that closely at this code but having done lots of variant handling with ostree in Python it shouldn't be too hard to catch. We should add a test for that, too.

pwithnall commented 4 years ago

I filed https://phabricator.endlessm.com/T29910 about it. I don’t think it’s hard to fix for any one particular metrics event, but would be a fair amount of work (including testing) to fix for all of them.

dbnicholson commented 4 years ago

Thanks. Without having looked at the code it seems that WrongPayloadError is being raised by the event processor, but I can't tell if that GVariant assertion is happening afterwards. So, maybe the error is not where I'm expecting. That said, what I think you'd want to do is catch the exception at a higher level of azafea so that an event processing error doesn't bring down the whole process. So, for event processing unit tests you'd still look for specific exceptions. But at the higher level that passes in the event for processing you'd catch the generic exception, log it, reject the event and carry on.

pwithnall commented 4 years ago

The assertion failures I’m talking about are GVariant assertions from within GLib, and they abort the python process; no Python exceptions are involved. There are definitely several different bugs at play in the error log in the 0th comment of this issue.