Open gbrownmozilla opened 2 years ago
I think this means https://github.com/mozilla-releng/balrog/blob/5cb0ab3c11b61d1252aab14f9deb7a9dbdb9bff4/src/auslib/services/releases.py#L486 has returned None.
Today there were some failures in Balrog submission-of-updates tasks in a staging release https://treeherder.mozilla.org/jobs?repo=try&group_state=expanded&selectedTaskRun=G35nLntMQvuFoCaKXaHQpw.0&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=53c5d2ecac3074646b91bee763382aaf83ef0a87 that seemed correlated with this old_data_version error in sentry. In this case, somehow two promote_devedition tasks had been spawned just a second apart, resulting in duplicate Balrog submission tasks; for each of those, the first task succeeded, the next failed (task descriptions seemed pretty much identical, and the submitted old_data_version seemed identical).
2022-07-05 18:26:36,308 - balrogclient.api - DEBUG - Balrog request to https://admin-stage.balrog.nonprod.cloudops.mozgcp.net/api/v2/releases/Devedition-104.0b1-build1 via POST
2022-07-05 18:26:36,308 - balrogclient.api - DEBUG - Data sent: {'blob': {'platforms': {'Linux_x86-gcc3': {'locales': {'en-CA': {'buildID': '20220705111051', 'appVersion': '104.0', 'displayVersion': '104.0 Beta 1', 'completes': [{'from': '*', 'filesize': 66371739, 'hashValue': '2dfa6daf912595ba6c82701ac25428163d3d9c9c28bc4853e9dc38ec3410000fe56f2dcac88e7fb7635715c083edd3ce097577ab12a487e4b957663c6851f119'}], 'partials': [{'from': 'Devedition-74.0b4-build4', 'filesize': 57319725, 'hashValue': 'fdecf6ac2d335a226e329ca013c1aedf7fdc1ca3855f87fb4f61945ddfa691cf6304f95a4cce0adaec6b7d9dbc91d0d81de2e96553667644382bc3b6c8439770'}]}}}}}, 'old_data_versions': {'platforms': {'Linux_x86-gcc3': {'locales': {}}}}}
2022-07-05 18:26:36,310 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): admin-stage.balrog.nonprod.cloudops.mozgcp.net:443
2022-07-05 18:26:36,362 - urllib3.connectionpool - DEBUG - https://admin-stage.balrog.nonprod.cloudops.mozgcp.net:443 "POST /api/v2/releases/Devedition-104.0b1-build1 HTTP/1.1" 500 107
2022-07-05 18:26:36,364 - balrogclient.api - ERROR - Caught HTTPError: b'{"type": "about:blank", "title": "Internal Server Error", "detail": "Internal Server Error", "status": 500}'
vs.
2022-07-05 18:20:07,830 - balrogclient.api - DEBUG - Balrog request to https://admin-stage.balrog.nonprod.cloudops.mozgcp.net/api/v2/releases/Devedition-104.0b1-build1 via POST
2022-07-05 18:20:07,830 - balrogclient.api - DEBUG - Data sent: {'blob': {'platforms': {'Linux_x86-gcc3': {'locales': {'en-CA': {'buildID': '20220705111051', 'appVersion': '104.0', 'displayVersion': '104.0 Beta 1', 'completes': [{'from': '*', 'filesize': 66372719, 'hashValue': 'b20305ef753793444771085df3973178c0648c47a48a8adcd48b6a9d9bff7459791dbd0bba60eba078c06f7349a131aae4f1d1eb4be2db5a8d29be0aef25d4f1'}], 'partials': [{'from': 'Devedition-74.0b4-build4', 'filesize': 57318301, 'hashValue': '3006aaa59668522f8a70e617e6a81d86ed0e8cc39f57106c72473a8e665bfbe51d81da91a6e1cf3193a249305d107c01979412c778ce308204ac7439bfcec287'}]}}}}}, 'old_data_versions': {'platforms': {'Linux_x86-gcc3': {'locales': {}}}}}
2022-07-05 18:20:07,832 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): admin-stage.balrog.nonprod.cloudops.mozgcp.net:443
2022-07-05 18:20:08,048 - urllib3.connectionpool - DEBUG - https://admin-stage.balrog.nonprod.cloudops.mozgcp.net:443 "POST /api/v2/releases/Devedition-104.0b1-build1 HTTP/1.1" 200 101
2022-07-05 18:20:08,050 - balrogclient.api - INFO - Data recieved: {"platforms": {"Linux_x86-gcc3": {"locales": {"en-CA": 1}}}}
2022-07-05 18:20:08,050 - balrogclient.api - DEBUG - REQUEST STATS: {"timestamp": 1657045208.0502152, "method": "POST", "url": "https://admin-stage.balrog.nonprod.cloudops.mozgcp.net/api/v2/releases/Devedition-104.0b1-build1", "status_code": 200, "elapsed_secs": 0.217107}
2022-07-05 18:20:08,053 - balrogscript.script - INFO - Taskcluster Release style Balrog submission
https://treeherder.mozilla.org/jobs?repo=try&revision=cc3b4c172fe9edd7eb04567a3c8aa3a00c0653a7 https://firefox-ci-tc.services.mozilla.com/tasks/GXrfm60gSyGhrcFNnuPzbQ/runs/0/logs/public/logs/live_backing.log
From sentry: https://sentry.io/organizations/mozilla/issues/3122737435/?project=6262499&statsPeriod=24h TypeError /v2.auslib_web_admin_views_releases_v2_update_release unsupported operand type(s) for +: 'NoneType' and 'int'
We have been hitting this TypeError occasionally for at least several months on staging, but I don't see it at all in production.