Failing notifications: google responds with 404

[missytake]

• Expected behavior: if we send notification tokens to FCM they accept them and wake up the device
• Actual behavior: fcm.googleapis.com returns an error message
• Steps to reproduce the problem: I don't know, it just appeared one morning
• Logs:

Nov 04 08:31:24 page notifiers[3151743]: {"level":40,"time":1730709084181,"msg":"Failed to deliver FCM notification to doALqBozT0O8NIHHMbftHW:APA91bHdKTY52mBWh8oYdVWYyKuCvNuEAlr2BRcI_ammxX6aLFJxg5HqJ3NuhMGkJAPGurtMaJ7ipYNlOdCnf3fpRNnQXS7qY1W_uRMJcZONlyWQc_rhzveyiGeVkYoFeXCtgfPDcQGV"}
Nov 04 08:31:24 page notifiers[3151743]: {"level":40,"time":1730709084181,"msg":"BODY: \"{\\\"message\\\":{\\\"token\\\":\\\"doALqBozT0O8NIHHMbftHW:APA91bHdKTY52mBWh8oYdVWYyKuCvNuEAlr2BRcI_ammxX6aLFJxg5HqJ3NuhMGkJAPGurtMaJ7ipYNlOdCnf3fpRNnQXS7qY1W_uRMJcZONlyWQc_rhzveyiGeVkYoFeXCtgfPDcQGV\\\",\\\"data\\\":{\\\"level\\\": \\\"awesome\\\"} } }\""}
Nov 04 08:31:24 page notifiers[3151743]: {"level":40,"time":1730709084181,"msg":"RES: Response { url: Url { scheme: \"https\", cannot_be_a_base: false, username: \"\", password: None, host: Some(Domain(\"fcm.googleapis.com\")), port: None, path: \"/v1/projects/delta-chat-fcm/messages:send\", query: None, fragment: None }, status: 404, headers: {\"vary\": \"X-Origin\", \"vary\": \"Referer\", \"vary\": \"Origin,Accept-Encoding\", \"content-type\": \"application/json; charset=UTF-8\", \"date\": \"Mon, 04 Nov 2024 08:31:24 GMT\", \"server\": \"scaffolding on HTTPServer2\", \"cache-control\": \"private\", \"x-xss-protection\": \"0\", \"x-frame-options\": \"SAMEORIGIN\", \"x-content-type-options\": \"nosniff\", \"alt-svc\": \"h3=\\\":443\\\"; ma=2592000,h3-29=\\\":443\\\"; ma=2592000\", \"accept-ranges\": \"none\", \"transfer-encoding\": \"chunked\"} }"}
Nov 04 08:31:24 page notifiers[3151743]: {"level":30,"time":1730709084271,"msg":"delivered notification for 6e34ddf6e631f2d6e257b804fc720ed07f8809f99e344da4aa4ed3ebb2fe48ae"}
Nov 04 08:31:24 page notifiers[3151743]: {"level":30,"time":1730709084271,"msg":"Sleeping for 3s 728ms 903us 446ns before next notification."}
Nov 04 08:31:24 page notifiers[3151743]: {"level":30,"time":1730709084271,"msg":"delivered notification for 708c05528b8f2a4cfd6f48d914a95c2cc3ba414e1229404ee5a44b03bb35b482"}
Nov 04 08:31:24 page notifiers[3151743]: {"level":30,"time":1730709084271,"msg":"Sleeping for 3s 728ms 775us 635ns before next notification."}
Nov 04 08:31:24 page notifiers[3151743]: {"level":30,"time":1730709084587,"msg":"Got direct notification for fcm-chat.delta.lite:doALqBozT0O8NIHHMbftHW:APA91bHdKTY52mBWh8oYdVWYyKuCvNuEAlr2BRcI_ammxX6aLFJxg5HqJ3NuhMGkJAPGurtMaJ7ipYNlOdCnf3fpRNnQXS7qY1W_uRMJcZONlyWQc_rhzveyiGeVkYoFeXCtgfPDcQGV."}
Nov 04 08:31:24 page notifiers[3151743]: {"level":40,"time":1730709084747,"msg":"Failed to deliver FCM notification to doALqBozT0O8NIHHMbftHW:APA91bHdKTY52mBWh8oYdVWYyKuCvNuEAlr2BRcI_ammxX6aLFJxg5HqJ3NuhMGkJAPGurtMaJ7ipYNlOdCnf3fpRNnQXS7qY1W_uRMJcZONlyWQc_rhzveyiGeVkYoFeXCtgfPDcQGV"}
Nov 04 08:31:24 page notifiers[3151743]: {"level":40,"time":1730709084747,"msg":"BODY: \"{\\\"message\\\":{\\\"token\\\":\\\"doALqBozT0O8NIHHMbftHW:APA91bHdKTY52mBWh8oYdVWYyKuCvNuEAlr2BRcI_ammxX6aLFJxg5HqJ3NuhMGkJAPGurtMaJ7ipYNlOdCnf3fpRNnQXS7qY1W_uRMJcZONlyWQc_rhzveyiGeVkYoFeXCtgfPDcQGV\\\",\\\"data\\\":{\\\"level\\\": \\\"awesome\\\"} } }\""}
Nov 04 08:31:24 page notifiers[3151743]: {"level":40,"time":1730709084747,"msg":"RES: Response { url: Url { scheme: \"https\", cannot_be_a_base: false, username: \"\", password: None, host: Some(Domain(\"fcm.googleapis.com\")), port: None, path: \"/v1/projects/delta-chat-fcm/messages:send\", query: None, fragment: None }, status: 404, headers: {\"vary\": \"X-Origin\", \"vary\": \"Referer\", \"vary\": \"Origin,Accept-Encoding\", \"content-type\": \"application/json; charset=UTF-8\", \"date\": \"Mon, 04 Nov 2024 08:31:24 GMT\", \"server\": \"scaffolding on HTTPServer2\", \"cache-control\": \"private\", \"x-xss-protection\": \"0\", \"x-frame-options\": \"SAMEORIGIN\", \"x-content-type-options\": \"nosniff\", \"alt-svc\": \"h3=\\\":443\\\"; ma=2592000,h3-29=\\\":443\\\"; ma=2592000\", \"accept-ranges\": \"none\", \"transfer-encoding\": \"chunked\"} }"}

[missytake]

It would help if we had heartbeat_failed_notifications metrics I think?

[link2xt]

Probably Google tries to push us to migrate away from v1 API: https://firebase.google.com/docs/cloud-messaging/migrate-v1

They want to return errors after July 2024, so guess it's happening. They promise to return 301 though, but maybe it is 301 and they redirected us somewhere where it returns 404: https://firebase.google.com/support/faq#deprecated-api-shutdown

[link2xt]

Threema has migrated it's relay at https://github.com/threema-ch/push-relay/pull/60