This morning, SBS sent 30k error mails in about 10 minutes. A user was trying to add users to a group, when somehting happend and SBS began to produce (client-side) errors at an incredible rate:
Jun 25 07:49:25 sbs-pf1 nginx[588]: 10.0.1.20 2a09:bac2:4d80:1478::20a:83 - [25/Jun/2024:07:49:25 +0000] "sram.surf.nl" "GET /collaborations/465/groups/1340 HTTP/1.1" 200 965 "https://engine.surfconext.nl/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15"
Jun 25 07:49:25 sbs-pf1 nginx[588]: 10.0.1.20 2a09:bac2:4d80:1478::20a:83 - [25/Jun/2024:07:49:25 +0000] "sram.surf.nl" "GET /config HTTP/1.1" 200 655 "https://sram.surf.nl/collaborations/465/groups/1340" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15"
Jun 25 07:49:25 sbs-pf1 nginx[588]: 10.0.1.20 2a09:bac2:4d80:1478::20a:83 - [25/Jun/2024:07:49:25 +0000] "sram.surf.nl" "GET /api/users/me HTTP/1.1" 200 26790 "https://sram.surf.nl/collaborations/465/groups/1340" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15"
Jun 25 07:49:25 sbs-pf1 nginx[588]: 10.0.1.20 2a09:bac2:4d80:1478::20a:83 - [25/Jun/2024:07:49:25 +0000] "sram.surf.nl" "GET /api/collaborations/access_allowed/465 HTTP/1.1" 200 18 "https://sram.surf.nl/collaborations/465/groups/1340" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15"
Jun 25 07:49:25 sbs-pf1 nginx[588]: 10.0.1.20 2a09:bac2:4d80:1478::20a:83 - [25/Jun/2024:07:49:25 +0000] "sram.surf.nl" "GET /api/user_tokens HTTP/1.1" 200 3 "https://sram.surf.nl/collaborations/465/groups/1340" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15"
Jun 25 07:49:25 sbs-pf1 gunicorn[68337]: INFO [base] user: cd3e0f328b75fa7177e965dd8fb4b8990c233303@sram.surf.nl, Path /api/users/error POST {"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15", "message": "Script error.", "url": "", "line": 0, "col": 0}
Jun 25 07:49:25 sbs-pf1 nginx[588]: 10.0.1.20 2a09:bac2:4d80:1478::20a:83 - [25/Jun/2024:07:49:25 +0000] "sram.surf.nl" "POST /api/users/error HTTP/1.1" 201 3 "https://sram.surf.nl/collaborations/465/groups/1340" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15"
Jun 25 07:49:25 sbs-pf1 gunicorn[68337]: ERROR [user] user: cd3e0f328b75fa7177e965dd8fb4b8990c233303@sram.surf.nl, {
Jun 25 07:49:25 sbs-pf1 gunicorn[68337]: "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15",
Jun 25 07:49:25 sbs-pf1 gunicorn[68337]: "message": "Script error.",
Jun 25 07:49:25 sbs-pf1 gunicorn[68337]: "url": "",
Jun 25 07:49:25 sbs-pf1 gunicorn[68337]: "line": 0,
Jun 25 07:49:25 sbs-pf1 gunicorn[68337]: "col": 0
Jun 25 07:49:25 sbs-pf1 gunicorn[68337]: }
Jun 25 07:49:25 sbs-pf1 gunicorn[68337]: NoneType: None
Jun 25 07:49:25 sbs-pf1 gunicorn[68337]: DEBUG [user] user: cd3e0f328b75fa7177e965dd8fb4b8990c233303@sram.surf.nl, Sending mail message to sram-logs@surf.nl with Message-id <cca56397.648b.4c0b.8d59.7c3cc606eced@sbs.pf1.internal.sram.surf.nl>
Jun 25 07:49:26 sbs-pf1 nginx[588]: 10.0.1.20 2a09:bac2:4d80:1478::20a:83 - [25/Jun/2024:07:49:26 +0000] "sram.surf.nl" "GET /api/images/collaborations/54f81bbf-4d2d-4999-8302-73fb66e034c2 HTTP/1.1" 200 20938 "https://sram.surf.nl/collaborations/465/groups/1340" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15"
Jun 25 07:49:26 sbs-pf1 nginx[588]: 10.0.1.20 2a09:bac2:4d80:1478::20a:83 - [25/Jun/2024:07:49:26 +0000] "sram.surf.nl" "GET /static/media/SourceSansPro-Bold.c06c1e4b0d684ef5c0b1.woff2 HTTP/1.1" 200 77684 "https://sram.surf.nl/collaborations/465/groups/1340" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15"
Jun 25 07:49:26 sbs-pf1 gunicorn[67977]: INFO [base] user: cd3e0f328b75fa7177e965dd8fb4b8990c233303@sram.surf.nl, Path /api/users/error POST {"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15", "message": "Script error.", "url": "", "line": 0, "col": 0}
Jun 25 07:49:26 sbs-pf1 nginx[588]: 10.0.1.20 2a09:bac2:4d80:1478::20a:83 - [25/Jun/2024:07:49:26 +0000] "sram.surf.nl" "POST /api/users/error HTTP/1.1" 201 3 "https://sram.surf.nl/collaborations/465/groups/1340" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15"
Jun 25 07:49:26 sbs-pf1 gunicorn[67977]: ERROR [user] user: cd3e0f328b75fa7177e965dd8fb4b8990c233303@sram.surf.nl, {
Jun 25 07:49:26 sbs-pf1 gunicorn[67977]: "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15",
Jun 25 07:49:26 sbs-pf1 gunicorn[67977]: "message": "Script error.",
Jun 25 07:49:26 sbs-pf1 gunicorn[67977]: "url": "",
Jun 25 07:49:26 sbs-pf1 gunicorn[67977]: "line": 0,
Jun 25 07:49:26 sbs-pf1 gunicorn[67977]: "col": 0
Jun 25 07:49:26 sbs-pf1 gunicorn[67977]: }
Jun 25 07:49:26 sbs-pf1 gunicorn[67977]: NoneType: None
Jun 25 07:49:26 sbs-pf1 gunicorn[67977]: DEBUG [user] user: cd3e0f328b75fa7177e965dd8fb4b8990c233303@sram.surf.nl, Sending mail message to sram-logs@surf.nl with Message-id <3867bc14.fc66.41ab.a80e.a4313b79a131@sbs.pf1.internal.sram.surf.nl>
...
This seems rather hard to debug, but we should at least rate-limit the error sending per user on the server side.
mrvanes
commented
6 days ago
This morning, SBS sent 30k error mails in about 10 minutes. A user was trying to add users to a group, when somehting happend and SBS began to produce (client-side) errors at an incredible rate:
This seems rather hard to debug, but we should at least rate-limit the error sending per user on the server side.