The crawler pipeline errors out and 500s, logging failed operations and exceptions during regular operation (e.g. when the underlying url is missing, or due to a transient failure at the target host). There is already a pattern for silently dealing with these failures - this is desirable as it reduces the noise in monitoring.
This change extends this pattern to underlying 404s and 503s, as well as adding additional telemetry when the target server responds in a way that triggers ServerProtocolViolation.
The crawler pipeline errors out and 500s, logging failed operations and exceptions during regular operation (e.g. when the underlying url is missing, or due to a transient failure at the target host). There is already a pattern for silently dealing with these failures - this is desirable as it reduces the noise in monitoring.
This change extends this pattern to underlying 404s and 503s, as well as adding additional telemetry when the target server responds in a way that triggers ServerProtocolViolation.