Add the capability for the CLI handle pages with server errors

[fellyph]

Feature Request

Description: Some URLs can return a server error. With the current version of the CLI, the audit continues without any alert, and using the CLI, the user cannot see if the audited URL has returned some error. Some error pages could use cookies for tracking. However, to simplify the audit process, when a URL returns an error, the CLI can skip extra steps from the audit.

This feature requests a warning to notify the user the report was taken from an error page, and observing the CLI over the error page, some steps could be skipped when an error page is analyzed, such as waiting 10 seconds to load the page completely and looking for a GDPR banner.

The CLI has three different scenarios where this functionally should handle with different strategies:

• Single page audit: for single page audit, the CLI should skip the audit and return a warning message
• CSV audit: for the list of URLs, the audit should skip the URLs with errors, notify the user each URL has encountered an error, and continue with the rest for the valid URLs.
• Sitemap audit: For sitemaps, the audit should skip the URLs with errors, notify the user when each URL has encountered an error, and continue with the rest of the valid URLs.

Some errors page to be considered to create a warning:

• 400 Bad request
• 401 Unauthorized
• 403 Forbidden
• 404 Not Found
• 408 Request Timeout
• 500 Internal server error
• 502 Bad Gateway
• 503 Service Unavailable

• 504 Gateway Timout

  UI suggestion

The report should be created when the audit is running for a list of URLs that still have some valid URLs. In this case, the CLI should create a report file; for URLs with an error, the CLI should warn the user about it. Currently, the dashboard has a component to display warnings for zero cookies pages, it can be reused to notify the server error response warning.

Audit process

If the server returns an error, skip the last steps from the audit:

For the list of URLs such as, csv and sitemaps, the user should be notified but the audit should continue looking for valid URLs.

Motivation: Avoid giving false positives to users. When the server returns an error, notify the users with a warning to provide feedback with the report that can be affected by the error returned by the server.

User Story: As a user, I want to get feedback from server errors so that I can understand the data returned more clearly.

Acceptance Criteria:

• [ ] Inspect a single URL, sitemap and CSV with the CLI
• [ ] Get a warning on the report
• [ ] Skip some steps when the server returns an error
• [ ] Audit for Sitemap and CSV should continue looking for valid URLs

Related : #802

[fellyph]

@amedina suggested skipping the audit if the server returns an error.

[fellyph]

When the user audits a sitemap URL that returns a 404 error, the CLI continues the audit and generates a report with a white screen:

We should include the verification for server errors for webpages and sitemaps