dart-lang / pub-dev

The pub.dev website
https://pub.dev
BSD 3-Clause "New" or "Revised" License
787 stars 147 forks source link

CCBot/2.0 is failing: Invalid HTTP date: expected " GMT" in "Thu, 19 Jul 2018 01:06:14 MSK" #1651

Closed sortie closed 6 years ago

sortie commented 6 years ago

The CCBot/2.0 crawler (https://commoncrawl.org/faq/) requests are dying with HTTP 500. This brings up the user error metric enough that our alerts trigger spuriously. This needs to be fixed.

pub: Handling request: https://pub.dartlang.org/documentation/modern_charts/latest/
pub: Request handler failed

Error:
    Error on line 1, column 26: Invalid HTTP date: expected " GMT".
    Thu, 19 Jul 2018 01:06:14 MSK
                             ^

Stack:
    package:http_parser/src/utils.dart 15:5                       wrapFormatException
    package:http_parser/src/http_date.dart 63:10                  parseHttpDate
    package:shelf/src/request.dart 78:29                          Request.ifModifiedSince
    package:pub_dartlang_org/shared/handlers.dart 104:35          isNotModified
    package:pub_dartlang_org/dartdoc/handlers.dart 103:9          documentationHandler
    package:pub_dartlang_org/frontend/handlers.dart 97:12         appHandler
    bin/service/frontend.dart 56:36                               _main.<fn>.<fn>
    package:pub_dartlang_org/shared/handler_helpers.dart 84:25    _userAuthParsingWrapper.<fn>
    package:pub_dartlang_org/shared/handler_helpers.dart 77:25    _sanitizeRequestWrapper.<fn>
    package:pub_dartlang_org/shared/handler_helpers.dart 95:27    _redirectToHttpsWrapper.<fn>
    package:pub_dartlang_org/shared/handler_helpers.dart 43:27    _logRequestWrapper.<fn>
    package:shelf/shelf_io.dart 88:29                             handleRequest
    package:pub_dartlang_org/shared/handler_helpers.dart 37:5     _handleRequest
    package:pub_dartlang_org/shared/handler_helpers.dart 32:48    runHandler.<fn>
    package:appengine/appengine.dart 65:12                        runAppEngine.<fn>
    package:appengine/src/appengine_internal.dart 62:16           runAppEngine.<fn>.<fn>.<fn>

It seems parsing the the date in the HTTP header is failing.

isoos commented 6 years ago