search

dart-lang / pub-dev

The pub.dev website
https://pub.dev
BSD 3-Clause "New" or "Revised" License
774 stars 145 forks source link

Crashing isolate due to uncaught exception: ClientException #7482

Open jonasfj opened 7 months ago

jonasfj commented 7 months ago

@isoos if we're using RetryClient from package:http am I correct in that it won't retry if partial response is received. Like in the case of ClientException?

I'm guessing this error isn't the end of the world, but we probably shouldn't crash due to this. I'm also guessing that this isn't the only place where we have this issue.

pub.analyzer: ERROR from index-builder isolate #1

Error:
    [Exception: Crashing isolate due to uncaught exception: ClientException: Connection closed before full header was received, uri=https://storage.googleapis.com/resumable/upload/storage/v1/b/dartlang-pub--search-snapshot/o?name=snapshot%2F2024.02.13.json.gz&uploadType=resumable&alt=json, #0      _wrapper.<anonymous closure> (package:pub_dev/service/entrypoint/_isolate.dart:316:9)
    #1      _RootZone.runBinary (dart:async/zone.dart:1666:54)
    #2      StackZoneSpecification._handleUncaughtError (package:stack_trace/src/stack_zone_specification.dart:161:20)
    #3      _Zone._processUncaughtError (dart:async/zone.dart:1081:14)
    #4      _CustomZone.handleUncaughtError (dart:async/zone.dart:1285:5)
    #5      Future._propagateToListeners (dart:async/future_impl.dart:760:16)
    #6      Future._completeError (dart:async/future_impl.dart:655:5)
    #7      Future._asyncCompleteError.<anonymous closure> (dart:async/future_impl.dart:745:7)
    #8      StackZoneSpecification._run (package:stack_trace/src/stack_zone_specification.dart:207:15)
    #9      StackZoneSpecification._registerCallback.<anonymous closure> (package:stack_trace/src/stack_zone_specification.dart:114:48)
    #10     _rootRun (dart:async/zone.dart:1399:13)
    #11     _CustomZone.run (dart:async/zone.dart:1301:19)
    #12     _CustomZone.runGuarded (dart:async/zone.dart:1209:7)
    #13     _CustomZone.bindCallbackGuarded.<anonymous closure> (dart:async/zone.dart:1249:23)
    #14     _microtaskLoop (dart:async/schedule_microtask.dart:40:21)
    #15     _startMicrotaskLoop (dart:async/schedule_microtask.dart:49:5)
    #16     _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:118:13)
    #17     _RawReceivePort._handleMessage (dart:isolate-patch/isolate_patch.dart:185:5)
    ]

Also related I think:

isolate.wrapper: Uncaught exception in isolate.

Error:
    ClientException: Connection closed before full header was received, uri=https://storage.googleapis.com/resumable/upload/storage/v1/b/dartlang-pub--search-snapshot/o?name=snapshot%2F2024.02.13.json.gz&uploadType=resumable&alt=json

Stack:
    package:http/src/io_client.dart 156:7                    IOClient.send
    ===== asynchronous gap ===========================
    dart:async                                               _Completer.completeError
    package:gcloud/src/storage_impl.dart 629:20              _MediaUploadStreamSink._completeError
    ===== asynchronous gap ===========================
    dart:async                                               Future.then
    package:gcloud/src/storage_impl.dart 659:10              _MediaUploadStreamSink._startResumableUpload
    package:gcloud/src/storage_impl.dart 541:9               new _MediaUploadStreamSink
    package:gcloud/src/storage_impl.dart 199:16              _BucketImpl.write
    package:pub_dev/shared/storage.dart 255:27               uploadWithRetry.<fn>
    package:pub_dev/shared/utils.dart 186:24                 retryAsync
    package:pub_dev/shared/storage.dart 253:9                uploadWithRetry
    package:pub_dev/shared/storage.dart 276:5                uploadBytesWithRetry
    package:pub_dev/shared/storage.dart 300:13               VersionedJsonStorage.uploadDataAsJsonMap
    package:pub_dev/search/backend.dart 212:32               SearchBackend.doCreateAndUpdateSnapshot
    ===== asynchronous gap ===========================
    dart:async                                               _CustomZone.registerUnaryCallback
    package:pub_dev/search/backend.dart 159:5                SearchBackend.doCreateAndUpdateSnapshot
    ===== asynchronous gap ===========================
    dart:async                                               _CustomZone.registerUnaryCallback
    package:pub_dev/search/backend.dart 159:5                SearchBackend.doCreateAndUpdateSnapshot
    package:pub_dev/search/backend.dart 106:17               SearchBackend.updateSnapshotInForeverLoop.<fn>
    package:pub_dev/task/global_lock.dart 81:22              GlobalLock.withClaim
    ===== asynchronous gap ===========================
    dart:async                                               _CustomZone.registerUnaryCallback
    package:pub_dev/task/global_lock.dart 40:15              GlobalLock.withClaim
    package:pub_dev/search/backend.dart 105:20               SearchBackend.updateSnapshotInForeverLoop
    ===== asynchronous gap ===========================
    dart:async                                               _CustomZone.registerUnaryCallback
    package:pub_dev/search/backend.dart 105:9                SearchBackend.updateSnapshotInForeverLoop
    package:pub_dev/service/entrypoint/analyzer.dart 78:23   _indexBuilderMain
    ===== asynchronous gap ===========================
    dart:async                                               _CustomZone.registerUnaryCallback
    package:pub_dev/service/entrypoint/analyzer.dart 77:3    _indexBuilderMain
    package:pub_dev/service/entrypoint/_isolate.dart 311:43  _wrapper.<fn>.<fn>
    package:pub_dev/service/services.dart 295:28             _withPubServices.<fn>.<fn>.<fn>
    dart:async                                               _CustomZone.run
    package:pub_dev/service/services.dart 291:10             _withPubServices.<fn>.<fn>
    package:gcloud/service_scope.dart 205:19                 _ServiceScope._fork.<fn>
    dart:async                                               runZoned
    package:gcloud/service_scope.dart 204:12                 _ServiceScope._fork
    package:gcloud/service_scope.dart 102:30                 fork
    package:pub_dev/service/services.dart 288:18             _withPubServices.<fn>
    ===== asynchronous gap ===========================
    dart:async                                               _CustomZone.registerUnaryCallback
    package:pub_dev/service/services.dart 225:7              _withPubServices.<fn>
    package:gcloud/service_scope.dart 205:19                 _ServiceScope._fork.<fn>
    dart:async                                               runZoned
    package:gcloud/service_scope.dart 204:12                 _ServiceScope._fork
    package:gcloud/service_scope.dart 102:30                 fork
    package:pub_dev/service/services.dart 222:10             _withPubServices
    package:pub_dev/service/services.dart 127:20             withServices.<fn>.<fn>
    ===== asynchronous gap ===========================
    dart:async                                               _CustomZone.registerUnaryCallback
    package:pub_dev/service/services.dart 84:26              withServices.<fn>.<fn>
    package:gcloud/service_scope.dart 205:19                 _ServiceScope._fork.<fn>
    dart:async                                               runZoned
    package:gcloud/service_scope.dart 204:12                 _ServiceScope._fork
    package:gcloud/service_scope.dart 102:30                 fork
    package:pub_dev/service/services.dart 82:18              withServices.<fn>
    package:appengine/src/appengine_internal.dart 37:51      withAppEngineServices.<fn>
    package:appengine/src/appengine_internal.dart 110:20     _withAppEngineServicesInternal.<fn>
    ===== asynchronous gap ===========================
    dart:async                                               _CustomZone.registerUnaryCallback
    package:appengine/src/appengine_internal.dart 102:45     _withAppEngineServicesInternal.<fn>
    package:gcloud/service_scope.dart 205:19                 _ServiceScope._fork.<fn>
    dart:async                                               runZoned
    package:gcloud/service_scope.dart 204:12                 _ServiceScope._fork
    package:gcloud/service_scope.dart 102:30                 fork
    package:appengine/src/appengine_internal.dart 101:13     _withAppEngineServicesInternal
    package:appengine/src/appengine_internal.dart 37:5       withAppEngineServices
    package:appengine/appengine.dart 150:29                  withAppEngineServices
    package:pub_dev/service/services.dart 78:10              withServices
    package:pub_dev/service/entrypoint/_isolate.dart 311:22  _wrapper.<fn>
    package:stack_trace                                      Chain.capture
    package:pub_dev/service/entrypoint/_isolate.dart 310:24  _wrapper
isoos commented 7 months ago

Normally this should have been caught by this block: https://github.com/dart-lang/pub-dev/blob/master/app/lib/search/backend.dart#L104-L110

The only thing so far that may be suspicious is the concurrent processing inside it: if some exception would not have been caught while using Pool + Future.wait...

isoos commented 7 months ago

The other potential cause could be the scheduled microtask in lock.withClaim: https://github.com/dart-lang/pub-dev/blob/master/app/lib/task/global_lock.dart#L45