search

filiph / linkcheck

Fast link checker
https://pub.dartlang.org/packages/linkcheck
MIT License
403 stars 51 forks source link

Linkcheck Internal errors and Unhandled exception #35

Closed zomdir closed 5 years ago

zomdir commented 5 years ago

Hi Filip,

Again compliments. Linkcheck is the fastest linkchecker I'm aware of. Here are some issues I've found.

Best regards, Hans

$ linkcheck wordpress.org Crawling: 459INTERNAL ERROR: Sorry! Please open https://github.com/filiph/linkcheck/issues/new in your favorite browser and copy paste the following output there:

Invalid argument(s): Text "" must be 73 characters long.

736INTERNAL ERROR: Sorry! Please open https://github.com/filiph/linkcheck/issues/new in your favorite browser and copy paste the following output there:

Invalid argument(s): Text "" must be 128 characters long.

$ linkcheck https://autorijschoolokido.nl/ Crawling: 96Unhandled exception: NoSuchMethodError: The getter 'primaryType' was called on null. Receiver: null Tried calling: primaryType

0 Object.noSuchMethod (dart:core/runtime/libobject_patch.dart:50:5)

1 DestinationResult.updateFromResponse (package:linkcheck/src/destination.dart:327:48)

2 checkPage (package:linkcheck/src/worker/worker.dart:127:11)

#3 worker. (package:linkcheck/src/worker/worker.dart:192:29) #4 _RootZone.runUnaryGuarded (dart:async/zone.dart:1314:10) #5 _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:336:11) #6 _BufferingStreamSubscription._add (dart:async/stream_impl.dart:263:7) #7 _SyncStreamController._sendData (dart:async/stream_controller.dart:764:19) #8 _StreamController._add (dart:async/stream_controller.dart:640:7) #9 _StreamController.add (dart:async/stream_controller.dart:586:5) #10 _RootZone.runUnaryGuarded (dart:async/zone.dart:1314:10) #11 _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:336:11) #12 _BufferingStreamSubscription._add (dart:async/stream_impl.dart:263:7) #13 _SyncStreamController._sendData (dart:async/stream_controller.dart:764:19) #14 _StreamController._add (dart:async/stream_controller.dart:640:7) #15 _StreamController.add (dart:async/stream_controller.dart:586:5) #16 _StreamSinkWrapper.add (dart:async/stream_controller.dart:858:13) #17 _RootZone.runUnaryGuarded (dart:async/zone.dart:1314:10) #18 CastStreamSubscription._onData (dart:_internal/async_cast.dart:81:11) #19 _RootZone.runUnaryGuarded (dart:async/zone.dart:1314:10) #20 _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:336:11) #21 _BufferingStreamSubscription._add (dart:async/stream_impl.dart:263:7) #22 _SyncStreamController._sendData (dart:async/stream_controller.dart:764:19) #23 _StreamController._add (dart:async/stream_controller.dart:640:7) #24 _StreamController.add (dart:async/stream_controller.dart:586:5) #25 _RawReceivePortImpl._handleMessage (dart:isolate/runtime/libisolate_patch.dart:171:12) 97Unhandled exception: NoSuchMethodError: The getter 'primaryType' was called on null. Receiver: null Tried calling: primaryType #0 Object.noSuchMethod (dart:core/runtime/libobject_patch.dart:50:5) #1 DestinationResult.updateFromResponse (package:linkcheck/src/destination.dart:327:48) #2 checkPage (package:linkcheck/src/worker/worker.dart:127:11) #3 worker. (package:linkcheck/src/worker/worker.dart:192:29) #4 _RootZone.runUnaryGuarded (dart:async/zone.dart:1314:10) #5 _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:336:11) #6 _BufferingStreamSubscription._add (dart:async/stream_impl.dart:263:7) #7 _SyncStreamController._sendData (dart:async/stream_controller.dart:764:19) #8 _StreamController._add (dart:async/stream_controller.dart:640:7) #9 _StreamController.add (dart:async/stream_controller.dart:586:5) #10 _RootZone.runUnaryGuarded (dart:async/zone.dart:1314:10) #11 _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:336:11) #12 _BufferingStreamSubscription._add (dart:async/stream_impl.dart:263:7) #13 _SyncStreamController._sendData (dart:async/stream_controller.dart:764:19) #14 _StreamController._add (dart:async/stream_controller.dart:640:7) #15 _StreamController.add (dart:async/stream_controller.dart:586:5) #16 _StreamSinkWrapper.add (dart:async/stream_controller.dart:858:13) #17 _RootZone.runUnaryGuarded (dart:async/zone.dart:1314:10) #18 CastStreamSubscription._onData (dart:_internal/async_cast.dart:81:11) #19 _RootZone.runUnaryGuarded (dart:async/zone.dart:1314:10) #20 _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:336:11) #21 _BufferingStreamSubscription._add (dart:async/stream_impl.dart:263:7) #22 _SyncStreamController._sendData (dart:async/stream_controller.dart:764:19) #23 _StreamController._add (dart:async/stream_controller.dart:640:7) #24 _StreamController.add (dart:async/stream_controller.dart:586:5) #25 _RawReceivePortImpl._handleMessage (dart:isolate/runtime/libisolate_patch.dart:171:12) 102 ^C Ctrl-C Terminating crawl.
filiph commented 5 years ago

Sorry for not seeing this earlier.

I'm afraid this is a known issue, it's to do with unicode in some places of HTML, and it's blocked on https://github.com/dart-lang/html/issues/70. I did not realize it's as widespread as being on wordpress.org, though. :/

b4stien commented 5 years ago

@filiph the upstream issue is fixed, do you think it's possible to release a version with the dependency pinned to https://github.com/dart-lang/html/commit/2b392a4b8d739edeac3552dff3f614219a733955?

filiph commented 5 years ago

This is amazing news! Unfortunately, I cannot publish a version on pub that depends on a git-only version. https://github.com/dart-lang/html/pull/109 needs to first make it onto pub.

One complication could be that package:html is also a transitive dependency of source_span (through which the unicode bug actually manifests). And that package may also need updating. But we'll worry about it when we get there, I guess.

nshahan commented 5 years ago

@filiph I published an updated version. html v0.14.0+3

filiph commented 5 years ago

Thank you @nshahan!

I just published version 2.0.11 of linkcheck that no longer has the unhandled exception issue.