search

filiph / linkcheck

Fast link checker
https://pub.dartlang.org/packages/linkcheck
MIT License
403 stars 51 forks source link

Kills Linux System Memory Runaway #20

Closed travis5491811 closed 6 years ago

travis5491811 commented 6 years ago

Tried linkcheck out on my system: Operating System: Linux Mint 19
Kernel: Linux 4.15.0-33-generic
Architecture: x86-64
4 GHz i5-8300H
15GB RAM
Dart VM version: 2.0.0
Pub 2.0.0
linkcheck version 2.0.4

Both times I tried it, it crashed my system. Both times was to try and find all dead links on a company website. When i ranlinkcheck <privateURL> system resources were at 3GB RAM with Load of less than 0.5. After about 700 links my CPU fans were going nuts and RAM was at 10GB in use. By 1200 links, RAM was maxed out at 15GB with an additional 2.5GB of SWAP. By 2193 links RAM still maxed out and SWAP was at 5.25GB and then whole system locked up. UI stopped responding and I could not do anything. I waited 8 minuets the last time after UI lock up before hard shutdown.

Has anybody tested this on a Linux system?

chalin commented 6 years ago

Linkcheck is regularly run on Travis (Linux) with every PR/commit submitted to GitHub for the two main Dart sites: site-www and site-webdev. For example, see https://travis-ci.org/dart-lang/site-www/jobs/424578417. I haven't seen linkcheck crash under Linux or MacOS before.

It might be an issue with the SDK under Linux Mint, you might want to take a look at the open issues.

travis5491811 commented 6 years ago

@chalin thanks for the info. Holy smokes there's a lot of open issues (over 4K) for the Dart dependency relative to the number of people following it (2394).

Has the team considered moving away from Dart to improve control over the user experience?

chalin commented 6 years ago

Many of those issues are quite old. Dart is alive and well not only for the web, but also mobile ... but that is a topic for another forum. I'll close this for now. Feel free to reopen if you can in point the issue to linkcheck.

filiph commented 6 years ago

Thanks for the report! Can you share the site so we can try to reproduce?

linkcheck could use a lot of optimization work — it's currently assuming the site size to be relatively small (in the hundreds to thousands of pages). It checks ~10K links in under a second on my machine so your issue might be something else. What I'm trying to say: we haven't been optimizing this for resource use. So there's work to be done there. But your machine does not look like an underresourced one, so this points to something else.

Holy smokes there's a lot of open issues (over 4K) for the Dart dependency relative to the number of people following it (2394).

Hmm, the # of open issues on Dart is just a function of the complexity of that project. Compare with Go, Rust, Scala, Kotlin. Not sure how the number of stars plays a role here. Dart is definitely not as popular as these other languages.

If the ratio of stars to open issues is an indication of anything, then projects like React Native (68326 stars vs 685 issues) and AngularJS 1 (59030 vs 408) are absolute gems — we should all drop every other alternative and use only them. :)