search

lycheeverse / lychee

⚡ Fast, async, stream-based link checker written in Rust. Finds broken URLs and mail addresses inside Markdown, HTML, reStructuredText, websites and more!
https://lychee.cli.rs
Apache License 2.0
2.13k stars 129 forks source link

Too many open files (os error 24) on macOS #1248

Open nathany opened 1 year ago

nathany commented 1 year ago

When running lychee README.md against this repository https://github.com/dariubs/GoBooks, I am seeing the following error on macOS 13.5.1, however these links are considered fine by GitHub Actions with Ubuntu.

✗ [ERR] https://github.com/thewhitetulip/web-dev-golang-anti-textbook | Failed: Network error: dns error: proto error: io error: Too many open files (os error 24)

It's happening on quite a few links.

Running against commit 1c71e7329e369b510afb806f484aa4e415b9aeb2 of https://github.com/dariubs/GoBooks with lychee installed via Homebrew.

❯ lychee --version lychee 0.13.0

nathany commented 1 year ago

Apologies, I didn't see an issue for it, but it's addressed in the README.

commit f2109a41a7b0ab404845d105729dc8a434b61f1c

Still, kind've wonder if the default max concurrency should be lower, at least on macOS, to avoid this out-of-the-box issue.

mre commented 1 year ago

Yeah, we should fix that. I will look at the repo to see what I can do.

mre commented 1 year ago

Similar discussion: https://github.com/RustScan/RustScan/issues/25

nathany commented 1 year ago

@mre Thanks. It is still occurring on main with that repo, with ~148 links to check.

Setting --max-concurrency 32 resolves it on macOS.

mre commented 8 months ago
reagle commented 6 months ago

I just encountered this on:

| lychee 0.14.3 | macOS 14.3.1 23D60 arm64

mre commented 6 months ago

I think rlimit is the way to go to fix this once and for all (famous last words). In the meantime, your best bet is --max-concurrency, unfortunately.