I've made three changes that address a memory leak that was causing FaviconFinder to take up infinite memory, and crash servers running on Linux.
I'd recommend making this first change, bumping async-http-client from 1.9.0 to 1.22.0. There have been a lot of changes since 1.9.0 that are worth pulling in, if not for stability alone.
I've also lowered the default response timeout duration from 30 to 15 seconds. You would have more evidence of what the best default is, but ideally this could be made configurable, because I don't think it's worth waiting 30 seconds for a favicon to load.
Previously response.body.collect(upTo:) was set to Int.max, I've defaulted it to 2048 * 1024 bytes (2MB). This is what was causing the runaway memory issues on some websites, because the Linux variant of FaviconFinder using async-http-client was requesting Int.max bytes, even on servers that did not have 4 GB of ram. (Testable with this link.)
will-lumley
commented
1 month ago
I've made three changes that address a memory leak that was causing FaviconFinder to take up infinite memory, and crash servers running on Linux.
I'd recommend making this first change, bumping async-http-client from 1.9.0 to 1.22.0. There have been a lot of changes since 1.9.0 that are worth pulling in, if not for stability alone.
I've also lowered the default response timeout duration from 30 to 15 seconds. You would have more evidence of what the best default is, but ideally this could be made configurable, because I don't think it's worth waiting 30 seconds for a favicon to load.
Previously
response.body.collect(upTo:)was set toInt.max, I've defaulted it to2048 * 1024bytes (2MB). This is what was causing the runaway memory issues on some websites, because the Linux variant of FaviconFinder using async-http-client was requestingInt.maxbytes, even on servers that did not have 4 GB of ram. (Testable with this link.)