Extract elfeed-curl into its own package

[DamienCassou]

Hi,

I have some problems with url-retrieve and thought I could reuse your curl code. Would it make sense to extract elfeed-curl into its own package?

[skeeto]

I really don't want to split Elfeed up into smaller packages. I like how simple it is with zero third-party dependencies, even its own. The curl backend is specialized specifically for Elfeed's needs, and no more. There would have to be a very significant benefit for this to change.

Fortunately, there already exists a standalone alternative to url-retrieve: request.el.

https://github.com/tkf/emacs-request

Give that a shot.

[DamienCassou]

Thanks for your answer. I understand your point. request.el seems to be a great solution but appears unmaintained.

[alphapapa]

@skeeto Chris, I wonder if we could convince you of the need for a new HTTP library for Emacs. request has a new maintainer now, but he's, admittedly, not an experienced elisp dev, and the deep bugs I've encountered and reported are unlikely to be fixed anytime soon, if ever. url.el seems to also have deep, mysterious bugs, especially regarding timeouts, sentinels, reaping buffers and processes, etc. And it feels like an impenetrable labyrinth of code, to me.

With the bugs I keep encountering, I'm almost to the point with matrix-client that I need a library that is neither request nor url, but there are no other choices. I'd be happy to help maintain such a library, but it's not my area of expertise, so I would prefer to follow the lead of someone like yourself, who knows what he's doing, so we could end up with a solid library that could serve as the standard Emacs HTTP library for a long time.

What do you think?

[skeeto]

Adam, I require no convincing that Emacs lacks a properly functioning HTTP request library. I've believed this since 2013. :-) I've never used request.el, but I had hoped it would fulfill this role.

I still believe this approach with a curl subprocess is by far the most practical option. The current alternatives are:

• Use a curl "competitor" for the backend (wget?). Curl is already the best at what it does, so there's no point to this.

• Write a pure Emacs Lisp implementation of an HTTP client. Since url-retrieve is just not well written, you could write a better one from scratch. However, good luck keeping up with HTTP/2, HTTP/3, etc. How will you handle TLS? Emacs' mediocre GnuTLS integration is one of the big problems with url-retrieve, so using it won't actually solve its problems. Don't even consider implementing TLS in pure Emacs Lisp. If you're going to call out to a native library, you might as well use an HTTP library while you're at it ... like curl.

• Integrate curl as a module. This would have the most flexibility and probably best performance, but then you have to manage a native module. How do you build it? Do you embed curl or link with one on the system? There's no reason to go through all this trouble when curl is very well-behaved as a command line program. It's also more likely to be available than whatever is needed to build a module (compiler, etc.).

So, ultimately, a well-functioning Emacs Lisp HTTP library is going to use curl subprocsesses to fetch content. There's just no way around it. That's what Elfeed prefers, and it's what request.el prefers.

I'm convinced there's a problem, and I know what the solution looks like. I'm just not personally interested in maintaining an Emacs Lisp HTTP library. I also care a lot more about Elfeed than I do about this problem such that I don't want to make Elfeed even a little worse off in order to solve it.

Perhaps you should "fork" elfeed-curl for your needs. Rip it out. Rename everything. Perhaps clean it up a little. Throw away unnecessary parts. It's all in the public domain for this reason.

[alphapapa]

So, ultimately, a well-functioning Emacs Lisp HTTP library is going to use curl subprocsesses to fetch content.

I agree. I wish Emacs could link with libcurl, but otherwise, that's the best option.

I'm convinced there's a problem, and I know what the solution looks like. I'm just not personally interested in maintaining an Emacs Lisp HTTP library. I also care a lot more about Elfeed than I do about this problem such that I don't want to make Elfeed even a little worse off in order to solve it.

Can't say I blame you.

Perhaps you should "fork" elfeed-curl for your needs. Rip it out. Rename everything. Perhaps clean it up a little. Throw away unnecessary parts. It's all in the public domain for this reason.

I feel like that's more than I would want to attempt by myself, but I'll keep it in mind. Thanks.

[alphapapa]

For future reference (including mine), I've listed the Emacs HTTP libraries I could find here: https://alphapapa.github.io/emacs-package-dev-handbook/#outline-container-Networking I mentioned elfeed-curl as a possibility for extracting into a separate package. Perhaps someday... :)

[clemera]

I'm thinking about a way to auto delegate url-retrieve to elfeed-curl (by translating the requests to curl commands, curl-to-elisp does it the other way around).