Elfeed does not update feeds anymore

[Steinou]

Hi, I've been using elfeed for a while without any issue. Very simple configuration, just imported opml file and add additional feeds directly in my .emacs. Since today, elfeed does not update my feeds anymore. When I hit G, I see "x feeds pending/16 in process" on top of the buffer, x decreases very fast and then nothing appears... In the elfeed-log buffer, I see all my feeds like: [time][error]: URL "(2) Failed to initialize." Any help would be much appreciated, thanks!

[skeeto]

It's strange that this issue has suddenly come up twice in this past week (see #276). In the other case it turned out to be a bizarre curl Windows build lacking libz support.

What operating system are you using? What version of Elfeed are you using? What version of Emacs are you using? What version of curl do you have installed? The most accurate way to get that one is evaluating elfeed-curl-get-capabilities or elfeed-curl-get-version (if you're not using the latest bleeding edge version).

The other thing to try is running curl standalone on the command line to make sure it works correctly. You can see the exact arguments Elfeed is passing for a particular by evaluating this:

(elfeed-curl--args URL nil)

Then try running it on the command line with those arguments. Hopefully you're getting the "Failed to initialize" error and you can narrow down to which argument is causing it.

[edmundmiller]

Alright, I'm having a similar issue, however, elfeed-search-update--force doesn't do anything. Doesn't work on 27 or 26 Ubuntu 18.04 curl version 7.58.0 eval:(elfeed-curl--args URL nil)

Debugger entered--Lisp error: (void-variable URL)
  (elfeed-curl--args URL nil)
  eval((elfeed-curl--args URL nil) nil)
  #f(compiled-function (exp &optional insert-value no-truncate char-print-limit) "Evaluate EXP and print value in the echo area.\nWhen called interactively, read an Emacs Lisp expression and\nevaluate it.  Value is also consed on to front of the variable\n`values'.  Optional argument INSERT-VALUE non-nil (interactively,\nwith a non `-' prefix argument) means insert the result into the\ncurrent buffer instead of printing it in the echo area.\n\nNormally, this function truncates long output according to the\nvalue of the variables `eval-expression-print-length' and\n`eval-expression-print-level'.  When NO-TRUNCATE is\nnon-nil (interactively, with a prefix argument of zero), however,\nthere is no such truncation.\n\nIf the resulting value is an integer, and CHAR-PRINT-LIMIT is\nnon-nil (interactively, unless given a positive prefix argument)\nit will be printed in several additional formats (octal,\nhexadecimal, and character).  The character format is only used\nif the value is below CHAR-PRINT-LIMIT (interactively, if the\nprefix argument is -1 or the value is below\n`eval-expression-print-maximum-character').\n\nRuns the hook `eval-expression-minibuffer-setup-hook' on entering the\nminibuffer.\n\nIf `eval-expression-debug-on-error' is non-nil, which is the default,\nthis command arranges for all errors to enter the debugger." (interactive #f(compiled-function () #<bytecode 0x500b69>)) #<bytecode 0x24dcfb>)((elfeed-curl--args URL nil) nil nil 127)
  apply(#f(compiled-function (exp &optional insert-value no-truncate char-print-limit) "Evaluate EXP and print value in the echo area.\nWhen called interactively, read an Emacs Lisp expression and\nevaluate it.  Value is also consed on to front of the variable\n`values'.  Optional argument INSERT-VALUE non-nil (interactively,\nwith a non `-' prefix argument) means insert the result into the\ncurrent buffer instead of printing it in the echo area.\n\nNormally, this function truncates long output according to the\nvalue of the variables `eval-expression-print-length' and\n`eval-expression-print-level'.  When NO-TRUNCATE is\nnon-nil (interactively, with a prefix argument of zero), however,\nthere is no such truncation.\n\nIf the resulting value is an integer, and CHAR-PRINT-LIMIT is\nnon-nil (interactively, unless given a positive prefix argument)\nit will be printed in several additional formats (octal,\nhexadecimal, and character).  The character format is only used\nif the value is below CHAR-PRINT-LIMIT (interactively, if the\nprefix argument is -1 or the value is below\n`eval-expression-print-maximum-character').\n\nRuns the hook `eval-expression-minibuffer-setup-hook' on entering the\nminibuffer.\n\nIf `eval-expression-debug-on-error' is non-nil, which is the default,\nthis command arranges for all errors to enter the debugger." (interactive #f(compiled-function () #<bytecode 0x2190c4d>)) #<bytecode 0x24dcfb>) ((elfeed-curl--args URL nil) nil nil 127))
  eval-expression((elfeed-curl--args URL nil) nil nil 127)
  funcall-interactively(eval-expression (elfeed-curl--args URL nil) nil nil 127)
  call-interactively(eval-expression nil nil)
  command-execute(eval-expression)

[skeeto]

@Emiller88, when you evaluate that expression, replace "URL" with the URL of one a feeds that doesn't work correctly:

(elfeed-curl--args "http://example.com/feed/" nil)

[edmundmiller]

My bad This is what I get. eval: Symbol’s function definition is void: elfeed-curl--args

[skeeto]

@Emiller88, you need to load Elfeed first since that function isn't (and shouldn't be) an autoload. Just bringing Elfeed up with "M-x elfeed" is sufficient to load it.

[edmundmiller]

@skeeto Thanks, I had it loaded for some reason elfeed-search-update and elfeed-search-update--force don't work but elfeed-search-fetch does

[grolfs]

I am experiencing this error as well. However, I am attempting to read atom feeds from Rational Team Concert that are secured by a form-based login. So I suspect I'm wandering far off the beaten path. Following the instructions above I get the following (URL value has been removed):

    ("--http1.1" "--compressed" "--silent" "--location" "-w(nil . %{size_header})" "-m30" "-D-" "--netrc --cookie /Users/<uid>/.private/elfeed_cookies.txt" <internal_url removed>)

If I call curl from the command line using the options above, I do get the feed xml to the terminal and at the end of the terminal output I see:

    </feed>(nil . 219)

I suspect the (nil . 219) is from the -w option. Is that expected?

Any suggestions on how I would debug this further?

P.S. I do not see any feed version information. This is the header information returned from the feed:

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:syndication="http://www.ibm.com/jazz/0.1.0/syndication" xmlns:process="com.ibm.team.process" xmlns:team.feed="http://www.ibm.com/team/Feed">

[skeeto]

@grolfs, it looks like you're using elfeed-curl-extra-arguments, but you've put multiple arguments into a single string. Split the --netrc, --cookie, and its argument into separate strings. Otherwise they're passed as one big argument to curl which won't work. There is no shell between Elfeed and curl to word split those strings.

[grolfs]

@skeeto, how did I miss that? Thank you. I am further and now see:

"unable to parse curl response"

in the elfeed-log. I also see in the *Messages* buffer:

Error running timer ‘elfeed-curl--call-callback’: (wrong-type-argument number-or-marker-p nil)

I took the output from the command line invocation of curl and followed the instructions from https://cweiske.de/tagebuch/atom-validation.htm. xmllint reported the feed validates.

Do you have any suggestions where to look?

emacs version:

GNU Emacs 26.1 (build 1, x86_64-apple-darwin17.5.0, NS appkit-1561.40 Version 10.13.4 (Build 17E202)) of 2018-06-18

elfeed version: 3.0.0

Thank you.

[skeeto]

You said you're using Elfeed 3.0.0. Is that from MELPA Stable or is it actually the bleeding edge version (e.g. from regular MELPA)? I've made a fix for a similar-sounding issue (40d3463) but it hasn't been put into an actual release yet.

What version of curl are you using and where did you get it? When you tested with the output of elfeed-curl--args, did you see the s-expression printed at the end?

[grolfs]

I'm currently using Melpa Stable. Should I switch to Melpa?

(use-package elfeed
    :ensure t
    :defer t
    :pin melpa-stable
 )

curl is the version shipped with macOS High Sierra:

$ curl --version
curl 7.54.0 (x86_64-apple-darwin17.0) libcurl/7.54.0 LibreSSL/2.0.20 zlib/1.2.11 nghttp2/1.24.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz HTTP2 UnixSockets HTTPS-proxy 

Yes, I see the s-expression at the end. I don't know if it's important, but the feed XML that is returned does not contain any line endings. It is one line that is 322,272 bytes long. Here's the end of my output with the s-expression:

</entry></feed>(nil . 221)

Thank you for all the help!

[skeeto]

I just made a new Elfeed release (3.1.0) which includes the recent curl compression fix. You should see the new version appear in the stable repository soon, and hopefully it fixes your issue.

[phikal]

I am using 3.1.0 (melpa stable) but the issue remains. More specifically, the top bar say "0 feeds pending, 5 in process" despite no curl processes being active (checked via ps aux | grep curl)! From what I see elfeed-curl-queue-active just doesn't change after one point. Interestingly I could "fix" this by setting elfeed-curl-queue-active to 0 and elfeed-use-curl to nil, updating (G) and then turning elfeed-use-curl on again.

This has happened a few times already since I started using elfeed, and usually I would resolve it by restarting Emacs, which is something I'd rather not do too often. Maybe it would be worth pointing out that I have a timer that runs elfeed-update periodically and that I let my device hibernate quite often.

[skeeto]

For large feeds (lots of entries), the majority of the processing time can easily be in parsing the feed rather than fetching it over the network. When it gets to "0 feeds pending" this is especially true since all the asynchronous curl processes have likely exited, and Emacs is just parsing all the returned content one feed after another. They're still "in process" but curl has completed its part.

As long as it eventually (e.g. within elfeed-curl-timeout) completes on its own, then it's working as intended. If you're using curl, and it jams up never finishing, then that's a bug that needs to be fixed. (The url-retrieve backend is known to jam up like this due to unavoidable bugs in Emacs.) There's an "elfeed-unjam" command to get it out of this state — so you don't need to restart Emacs nor mess with the internal queue yourself — but you shouldn't need to use it with the curl backend.

If it gets jammed, check Messages for errors. That will give us a clue as to what's going wrong. Also report exactly what version of curl and Emacs you're using.

[phikal]

For large feeds (lots of entries), the majority of the processing time can easily be in parsing the feed rather than fetching it over the network

I've realized that, when updates go well (which is the case 97% of the time) it usually requires a few 1-2 extra seconds to process some feeds. The thing just is that I don't think I have any feeds that need more than a day to process.

If you're using curl, and it jams up never finishing, then that's a bug that needs to be fixed.

This is what I am experiencing. Haven't quite waited long enough to confirm that it "never" finishes, but it certainly takes much longer than it should by any reasonable standard. I'll try using elfeed-unjam the next time it happens.

If it gets jammed, check Messages for errors.

All I could find related to elfeed has been:

    428:Error running timer ‘elfeed-curl--call-callback’: (wrong-type-argument stringp ("https://archive.org/services/collection-rss.php?collection=retroist-podcast"))
    551:Error running timer ‘elfeed-curl--call-callback’: (wrong-type-argument stringp ("https://www.romanzolotarev.com/rss.xml"))
    715:Error running timer ‘elfeed-curl--call-callback’: (wrong-type-argument stringp ("https://eklitzke.org/index.rss"))
    727:Error running timer ‘elfeed-curl--call-callback’: (wrong-type-argument stringp ("https://bits.debian.org/feeds/atom.xml"))
    824:Error running timer ‘elfeed-curl--call-callback’: (wrong-type-argument stringp ("https://archive.org/services/collection-rss.php?collection=retroist-podcast"))
    857:Error running timer ‘elfeed-curl--call-callback’: (wrong-type-argument number-or-marker-p nil) [10 times]

Also report exactly what version of curl and Emacs you're using.

$ curl --version
curl 7.52.1 (x86_64-pc-linux-gnu) libcurl/7.52.1 OpenSSL/1.0.2l zlib/1.2.8 libidn2/0.16 libpsl/0.17.0 (+libidn2/0.16) libssh2/1.7.0 nghttp2/1.18.1 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL

$ emacs --version
GNU Emacs 25.1.1
Copyright (C) 2016 Free Software Foundation, Inc.
GNU Emacs comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GNU Emacs
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING.

Both installed from the Debian stable repositories.

[skeeto]

Thank you for the thorough report. I also run Debian Stable, so I'm using the exact same software. This particular error has been reported before, but I've never been able to reproduce it nor get enough information to figure out what's going on. The URL somehow gets nested inside a list, causing a simple type error early in the callback. Since it's asynchronous, a simple backtrace doesn't tell enough of the story.