How reliable is elfeed-tube-fill-feeds?

[agzam]

I've been trying to backfill the feeds using elfeed-tube-fill-feeds, but not very successfully. This is what I've been trying to do:

• I would set the filter - (elfeed-search-live-filter) to a specific channel, e.g., I set it to "=MIT" for "MIT OpenCourseWare" channel
• It shows initial 15 videos
• I would execute elfeed-tube-fill-feeds, it opens the confirmation buffer where the channel is listed
• I would press "C-c C-c"
• *elfeed-search* buffer header changes to "Filling Feed" MIT OpenCourseWare ... fetch metadata (4500 pending)
• It spins up a bunch of hidden *curl* buffers, there's also some commotion in a hidden *code-conversion-work* buffer
• Understandably, it's takes long time. For some reason the number of pending operations during the whole process only increases

In the end - results are the same - the feed either adds only a few vids, or doesn't get updated at all, no matter how long I wait. I even tried leaving it overnight and it still didn't work. Is it possible that I'm missing something in my setup?

Oh, I also see some errors like this in the Messages buffer:

Error running timer ‘elfeed-curl--call-callback’: (wrong-type-argument stringp ("invidious.perennialte.ch/api/v1/videos/oEskbXrhkkk?fields=published" "invidious.perennialte.ch/api/v1/videos/q0PxCQWG3ic?fields=published" "invidious.perennialte.ch/api/v1/videos/zmzyW1rP-hk?fields=published" "invidious.perennialte.ch/api/v1/videos/qbyeQum8qTE?fields=published"))

[karthink]

Speaking generally, I've filled feeds for channels with 4500 videos before and it's worked.

When I have time I'll try to reproduce this error using the MIT OpenCourseWare channel and get back to you.

[karthink]

BTW for more diagnostic information, you can run (setq elfeed-log-level 'debug), and check the *elfeed-tube-log* buffer.

[agzam]

It's not only MIT OpenCoursWare channel. It doesn't work for other channels as well. I think I'm getting rate-limited. (setq elfeed-log-level 'debug) shows a bunch of 500 errors in the log.

elfeed-tube-log

[2024-06-09 19:05:09] [debug]: [(fill-confirm-feeds): ("Fireship")] [2024-06-09 19:05:09] [debug]: [(fill-feeds): Backfilling feed: Fireship] [2024-06-09 19:05:41] [error]: [Search][(28) Operation timeout.]: invidious.privacydev.net/api/v1/channels/UCsBjURrPoezykLs9EqgamOA/videos?fields=videos(title,videoId,author,published),continuation&sort_by=newest ((attempt 0/2)) [2024-06-09 19:05:46] [debug]: [Backfilling...][Fetched: 60 entries] [2024-06-09 19:05:47] [debug]: [Backfilling...][Fetched: 60 entries] [2024-06-09 19:05:48] [debug]: [Backfilling...][Fetched: 60 entries] [2024-06-09 19:05:49] [debug]: [Backfilling...][Fetched: 60 entries] [2024-06-09 19:05:50] [debug]: [Backfilling...][Fetched: 60 entries] [2024-06-09 19:05:51] [debug]: [Backfilling...][Fetched: 60 entries] [2024-06-09 19:05:52] [debug]: [Backfilling...][Fetched: 60 entries] [2024-06-09 19:05:53] [debug]: [Backfilling...][Fetched: 60 entries] [2024-06-09 19:05:54] [debug]: [Backfilling...][Fetched: 60 entries] [2024-06-09 19:05:55] [debug]: [Backfilling...][Fetched: 51 entries] [2024-06-09 19:05:55] [debug]: [Fixing publish dates] [2024-06-09 19:05:58] [error]: [Search][HTTP 500]: invidious.lunar.icu/api/v1/videos/8BrLNgKLWzs?fields=published ((attempt 0/2)) [2024-06-09 19:05:58] [error]: [Search][HTTP 500]: invidious.lunar.icu/api/v1/videos/M7uo5jmFDUw?fields=published ((attempt 0/2)) [2024-06-09 19:05:58] [error]: [Search][HTTP 500]: invidious.lunar.icu/api/v1/videos/2hlD7dWp09M?fields=published ((attempt 0/2)) [2024-06-09 19:05:58] [error]: [Search][HTTP 500]: invidious.lunar.icu/api/v1/videos/AgyJv2Qelwk?fields=published ((attempt 0/2)) [2024-06-09 19:05:58] [error]: [Search][HTTP 500]: invidious.lunar.icu/api/v1/videos/rIrNIzy6U_g?fields=published ((attempt 0/2)) [2024-06-09 19:05:58] [error]: [Search][HTTP 500]: invidious.lunar.icu/api/v1/videos/wbQwD3QS19I?fields=published ((attempt 0/2)) [2024-06-09 19:05:58] [error]: [Search][HTTP 500]: invidious.lunar.icu/api/v1/videos/pPStdjuYzSI?fields=published ((attempt 0/2)) [2024-06-09 19:05:58] [error]: [Search][HTTP 500]: invidious.lunar.icu/api/v1/videos/m_xoN8KlP3w?fields=published ((attempt 0/2)) [2024-06-09 19:05:58] [error]: [Search][HTTP 500]: invidious.lunar.icu/api/v1/videos/KbzGy3whpy0?fields=published ((attempt 0/2)) etc.

[karthink]

I tried out filling both channels:

1. Fireship: It worked without issues.
2. MIT OpenCourseWare: It failed with the same errors that you got. I traced the errors to possible bugs in elfeed-curl (part of Elfeed). This appears to be a timing based bug, which is supposed to be impossible in Emacs, so I'm not sure yet what's going on.

Also, skeeto is not accepting PRs for Elfeed (as he's moved away from elisp for now) so fixing it for everyone is going to be tough even if I can pinpoint the bug.

As a result, I'm considering writing my own fetcher and avoiding elfeed-curl, but it will be a while before I can work on elfeed-tube. Let's leave this issue open for now.

I hope the other elfeed-tube features are working to your satisfaction.

[agzam]

Hmm. I still can't fetch for Fireship channel. I even tried (setq elfeed-use-curl nil). I'm using latest elfeed and emacs-version GNU Emacs 30.0.50 (build 7, x86_64-pc-linux-gnu, GTK+ Version 3.24.42, cairo version 1.18.0) of 2024-06-10 and curl 8.8.0. I didn't know @skeeto is not accepting PRs for Elfeed. Thank you for looking into this. I will try to debug elfeed-curl, maybe I'll get to the bottom of this. Once again, thank you!

[SohumB]

I'm pretty sure you were running into the bug I fixed in https://github.com/skeeto/elfeed/pull/524 .

[agzam]

@SohumB I tried your branch, it didn't fix this specific problem for me, at least on one of the machines. I'm getting 403s. It could be due to the corporate firewall settings and other shit. I'll try on my Linux machine sometime later, but tbh, I'm skeptical.

[SohumB]

Hm, that's strange. If you're getting 403s, well, obviously there's nothing elfeed or curl can do about that, but I don't see any other way this error you posted in the initial message can happen:

Error running timer ‘elfeed-curl--call-callback’: (wrong-type-argument stringp ("invidious.perennialte.ch/api/v1/videos/oEskbXrhkkk?fields=published" "invidious.perennialte.ch/api/v1/videos/q0PxCQWG3ic?fields=published" "invidious.perennialte.ch/api/v1/videos/zmzyW1rP-hk?fields=published" "invidious.perennialte.ch/api/v1/videos/qbyeQum8qTE?fields=published"))

That callback does a bunch of things, but that wrong-type-argument can only really come (I believe) from the start-process call, and that can only be wrong if elfeed-curl-queue is of the wrong shape. If you're running my branch, there's now no code that writes to that var that doesn't go through type checks, so I don't really understand how that specific error could happen. (Of course, other errors could be happening.)

Have you tried toggle-debug-on-error and looking at the backtrace?