search

fasheng / elfeed-protocol

Provide extra protocols to make like Fever, NewsBlur, Nextcloud/ownCloud News and Tiny Tiny RSS work with elfeed
GNU General Public License v3.0
100 stars 18 forks source link

ttrss api limit? #17

Closed punassuming closed 5 years ago

punassuming commented 5 years ago

I am trying to sync ~6k tt-rss articles to elfeed-protocol. Currently, after elfeed-update, when I set log-level to debug I see the following:

[2019-03-11 00:18:04] [debug]: elfeed-protocol-ttrss: parsing entries, first-entry-id: 1 last-entry-id: 3407
[2019-03-11 00:18:04] [debug]: elfeed-protocol-ttrss: parsed 200 entries with 0.129346s, first-entry-id: 1 last-entry-id: 3608

After the update, I see no additional entries (stuck at ~1900 entries at this point, and I never see more than 200 entries parsed on an update.

Was wondering if there was something in the tt-rss configuration that i need to change. I have already changed elfeed-protocol-ttrss-maxsize to 10000.

fasheng commented 5 years ago

The latest commit should fix your issue, please have a try :)

At first I want to implement a elfeed-protocol-ttrss-update-older like owncloud protocol did, but failed. So just always fetch all article instead unread this time. https://github.com/fasheng/elfeed-protocol/blob/1853b53a6d8404f8dc5a4d0deb70d3a5c95da850/elfeed-protocol-ttrss.el#L466-L470

punassuming commented 5 years ago

Awesome! Will try it tonight.

fasheng commented 5 years ago

Oh, forgot to say, this change only effect for the first update operation, so you need run elfeed-protocol-ttrss-reinit one time or just clear your elfeed-db-directory. And to parse the huge json buffer may block a few seconds.

punassuming commented 5 years ago

Tried it, still only parsing 200 a time. Trying a loop to see if I can force it that way. Will also look into ttrss API to see if I can get the full json through my browser.

fasheng commented 5 years ago

Weird, here is the curl command to fetch articles for debug, sid could be found in elfeed-protocol-ttrss-sid. Wait for your good news~

curl -d '{"sid":"xxx","op":"getHeadlines","feed_id":-4, "view_mode":"all_articles","show_content":"1", "include_attachments":"1", "order_by":"date_reverse", "limit":10}' https://yourttrss.com/api/
punassuming commented 5 years ago

I found the reason:

Limit:

Before API level 6 maximum amount of returned headlines is capped at 60, API 6 and above sets it to 200.

is there a way to iterate through the missing entries automatically?

I am doing this: (loop repeat 10 do (elfeed-update)) but it seems that the order or retrieval is not consistent. Sometimes I will get 100 new entries, other times, no change.

punassuming commented 5 years ago

One thought: use getFeeds to get full list, and then utilize getHeadlines in recurring job of 200 to get the entire list. Do you think that would work?

fasheng commented 5 years ago

I see, personal prefer to use Nextcloud News so didn't realize this limit before.

But update multiple times should works because ("order_by" . "date_reverse") was set for each request. Please wait some minutes, I will try to find the reason...

fasheng commented 5 years ago

Well, after some big changes, I think it works for you, now.

By default, first time update operation only fetch the starred and unread articles like before. Then you have two methods to fetch the older articles:

  1. Run elfeed-protocol-ttrss-update-older multiple times manually
  2. Run elfeed-protocol-ttrss-set-entry-skip set the update opeartion's skip param to 0, so next time elfeed-update will fetch ttrss article from the oldest one.

And the update operations could not executed in the same time, so run-at-time with some delays will help you. Enjoy!

punassuming commented 5 years ago

Awesome, I really appreciate the work.

fasheng commented 5 years ago

Close now, any problem just report later~