search

FreshRSS / FreshRSS

A free, self-hostable news aggregator…
https://freshrss.org
GNU Affero General Public License v3.0
10.06k stars 858 forks source link

[BUG] Importing favourites from TT-RSS #5583

Open arkhi opened 1 year ago

arkhi commented 1 year ago

Hello.

First of all, the bug might be sitting on the chair, so please bear with me. :)

Summary

I understand some PR say it is possible to import TT-RSS exports of favourites:

I can’t find a documentation about how this feature works in the context of TT-RSS files being imported. What I’m trying does not show any favourites after importing them successfully.

To Reproduce

  1. Export favourites with the json resulting of a TT-RSS Generated feed.
  2. Go to the import Section of FreshRSS: /i/?c=importExport.
  3. Select the json file generated previously.
  4. Click the “Import” button.
  5. FreshRSS shows a banner saying “Your feeds have been imported and will now be updated”.
  6. Check the favourites: /i/?a=normal&get=s.
  7. No new favourite appears.

Expected behaviour

  1. Steps 1 to 6 above.
  2. New favourites are shown.

Environment information

Additional context

I saw in the previous PR and comments people had xml files (TinyTinyRSS_exported_2018-10-24.xml is the given example), but could not find where they had those files from.

I wanted to migrate from TT-RSS to FreshRSS for some months now and I’m now giving it a go.

Tiny Tiny RSS (TT-RSS) related resources

Alkarex commented 1 year ago

Hello, If you could share a sample of your export containing some of your favourites, I can try to check

arkhi commented 1 year ago

Here you go; thank you! :)

2023-08-13-starred-articles.json.zip

This is one of a batch exported with a curl command:

curl -o export-favourites-#1.json "https://syndication.example.com/public.php?op=rss&limit=100&offset=[0-236]00&format=json&id=-1&is_cat=0&q=&key=BLABLABLA"
Alkarex commented 1 year ago

Would you be able to test this patch? https://github.com/FreshRSS/FreshRSS/pull/5584 (on the current edge branch of FreshRSS) Note that your JSON file must contain starred or favourites or favorites in its name. Furthermore, it does not look like the export contains the original feed URL or Web site URL for the items, so they are all imported in a placeholder http://import.localhost/import.xml. If you have a way to export the feed information, I can try to import it. Last thing, could you export an XML version instead of JSON version, to check whether that format still works?

arkhi commented 1 year ago

Thanks!

Would you be able to test this patch? https://github.com/FreshRSS/FreshRSS/pull/5584 (on the current edge branch of FreshRSS)

I tried and it imported the batch properly, even without some of the feeds present in the database.

I understand this is related to the following point. :) I was wondering if it would import starred articles which no longer have their matching feeds (The feed died or I stopped following it), and it looks like the import works regardless of the presence of the feed; good news!

This is what I tried:

Note that your JSON file must contain starred or favourites or favorites in its name.

Do you mean the upmost title property at the root of the file? If yes, we’re good.

Also, “favourites” was actually my bias. TT-RSS calls them “Starred” articles, so no worries if you want to keep it simple and not start to handle every single possible naming (thanks though :) ).

Furthermore, it does not look like the export contains the original feed URL or Web site URL for the items, so they are all imported in a placeholder http://import.localhost/import.xml. If you have a way to export the feed information, I can try to import it.

Ha yes… I don’t know how to do that at the moment but that would be a sensitive thing to do.

The thing is some favourites are around 15 years old and I think TT-RSS did not have that information at the time. Furthermore, some links are published articles, which are not actual part of syndicated feeds but URLs pushed to TT-RSS via its API (The “published” status allows to create a feed of any URL we’d like to share; something I would love to see on FreshRSS and I plan to emulate via a manual label here’s an example: https://arkhi.org/feed/juji)

The json export does not seem to provide this information but the xml does. I’m gonna check if I can hack something on the TT-RSS side.

Last thing, could you export an XML version instead of JSON version, to check whether that format still works?

Here you go! I updated the previous file and link it here too, so that their content match.

arkhi commented 1 year ago

The json export does not seem to provide this information but the xml does. I’m gonna check if I can hack something on the TT-RSS side.

I was not running TT-RSS with the latest code. The lack of source in the json export has been fixed in the meantime.

Here are the latest version of the exports based on a freshly pulled TT-RSS:

arkhi commented 1 year ago

(Based on what you did and from what I understand, a tool for migrating the whole content from a TT-RSS instance to FreshRSS does not seem like an impossible task. Just in case, I can provide you with a full export in private from TT-RSS if you’re interested.)

arkhi commented 1 year ago

What would be the best way to document the issue being addressed here? I could help writing.

Alkarex commented 1 year ago

I have added the support for the source in https://github.com/FreshRSS/FreshRSS/pull/5584 New tests welcome for JSON import. The XML format has changed a lot since the last type of TT-RSS export I saw, and it will require some more work. You are welcome to mail me another example of export if you like.

arkhi commented 1 year ago

I have added the support for the source in https://github.com/FreshRSS/FreshRSS/pull/5584

I emptied then removed the existing http://import.localhost/import.xml feed and imported zipped json files again:

screenshot 1

screenshot 2

screenshot 3

arkhi commented 1 year ago

About the json obtained with the TT-RSS data migration plugin, it is a different format than the exported files through Generated feeds.

Thank you; I’ll send you something.

arkhi commented 1 year ago

The XML format has changed a lot since the last type of TT-RSS export I saw, and it will require some more work.

I also do not know which tool was used to generate the xml mentioned in the previous issues (#2018, #2079 and #2099). That could explain the differences between the formats.

Maybe it was an entirely different tool?

arkhi commented 1 year ago

There was a TT-RSS import_export preference before which was discontinued: https://community.tt-rss.org/t/found-lot-of-orphaned-records-in-database/2657/8

Alkarex commented 1 year ago

One of the problems is that in source info attached to the items is the Web site URL, not the feed URL... This also makes it challenging to detect which feeds we already have, but I can try something (maybe not for the upcoming FreshRSS release though, which is due soon).

arkhi commented 1 year ago

For what it’s worth, I cleared all feeds and did the following again:

  1. Import OMPL file (screenshot 1).
  2. Import all zipped json files (screenshot 2).

screenshot 1 screenshot 2

Alkarex commented 1 year ago

For what it’s worth, I cleared all feeds and did the following again

That seems to look good, doesn't it?

arkhi commented 1 year ago

That seems to look good, doesn't it?

It depends what you mean by looking good. :)

Articles are imported but the previous hiccups still apply (articles appearing in multiple places, etc).