[Question]: import/export format

iBicha commented 9 months ago

Guidelines

[X] I have searched the issue tracker for open and closed issues that are similar to the feature request I want to file, without success.
[X] I have searched the documentation for information that matches the description of the feature request I want to file, without success.
[X] This issue contains only one feature request.

Problem Description

I'm working on yt2alt a tool to simplify importing your profile from Youtube to an alternative platform, and I'd like to add support for FreeTube.

yt2alt exports the following things

Subscriptions
Watch history
Liked videos
Watch later
Recommended videos
Playlists

I would like to understand better the file format used by FreeTube. Specifically:

It looks like FreeTube can import 3 files: Subscriptions, History, and Playlists
FreeTube does support certain special types of Playlists, like the "WatchLater"

Proposed Solution

I'm looking for guidance on how to generate profile files (subscriptions, history, playlists) that will match Youtube profile (as close as I can).

Alternatives Considered

N/A

Issue Labels

support for external software

Additional Information

If you have suggestions on which apps to support as well, let me know! Thanks

absidue commented 9 months ago

Is your goal to generate files that you can import into FreeTube (e.g. similarly to how you can import the google take out subscriptions and watch history files) or files that you can place in the location in the file system where FreeTube saves it's data, so they get used the next time you start FreeTube?

iBicha commented 9 months ago

Is your goal to generate files that you can import into FreeTube (e.g. similarly to how you can import the google take out subscriptions and watch history files) or files that you can place in the location in the file system where FreeTube saves it's data, so they get used the next time you start FreeTube?

My primary goal is to make "migration" as frictionless and complete as possible. So both options might be valid, and might even add them both separately if that's a good idea.

For example, I have two options for Invidious: save to file, and use the import API. The file approach requires that you import into Invidious manually. The API approach does the import for you, but you are "tied" to that instance where the import happens. If you would like to keep the file longer (say, to import to multiple instances) then using a file is more appropriate.

I'm open to both as I said, given that I'm using a public facing functionality, and not hacking into an implementation detail that would break later if the format or location changes.

iBicha commented 9 months ago

To kinda iterate on the goal: lots of people feels like they are starting from scratch when they move from Youtube to a new platform (empty subscription list, no playlists, no history, etc). I want this tool to make the new platform feel like home, by bringing as many parts of the Youtube profile as possible, and match it with available features in the new platform

absidue commented 9 months ago

Okay so your best bet is to generate FreeTube database files, but direct your users to import them through the app, that way they won't have to worry about messing around in the file system.

If you don't want to worry about format changes, then you should stick to only supporting subscriptions and watch history for the moment, as the latest release 0.19.1 only supports a single "Favorites" playlist, whereas the upcoming release and the nightlies support multiple playlists with a different format.

FreeTube uses nedb databases (https://github.com/seald/nedb), the records are JSON with each record taking up one line in the file, so if one database contains 3 records, the file would have 3 lines with each one containing a JSON object and one final empty line. The file extension needs to be .db for FreeTube's import to treat them as FreeTube databases.

Subscriptions

https://docs.freetubeapp.io/usage/importing-subscriptions/ (FreeTube currently supports importing: FreeTube subscriptions exports, Google Takeout CSV and JSON, Invidious OPML and NewPipe JSON) For subscriptions FreeTube supports multiple profiles, as YouTube only supports one, I would recommend that you follow the same approach that FreeTube does when someone imports their Google Takeout subscriptions and put all of them in the special all channels profile. Alternatively you can generate a file in the same format as Google Takeout and let FreeTube handle that for you.

Here is an example of what it looks like if you launch FreeTube for the first time, subscribe to the official @YouTube channel and then export your subscriptions database:

{"_id":"allChannels","name":"All Channels","bgColor":"#BD93F9","textColor":"#000000","subscriptions":[{"id":"UCBR8-60-B28hp2BmDPdntcQ","name":"YouTube","thumbnail":"https://yt3.googleusercontent.com/584JjRp5QMuKbyduM_2k5RlXFqHJtQ0qLIPZpwbUjMJmgzZngHcam5JMuZQxyzGMV5ljwJRl0Q=s176-c-k-c0x00ffffff-no-rj"}]}

The background color (bgColor) is randomly selected from one of FreeTube's colors (https://github.com/FreeTubeApp/FreeTube/blob/development/src/renderer/helpers/colors.js#L2-L38) and the text color textColor is either #000000 (black) or #FFFFFF (white), whichever has the higher contrast with the background color. For your use case you probably want to hard code both to specific values, to make your code simpler, that way the only dynamic thing in your code would be the subscriptions that go in the profile.

subscriptions is an array of channels. Here is a TypeScript type definition for a channel entry (FreeTube doesn't use TypeScript, but the syntax is well known, so I decided to use it here):

type SubscriptionChannel = {
  id: string, // channel ID (this has to be the actual id, not a username like `@YouTube`)
  name: string, // channel name
  thumbnail: string // channel thumbnail URL
}

Watch history

https://docs.freetubeapp.io/usage/importing-history/ (FreeTube currently supports importing: FreeTube watch history exports and Google Takeout JSON)

Please filter out ads, private videos and member only videos. As FreeTube is the equivalent of using YouTube logged out, private videos and member only videos cannot be watched inside FreeTube, so keeping them in the history would be pointless. Keeping them in there will likely just result in user frustration, if the user tries to watch them (FreeTube doesn't handle them well either, but it's a low priority bug, as it's very unlikely that a user would encouter them during normal use anyway).

Here is an example of what the FreeTube database export looks like after watching these two videos:

"Me at the zoo" https://youtu.be/jNQXAC9IVRw
"lofi hip hop radio 📚 - beats to relax/study to" https://youtu.be/jfKfPfyJRdk

{"videoId":"jfKfPfyJRdk","title":"lofi hip hop radio 📚 - beats to relax/study to","author":"Lofi Girl","authorId":"UCSJ4gkVC6NrvII8umztf0Ow","published":1657627949000,"description":"🎼 | Listen on Spotify, Apple music and more\n→  https://fanlink.to/lofigirl-music\n\n🌎 | Lofi Girl on all social media\n→  https://fanlink.to/lofigirl-social\n\n👕 | Lofi Girl merch\n→  https://bit.ly/Iofigirl-shop?utm_medium=product_shelf&utm_source=youtube&utm_content=YT-APM2gMBZ2lHbH_8duEeNoqM8TYmxMhpPuSxqQum6aVrALareZf8J-lqrO5lPPfx45Xo305Bdd2BxYnHp1qwlkFRMvIjibTJQ7U68tEmOHYDMtsX6d6fE1iwD_sW0Hby8tNnPT5XPu-Rfl6JlzTq9F2PzoagBAZty3hybvrGSD7BbrhBqmyjf19VL\n\n🎭 | Create your lofi avatar now\n→  https://lofigirl.com/generator/\n\n💬 | Join the Lofi Girl community\n→   https://bit.ly/lofigirl-discord\n→   https://bit.ly/lofigirl-reddit\n\n🎶 | Radio tracklist\n→  https://bit.ly/lofi-tracklist\n\n🎨 | Art by Lofi Studio (full list of artists here)\n→  https://www.instagram.com/p/CuChqFXs08M/\n\n📝 | Submit your music / art\n→  https://bit.ly/lofi-submission\n\n🤗 Thank you for listening, I hope you will have a good time here","viewCount":332857195,"lengthSeconds":0,"watchProgress":0,"timeWatched":1705512531034,"isLive":false,"type":"video"}
{"videoId":"jNQXAC9IVRw","title":"Me at the zoo","author":"jawed","authorId":"UC4QobU6STFB0P71PMvOGN5A","published":1114313512000,"description":"00:00 Intro\n00:05 The cool thing\n00:17 End","viewCount":303366518,"lengthSeconds":19,"watchProgress":4.404224,"timeWatched":1705512414087,"isLive":false,"type":"video"}

Here is a TypeScript type definition for one of those video records (FreeTube doesn't use TypeScript, but the syntax is well known, so I decided to use it here):

type HistoryEntry = {
  videoId: string, // video ID
  title: string, // video title
  author: string, // channel name/uploader name
  authorId: string, // channel id/uploader id
  published: number, // timstamp of when the video was published/uploaded in milliseconds since 1970-01-01 00:00:00 UTC
  description: string, // video description
  viewCount: number, // number of views that the video has
  lengthSeconds: number, // duration of the video in seconds, can be decimal
  watchProgress: number, // how much of the video has been watched in seconds, can be decimal, use 0 if unknown
  timeWatched: number, // timstamp of when the video was last watched in milliseconds since 1970-01-01 00:00:00 UTC, used by FreeTube to chronologically sort entries by how recently they were watched
  isLive: boolean, // Whether the video is live/was live at the time it was watched by the user
  type: 'video' // always the string "video"
}

iBicha commented 9 months ago

Thanks a lot for the detailed response, much appreciated. At a glance I think this should be enough to get me going. I'll close the issue, and follow up if there's any detail to iron out.

iBicha commented 9 months ago

@absidue how important are all the fields in the history videos? Some fields might not be available, and it would be super slow to make an additional web call for each video to verify more data.

Please filter out ads, private videos and member only videos.

This also might require a web request per video, which might take very long...

absidue commented 9 months ago

Which fields are unavailable?

iBicha commented 9 months ago

a ReelItem for example is missing a lot of things - but I think the list is author, authorId, published, viewCount, lengthSeconds, watchProgress, timeWatched, isLive. Any of these CAN be missing, but not always.

I did try importing, and while FreeTube complained about some items with missing info, it did import the rest. I feel like this is in the territory of "Good enough" since history does not have to be accurate. Although it does show that videos are published on 1969 (since I'm setting the timestamp to 0 as it is missing)

Perhaps this could be an initial implementation and improved later.

If you do feel like trying this, it's here https://github.com/iBicha/yt2alt/pull/12 (just an npm i and npm start to run)

absidue commented 9 months ago

FreeTube will skip importing items without the required keys, so ideally you want to try filling all of them whenever possible.

isLive: ReelItems/shorts are never live (they are limited to 1 minute in length by YouTube), so you can always set it to false for them.
watchProgress: As mentioned in my type definition you should set it to 0 if it is unknown (that's what FreeTube does if you manually mark it as watched from anywhere other than the watch page)
lengthSeconds: You can set that to an empty string, then FreeTube will handle it correctly '' (that's what FreeTube does when it is parsing shorts on the shorts in the channel shorts tab)
viewCount: is there for ReelItems, you just need to parse the compact number. FreeTube does have code for parsing en-US compact numbers, but if you copied that, you would need to change the license of your project to AGPL-3.0, as AGPL-3.0 is infective, so you probably want to write your own (YouTube.js defaults to requesting the page in en-US, but as you are dealing with logged in users, I don't know what language YouTube uses)
published: For non-ReelItems you can parse the relative date string, to get an approximate published date, otherwise you would need to set it to undefined, but as JSON.stringify skips undefined fields, I'm not sure what you could do.
timeWatched: If it's missing you could do an approximation, e.g. item 1 was watched on 2024-01-03 00:00:00, item 2 is unknown, item 3 was watched on 2024-01-01 00:00:00, then you would set item 2 to 2024-01-02 00:00:00, that way at least it would be in the correct position in the watch history (freetube uses timeWatched to sort by most recently watched, which is why accuracy isn't that important as long as items are sort of in the correct place)
author and authorId: If these are unknown you would set them to undefined, but you have the same problem as above, that JSON.stringify skips undefined fields (You want the field to exist so that the import doesn't skip the item, but you want it to be undefined so that the rest of the app know's it's unknown).

For the values that need to be undefined, I'm not sure off the top of my head if you could set them to null, as I know that FreeTube's code generally has an inconsistency problem (some places explicitly check for undefined others explicitly for null and some places also just to a truthy check), it's a side-effect of how it has grown over time with lots of different people working on it.

FreeTubeApp / FreeTube