iv-org / invidious

Invidious is an alternative front-end to YouTube
https://invidious.io
GNU Affero General Public License v3.0
16.19k stars 1.79k forks source link

Fix youtube CSV playlist importer #4787

Open ThatMatrix opened 2 months ago

ThatMatrix commented 2 months ago

This PR is a follow up of this issue https://github.com/iv-org/invidious/issues/3954 .

tldr: This PR aims to fix the Youtube CSV Importer due to Google Takeout changing their export format

I've recently used a invidious account, exported all my playlists data from Google Takeout. However, it seems that they changed their format again by putting most of the datas about a playlist in playlists.csv. Only to leave the bare minimum in the playlist specific files.

This PR changes this by putting default values where info could be extracted before and fixing the out of bound error that could be happening depending on the playlist format.

Here is the error encountered:

Title: `Index out of bounds (IndexError)`
Date: `2024-07-11T01:36:38Z`
Route: `/data_control?referer=%2Ffeed%2Fplaylists`
Version: `2024.04.27-eda7444 @ master`

<details>
<summary>Backtrace</summary>
<p>

Index out of bounds (IndexError)
  from /usr/share/crystal/src/json/any.cr:237:3 in 'update_data_control'
  from lib/kemal/src/kemal/route.cr:12:9 in '->'
  from src/invidious/helpers/handlers.cr:30:37 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:94:12 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from src/ext/kemal_static_file_handler.cr:112:11 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/request_processor.cr:51:11 in 'handle_client'
  from /usr/share/crystal/src/fiber.cr:146:11 in 'run'
  from ???

</p>
</details>

I also give you an extract of what a Google Takeout playlist file looks like

Video ID,Playlist video creation timestamp
dQw4w9WgXcQ,2024-06-24T23:49:51+00:00
-tJYN-eG1zk,2024-06-23T20:30:24+00:00
ThatMatrix commented 2 months ago

I have tested importing a playlist with the final commit version and the provided docker-compose.yml file. It works with one of my GoogleTakeout playlist file.

ThatMatrix commented 2 months ago

I took the liberty to add a default value/placeholders for the description as well as privacy fields in the meantime but I would like to receive advice on what to do with them if possible. Thanks a lot

SamantazFox commented 2 months ago

Can you make a separate function for the new format? Users who have exported when v1 was around should still be able to import their data!

PassiveLemon commented 2 weeks ago

Importing works for me with this PR