Schmuppes commented 3 years ago

[x] I am using the latest version - 0.21.7
[x] I checked, but didn't find any duplicates (open OR closed) of this issue in the repo.
[x] This issue contains only one bug. I will open one issue for every bug report I want to file.

I wanted to import my YouTube subscriptions and followed the instructions. The thing is that the file that YouTube creates in the .zip is a .csv file. That cannot be imported to NewPipe. I've also tried changing the file type to .json, but that does not work either.

Device info

Android version/Custom ROM version: 10
Device model: Redmi Note 5 Pro

ed1d1a8d commented 3 years ago

Also running into this. I believe https://github.com/TeamNewPipe/NewPipeExtractor/blob/dev/extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeSubscriptionExtractor.java is the file to modify to support csv exports.

Here is a python script that converts from the csv format to the old json format:

import pandas as pd
df = pd.read_csv('subscriptions.csv')

subs_list = [
    dict(snippet=dict(
        resourceId=dict(
            channelId=r["Channel Id"]
        ),
        title=r["Channel Title"]
    ))
    for _, r in df.iterrows()
]

import json
with open('subscriptions.json', 'w') as f:
    json.dump(subs_list, f)

triallax commented 3 years ago

There have been several reports of this on IRC, so I think this should be given some priority.

Kommynct commented 3 years ago

I can't seem to run that python script, i installed pandas but i get this:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3361, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 76, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5198, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5206, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'Channel Id'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/communist/Downloads/Telegram Desktop/Takeout/YouTube and YouTube Music/subscriptions/pythonscript.py", line 4, in <module>
    subs_list = [
  File "/home/communist/Downloads/Telegram Desktop/Takeout/YouTube and YouTube Music/subscriptions/pythonscript.py", line 7, in <listcomp>
    channelId=r["Channel Id"]
  File "/usr/lib/python3.9/site-packages/pandas/core/series.py", line 942, in __getitem__
    return self._get_value(key)
  File "/usr/lib/python3.9/site-packages/pandas/core/series.py", line 1051, in _get_value
    loc = self.index.get_loc(label)
  File "/usr/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3363, in get_loc
    raise KeyError(key) from err
KeyError: 'Channel Id'

ed1d1a8d commented 3 years ago

That is a weird error. Could you post the first few lines of your subscriptions.csv file? I wonder if it has a different format than mine...

Kommynct commented 3 years ago

Channel ID,Channel URL,Channel title
UC-9mlH6az1Q_XUP3fw1N4Fg,http://www.youtube.com/channel/UC-9mlH6az1Q_XUP3fw1N4Fg,Steven Wilson
UC-tLyAaPbRZiYrOJxAGB7dQ,http://www.youtube.com/channel/UC-tLyAaPbRZiYrOJxAGB7dQ,Pursuit of Wonder

triallax commented 3 years ago

@Schmuppes why did you close this?

Schmuppes commented 3 years ago

@Schmuppes why did you close this?

It was an accident, I was gonna delete a half-typed reply. Can you open it again?

Edit: Thanks, I'm sorry.

ed1d1a8d commented 3 years ago

@Kommynct That is bizarre. Your column labels are slightly different from mine:

Channel Id,Channel Url,Channel Title
UC-3SbfTPJsL8fJAPKiVqBLg,http://www.youtube.com/channel/UC-3SbfTPJsL8fJAPKiVqBLg,Deep Look

Here's a fixed version of the script (it normalizes the column names to be all lowercase first):

import pandas as pd
df = pd.read_csv('subscriptions.csv')
df.columns= df.columns.str.lower()

subs_list = [
    dict(snippet=dict(
        resourceId=dict(
            channelId=r["channel id"]
        ),
        title=r["channel title"]
    ))
    for _, r in df.iterrows()
]

import json
with open('subscriptions.json', 'w') as f:
    json.dump(subs_list, f)

Schmuppes commented 3 years ago

@Kommynct That is bizarre. Your column labels are slightly different from mine:
Channel Id,Channel Url,Channel Title
UC-3SbfTPJsL8fJAPKiVqBLg,http://www.youtube.com/channel/UC-3SbfTPJsL8fJAPKiVqBLg,Deep Look

Which makes me wonder if that makes a difference. My German file has the translated labels of course, but the actual information below it is the same.

kachapman commented 3 years ago

I made both scripts, and I get this error when I try running them. Any ideas?

Traceback (most recent call last): File "/home/chapmanfamily/Desktop/takeout-20210726T005738Z-001/Takeout/YouTube and YouTube Music/subscriptions/newpipe1.py", line 16, in with open('subscriptions.json', 'w') as f: PermissionError: [Errno 13] Permission denied: 'subscriptions.json

hfmkfj commented 3 years ago

i got the same problem yesterday and i also tried the online csv to json but didn't work... the last import was in May version: 0.21.4 Device: Redmi K30 Ultra OS: MIUI 12.5.4 (Android 11)

kachapman commented 3 years ago

I made both scripts, and I get this error when I try running them. Any ideas?

Traceback (most recent call last): File "/home/chapmanfamily/Desktop/takeout-20210726T005738Z-001/Takeout/YouTube and YouTube Music/subscriptions/newpipe1.py", line 16, in with open('subscriptions.json', 'w') as f: PermissionError: [Errno 13] Permission denied: 'subscriptions.json

Nm! I changed users and it worked.

Mylloon commented 3 years ago

beware, the names of the columns change according to the languages from now on, it is better not to do so

alecStewart1 commented 3 years ago

The PermissionError y'all are getting I believe is because the files given to you from the Google Takeout are set to be read-only. You may have to have set read-write permissions for those files. When I tried importing just the CSV file alone into NewPipe, I got an traceback error from Java said Media is read-only or something like that.

alecStewart1 commented 3 years ago

I just tried the above script from @ed1d1a8d and I had no troubles since I made sure to give read-write permissions to the subscriptions.csv file. Should work for others as well, so long as you do that.

However, in that script, what is this snippet field?

ed1d1a8d commented 3 years ago

@alecStewart1 Not sure, but going through the existing NewPipe code, it seems it is a required part of the json structure.

DerLexx commented 3 years ago

Hey.

I'd also face the issue. I'm new to Github and also not really aware of how the scripts and all that works.

I just wanted to leave my voice here to, the issue is present and I tried to roll back to an older version of the .apk, but then I found out it's about .csv has to be .jason

Is someone able to explain for a beginner windows user how those script work and if it works?

Excuse me if this is a dumb question.

talanc commented 3 years ago

@TobiGr or the other main devs: From what I can see, NewPipe now needs to import using the CSV files instead of JSON files (unless its the older format). Is this bug up for grabs?

diogotito commented 3 years ago

Hey.

I'd also face the issue. I'm new to Github and also not really aware of how the scripts and all that works.

I just wanted to leave my voice here to, the issue is present and I tried to roll back to an older version of the .apk, but then I found out it's about .csv has to be .jason

Is someone able to explain for a beginner windows user how those script work and if it works?

Excuse me if this is a dumb question.

The script ed1d1a8d shared worked for me. I was able to import all my subscriptions just yesterday!

For this specific snippet, you'll need to have two things installed on your Windows system:

A Python interpreter, as this script is written in the Python programming language. It doesn't come pre-installed on Windows.
The Pandas library for Python, which this script uses to comfortably read the subscriptions from the CSV file and transform them to be written in JSON.

There are a few ways you can go about this, but I'll try to explain what I think is the simplest method.

To install Python on Windows 10, you can type python in the Start menu, and it will open the Microsoft Store page where you can install it easily (alternatively, run the installer from https://www.python.org/).

This won't come with the Pandas library, however! (unless you installed Anaconda Python, which happens to be the recommended method in Pandas' website) For that, open a Command Prompt window (⊞ Win+R cmd) and type

pip install pandas

Hopefully, that will install Pandas without any hiccups.

Now, to use ed1d1a8d's script, copy it and save it to a file with a .py extension next to your subscriptions.csv file. Double click your new Python script file, and a subscriptions.json should appear! If it doesn't, something went wrong, probably Pandas will be to blame.

Then try to import that in NewPipe.

Good luck!

DerLexx commented 3 years ago

Hey.

I'd also face the issue. I'm new to Github and also not really aware of how the scripts and all that works.

I just wanted to leave my voice here to, the issue is present and I tried to roll back to an older version of the .apk, but then I found out it's about .csv has to be .jason

Is someone able to explain for a beginner windows user how those script work and if it works?

Excuse me if this is a dumb question.

The script ed1d1a8d shared worked for me. I was able to import all my subscriptions just yesterday!

For this specific snippet, you'll need to have two things installed on your Windows system:

A Python interpreter, as this script is written in the Python programming language. It doesn't come pre-installed on Windows.

The Pandas library for Python, which this script uses to comfortably read the subscriptions from the CSV file and transform them to be written in JSON.

There are a few ways you can go about this, but I'll try to explain what I think is the simplest method.

To install Python on Windows 10, you can type python in the Start menu, and it will open the Microsoft Store page where you can install it easily (alternatively, run the installer from https://www.python.org/).

This won't come with the Pandas library, however! (unless you installed Anaconda Python, which happens to be the recommended method in Pandas' website) For that, open a Command Prompt window (⊞ Win+R cmd) and type
pip install pandas
Hopefully, that will install Pandas without any hiccups.

Now, to use ed1d1a8d's script, copy it and save it to a file with a .py extension next to your subscriptions.csv file. Double click your new Python script file, and a subscriptions.json should appear! If it doesn't, something went wrong, probably Pandas will be to blame.

Then try to import that in NewPipe.

Good luck!

Thank you so much and the reply was really helpful. I'll try it by tomorrow!

Cheers

tbjgolden commented 3 years ago

also, for anyone having issues with python/pip/pandas I've also made this small node script to do the same thing: https://gist.github.com/tbjgolden/72bf7595a8338772516cf6b3968d69c9

lmk if this ends up being too tricky for you and I'll set a web page up or something

triallax commented 3 years ago

@tbjgolden could you at least post the source code? It's not really feasible to see how the script works, as it's minified.

tbjgolden commented 3 years ago

@tbjgolden could you at least post the source code? It's not really feasible to see how the script works, as it's minified.

updated: https://gist.github.com/tbjgolden/72bf7595a8338772516cf6b3968d69c9#gistcomment-3842080

diogotito commented 3 years ago

I remembered that for Windows users, PowerShell comes pre-installed and can work with CSV and JSON out of the box. I wrote this short pipeline to try it out:

Import-Csv .\subscriptions.csv | ForEach-Object {
    @{
        snippet = @{
            resourceId = @{
                channelId = $_.'Channel Id'
            };
            title = $_.'Channel Title'
        }
    }
} | ConvertTo-Json -Depth 100 | Out-File -Encoding utf8 .\subscriptions.json

To use this:

In File Explorer, navigate to the folder where your ''subscriptions.csv'' lives
File → Open Windows PowerShell
Paste the above snippet (Ctrl+V or just Right Click) and press Enter

Easy! NewPipe seems to handle it fine the resulting ''subscriptions.json'' for me. Let me know if it works for you if you try it out!

ens4dz commented 3 years ago

Without Pc You can use Google colab for easy use: Link

1-click top left corner -> show file browser 2-click at file upload icon, then choose your csv file 3-Run code 4-download the json file from file browser

talanc commented 3 years ago

@mhmdanas Is anyone working on a fix for this? If not I'll try to fix it.

xdestviz commented 3 years ago

Without Pc You can use Google colab for easy use: Link

1-click top left corner -> show file browser 2-click at file upload icon, then choose your csv file 3-Run code 4-download the json file from file browser

When I try to run this, I get the following error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2897             try:
-> 2898                 return self._engine.get_loc(casted_key)
   2899             except KeyError as err:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'Channel Id'

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
4 frames
/usr/local/lib/python3.7/dist-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2898                 return self._engine.get_loc(casted_key)
   2899             except KeyError as err:
-> 2900                 raise KeyError(key) from err
   2901 
   2902         if tolerance is not None:

KeyError: 'Channel Id'

Schmuppes commented 3 years ago

Easy! NewPipe seems to handle it fine the resulting ''subscriptions.json'' for me. Let me know if it works for you if you try it out!

I'm afraid it doesn't work for me. NewPipe shows an error message to the tune of "Subscriptions could not be imported".

ens4dz commented 3 years ago

beware, the names of the columns change according to the languages from now on, it is better not to do so @xdestviz : assert that columns of your csv file is "Channel Id" and "Channel Title"

diogotito commented 3 years ago

Yes, the 3 scripts here have the column names "Channel Id" and "Channel Title" hardcoded in them. They assume the 1st line of the subscriptions.csv is

Channel Id,Channel Url,Channel Title

Open the subscriptions.csv in a text editor and check if the 1st line matches the above.

Otherwise

The Python script (either locally or in Google Collab) will spit out an error message with Key Error in it, meaning that it tried to access a column named "Channel Id" that doesn't exist.
The Node script will create a valid subscriptions.json document with 0 channels inside. If you open it in a text editor you will see [] in it.
The PowerShell Script will create a subscriptions.json with as many channels as the csv, but all of them will have nulls instead of the correct channel data. NewPipe will rightly fail to read this file.

neodreen commented 3 years ago

If someone's wondering about it, you need to change the name of your file and the first 3 lines of the .csv file to comply with any English variant. My .csv was in Spanish and I had to change the name of the file from suscripciones.csv to subscriptions.csv and the 3 first lines were in Spanish too, but changing them to what's shown on the examples above fixed it.

NioAntoVenice commented 3 years ago

Hi, I'm from Italy and ofc using youtube italian language and the exported file is named "iscrizioni.csv"... ho can i convert it on the right way? Pls give me an usable/woeking link to convert the csv file to ,json (but i think it should be easier to fix NewPipe using the csv exported file...

talanc commented 3 years ago

@KuroInside @NioAntoVenice (and anyone else who wants to help) Would you be able to upload your takeout zip file? OR Take a screenshot(s) that shows the following:

The path to the csv file from the ZIP file
The name of the subscriptions.csv (as it's different in different languages)
The header / first line of the csv file

Here's an ideal example:

neodreen commented 3 years ago

takeout-20210805T122423Z-001.zip

@talanc Here's my takeout zip, hope this can help. Sorry for not uploading just a screenshot but it's really late here and I'm exhausted. XD

JCGdev commented 3 years ago

I've coded a script for dealing with this,

https://github.com/JCGdev/Newpipe-CSV-Fixer

the temporary solution is:

Replacing the 3 column names with service_id,url,name.
Converting the CSV to a JSON and appending it in the empty list of the "NewPipe headers" {"app_version":"x.xx.x","app_version_int":xxx,"subscriptions":[]}

I got this info from redddit on here

tbjgolden commented 3 years ago

takeout-20210805T122423Z-001.zip

@talanc Here's my takeout zip, hope this can help. Sorry for not uploading just a screenshot but it's really late here and I'm exhausted. XD

This seems to be a pretty active issue so I'll set something up to fix that bug and make it easier :D

tbjgolden commented 3 years ago

https://tbjgolden.github.io/newpipe-csv-json-fixer/

Simple solution for non-technical people - converts your subscriptions.csv files to subscriptions.json on the web page.

@talanc @KuroInside @NioAntoVenice this solution should work for non-English language YouTube users 😄

Source code: https://github.com/tbjgolden/newpipe-csv-json-fixer

talanc commented 3 years ago

I've made the following PRs to fix the issue. Hopefully the devs can review and (if all is good) approve it soon. https://github.com/TeamNewPipe/NewPipe/pull/6882 https://github.com/TeamNewPipe/NewPipeExtractor/pull/709

It allows the user to select either a CSV file or ZIP file for importing subscriptions.

pixie1259 commented 3 years ago

Thanks diogotito for the PowerShell script and instructions. I made a copy of the csv, moved it to the User folder and opened it from there, opened PowerShell, pasted the snippet and voila! Done! Then moved it back to my phone and imported from there. If I remember correctly, I think I even forgot to press Enter and just closed PowerShell after seeing the json file next to the csv in all the excited state my brain was in after fruitlessly looking for a solution for a few hours.

Cheers!

nicholsmm commented 3 years ago

I remembered that for Windows users, PowerShell comes pre-installed and can work with CSV and JSON out of the box. I wrote this short pipeline to try it out:
Import-Csv .\subscriptions.csv | ForEach-Object {
    @{
        snippet = @{
            resourceId = @{
                channelId = $_.'Channel Id'
            };
            title = $_.'Channel Title'
        }
    }
} | ConvertTo-Json -Depth 100 | Out-File -Encoding utf8 .\subscriptions.json
To use this:

In File Explorer, navigate to the folder where your ''subscriptions.csv'' lives

File → Open Windows PowerShell

Paste the above snippet (Ctrl+V or just Right Click) and press Enter

Easy! NewPipe seems to handle it fine the resulting ''subscriptions.json'' for me. Let me know if it works for you if you try it out!

Hi Dio, this worked for me, thanks. One wrinkle in Win 11 - I could not figure out step 2. So I moved subscriptions.csv into my home user folder (C:/User/Myname), opened Powershell, ran your script and swept the .csv & .json files back into my docs folder before transferring it to my phone via usb. Import into NewPipe worked a charm then. Thanks again.

NioAntoVenice commented 3 years ago

Ore maybe more "easy!" install vanced ;)Ahahh thank you so much anyway ! Il domenica 29 agosto 2021, 10:18:42 CEST, nicholsmm @.***> ha scritto:

I remembered that for Windows users, PowerShell comes pre-installed and can work with CSV and JSON out of the box. I wrote this short pipeline to try it out: Import-Csv .\subscriptions.csv | ForEach-Object { @{ snippet = @{ resourceId = @{ channelId = $.'Channel Id' }; title = $.'Channel Title' } } } | ConvertTo-Json -Depth 100 | Out-File -Encoding utf8 .\subscriptions.json To use this:

In File Explorer, navigate to the folder where your ''subscriptions.csv'' lives
File → Open Windows PowerShell
Paste the above snippet (Ctrl+V or just Right Click) and press Enter

Easy! NewPipe seems to handle it fine the resulting ''subscriptions.json'' for me. Let me know if it works for you if you try it out!

Hi Dio, this worked for me, thanks. One wrinkle in Win 11 - I could not figure out step 2. So I moved subscriptions.csv into my home user folder (C:/User/Myname), opened Powershell, ran your script and swept the .csv & .json files back into my docs folder before transferring it to my phone via usb. Import into NewPipe worked a charm then. Thanks again.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

juandarr commented 3 years ago

For anyone having problems with the subscription export, I built this webpage that allows to export your subscription information to valid JSON using two different methods: 1) Export your YouTube subscriptions from Google takeout. You get a CSV file which can be converted with this tool to a valid JSON format. 2) (Faster and less troublesome) Export your YouTube subscriptions using the Chrome extension YouTube Subscriptions Exporter. Your YouTube subscription page will be scrapped and the subscriptions will be copied on the clipboard. The tool can be used to change this data to JSON.

Since the tool is web based, you can use it from mobile and desktop. You just need a web browser. If you have questions, problems, I am here to help. Cheers

cangSDARM commented 3 years ago

My solution was successful on version 0.21.8 https://github.com/SkyTubeTeam/SkyTube/issues/991#issuecomment-910112869

Simon311 commented 3 years ago

Importing the CSV file doesn't work in the latest NewPipe available from F-Droid. But perhaps more importantly, my Google export only returned 6 channels out of hundreds. I know there's nothing you guys can do about it, but has anyone had this same issue? Is there a way to report this to Google, since this is a clear GDPR violation?

juandarr commented 3 years ago

Importing the CSV file doesn't work in the latest NewPipe available from F-Droid. But perhaps more importantly, my Google export only returned 6 channels out of hundreds. I know there's nothing you guys can do about it, but has anyone had this same issue? Is there a way to report this to Google, since this is a clear GDPR violation?

If you are having issues with Google takeout use solution (2) from my previous comment: Export YouTube subscription data.

Chiryoja commented 3 years ago

Ore maybe more "easy!" install vanced ;)Ahahh thank you so much anyway ! Il domenica 29 agosto 2021, 10:18:42 CEST, nicholsmm @.***> ha scritto: I remembered that for Windows users, PowerShell comes pre-installed and can work with CSV and JSON out of the box. I wrote this short pipeline to try it out: Import-Csv .\subscriptions.csv | ForEach-Object { @{ snippet = @{ resourceId = @{ channelId = $.'Channel Id' }; title = $.'Channel Title' } } } | ConvertTo-Json -Depth 100 | Out-File -Encoding utf8 .\subscriptions.json To use this: - In File Explorer, navigate to the folder where your ''subscriptions.csv'' lives - File → Open Windows PowerShell - Paste the above snippet (Ctrl+V or just Right Click) and press Enter Easy! NewPipe seems to handle it fine the resulting ''subscriptions.json'' for me. Let me know if it works for you if you try it out! Hi Dio, this worked for me, thanks. One wrinkle in Win 11 - I could not figure out step 2. So I moved subscriptions.csv into my home user folder (C:/User/Myname), opened Powershell, ran your script and swept the .csv & .json files back into my docs folder before transferring it to my phone via usb. Import into NewPipe worked a charm then. Thanks again. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

WoW... i've been trying python scripts all afternoon, and then came across your post. It worked first try... thanks so much!

TeamNewPipe / NewPipe

YouTube subscriptions cannot be imported (seem to be exported as .csv now) #6757

Device info

https://tbjgolden.github.io/newpipe-csv-json-fixer/