[BUG] Subscriptions and Playlists Have Disappeared

[rockbo842]

Describe the bug All of my subscriptions have disappeared.

I have the container running internet access through another container to provide a VPN for the Youtube downloads. I tried to add a Brave browser container and have it route through the same VPN container. There was an error in the VPN container which required me to reinstall it. Prior to realizing the error took place, I tried to access to YoutubeDL-Material container but it was down due to the VPN container issue. Once the VPN container was reinstalled, I went to YoutubeDL-Material and all of the subscriptions and playlists were gone.

My files are still on my server and the YoutubeDL-Material home page for the container shows recently downloaded files. However, all of my subscriptions and playlists are now gone.

During this situation, no changes were made to the container's settings.

I am running the container on Unraid 6.9.2 and the container is set to pull the nightly.

To Reproduce Steps to reproduce the behavior: Unknown

Expected behavior Subscriptions and playlists should be there.

Screenshots If applicable, add screenshots to help explain your problem.

Environment

• YoutubeDL-Material 4.2?
• Unraid 6.9.2

[rockbo842]

The logs show this error repeatedly:

2022-02-23T03:49:29.011Z INFO: Found new update for yt-dlp. Updating binary... 2022-02-23T03:49:30.699Z INFO: YoutubeDL-Material v4.2 started on PORT [OBFUSCATED]

[samssausages]

I'm also having this problem on Unraid docker. Looks like the update caused that. I lost all my subscriptions and can't see the videos in the UI

Log 2022-02-25T08:54:02.368Z INFO: Beginning migration: 4.1->4.2+ 2022-02-25T08:54:02.483Z INFO: 4.1->4.2+ migration complete! 2022-02-25T08:54:04.777Z INFO: youtube-dl successfully updated! 2022-02-25T08:54:04.784Z INFO: YoutubeDL-Material v4.2 started on PORT xxxxx 2022-02-26T08:51:37.641Z INFO: youtube-dl successfully updated! 2022-02-26T08:51:37.649Z INFO: YoutubeDL-Material v4.2 started on PORT xxxxx 2022-02-27T03:28:31.133Z INFO: youtube-dl successfully updated! 2022-02-27T03:28:31.140Z INFO: YoutubeDL-Material v4.2 started on PORT xxxxx

[rockbo842]

I was starting to believe that was the issue. I wonder if there will be a way to get them back.

[l-ucky]

Hello. I am having the same problems. I have Youtube-DL Material running in Docker and managed with Portainer. I can still download videos fine, but I cannot see my subscriptions to playlists or channels. I've fixed this before by just recreating the subscriptions but is there another way to fix this?

[l-ucky]

Hello again. I've tried downloading the nightly build from docker-hub and my subscriptions still won't show up. I can see the raw database files in the YTDL config but it's not identifying the data written about what videos are downloaded.

[rockbo842]

The entire container was failing to work for me until today. Now the subscriptions I added since the purge are working, however, the old subscriptions settings seem to be lost.

[l-ucky]

Me too. I can add new subscriptions, but it just over writes the old subscriptions which burns my disks out because it's not the first time the subscriptions failed after a redployment. There are database options but I don't see anywhere to import my old database files. Maybe I could rebuild youtube-dl material with the old database file since I noticed it keeps making new ones.

[rockbo842]

I was curious about the database situation as well. It would be great to be able to import an old database or update an existing database by adding files to the new subscription folder. That way the new subscription won't try to download the same videos again. I wish I had the know-how to develop something like that.

[samssausages]

I haven't been able to use YoutubeDL-Material since this update, I don't want to re-subscribe and then have it download everything all over again. (Is that what would happen if I did?)

Hate that this update pretty much broke it for me and I haven't been able to get any of my Subs downloaded since.

[rockbo842]

That is exactly what will happen unless you set the new subscription not to "Download All Uploads". If you uncheck that when setting up the subscription, it will give you some parameters to limit the downloads. The issue is that you won't be able to use the UI to watch the videos from the old subscription within the new subscription page. The old videos will be available from the home page, though.

[l-ucky]

I tried doing the solution above and just got duplicates. I hope this issue is being looked at.

[l-ucky]

I just tried again with the downloading videos starting from XX days ago. Nothing downloaded into the proper custom folder name and when I restarted the youtube-dl material container it deleted the subscription from the WebUI. I'm using the stable 4.2 version.

[l-ucky]

2022-02-28T13:36:10.202Z INFO: youtube-dl successfully updated! 2022-02-28T13:36:10.207Z INFO: YoutubeDL-Material v4.2 started on PORT 17442 2022-02-28T13:41:05.069Z ERROR: ERROR: u'test' is not a valid URL. Set --default-search "ytsearch" (or run youtube-dl "ytsearch:test" ) to search YouTube 2022-02-28T13:41:05.069Z ERROR: Subscribe: Failed to get subscription info. Subscribe failed. 2022-03-01T13:08:21.411Z INFO: youtube-dl successfully updated! 2022-03-01T13:08:21.418Z INFO: YoutubeDL-Material v4.2 started on PORT 17442 2022-03-01T13:15:09.444Z INFO: Backing up your non-video/audio files to appdata/backups/backup-1646140509444.zip. This may take up to a few seconds/minutes. 2022-03-01T13:15:36.998Z INFO: Downloading new files... 2022-03-01T13:15:37.500Z INFO: Installing update v4.1... 2022-03-01T13:15:47.971Z INFO: Update complete! Restarting server... 2022-03-01T13:51:41.411Z WARN: Cannot find config with key 'ytdl_max_concurrent_downloads'. Creating one with the default value... 2022-03-01T13:51:41.415Z WARN: Cannot find config with key 'ytdl_download_rate_limit'. Creating one with the default value... 2022-03-01T13:51:41.416Z WARN: Cannot find config with key 'ytdl_allow_autoplay'. Creating one with the default value... 2022-03-01T13:51:41.417Z WARN: Cannot find config with key 'ytdl_use_sponsorblock_api'. Creating one with the default value... 2022-03-01T13:51:41.418Z WARN: Cannot find config with key 'ytdl_generate_nfo_files'. Creating one with the default value... 2022-03-01T13:51:41.419Z WARN: Cannot find config with key 'ytdl_use_local_db'. Creating one with the default value... 2022-03-01T13:51:41.419Z WARN: Cannot find config with key 'ytdl_mongodb_connection_string'. Creating one with the default value... 2022-03-01T13:51:41.449Z INFO: Beginning migration: 4.2->4.3+ 2022-03-01T13:51:41.462Z INFO: 4.2->4.3+ migration complete! 2022-03-01T13:51:42.463Z INFO: Found new update for youtube-dl. Updating binary... 2022-03-01T13:51:43.192Z INFO: YoutubeDL-Material v4.2 started on PORT 17442 2022-03-01T14:19:35.308Z ERROR: connect ECONNREFUSED 127.0.0.1:27017 2022-03-01T14:19:35.309Z ERROR: Failed to connect to MongoDB. Verify your connection string is valid. 2022-03-01T14:27:48.268Z INFO: Config items set using ENV variables. 2022-03-01T14:27:49.378Z INFO: Found new update for youtube-dl. Updating binary... 2022-03-01T14:27:50.094Z INFO: YoutubeDL-Material v4.2 started on PORT 17442 2022-03-01T14:28:45.819Z ERROR: Failed to transfer DB as it cannot transition into its current status: MongoDB 2022-03-01T14:28:50.989Z ERROR: Failed to transfer DB as it cannot transition into its current status: MongoDB 2022-03-03T14:50:07.000Z ERROR: Failed to transfer DB as it cannot transition into its current status: MongoDB 2022-03-03T15:06:28.339Z ERROR: ERROR: This video is unavailable on this device. ERROR: This video is unavailable on this device. 2022-03-03T15:06:28.339Z INFO: An error was encountered with at least one video, backup method will be used. 2022-03-03T15:11:43.581Z INFO: Config items set using ENV variables. 2022-03-03T15:11:45.029Z INFO: YoutubeDL-Material v4.2 started on PORT 17442 2022-03-03T15:26:04.086Z INFO: youtube-dl successfully updated! 2022-03-03T15:26:04.092Z INFO: YoutubeDL-Material v4.2 started on PORT 17442

[l-ucky]

Hello thread,

I clicked into the YoutubeDL-Material settings within the subscription section. I clicked on Args and sorted by category. I found that the argument -w is just what you need. -w means do not overwrite, so you will skip what you have and if you are up-to-date, then it won't download a thing but be ready for new stuff. You have to relink the subscription, but just use the argument when you make a replacement subscription. If you used the last solution where you set the new subscription to only download videos starting at the day of failure (e.g. only download videos in the last 13 days...because my failure was 13 days ago so I need only the last 13 days and going forward's videos.) I hope this post helps, here's a picture to make it clearer, I squared the important points in red boxes.

https://i.imgur.com/gC60oQS.png

[rockbo842]

Thanks for running this down! I will try it out later today and see how it goes.

[samssausages]

Thanks l-ucky! That works as a good work around. I still had to re-create my subs, but it pulled in what I already had downloaded and essentially helped rebuild the database for me. Just give it some time for it to query your subs and re-build everything. I let it run overnight and looks like everything is as it was before now.

[samssausages]

Son of a Gun... It happened again with the last update. Getting a bit annoying losing and having to redo all my subs...

[rockbo842]

Mine seem to be ok. It looks like the creator is gone so I don't understand why there are updates taking place.

[GlassedSilver]

Mine seem to be ok. It looks like the creator is gone so I don't understand why there are updates taking place.

I did some dependency updates in the last few days to fix the aging backend.

The thing is, I would strongly advise using the nightly builds as before, the latest stable is everything but stable. (oh the irony, we're aware :P)

Also make sure to use archive options. That makes sure that YTDL-M will remember what's already been downloaded and not retry those downloads. (one exception being to redownload very recent downloads. This setting got introduced to pull videos again after some safety period in case YT's processing only had mediocre quality files available at download time)

The archive option can be set in: Settings > Downloader > Use youtube-dl archive

Strong recommendation to set that!

The redownload function for refreshing potentially mediocre source material can be set in: Settings > Main > Redownload fresh uploads

And last but not least I URGE everyone to use a MongoDB backend to store the database of YoutubeDL-Material. It fixes stability, performance, scalability and data loss issues. Ever since I activated it on my instance it's been running without losing data.

Here's a little guide how to do it.

Sadly playlists and subscriptions don't reappear without having a manual backup at hand. The video and audio files (if you have any audio-only downloads) should reappear as normal downloads without playlist/subscription association after some time automagically, the app scans the file system for any possibly missing media on every startup.

So... subscriptions need to be recreated if lost and not backed up, same for playlists, but with the MongoDB backend it shouldn't happen again, at least not for the same reason as it did before.

If any further help is needed, don't hesitate to reach out again. I'll close this issue now, but I can reopen it if there's a need for it. :)

[l-ucky]

Hello thanks for the work you put in. I removed my standalone youtubedl-material docker container and used your docker-compose.yml stack. I entered into the settings to follow your instructions, and I get a "Connection Successful!" when clicking the "test connection string" when I use the container name. However, If I switch to my local-IP and port number for MongoDB it fails.

How do I know everything has been transfered? I dont hear any writing on my server, its at my desk with me, and I have around 1600gb for MongoDB to sort through.

In the error logs I see this:

2022-03-25T14:03:09.134Z ERROR: connection timed out
2022-03-25T14:03:09.135Z ERROR: Failed to connect to MongoDB. Verify your connection string is valid.
2022-03-25T14:05:51.443Z ERROR: connection timed out
2022-03-25T14:05:51.443Z ERROR: Failed to connect to MongoDB. Verify your connection string is valid.
2022-03-25T14:10:18.219Z ERROR: connection timed out
2022-03-25T14:10:18.220Z ERROR: Failed to connect to MongoDB. Verify your connection string is valid.

Last update, if I put in the MongoDB local-IP from portainer (e.g. 172.25.0.3) then the connection string is "successful."

[rockbo842]

Thanks, @GlassedSilver! Good to see you and hear that you are still working on this! I'll work on setting up the MongoDB as you suggested. Is there a way to have MongoDB look at an existing folder and create the associated database from the content on its own?

[GlassedSilver]

Did you hit save before transferring? (sorry if that sounds like a stupid question)

Also we had a different mongoDB-related question recently, they self-solved it, maybe it can help you too? https://github.com/Tzahi12345/YoutubeDL-Material/issues/508

Possibly no, because you don't seem to even attempt to use authentication, but... That's just something I pondered.

(note to self: possibly duplicate: #510)

And you're giving me too much praise, I'm just trying to keep the ship sailing, because I love this application and I'm very happy to see that at least me giving support seems to be of some value to folks. As long as others can pitch in some PRs and dependency-bot can keep the backend cleaned we're good for now. :)

[l-ucky]

One day I will do a commit to github, but I'm just a lowly datahorder, a degenerate in the computer world.

[l-ucky]

Did you hit save before transferring?

I did, and I also don't have to transfer local to DB, I only have transfer DB to local and Im nervous it will wipe my data.

[GlassedSilver]

As for when to know if everything transferred... You can do this:

Compare the stats of your Local DB before moving in that Settings > Database page (most notably the files value) against the mongoDB instance's listing. You can view that with MongoDBCompass, MongoDB's official database inspector and editor.

Here's a visualization:

Thanks, @GlassedSilver! Good to see you and hear that you are still working on this! I'll work on setting up the MongoDB as you suggested. Is there a way to have MongoDB look at an existing folder and create the associated database from the content on its own?

No, that's not possible. That's what we'd call magic and until I find my fairy dust that I misplaced again I won't be able to make it work like that. ;) :P hehe

One day I will do a commit to github, but I'm just a lowly datahorder, a degenerate in the computer world.

We all start small, at one point we were all not even a centimeter in size. Your interest in this project and that you help with feedback and such is already a value of its own! :)

Did you hit save before transferring?

I did, and I also don't have to transfer local to DB, I only have transfer DB to local and Im nervous it will wipe my data.

Then it should be done transfering. Feel free to verify by connecting to your mongoDB with Compass and check if you can see this in your view:

[l-ucky]

For a moment I thought you had to get MongoDBCompass on your server, but I used their GUI form the AUR and I'm in looking at the database.

Thanks, @GlassedSilver! Good to see you and hear that you are still working on this! I'll work on setting up the MongoDB as you suggested. Is there a way to have MongoDB look at an existing folder and create the associated database from the content on its own?

No, that's not possible. That's what we'd call magic and until I find my fairy dust that I misplaced again I won't be able to make it work like that. ;) :P hehe

Since I have 130 entries and thats it, do I delete those DB entries so I can reimport or is there a force import function? I went ahead and deleted the 130 entries and other DB entires other than the Admin one, its protected. I'm not redeploying the containers mangodb and ytdl-material.

With mongoDB I'm still stuck at only seeing 130 file entries. I recall having 23000 videos or around there and with the nightly build stack I only have 130 showing. This might be the problem instead of the DB. I was using the latest tag before, and not the nightly tag. How can I get around transfering the DB over to MongoDB? I don't have the option to import the json database to MongoDB, I am only able to transfer from MongoDB to a json, which I'm nervous may overwrite my file.

When looking at htop I can see that mongoDB is doing work, but I dont see updated reflecting in the MongoDB Compass app. However in portainer I can see the DB container is writing a lot and exchanging information over the network a lot, so it's working but I don't see it.

Sadly playlists and subscriptions don't reappear without having a manual backup at hand. I'll get to work lol.