Cannot delete source

[MatthK]

As the channel I added is way too big, I wanted to delete it.

Unfortunately, TubeSync seems to constantly time out. How can I increase the timeout? Or is there a manual way to delete a source?

2021/09/04 12:33:07 [error] 244#244: *15 upstream prematurely closed connection while reading response header from upstream, client: 192.168.7.10, server: _, request: "POST /source-delete/2cb544f2-510e-4131-ab4a-78f28bb5ad88 HTTP/1.1", upstream: "http://127.0.0.1:8080/source-delete/2cb544f2-510e-4131-ab4a-78f28bb5ad88", host: "192.168.7.14:4848", referrer: "http://192.168.7.14:4848/source-delete/2cb544f2-510e-4131-ab4a-78f28bb5ad88"
192.168.7.10 - - [04/Sep/2021:12:33:07 +0800] "POST /source-delete/2cb544f2-510e-4131-ab4a-78f28bb5ad88 HTTP/1.1" 502 166 "http://192.168.7.14:4848/source-delete/2cb544f2-510e-4131-ab4a-78f28bb5ad88" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0"
[2021-09-04 12:33:07 +0800] [303] [INFO] Worker exiting (pid: 303)
[2021-09-04 12:33:08 +0800] [311] [INFO] Booting worker with pid: 311
[2021-09-04 12:33:48 +0800] [226] [CRITICAL] WORKER TIMEOUT (pid:295)
2021/09/04 12:33:48 [error] 244#244: *15 upstream prematurely closed connection while reading response header from upstream, client: 192.168.7.10, server: _, request: "POST /source-delete/2cb544f2-510e-4131-ab4a-78f28bb5ad88 HTTP/1.1", upstream: "http://127.0.0.1:8080/source-delete/2cb544f2-510e-4131-ab4a-78f28bb5ad88", host: "192.168.7.14:4848", referrer: "http://192.168.7.14:4848/source-delete/2cb544f2-510e-4131-ab4a-78f28bb5ad88"
192.168.7.10 - - [04/Sep/2021:12:33:48 +0800] "POST /source-delete/2cb544f2-510e-4131-ab4a-78f28bb5ad88 HTTP/1.1" 502 166 "http://192.168.7.14:4848/source-delete/2cb544f2-510e-4131-ab4a-78f28bb5ad88" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0"
[2021-09-04 12:33:48 +0800] [295] [INFO] Worker exiting (pid: 295)
[2021-09-04 12:33:49 +0800] [319] [INFO] Booting worker with pid: 319
192.168.7.10 - - [04/Sep/2021:12:34:04 +0800] "GET / HTTP/1.1" 200 3694 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0"
192.168.7.10 - - [04/Sep/2021:12:34:04 +0800] "GET /static/styles/tubesync.css HTTP/1.1" 200 34505 "http://192.168.7.14:4848/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0"

[meeb]

Currently editing or deleting a channel as an operation has no choice but to recursively check the media associated with a source one at a time which is why it's slow (because, for example, if you changed the date range or quality requirements your media needs to be individually checked to make sure it's still valid - or for deleting it has to check if the thumbnail exists and delete it and notify the media server the item is gone). There is some work ongoing to make this sensible with #94 but this is now pending replacing the worker library entirely now.

You can manually delete the source by UUID in the database which would do what you want, then delete any media items in the database which is linked to the source UUID. This would leave any downloaded thumbnails or media kicking about on disk though to manually sort out.

If you want to add large channels I would strongly advise you use PostgreSQL as a back-end as well if you aren't already.

[meeb]

Pending the major refactoring work, I've just pushed a couple of quick helper commands to delete massive channels on the command line. Wait for the image to build, pull :latest and then you should be able to use something like:

$ docker exec -ti tubesync python3 /app/manage.py list-sources

Note the source UUID and then use:

$ docker exec -ti tubesync python3 /app/manage.py delete-source --source source-uuid-here

It has no confirmation so make sure you get the UUID right. It'll take some time to delete extremely large sources.

[MatthK]

Wow, thank you for the very quick response. That is incredible.

And it worked. Pulled the latest image, rebuild the container and then could delete the tasks. It ran for almost 10 minutes. I assume that did not delete the thumbnails, no?

Is there an easy way to migrate from the standard DB to PostgreSQL?

[meeb]

That should have deleted thumbnails, media and cleaned everything up properly.

As for migrating, there's a guide here:

https://github.com/meeb/tubesync/blob/main/docs/other-database-backends.md

Basically you're just using Django's built in "dumpdata" to dump from SQLite into JSON, swapping over the backend database connection and then using "loaddata" to import the JSON dump into Postgres. Keep the SQLite database backed up though as you might well encounter issues.

[MatthK]

Yes, I just saw it at the end of the messages that it did delete the thumbnails. I missed that in the beginning. I managed to migrate the DB to MySql, which I had another docker container already running. The SQLite DB is 1.2GB already, so no wonder it was struggling. The migration worked quite well, although the user needs Superuser permission on the server to successfully load the data. That's quite strange. And I am amazed. The performance of the page is now like day and night. Previously the system was quite slow, it took a few seconds to load any page, while now it is showing pages almost immediately. Thanks again for the very quick and perfect support.

[meeb]

No problem. SQLite works really well... until the database gets quite busy then you get write locks and it starts to trudge a bit. MySQL can have issues due to MySQL not handling very long running database connections very well, or at all, so if after a day you get "MySQL has gone away" errors please open an issue with the details.