Open pghpete opened 1 year ago
I do see that the bug report lists youtube-dl but I've been using yt-dlp instead as its far more advanced and less buggy. Does't ytcc now use yt-dlp?
Ytcc uses yt-dlp over youtube-dl, if both are found on your system. The error is probably related to something else. It seems there is a problem in how ytcc or yt-dlp calculate unique identifiers for some videos. Somehow it calculates the same id for two different URLs. I cannot reproduce the issue though. To reproduce, I'd need a list of subscriptions that trigger the error. Did you add new subscriptions recently (especially something not from youtube) that cause the issue?
hum, not that I know of. I did remove the very last sub I performed and re-ran it but it produced the same issue. I then removed the last sub before the error and that didn't help. I ended exporting my existing subscriptions, moving the DB out of the way, and resubscribing to all via my exported list.
I wasn't able to find out what specifically was the issue so I ended up using a heavy hand and moving the DB out of the way and I still ended up having the same error happen.
I have attached a text file of the subscriptions where this is happening.
Please let me know if there is anything I can do to assist with tracking down what might be the issue here.
I can reproduce the issue with an empty DB and adding only one subscription:
$ ytcc bug-report
---ytcc version---
2.6.1
---youtube-dl version---
youtube-dl not found
---yt-dlp version---
2023.07.06
---Click version---
8.1.6
---SQLite version---
SQLite system library version: 3.42.0
Python module version: 2.6.0
---python version---
3.11.3 (main, Jun 5 2023, 09:32:32) [GCC 13.1.1 20230429]
---mpv version---
mpv is not installed
---config dump---
[ytcc]
download_dir = /mnt/stufz/Videos/Youtube
download_subdirs = true
mpv_flags = --ytdl --ytdl-format=bestvideo[height<=?1080]+bestaudio/best
order_by = playlists:asc, publish_date:desc
video_attrs = id, title, publish_date, duration, playlists
playlist_attrs = name, url, reverse, tags
db_path = ~/.local/share/ytcc/ytcc.db
date_format = %Y-%m-%d
max_update_fail = 5
max_update_backlog = 20
age_limit = 1000
[tui]
alphabet = sdfervghnuiojkl
default_action = download_video
[theme]
prompt_download_audio = 2
prompt_download_video = 4
prompt_play_audio = 2
prompt_play_video = 4
prompt_mark_watched = 1
table_alternate_background = 245
plain_label_text = 244
[youtube_dl]
format = (bestvideo[height<=?1080][vcodec^=avc1]+bestaudio/best)/bestvideo[height<=?1080]+bestaudio/best
output_template = %(upload_date)s - %(title)s.%(ext)s
ratelimit = 0
retries = 0
subtitles = en, fr
thumbnail = true
skip_live_stream = true
merge_output_format = mp4
max_duration = 9000
restrict_filenames = false
[storm@velona ~]$ rm ~/.local/share/ytcc/ytcc.db
[storm@velona ~]$ ytcc -l debug subscribe "HappilyEverHanks" "https://www.youtube.com/@HappilyEverHanks" [1692413350.220437] [MainProcess/MainThread] ytcc.config.DEBUG: Trying to read config from following locations: [PosixPath('/etc/ytcc/ytcc.conf'), PosixPath('/home/storm/.config/ytcc/ytcc.conf'), PosixPath('/home/storm/.ytcc.conf')][1692413350.222454] [MainProcess/MainThread] ytcc.config.DEBUG: Config was read from following locations: ['/home/storm/.config/ytcc/ytcc.conf'][1692413353.930687] [MainProcess/MainThread] ytcc.core.INFO: Performing update check on 10 playlist items[1692413353.931664] [MainProcess/MainThread] asyncio.DEBUG: Using selector: EpollSelector
[1692413354.171524] [MainProcess/MainThread] ytcc.updater.INFO: Checking playlist 'HappilyEverHanks'...
[1692413360.430457] [MainProcess/MainThread] ytcc.updater.WARNING: Publication date of video 'HappilyEverHanks - Live' of playlist 'HappilyEverHanks' is unknown
[1692413360.430896] [MainProcess/MainThread] ytcc.updater.WARNING: Duration of video 'HappilyEverHanks - Live' of playlist 'HappilyEverHanks' is unknown
[1692413360.431155] [MainProcess/MainThread] ytcc.updater.INFO: Processed video 'HappilyEverHanks - Live' of playlist 'HappilyEverHanks'
[1692413394.533273] [MainProcess/MainThread] ytcc.updater.WARNING: Publication date of video 'HappilyEverHanks - Videos' of playlist 'HappilyEverHanks' is unknown
[1692413394.533809] [MainProcess/MainThread] ytcc.updater.WARNING: Duration of video 'HappilyEverHanks - Videos' of playlist 'HappilyEverHanks' is unknown
[1692413394.534087] [MainProcess/MainThread] ytcc.updater.INFO: Processed video 'HappilyEverHanks - Videos' of playlist 'HappilyEverHanks'
[1692413394.573304] [MainProcess/MainThread] ytcc.database.DEBUG: PRAGMA foreign_keys = ON;
[1692413394.574256] [MainProcess/MainThread] ytcc.database.DEBUG: CREATE TABLE tag ( name VARCHAR NOT NULL, playlist INTEGER REFERENCES playlist (id) ON DELETE CASCADE, CONSTRAINT tagKey PRIMARY KEY (name, playlist) );
[1692413394.614747] [MainProcess/MainThread] ytcc.database.DEBUG: CREATE TABLE playlist ( id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, name VARCHAR UNIQUE, url VARCHAR UNIQUE, reverse BOOLEAN );
[1692413394.672630] [MainProcess/MainThread] ytcc.database.DEBUG: CREATE TABLE content ( playlist_id INTEGER NOT NULL REFERENCES playlist (id) ON DELETE CASCADE, video_id INTEGER NOT NULL REFERENCES video (id) ON DELETE CASCADE, CONSTRAINT contentKey PRIMARY KEY (playlist_id, video_i
d) );
[1692413394.724542] [MainProcess/MainThread] ytcc.database.DEBUG: CREATE TABLE video ( id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, title VARCHAR NOT NULL, url VARCHAR UNIQUE NOT NULL, description VARCHAR, duration FLOAT, publish_date FLOAT, watch_date FLOAT, thumbnail_url VARCHAR,
extractor_hash VARCHAR UNIQUE );
[1692413394.755450] [MainProcess/MainThread] ytcc.database.DEBUG: CREATE TABLE extractor_meta ( extractor_hash VARCHAR PRIMARY KEY, failure_count INTEGER );
[1692413394.807044] [MainProcess/MainThread] ytcc.database.DEBUG: PRAGMA user_version = 5;
[1692413394.830349] [MainProcess/MainThread] ytcc.database.DEBUG: PRAGMA user_version;
[1692413394.830911] [MainProcess/MainThread] ytcc.database.DEBUG: BEGIN
[1692413394.831041] [MainProcess/MainThread] ytcc.database.DEBUG: INSERT INTO playlist (name, url, reverse) VALUES ('HappilyEverHanks', 'https://www.youtube.com/@HappilyEverHanks', 0);
[1692413394.831681] [MainProcess/MainThread] ytcc.database.DEBUG: COMMIT
[storm@velona ~]$ ytcc -l debug update
[1692413400.438851] [MainProcess/MainThread] ytcc.config.DEBUG: Trying to read config from following locations: [PosixPath('/etc/ytcc/ytcc.conf'), PosixPath('/home/storm/.config/ytcc/ytcc.conf'), PosixPath('/home/storm/.ytcc.conf')]
[1692413400.440528] [MainProcess/MainThread] ytcc.config.DEBUG: Config was read from following locations: ['/home/storm/.config/ytcc/ytcc.conf']
[1692413400.444888] [MainProcess/MainThread] ytcc.database.DEBUG: PRAGMA foreign_keys = ON;
[1692413400.445291] [MainProcess/MainThread] ytcc.database.DEBUG: PRAGMA user_version;
[1692413400.445850] [MainProcess/MainThread] asyncio.DEBUG: Using selector: EpollSelector
[1692413400.447014] [MainProcess/MainThread] ytcc.database.DEBUG: SELECT p.id AS id, p.name AS name, p.url AS url, p.reverse AS reverse, t.name AS tag FROM playlist AS p LEFT OUTER JOIN tag AS t ON p.id = t.playlist;
[1692413400.452170] [MainProcess/MainThread] ytcc.database.DEBUG: WITH ids as ( SELECT v.id FROM video AS v JOIN content c ON v.id = c.video_id JOIN playlist p ON p.id = c.playlist_id LEFT JOIN tag AS t ON p.id = t.playlist WHERE v.publish_date > 0 AND v.publish_date < Inf AND p.name
IN ('HappilyEverHanks') ) SELECT v.id AS id, v.title AS title, v.url AS url, v.description AS description, v.duration AS duration, v.publish_date AS publish_date, v.watch_date AS watch_date, v.thumbnail_url AS thumbnail_url, v.extractor_hash AS extractor_hash, p.name AS playlist_nam
e, p.url AS playlist_url, p.reverse AS playlist_reverse FROM video AS v JOIN content c ON v.id = c.video_id JOIN playlist p ON p.id = c.playlist_id WHERE v.id in ids
[1692413401.578072] [MainProcess/MainThread] ytcc.updater.INFO: Checking playlist 'HappilyEverHanks'...
[1692413403.952282] [MainProcess/MainThread] ytcc.database.DEBUG: SELECT failure_count FROM extractor_meta WHERE extractor_hash = 'youtubetab UChEzR-bdxoEpWdDLVC9wWew'
[1692413403.952661] [MainProcess/MainThread] ytcc.database.DEBUG: SELECT failure_count FROM extractor_meta WHERE extractor_hash = 'youtubetab UChEzR-bdxoEpWdDLVC9wWew'
[1692413409.173781] [MainProcess/MainThread] ytcc.updater.WARNING: Publication date of video 'HappilyEverHanks - Live' of playlist 'HappilyEverHanks' is unknown
[1692413409.173978] [MainProcess/MainThread] ytcc.updater.WARNING: Duration of video 'HappilyEverHanks - Live' of playlist 'HappilyEverHanks' is unknown
[1692413409.174135] [MainProcess/MainThread] ytcc.updater.INFO: Processed video 'HappilyEverHanks - Live' of playlist 'HappilyEverHanks'
[1692413479.361800] [MainProcess/MainThread] ytcc.updater.WARNING: Publication date of video 'HappilyEverHanks - Videos' of playlist 'HappilyEverHanks' is unknown
[1692413479.362046] [MainProcess/MainThread] ytcc.updater.WARNING: Duration of video 'HappilyEverHanks - Videos' of playlist 'HappilyEverHanks' is unknown
[1692413479.362200] [MainProcess/MainThread] ytcc.updater.INFO: Processed video 'HappilyEverHanks - Videos' of playlist 'HappilyEverHanks'
[1692413479.362614] [MainProcess/MainThread] ytcc.database.DEBUG: SELECT id FROM playlist WHERE name = 'HappilyEverHanks'
[1692413479.363093] [MainProcess/MainThread] ytcc.database.DEBUG: BEGIN
[1692413479.363455] [MainProcess/MainThread] ytcc.database.DEBUG: INSERT INTO video (title, url, description, duration, publish_date, watch_date, thumbnail_url, extractor_hash) VALUES ('HappilyEverHanks - Videos', 'https://www.youtube.com/@HappilyEverHanks/videos', 'WELCOME to HappilyEverHanks! We are Kyle (Carl), Renee (Ney Ney), & Dexter (Sexy Dexy) 😸 In 2017 we started traveling as ER nurses and soon realized that life had bigger plans for us. We have been living on the road full-time for over 5 years now and WOW...what a journey it has been! We are here to show you that life can be anything that you make it. We keep it real and fun every Thursday, and we encourage you to subscribe along for the journey! #ThanksHanks Learn more about our craziness below! ⬇️ ', -1, 169201.0, NULL, 'https://yt3.googleusercontent.com/8D4cgAbhYS-VsJ9IhejI2b7VA6pA2WOLnQ7UWrkgmSaDgaYrx7s_-TunpTHN9lwrjg3bH9jBMA=w2120-fcrop64=1,00000000ffffffff-k-c0xffffffff-no-nd-rj', 'youtubetab UChEzR-bdxoEpWdDLVC9wWew') ON CONFLICT (url) DO UPDATE SET title = 'HappilyEverHanks - Videos', url = 'https://www.youtube.com/@HappilyEverHanks/videos', description = 'WELCOME to HappilyEverHanks! We are Kyle (Carl), Renee (Ney Ney), & Dexter (Sexy Dexy) 😸 In 2017 we started traveling as ER nurses and soon realized that life had bigger plans for us. We have been living on the road full-time for over 5 years now and WOW...what a journey it has been! We are here to show you that life can be anything that you make it. We keep it real and fun every Thursday, and we encourage you to subscribe along for the journey! #ThanksHanks Learn more about our craziness below! ⬇️ ', duration = -1, publish_date = 169201.0, thumbnail_url = 'https://yt3.googleusercontent.com/8D4cgAbhYS-VsJ9IhejI2b7VA6pA2WOLnQ7UWrkgmSaDgaYrx7s_-TunpTHN9lwrjg3bH9jBMA=w2120-fcrop64=1,00000000ffffffff-k-c0xffffffff-no-nd-rj', extractor_hash = 'youtubetab UChEzR-bdxoEpWdDLVC9wWew'
[1692413479.364290] [MainProcess/MainThread] ytcc.database.DEBUG: INSERT OR IGNORE INTO content (playlist_id, video_id) VALUES (1,(SELECT id FROM video WHERE url = 'https://www.youtube.com/@HappilyEverHanks/videos'));
[1692413479.364528] [MainProcess/MainThread] ytcc.database.DEBUG: COMMIT
[1692413479.416085] [MainProcess/MainThread] ytcc.database.DEBUG: SELECT id FROM playlist WHERE name = 'HappilyEverHanks'
[1692413479.416528] [MainProcess/MainThread] ytcc.database.DEBUG: BEGIN
[1692413479.416800] [MainProcess/MainThread] ytcc.database.DEBUG: INSERT INTO video (title, url, description, duration, publish_date, watch_date, thumbnail_url, extractor_hash) VALUES ('HappilyEverHanks - Live', 'https://www.youtube.com/@HappilyEverHanks/streams', 'WELCOME to HappilyEverHanks! We are Kyle (Carl), Renee (Ney Ney), & Dexter (Sexy Dexy) 😸 In 2017 we started traveling as ER nurses and soon realized that life had bigger plans for us. We have been living on the road full-time for over 5 years now and WOW...what a journey it has been! We are here to show you that life can be anything that you make it. We keep it real and fun every Thursday, and we encourage you to subscribe along for the journey! #ThanksHanks Learn more about our craziness below! ⬇️ ', -1, 169201.0, NULL, 'https://yt3.googleusercontent.com/8D4cgAbhYS-VsJ9IhejI2b7VA6pA2WOLnQ7UWrkgmSaDgaYrx7s_-TunpTHN9lwrjg3bH9jBMA=w2120-fcrop64=1,00000000ffffffff-k-c0xffffffff-no-nd-rj', 'youtubetab UChEzR-bdxoEpWdDLVC9wWew') ON CONFLICT (url) DO UPDATE SET title = 'HappilyEverHanks - Live', url = 'https://www.youtube.com/@HappilyEverHanks/streams', description = 'WELCOME to HappilyEverHanks! We are Kyle (Carl), Renee (Ney Ney), & Dexter (Sexy Dexy) 😸 In 2017 we started traveling as ER nurses and soon realized that life had bigger plans for us. We have been living on the road full-time for over 5 years now and WOW...what a journey it has been! We are here to show you that life can be anything that you make it. We keep it real and fun every Thursday, and we encourage you to subscribe along for the journey! #ThanksHanks Learn more about our craziness below! ⬇️ ', duration = -1, publish_date = 169201.0, thumbnail_url = 'https://yt3.googleusercontent.com/8D4cgAbhYS-VsJ9IhejI2b7VA6pA2WOLnQ7UWrkgmSaDgaYrx7s_-TunpTHN9lwrjg3bH9jBMA=w2120-fcrop64=1,00000000ffffffff-k-c0xffffffff-no-nd-rj', extractor_hash = 'youtubetab UChEzR-bdxoEpWdDLVC9wWew'
[1692413479.417094] [MainProcess/MainThread] ytcc.database.DEBUG: ROLLBACK
[1692413479.422798] [MainProcess/MainThread] ytcc.cli.ERROR: Cannot connect to the database or query failed unexpectedly
[1692413479.423356] [MainProcess/MainThread] ytcc.cli.DEBUG: Unknown database error
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/ytcc/cli.py", line 742, in main
exit_code = cli.main(standalone_mode=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/click/decorators.py", line 92, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/ytcc/cli.py", line 357, in update
ytcc.update(max_fail, max_backlog)
File "/usr/lib/python3.11/site-packages/ytcc/core.py", line 152, in update
updater.update()
File "/usr/lib/python3.11/site-packages/ytcc/updater.py", line 246, in update
asyncio.run(self.do_update())
File "/usr/lib/python3.11/asyncio/runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/ytcc/updater.py", line 243, in do_update
await asyncio.gather(*map(self.update_playlist, playlists))
File "/usr/lib/python3.11/site-packages/ytcc/updater.py", line 237, in update_playlist
self.database.add_videos([video], playlist)
File "/usr/lib/python3.11/site-packages/ytcc/database.py", line 349, in add_videos
cursor.execute(insert_video, asdict(video))
sqlite3.IntegrityError: UNIQUE constraint failed: video.extractor_hash
Note that the issue still occurs even when using the channel id (UChEzR-bdxoEpWdDLVC9wWew
) instead of the @
version.
Issue does not occur with eg. this channel: https://www.youtube.com/channel/UCSqUTKab5R1I4B5frDEjXzA
The only notable difference I can see with a quick glance is that the channel causing the bug has the "Live" tab while the second one has not.
I've dig a bit in the code and seems to be indeed linked to a difference in what is returned by yt-dlp for the 2 channels:
The info
structure retrieved in https://github.com/woefe/ytcc/blob/master/ytcc/updater.py#L82 contains an entries
field that is a generator that returns object of _type
set to url
and whose url
field is indeed the video.
The entries
field of the info
structure contains 2 objects of type playlist
, one for the "Videos" tab and one for the "Live" tab. Obviously, the url
field for these only points to the channel page's tab, not to any video. These objects then follow the same pattern as for the other channel, ie. the entries
field is a generator that does return a list of objects with the url
corresponding to actual video url.
There is an easy fix: update the subscription to use the playlist/link to the "videos" tab. Unfortunately, for some users that means having to update multiple subscriptions and there is currently no way of being notified that this even needs to be done.
I've tried to do a small workaround in #118 so it's possible to support channels that were already added to ytcc and that enabled the new features at some points, making them use "tabs".
Let me know what you think!
ytcc version 2.6.1 is failing its update process with an error message "Cannot connect to the database or query failed unexpectedly"
Running ytcc with a log level of debug...
bug report info...