search

bibanon / tubeup

Use yt-dlp to download video/metadata and upload to the Internet Archive.
https://pypi.python.org/pypi/tubeup/
GNU General Public License v3.0
417 stars 70 forks source link

Regression: Uploads with null description fail #154

Closed puigru closed 3 years ago

puigru commented 3 years ago

Uploading items with a null description causes TubeUp to crash since #141 This seems to affect Twitch VODs in particular

[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.02.04.1
[debug] Python version 3.7.3 (CPython) - Linux-4.19.75-v7+-armv7l-with-debian-10.1
[debug] exe versions: ffmpeg 4.1.6-1, ffprobe 4.1.6-1
[debug] Default format spec: bestvideo+bestaudio/best

An exception just occured, if you found this exception isn't related with any of your connection problem, please report this issue to https://github.com/bibanon/tubeup/issues
Traceback (most recent call last):
  File "/home/joel/.local/lib/python3.7/site-packages/tubeup/__main__.py", line 96, in main
    use_download_archive):
  File "/home/joel/.local/lib/python3.7/site-packages/tubeup/TubeUp.py", line 370, in archive_urls
    identifier, meta = self.upload_ia(basename, custom_meta)
  File "/home/joel/.local/lib/python3.7/site-packages/tubeup/TubeUp.py", line 294, in upload_ia
    vid_meta)
  File "/home/joel/.local/lib/python3.7/site-packages/tubeup/TubeUp.py", line 475, in create_archive_org_metadata_from_youtubedl_meta
    description_text = re.sub('\r?\n', '<br>', description_text)
  File "/usr/lib/python3.7/re.py", line 192, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or bytes-like object
vxbinaca commented 3 years ago

does this affect the JSON? IE can I ditch the empty. description file and fix this or is it some other part of the program that's choking?

vxbinaca commented 3 years ago

Twitch videos are still broken, you have to find what didn't upload and manually re-upload each file, just like before.

puigru commented 3 years ago

Twitch videos are still broken, you have to find what didn't upload and manually re-upload each file, just like before.

Could you provide an example? I uploaded a VOD earlier which used to crash before #155 without issue.

brandongalbraith commented 3 years ago

@vxbinaca Have you hit this issue again with Twitch? If so, do you have a URL I can run through the code path involved? Going to try to knock this out if it's ongoing, or close if it seems resolved.

vxbinaca commented 3 years ago

@vxbinaca Have you hit this issue again with Twitch? If so, do you have a URL I can run through the code path involved? Going to try to knock this out if it's ongoing, or close if it seems resolved.

I've had to make scripts that get failed uploads, and then use VIM magic to list out Twitch VideoIDs and then insert commands into a batch file to then upload the failed rips on disk.

Pick any Twitch URL, Most do not work.

vxbinaca commented 3 years ago

I could have sworn we have safety checks in place for null metadata of certain types. I feel like I've seen something like this before.

puigru commented 3 years ago

I'm going to close this given that the particular issue reported here was patched and I haven't run into it since. If a similar issue exists it might be better to open a new issue with an updated stacktrace.

brandongalbraith commented 3 years ago

Sounds good @XXLuigiMario, feel free to open a new issue with a fresh stacktrace if you encounter again (likewise @vxbinaca if you're running into Twitch videos it errs out on).

vxbinaca commented 3 years ago

I'm re-opening because I've had to set up a semi-automatic method to save failed Twitch rips.

vxbinaca commented 3 years ago

Closing again because switching to YT-DLP has fixed this issue and now all Twitch rips upload as they should.

brandongalbraith commented 3 years ago

@vxbinaca This is an interesting observation! This means that there is a material difference between how ytdl and ytdlp rip Twitch content. Glad to hear the library swap worked!

vxbinaca commented 3 years ago

@vxbinaca This is an interesting observation! This means that there is a material difference between how ytdl and ytdlp rip Twitch content. Glad to hear the library swap worked!

I ran a test rip this morning and everything that could be ripped was uploaded. Theres still errors in getting some videos but Twitch streams fully processed and uploaded are no longer sitting on disk. I'm doing a test run of youtube right now, Vimeo works too it's solid. Haven't tried other extractors.

If you join YT-DLPs discord they're actually responsive and will merge PRs quickly if they're good.