Closed vxbinaca closed 4 years ago
vxbinaca
commented
4 years ago @Coloradohusky @brandongalbraith
I found a clue: Delete the failed uploads .info.json files and the videos upload.
So the problem is in the metadata for sure. The re-rip appears to grab the metadata again and the handoff between youtube-dl and internetarchive doesn't bug out.
vxbinaca
commented
4 years ago @antonizoon I really hate to tag you in this but this is beyond my technical capabilities to fix.
Some element in metadata parsing is killing Tubeup rips with increasing frequency as Youtube rolls out a system wide change. This will only increase and the tool will become useless as it fails to handle this new element, until no videos made after the last few months work anymore.
vxbinaca
commented
4 years ago So lets tally where we're at:
Tubeup is decomposing. It is beyond my abilities to fix. It needs TLC and careful attention. I can manage the flags for YTDL and IA, but thats it.
As time goes on my ability (and others) to mirror videos to Archive.org will decrease until there is (almost) none, and then thats it.
It's been a nice ride, boys. I think it's over finally though.
Coloradohusky
commented
4 years ago Everything works fine for me (.info.json and all), but that sucks that things are breaking. And yeah, YTDL is a huge mess lol, over 600 pull requests? That's insane
vxbinaca
commented
4 years ago I finally managed to get that video to upload but I'm unsure why it's failing in the first place but others are failing and it's becoming an administrative nightmare.
antonizoon
commented
4 years ago Looking at the code for that specific video at least, the problem seems very simple to me.
https://github.com/bibanon/tubeup/blob/master/tubeup/TubeUp.py#L462
In this element I tried to avoid the situation of iterating a nonexistent tags array by checking if the tags key exists. However, I suppose tags = None in some recent cases due to changes and we have to handle that case properly (whereby it accidentally tries to iterate over a NoneType, thus the exception).
Regarding the rest of the metadata function, the entire metadata function can be designed to be more resilient to metadata failures and if they occur just leave those areas blank.
We can also set up a final failsafe by doing a try except Exception: block which prints the error, but goes on without halting the script on malformed videos. This way users could accumulate errors in maybe a log file to take manual action (such as manually populating the video metadata to the item), without standing in the way of further downloads and uploads.
@brandongalbraith I can describe to you better ways of designing the script now that I have achieved much more skill than I had initially been able to apply to this script. We need to all discuss this weekend or something.
vxbinaca
commented
4 years ago videos that didn't work eventually sometimes work, and it may have to do with youtube auto generating tags.
Resiliancy is a good idea. If tags aren't set and IA needs them, let's just dump in the siteID and "video" and move the hell on and quash these rip killing bugs.
vxbinaca
commented
4 years ago Yup, we found it.
https://www.youtube.com/watch?v=g7atkk3UseA
"tags": null
@antonizoon if you could get a fix in for this I'll push it up. Fixing this will fix a ton of sites that are broken right now like Periscope. If no tags are found just dump in the siteID and "video" because IA requires two tags, and we can squash this bug.
Edit 2:
Removing the null tags from the file allows uploads again. This is for sure the probem.
vxbinaca
commented
4 years ago @antonizoon @brandongalbraith Please someone do a fix a majority of the videos I rip stay on the hard disk and don't get uploaded. 86 / 96 videos. One Null tags video breaks a chain of good videos, and it's effecticly broken Tubeup.
I wish I didn't have a shitty low skill job that demanded 10 hours of my day so I could learn python and fix theis myself but I can't.
I'm getting ready to quit archival because the bugs are winning at preventing history from being saved. Right now I'm having to use VIM to hand edit JSON on my VPS to clear out the logjam of failed uploads.
brandongalbraith
commented
4 years ago @vxbinaca Currently traveling far from civilization but will attempt to get a hot fix for this in tomorrow. Sorry you’ve been impacted for as long as you have
cc @antonizoon
Coloradohusky
commented
4 years ago Submitted a pull request, just a quick attempt to fix it, not exactly sure if it'll work or not cause I haven't tested it yet
vxbinaca
commented
4 years ago @vxbinaca Currently traveling far from civilization but will attempt to get a hot fix for this in tomorrow. Sorry you’ve been impacted for as long as you have
cc @antonizoon
With the state of things in the world you made a good choice, traveling far from civilization.
Submitted a pull request, just a quick attempt to fix it, not exactly sure if it'll work or not cause I haven't tested it yet
Yeah it's syntax doesn't work. Try to submit PRs for code you've tested from now on.
vxbinaca
commented
4 years ago Fixed! Thanks again Brandon.
This bug has returned and the clue is in this jungle of output:
tubeup --metadata=collection:tubeup TL470fJMi7w --debugOriginally posted by @vxbinaca in https://github.com/bibanon/tubeup/issues/114#issuecomment-645684350