Error: "warning: s3 is overloaded, sleeping for 30 seconds and retrying. 9001 retries left."

[prof-frink]

I keep getting the following error whenever I try to upload videos by the Youtuber Shirrako (and only him):

prof_frink@DESKTOP-1PS5GGT:~$ tubeup https://www.youtube.com/watch?v=oTcwl232id8
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.11.07
[debug] Python version 3.6.6 (CPython) - Linux-4.4.0-17134-Microsoft-x86_64-with-Ubuntu-18.04-bionic
[debug] exe versions: ffmpeg 3.4.4-0ubuntu0.18.04.1, ffprobe 3.4.4-0ubuntu0.18.04.1
[debug] Proxy map: {}
video doesn't have subtitles
 uploading Red_Dead_Redemption_2_-_Bringing_Black_Man_To_KKK-oTcwl232id8.annotations.xml: [                              uploading Red_Dead_Redemption_2_-_Bringing_Black_Man_To_KKK-oTcwl232id8.annotations.xml: [############################# uploading Red_Dead_Redemption_2_-_Bringing_Black_Man_To_KKK-oTcwl232id8.annotations.xml: [################################] 1/1 - 00:00:00
 warning: s3 is overloaded, sleeping for 30 seconds and retrying. 9001 retries left.
 uploading Red_Dead_Redemption_2_-_Bringing_Black_Man_To_KKK-oTcwl232id8.annotations.xml: [                              uploading Red_Dead_Redemption_2_-_Bringing_Black_Man_To_KKK-oTcwl232id8.annotations.xml: [############################# uploading Red_Dead_Redemption_2_-_Bringing_Black_Man_To_KKK-oTcwl232id8.annotations.xml: [################################] 1/1 - 00:00:00
 warning: s3 is overloaded, sleeping for 30 seconds and retrying. 9001 retries left.
 uploading Red_Dead_Redemption_2_-_Bringing_Black_Man_To_KKK-oTcwl232id8.annotations.xml: [                              uploading Red_Dead_Redemption_2_-_Bringing_Black_Man_To_KKK-oTcwl232id8.annotations.xml: [############################# uploading Red_Dead_Redemption_2_-_Bringing_Black_Man_To_KKK-oTcwl232id8.annotations.xml: [################################] 1/1 - 00:00:00
 warning: s3 is overloaded, sleeping for 30 seconds and retrying. 9001 retries left.
 uploading Red_Dead_Redemption_2_-_Bringing_Black_Man_To_KKK-oTcwl232id8.annotations.xml: [                              uploading Red_Dead_Redemption_2_-_Bringing_Black_Man_To_KKK-oTcwl232id8.annotations.xml: [############################# uploading Red_Dead_Redemption_2_-_Bringing_Black_Man_To_KKK-oTcwl232id8.annotations.xml: [################################] 1/1 - 00:00:00
 warning: s3 is overloaded, sleeping for 30 seconds and retrying. 9001 retries left.
 uploading Red_Dead_Redemption_2_-_Bringing_Black_Man_To_KKK-oTcwl232id8.annotations.xml: [                              uploading Red_Dead_Redemption_2_-_Bringing_Black_Man_To_KKK-oTcwl232id8.annotations.xml: [############################# uploading Red_Dead_Redemption_2_-_Bringing_Black_Man_To_KKK-oTcwl232id8.annotations.xml: [################################] 1/1 - 00:00:00
 warning: s3 is overloaded, sleeping for 30 seconds and retrying. 9001 retries left.
 uploading Red_Dead_Redemption_2_-_Bringing_Black_Man_To_KKK-oTcwl232id8.annotations.xml: [                              uploading Red_Dead_Redemption_2_-_Bringing_Black_Man_To_KKK-oTcwl232id8.annotations.xml: [############################# uploading Red_Dead_Redemption_2_-_Bringing_Black_Man_To_KKK-oTcwl232id8.annotations.xml: [################################] 1/1 - 00:00:00
 warning: s3 is overloaded, sleeping for 30 seconds and retrying. 9001 retries left.

It's weird, because all other videos upload just fine, and two of his videos were already posted to Mirrortube:

https://archive.org/details/youtube-MPYAM9AfRHo https://archive.org/details/youtube-mZ7RicfseRU

Also, I'm new to this, so I'm wondering: is there something else I should be doing other than just 'tubeup \<url>'? I can't add to any collections, but I noticed Sketch the Cow is adding them to Mirrortube. Just wondering if this is the proper procedure, or if I'm creating unnecessary work for someone.

As well, I noticed that the download page mentioned not uploading entire channels. Is this an absolute, or does it depend on how many videos are in the channel?

Thanks!

[vxbinaca]

• Use Github markdown to properly render the command that's hard to read, IE wrap it in "`"
• You didn't read the front page or #37 :

Upload taking forever? Getting s3 throttling on upload? Tubeup has specifically been tailored to wait the longest possible time before failing, and we've never seen a S3 outage that outlasted the insane wait times set in Tubeup.

I should update that because there was exactly one and it lasted 2 days. That's an incredible amount of time. All that's happening is Tubeup just sits there and waits in like patiently to upload to the item. It's not failing until it returns you to a command prompt. We - I - saw this years ago and fixed for it.

• Sketch had a bot on it that moves your items based on certain tags to keep them out of opensource_)movies because me and a few others shoved about 100,000 videos into it and it was causing a huge mess even with my attempts to clean it up.

• IA staff doesn't like us filling their service with stuff that's not at risk of loss. Please don't rock the boat there. Is what you're ripping really in danger? Yes I know about that guy and the suffragettes video.

Closing because this isn't a bug it's a safety feature working as intended.

[prof-frink]

• Sorry, I'm not technically-inclined, and I'm new to Github.
• I did read both, but I didn't think they applied in my case because (1) the issue only seemed to happen in my case with certain videos, not all of them (i.e., most uploaded almost instantly, but only these were having issues), and (2) in her issue, the program didn't appear to be even trying to upload anything, so I wasn't sure if it was the same problem (again, I'm not technically-inclined).
• I see, so it's a bot. In other words, I'm not causing him more work (at least I hope not!).
• One channel I'm looking at, Know More News, is currently the target of a mass-flagging campaign. He has 242 videos (only one of which has been archived).

[vxbinaca]

Theres a ton of people (and bots) - thousands - who upload to IA constantly. Sometimes theres brief S3 overloads because of the amount of incoming dta or a hard drive needs to be changed or some big operation is happening at IA. Tubeup has been configured to wait for these to clear up - figure way longer than most S3 outages figure the max Tubeup will wait is 36 hours for ONE video before failing.

No your case isn't special. I'm really not trying to be rude either. Yes Sketch aka Jason Scotts alt account is a bot. You don't upload 127,000 videos and have a life without automation. You also don't do that much without running into bugs and given I test Tubeup daily and I have crontabs that cycle scripts for me 24/7 I will hit a bug first 99 percent of the time. I also from time to time find bugs in other place like youtube-dl and internetarchive.

When Tubeup returns to command prompt, that's when it failed or succeeded - in most cases including this one.

[brandongalbraith]

@vxbinaca Whatcha think about extending the S3 timeout notice in tubeup to say "THIS IS EXPECTED BEHAVIOR" or something of the sort? Might save an issue or two from being created in the future.

[prof-frink]

Heh, that might not be a bad idea for dum-dums like myself 😅

Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Sunday, November 11, 2018 5:12 PM, Brandon Galbraith notifications@github.com wrote:

@vxbinaca Whatcha think about extending the S3 timeout notice in tubeup to say "THIS IS EXPECTED BEHAVIOR" or something of the sort? Might save an issue or two from being created in the future.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[vxbinaca]

It's already on the front page, isn't an issue for most people and complicates the code. After major bugs are fixed for the last time maybe. Not it's not needed right now.

[prof-frink]

Yeah, true enough. I just gotta learn patience. I'm uploading a video right now—if it takes more than two days to finish, I'll let you know.

p-f

Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Sunday, November 11, 2018 5:17 PM, Paul Henning notifications@github.com wrote:

It's already on the front page, isn't an issue for most people and complicates the code. After major bugs are fixed for the last time maybe. Not it's not needed right now.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[vxbinaca]

It will fail if it takes 2 days. Likely whatever log jam is happening right now will clear within the next few hours

[prof-frink]

Understood. Thanks!

[prof-frink]

Just an addendum: it did finally upload the video in question, although it took around 24 hours to do so. Now it seems to be doing this for just about every video I try to upload.

Not trying to be offensive, but I'm beginning to question whether this is an efficient tool for preserving verboten material if it takes an entire day to upload a single video, whereas the HTML uploader/regular Python uploader works instantly. It could take an entire year just to preserve one channel. I'm not a programmer, but basic logic tells me that if a program were trying to upload to a server, it makes no sense that one video would upload instantly, whereas another video (perhaps even smaller in size) should struggle to upload.

Could it be that certain items/identifiers are being pegged for certain servers, and the wait time depends on how busy that particular server is? Am I far off in my theory?

[vxbinaca]

The HTML interface is used by people who don't care about metadata, and the metadata matters just as much as the video. I just uploaded 5 channels today in a collection, and a 28 gigabyte stream. They all uploaded without S3 timeouts within 10 minutes in the case of the large stream.

It doesn't have this type of problem normally it goes very fast especially on the VPS I use.

@jjjake he's claiming theres insane S3 timeouts and he's having issues uploading video in a timely manner. Is something going on at IA? I checked Twitter and there isn't any recent reported outages or problems.

[prof-frink]

Yeah, a video I tried to upload overnight took 12 hours and was still going when I terminated it.

FWIW, I'm using Ubuntu for Windows downloaded from the Windows Store. Not sure if that has any bearing. Please let me know if you need any more information/want me to try something.

[vxbinaca]

Just leave it and let it fail. If it fails email info@archive.org, tell them what item is S3 erroring. I'm not the internet archive, this isn't a problem experienced by anyone else here and they'd be able to help you better than me and that email will get it fixed faster.

It's not a problem with Tubeup it's on IAs end.

[vxbinaca]

I reviewed the uploads you mentioned and they look normal. They're both muxed as Matrakoska and were derived properly so they uploaded.

Contact IA staff from your IA accounts email and ask them about the S3 problems.

If you have more problems with Tubeup don't hesitate to make an issue I'll either fix something or close with an explanation. I try to be helpful with users even if I come across as short.

[vxbinaca]

My last reply for a while unless @prof-frink needs more help:

• You said something about a collection. You need to email staff about creating one. This FAQ will tell you everything you need to know. Collections give you admin powers over your collection. Do not abuse your powers.

• I noticed one of your uploads was Sargon Of Akkad. In short you don't need to do that because it's already taken care of automatically every hour. Consider looking at my other collections and keeping it in mind that I automatically track all those subjects, just so you don't make another issue complaining about write errors when you try to back up a Louis La Vau video.

• Archival is a responsibility and you should continue to back up that channel until it dies. Mister Metokur and Lauren Southern were a mess I had to clean up from others getting bits and pieces. Now they're tracked by me and are complete(ish in MMs case).

[prof-frink]

Understood. I'll do as you suggested.

• I actually have two accounts, one that's already an Archivist account, @the_professor, and the account I was referring to, @christ_chan64. I prefer to keep them separate for personal reasons. I have no desire to create a collection in my second account at the present time—I was just inquiring how to go about uploading to the archiveteam_youtube collection. I'm a librarian by profession, and take my role as a neutral (as much as possible) archivist of mankind's records very seriously. (Well, I've doxxed myself enough for one day.)

• Hmm, don't see any videos by him in my account, unless I'm mistaken. But yeah, I generally try not to step on someone else's toes, unless it's obvious that their collecting has been abandoned (for many years in some cases).

• Oh, I know. Got a couple of channels I'm trying to keep on top of, while keeping the Lichess Archive up-to-date. Keeps me busy, especially when there's downtime at work, which is often (again, librarian, lol).

I had never even considered using a VPS. May be a useful tool for my archival work.

At any rate, it was good to make your acquaintance!

[vxbinaca]

@prof-frink Join irc.rizon.net #bibanon we all hang out there. We do 4chan archival too and have another neat tool for that. I'm a locksmith by trade.