Open mariocynicys opened 2 years ago
My -H "Authorization=Bearer $(gcloud auth print-access-token)"
hack failed after about one hour of running. I started getting HTTP 429 responses from Google Cloud, which makes me think it wasn't an issue with the token itself, but something more fundamental we will need to resolve.
Google's docs say:
If you run into any issues such as increased latency or error rates, pause your ramp-up or reduce the request rate temporarily in order to give Cloud Storage more time to scale your bucket. You should use exponential backoff to retry your requests when:
Receiving errors with 408 and 429 response codes. Receiving errors with 5xx response codes.
This PR adds an upload proxy node. A state aware node that handles the work of uploading content to a remote location on behalf of Shaka Packager. This PR adds no new dependencies.
HTTPUpload
which runs as a python thread and manages anHTTPServer
that Shaka Packager should send all its requests to.HTTPServer
instantiates request handlers for requests coming from Shaka Packager, these request handlers send the incoming requests to the upload location, and they might add more headers to the request.HTTPUpload
are there to support different protocols -other than http/https- for cloud providers and to extend theHTTPUpload
's capabilities, but they should all use an http/https API after processing the upload url.RequestBodyAsFileIO
is a class that digests a request body incrementally without writing it to the disk. It is used by the request handlers to send the request body that lacks anEOF
to the upload location.Issues:
somepath\stream_xx.m3u8
which we should be able to control using the argument--playlist_name
but for some reason Shaka Packager doesn't recognize it.