search

Ascoware / get-iplayer-automator

The goal of Get iPlayer Automator is to allow iTunes and your Mac to become the hub for your British Television experience regardless of where in the world you are. Currently, Get iPlayer Automator allows you to download and watch BBC and ITV shows on your Mac. Series-Link/PVR functionality ensures you will never miss your favourite shows. Programmes are fully tagged and added to iTunes automatically upon completion. It is simple and easy to use, and runs on any machine running Mac OS X 10.7 or later. And since the shows are in iTunes, it is extremely easy to transfer them to your iPod, iPhone, or Apple TV allowing you to enjoy your shows on the go or on your television.
GNU General Public License v3.0
144 stars 27 forks source link

ITV Download Fail GIA 1.20 #303

Closed sxmpete closed 3 years ago

sxmpete commented 3 years ago

Error trying to download Doc Martin Episode, log follows: INFO: Loading proxy settings... INFO: No proxy to load INFO: Proxy load complete. INFO: No proxy will be used AppController: Starting Downloads

Downloading Show 1/1:

ITV Download (ID=1a7665a0025): Downloading Doc Martin - Mon Dec 07 INFO: Preparing Request for Auth Info INFO: Requesting Metadata. DEBUG: Metadata response status code: 200 INFO: Metadata processed. DEBUG: youtube-dl args:["https://www.itv.com/hub/doc-martin/1a7665a0025", "-f", "mp4/best", "-o", "/Users/peterdonlevy2/Movies/TV Shows/Doc Martin/Doc Martin.s07e01.Rescue Me.%(ext)s", "--verbose"] DEBUG: youtube-dl environment: ["PATH": "/Applications/Get iPlayer Automator v1.19.3.app/Contents/MacOS:/Applications/Get iPlayer Automator v1.19.3.app/Contents/Resources/get_iplayer/utils/bin:/usr/bin", "PYTHONPATH": "/Applications/Get iPlayer Automator v1.19.3.app/Contents/Resources"] [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: [u'https://www.itv.com/hub/doc-martin/1a7665a0025', u'-f', u'mp4/best', u'-o', u'/Users/peterdonlevy2/Movies/TV Shows/Doc Martin/Doc Martin.s07e01.Rescue Me.%(ext)s', u'--verbose'] [debug] Encodings: locale US-ASCII, fs utf-8, out None, pref US-ASCII [debug] youtube-dl version 2020.12.12 [debug] Python version 2.7.10 (CPython) - Darwin-18.7.0-x86_64-i386-64bit [debug] exe versions: ffmpeg 4.3, rtmpdump 2.4-102-ga3a600d-get_iplayer [debug] Proxy map: {} [debug] Using fake IP 25.99.254.71 (GB) as X-Forwarded-For. [ITV] 1a7665a0025: Downloading webpage [ITV] 1a7665a0025: Downloading JSON metadata ERROR: Unable to download JSON metadata: <urlopen error EOF occurred in violation of protocol (_ssl.c:590)> (caused by URLError(SSLEOFError(8, u'EOF occurred in violation of protocol (_ssl.c:590)'),)) File "/Applications/Get iPlayer Automator v1.19.3.app/Contents/Resources/youtube-dl/youtube_dl/extractor/common.py", line 632, in _request_webpage return self._downloader.urlopen(url_or_request) File "/Applications/Get iPlayer Automator v1.19.3.app/Contents/Resources/youtube-dl/youtube_dl/YoutubeDL.py", line 2248, in urlopen return self._opener.open(req, timeout=self._socket_timeout) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 431, in open response = self._open(req, data) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 449, in _open '_open', req) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain result = func(*args) File "/Applications/Get iPlayer Automator v1.19.3.app/Contents/Resources/youtube-dl/youtube_dl/utils.py", line 2736, in https_open req, **kwargs) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1197, in do_open raise URLError(err) ITV Download (ID=1a7665a0025): youtube-dl finished downloading AppController: Downloads Finished

skovatch commented 3 years ago

Try it again? I'm downloading it right now without any problems. What version of macOS are you running?

davidch119 commented 3 years ago

Not working for me either I have High Sierra 10.13.6 and Nord VPN Downloads a bit maybe 20 percent of file and then halts with unknown error When I try again it sometimes takes off where it left off and continues for a little bit more. At the moment I'm experimenting with repeating the download every 5 minutes but so far no luck. It seems a bit random sometimes it fails immediately, sometimes continues with the download. BBC is fine I wonder if your program is too ready to accept a download failure ? Maybe it should repeat the request several times ? I say this because I seem to be downloading a bit more of "Grantchester" every time I try although sometimes the download won't start again but sometimes its OK again after the third attempt. I have now succeeded in downloading one episode of "Grantchester" by repeating the download around 10 times. This is quite slow because it refreshes the programme index before each download

skovatch commented 3 years ago

First off, I think ITV has gotten more aggressive in their blocking lately. I will randomly get a 403 error at the start, but the next time I try it it works.

I wonder if your program is too ready to accept a download failure ? Maybe it should repeat the request several times ? I say this because I seem to be downloading a bit more of "Grantchester" every time I try although sometimes the download won't start again but sometimes its OK again after the third attempt.

GiA has an auto-retry timer, but the default is 30 minutes. For individual shows, youtube-dl is doing the work, so you should compare it against youtube-dl on the command-line. If it's still slow/failing there I can't do anything about it.

This is quite slow because it refreshes the programme index before each download

This shouldn't be necessary -- are you quitting and restarting the application? You should only need to stop and start again.

hugofalkman commented 3 years ago

I have set the auto-retry in preferences to 0.25 minutes, Then I forget about it, and sooner or later GiA finishes the ITV downloads.

davidch119 commented 3 years ago

I am not quitting the program just rerunning it It does refresh the index every time it does a download. I managed to bypass this by deselecting indexing for BBC, ITV and Radio. Not ideal of course. As hugofalkman says if you keep repeating the download eventually you get a downloaded programme. but surely it would be possible to automate this ? When the download fails mid download just repeat the download x times ?

hugofalkman commented 3 years ago

@davidch119 thats what setting the auto-retry to 0.25 minutes does, automating the process.

davidch119 commented 3 years ago

hugofalkman I think we are at cross purposes here. I set auto retry to 5 minutes and then 1 minutes. But every time it went through the lengthy process of refreshing the index each time(until I disabled it). Are you saying setting it to exactly 0.25 minutes makes a difference ?

sxmpete commented 3 years ago

I am running macOS Mojave and just failed download attempts on 2 different vpn servers. BTW, attached verbose log shows failure to download m3u8 info. This sounds similar to Issue 277.

Pete

On Sat, Jan 2, 2021 at 12:30 PM Scott Kovatch notifications@github.com wrote:

Try it again? I'm downloading it right now without any problems. What version of macOS are you running?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Ascoware/get-iplayer-automator/issues/303#issuecomment-753505174, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGZX3OOTNUZLHZ4QENRPCHTSX5J2BANCNFSM4VRFAW4Q .

-- Regards, Peter

hugofalkman commented 3 years ago

@davidch119 Yes sorry, see the problem got worse in 1.20. It no longer fails, just hangs. And once you hit the stop button to stop that, the refreshing of the cache starts. In earlier releases of GiA, it just failed and then restarted after the auto-retry period.

@skovatch Scott, you must be able to simulate an ITV network failure, to see that once you hit the stop button, the refresh of the cache starts.

skovatch commented 3 years ago

I'm downloading Doc Martin, and stopping the download part of the way through with the Stop button. Then I click the Start button. I don't see any reloading of the ITV cache. Are you describing something else? Maybe I need to shut off my wifi while it's downloading to reproduce it? If you have a screen recorder try capturing the footage and uploading it here.

I'm still trying to understand why the cache would refresh every time you start up a download. If you hit Command-R, yes, it will reload the caches.

Sorry for sounding obtuse here, but I must be missing something about your usage to reproduce it.

hugofalkman commented 3 years ago

Yes, you need to simulate a network error, don't know if turning off wifi is enough. Before 1.20 this caused the download to fail, but it seems now GiA hangs instead. If you now hit stop, the cache refresh starts. Note the refresh starts when you hit stop, NOT when you start. The refresh has nothing to do with starting a download.

davidch119 commented 3 years ago

skovatch Maybe this will help ? I had switched off updating for BBC TV and RADIO because it takes such a long time can produce a longer log. I think I was using an earlier version of your program but it seems to make no difference which version I use. Also (use current webpage has stopped working for ITV) It gives "invalid PID" for older programmes and then an additional error before each attempt to download. But if you persist after 10 or so attempts the programme downloads. The seems to be no logic to what is happening sometimes you get an error immediately, sometimes it downloads quite a bit and other times it downloads for just a second. So I assume the failure must be triggered by something from ITV.Downloading Show 1/1:

ITV Download (ID=1a7665a0032): Downloading Doc Martin - Tue Dec 15 INFO: Preparing Request for Auth Info INFO: Metadata processed. ITV Download (ID=1a7665a0032): youtube-dl finished downloading AppController: Downloads Finished GetITVShows: ITV Cache Update Starting GetITVShows (Info): Episodes: 2554 Today Programmes: 372 <AppController: 0x7fb576d22180>: Index Updated. GetITVShows: Update Finished

INFO: Loading proxy settings... INFO: No proxy to load INFO: Proxy load complete. INFO: No proxy will be used AppController: Starting Downloads

Downloading Show 1/1:

ITV Download (ID=1a7665a0032): Downloading Doc Martin - Tue Dec 15 INFO: Preparing Request for Auth Info INFO: Metadata processed. ITV Download (ID=1a7665a0032): Download Cancelled ITV Download (ID=1a7665a0032): youtube-dl finished downloading

davidch119 commented 3 years ago

skovatch I notice you are not in the UK either. I thought I would try to run a more up to date version of osx from a usb drive. Can you tell me which version of OSX you are using and I will try that.

hugofalkman commented 3 years ago

@skovatch Ok Scott, finally managed to catch the error and the cache refresh in the video attached. (Turning off wifi didn't generate this behaviour, had to wait for the real ITV network error.) When the first refresh ends, the auto-retry kicks in (set to 0.25 minutes), gets a second failure followed by another refresh, where I stopped the video. Running Big Sur 11.1.

https://user-images.githubusercontent.com/10969068/103485979-b9d77500-4dfa-11eb-8e7d-cdb0da318529.mov

skovatch commented 3 years ago

OK, I got this to happen once, and I think I know what's going on. I can push what I think is a fix -- I haven't seen it again after applying it.

hugofalkman commented 3 years ago

Cache still refreshing after failure. Also, this time the BBC refresh hangs (only the ITV refresh finishes).

This time the ITV download hanged. And when restarted the download then failed, and the cache refresh started.

Screenshot 2021-01-04 at 20 20 39

GiAlog.txt

skovatch commented 3 years ago

OK, some rambling thoughts here: the problem is that the auto-retry code is scheduling a time for the download to start again, but when the timer fires to start the downloads it's calling forceUpdate on the cache. The auto-retry code shares the delayed start implementation with the "Schedule Start.." menu item, which lets you pick a time in the future to start downloads. And, as the text of the 'Pick a time' window indicates, this is deliberate behavior. I'm still not clear on why the reload happens right away on failure, though.

In any event, I think I'm going to separate the two use cases so the auto-restart doesn't mess with the cache. I can't see a better way right now.

davidch119 commented 3 years ago

Wonder if it has to do with pids ? The PID for a programme seems to change in time. What I have noticed is this If you deselect ITV from indexing then subsequent retries at downloading of ITV programmes no longer work - you get an immediate error. However if you deselect indexing for BBC TV and Radio as a temporary fix this considerably speeds things up because indexing ITV programmes takes just 60 seconds whereas indexing of BBC takes several minutes.

davidch119 commented 3 years ago

Hi skovatch your changes haven't stopped the ITV index being refreshed each time (I had disabled BBC because it takes such a long time)Downloading Show 1/1:

ITV Download (ID=1a7665a0030): Downloading Doc Martin - Mon Dec 14 INFO: Preparing Request for Auth Info INFO: Metadata processed. ITV Download (ID=1a7665a0030): youtube-dl finished downloading AppController: Downloads Finished GetITVShows: ITV Cache Update Starting GetITVShows (Info): Episodes: 2590 Today Programmes: 394 <AppController: 0x7fe4ba445be0>: Index Updated. GetITVShows: Update Finished

INFO: Loading proxy settings... INFO: No proxy to load INFO: Proxy load complete. INFO: No proxy will be used AppController: Starting Downloads

Downloading Show 1/1:

ITV Download (ID=1a7665a0030): Downloading Doc Martin - Mon Dec 14 INFO: Preparing Request for Auth Info INFO: Metadata processed. ITV Download (ID=1a7665a0030): youtube-dl finished downloading AppController: Downloads Finished GetITVShows: ITV Cache Update Starting GetITVShows (Info): Episodes: 2590 Today Programmes: 394 <AppController: 0x7fe4ba445be0>: Index Updated. GetITVShows: Update Finished

INFO: Loading proxy settings... INFO: No proxy to load INFO: Proxy load complete. INFO: No proxy will be used AppController: Starting Downloads

Downloading Show 1/1:

ITV Download (ID=1a7665a0030): Downloading Doc Martin - Mon Dec 14 INFO: Preparing Request for Auth Info INFO: Metadata processed. ITV Download (ID=1a7665a0030): youtube-dl finished downloading AppController: Downloads Finished GetITVShows: ITV Cache Update Starting GetITVShows (Info): Episodes: 2590 Today Programmes: 394 <AppController: 0x7fe4ba445be0>: Index Updated. GetITVShows: Update Finished

INFO: Loading proxy settings... INFO: No proxy to load INFO: Proxy load complete. INFO: No proxy will be used AppController: Starting Downloads

Downloading Show 1/1:

ITV Download (ID=1a7665a0030): Downloading Doc Martin - Mon Dec 14 INFO: Preparing Request for Auth Info INFO: Metadata processed. ITV Download (ID=1a7665a0030): youtube-dl finished downloading INFO: Downloading thumbnail INFO: Thumbnail download completed to file:///var/folders/tk/pxdzgc6d5mj4s777h7ghldwc0000gn/T/CFNetworkDownload_PYcqw3.tmp INFO: Beginning AtomicParsley Tagging. INFO: AtomicParsley Tagging finished. AppController: Downloads Finished

skovatch commented 3 years ago

This is now fixed in 1.20.2.