search

Ascoware / get-iplayer-automator

The goal of Get iPlayer Automator is to allow iTunes and your Mac to become the hub for your British Television experience regardless of where in the world you are. Currently, Get iPlayer Automator allows you to download and watch BBC and ITV shows on your Mac. Series-Link/PVR functionality ensures you will never miss your favourite shows. Programmes are fully tagged and added to iTunes automatically upon completion. It is simple and easy to use, and runs on any machine running Mac OS X 10.7 or later. And since the shows are in iTunes, it is extremely easy to transfer them to your iPod, iPhone, or Apple TV allowing you to enjoy your shows on the go or on your television.
GNU General Public License v3.0
146 stars 26 forks source link

Unable to Download BBC Programs #54

Open miffi1987 opened 7 years ago

miffi1987 commented 7 years ago
  1. Tried the latest version of GIA. The program indexing update is not working with below log.

Get iPlayer Automator 1.9 Initialized. GetITVShows: ITV Cache Update Starting AppController: Updating Programme Index Feeds...

ERROR: Programe Type '' does not exist. Try using --refresh ... ERROR: Cannot update new programmes history from cache file: /Users/FelixM/Library/Application Support/Get iPlayer Automator/radio.cache GetITVShows: Update Finished

  1. Unable to download BBC programs even when using "use current webpage" function.

Downloading Show 1/1:

BBC Download (ID=p057nfy7): Downloading Unknown: PID Not Found ERROR: Programe Type '' does not exist. Try using --refresh BBC Download (ID=p057nfy7): Unknown: PID Not Found Failed

  1. But for ITV Hub, GIA still works well when apply "use current webpage" .

I've upgraded Mac Sierra to latest 10.12.6 version. Will this be the problem? Thanks for your help.

MacInWin commented 7 years ago

It's working for me on 10.12.6 just fine. And after updating the cache, it's searching on BBC just fine, too. Maybe force a cache update?

Lordsnooty05 commented 7 years ago

Yes it is working for me as well. Maybe a wee bit slow at updating about 2 or 3 minutes but it downloads much quicker than ever before......really fast

ofiucus commented 7 years ago

I'm having the same problem with Show Name 'Unknown: PID not found'. Console says this: Get iPlayer Automator 1.9 Initialized.

AppController: Updating Programme Index Feeds...

ERROR: Programe Type '' does not exist. Try using --refresh

AppController: Index was Up-To-Date.

INFO: Loading proxy settings... INFO: No proxy to load INFO: Proxy load complete. INFO: No proxy will be used

AppController: Starting Downloads

Downloading Show 1/2:

BBC Download (ID=p057nmkt): Downloading Unknown: PID Not Found ERROR: Programe Type '' does not exist. Try using --refresh BBC Download (ID=p057nmkt): Unknown: PID Not Found Failed

Downloading Show 2/2:

BBC Download (ID=p0578x02): Downloading Unknown: PID Not Found ERROR: Programe Type '' does not exist. Try using --refresh BBC Download (ID=p0578x02): Unknown: PID Not Found Failed

AppController: Downloads Finished

ghost commented 7 years ago

I'm getting exactly the same

(BBC Download (ID=b091w29y): Downloading Unknown: PID Not Found ERROR: Programe Type '' does not exist. Try using --refresh BBC Download (ID=b091w29y): Unknown: PID Not Found Failed)

It's extremely frustrating. It also won't let me submit a bug report.

Any help would be greatly appreciated!

Lordsnooty05 commented 7 years ago

Hannah are you downloading from Safari the BBC programs? I just downloaded about 10 different programs about an hour ago and it was working fine. It takes about 5 to 10 minutes to cache but once it does that it downloads right away for me

gec118 commented 7 years ago

I get exactly the same:

Downloading Show 8/8:

BBC Download (ID=b0074nq6): Downloading Unknown: PID Not Found ERROR: Programe Type '' does not exist. Try using --refresh BBC Download (ID=b0074nq6): Unknown: PID Not Found Failed

kw4c commented 7 years ago

Well this can't be good and I'm using privatetunnel...

WARNING: No media streams found for requested programme versions and recording modes. WARNING: The BBC has blocked access to this programme because it has determined that you are using get_iplayer outside the UK. ERROR: Could not get programme metadata

marco-brandizi commented 7 years ago

I've started from "Use current Web Page". The Process ID gave me "PID not found". Then I clicked on the Start button anyway and boom! It downloaded it! So, maybe there is some confusion with messages?

skovatch commented 6 years ago

@marco-brandizi if you haven't cached anything the app won't be able to find a PID in the cache, and that's why it says 'PID not found'. The name of the show isn't required for the download, just the PID, so the download will work whether the metadata is cached or not.

skovatch commented 6 years ago

You could try deleting the contents of ~/Library/Application Support/Get iPlayer Automator, make sure you have 'BBC TV' selected in the preferences, and launch the application again, or force the cache to refresh. Somehow it thinks a " is a program type.

Also, in the preferences, click "Restore Defaults".

MacInWin commented 6 years ago

I did the "Restore Defaults," but still get the same error. Here is the log:

Get iPlayer Automator 1.9.2 Initialized.

GetITVShows: ITV Cache Update Starting AppController: Updating Programme Index Feeds... GetITVShows (Info): Merge complete B/F Programmes: 1273 C/F Programmes: 1262 Today Programmes: 173 AppController: Index Updated. GetITVShows: Update Finished

INFO: Loading proxy settings... INFO: No proxy to load INFO: Proxy load complete. INFO: No proxy will be used

AppController: Starting Downloads

Downloading Show 1/1: BBC Download (ID=b01n3jz9): Downloading How We Won the War - Northern Ireland ERROR: Programe Type '' does not exist BBC Download (ID=b01n3jz9): How We Won the War - Northern Ireland Failed

AppController: Downloads Finished

crgmrgn commented 6 years ago

Similar issue here with 1.9.3, adding a BBC download from its webpage (or via search in the app) is fine, I can process the PID without error, so the cache seems fine, but the download fails with the warning above of a programme type ".

I've forced the cache clear and also erased all the app settings a few times to verify that its not a lingering setup problem, but 1.9.2 and 1.9.3 seem to produce this issue (in fact I need to return to a version pre-3.0.2 inclusion for successful operation to resume), but this obviously means suffering serialised cache updates ;-)

Happy to provide logs if it helps ...

crgmrgn commented 6 years ago

Issue is still present in v1.9.6. Just to verify that the rest of the install is correct I checked a download from ITV hub and "Use Current Webpage" and that is working perfectly. So it seems to be something specifically with the search terms for the BBC download scripts.

skovatch commented 6 years ago

Have any of you deleted the entire application support folder? In the Finder, open a new window. Then from the menu, choose Go, then Go To Folder... Type in "~/Library/Application Support", then find "Get iPlayer Automator" and throw it in the trash. Then launch the application again.

WilliamOfGaunt commented 6 years ago

I just did that and lost my PVR list – there were quite a few programmes there!

From: Scott Kovatch Sent: Saturday, October 7, 2017 11:08 PM To: Ascoware/get-iplayer-automator Cc: Subscribed Subject: Re: [Ascoware/get-iplayer-automator] Unable to Download BBC Programs (#54)

Have any of you deleted the entire application support folder? In the Finder, open a new window. Then from the menu, choose Go, then Go To Folder... Type in "~/Library/Application Support", then find "Get iPlayer Automator" and throw it in the trash. Then launch the application again.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

skovatch commented 6 years ago

Eek! Don't do it unless you're specifically having this problem! It will clobber all of your download history, PVR links, and cache.

The errors hint at not being able to write out a file in the cache location, and I'm hoping there's a bogus file in there causing problem.

WilliamOfGaunt commented 6 years ago

Not to worry – I hooked the folder back from Trash just now!

Thanks for all your work on this application, which is invaluable to many.

I hope your eyes are coping OK. (My sight is pretty much heading south now, so I sympathize.)

From: Scott Kovatch Sent: Sunday, October 8, 2017 12:37 AM To: Ascoware/get-iplayer-automator Cc: WilliamOfGaunt ; Comment Subject: Re: [Ascoware/get-iplayer-automator] Unable to Download BBC Programs (#54)

Eek! Don't do it unless you're specifically having this problem! It will clobber all of your download history, PVR links, and cache.

The errors hint at not being able to write out a file in the cache location, and I'm hoping there's a bogus file in there causing problem.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

heroineinabook commented 6 years ago

Similar issue. I've dumped the ~/library folder and it's a clean install of 1.9.7. I've also forced clean my cache.

I am getting this:

INFO: tv episode PID detected (b097rvy8) INFO: Trying to download PID: b097rvy8 using type: tv INFO: PID found in cache: b097rvy8 INFO: 1 matching programmes WARNING: No media streams found for requested programme versions and recording modes. BBC Download (ID=b097rvy8): The Last Post: Series 1 - Episode 1 Failed

crgmrgn commented 6 years ago

Decided to purge all signs of previous GIA or related apps, dumped them all, cleaned up any prefs in system to individual home directories and did a clean reboot just for kicks. Downloaded v1.9.6 zip from the releases section and did a clean install. Looks like I've got BBC downloading operational again.

So, looks like I had some cruft causing issues, where I don't know as I'd previously tried deleting the prefs dir for my account, but its all good now thanks! BTW, in app updated to v1.9.7 successfully over this as well since ...

vertigo4 commented 6 years ago

For the last few days I have been unable to download any BBC programmes. This is the error message in the log:

WARNING: The BBC has blocked access to this programme because it has determined that you are using get_iplayer outside the UK.

-

I am indeed outside the UK but have not had problems before, either using the custom proxy in Get iPlayer or none.

I should also say that even using a previously reliable VPN offering UK IP addresses I am also now unable to watch BBC iPlayer live.

Any thoughts, suggestions, workarounds?

MacInWin commented 6 years ago

I read somewhere that the BBC is sending that error reply for any connection they determine is a VPN, even if the VPN originated within the UK (not that they can tell, they just block ALL VPN connections). I don't think there is a workaround.

skovatch commented 6 years ago

Please note that this is not the place for tech support on VPN services if you are outside the UK. Unfortunately there is no forum right now for discussions.

I will, however, recommend you investigate a DNS-based proxy service, rather than VPN.

vertigo4 commented 6 years ago

Thanks, Scott.

So that means that Get iPlayer Automator will no longer work outside the UK unless - maybe - one is using a DNS-based proxy. What a drag! A lot of very disappointed people will miss a great service.

hugofalkman commented 6 years ago

https://www.fastestvpnguide.com/workaround-for-bbc-iplayer-vpn-not-working/

gec118 commented 6 years ago

I disagree that GiA can’t work with a vpn. I have watched several BBC programmes with a vpn this weekend. If I can watch them, why can’t Automator download them?

sciencefish commented 6 years ago

@gec118 please read the article posted previously by @hugofalkman, it explains that VPN providers are having individual IPs blacklisted. You are fortunate that yours hasn’t yet been blacklisted.

gec118 commented 6 years ago

@sciencefish Yes, I did - after I posted, and I totally agree with you, that I have a "good" VPN (It should be as it isn't cheap!). But my point is still valid. I can watch BBC programmes with my VPN but GiA won't download them. Therefore it would not appear to be a problem with my VPN but something else.

Sircam19 commented 6 years ago

The difference is usage of RTMP dump. Not used when watching/ streaming but when downloading.

kw4c commented 6 years ago

And the BBC has specifically identified you are using get_iplayer to do the download over VPN.

Vangelis66 commented 6 years ago

@Sircam19 wrote:

The difference is usage of RTMP dump. Not used when watching/ streaming but when downloading.

It would be appreciated if people not post inaccuracies...

Latest version(s) of GiA uses a modified 3.02 version of the get_iplayer (GiP) perl script; GiP no longer supports the download of the older RTMP (aka Flash) streams, but implements a pure perl native HLS/MPEG-DASH downloader for beeb streams; RTMPdump in GiA is used for fetching ITV streams, not BBC content.

You are correct though that watching a BBC stream inside a desktop browser is (somewhat) different to downloading the programme with GiA; in the first case, you are viewing either an AdobeHDS or MPEG-DASH stream (depends on OS/browser/Flash plugin), while in the second case you're downloading probably an AppleHLS stream.

Likewise with ITVHub; watch a stream in a browser, it's (encrypted) MPEG-DASH (or AppleHLS in some mobile devices); download with GiA, it's an RTMP stream.

So people shouldn't compare how their VPS/VPN/SmartDNS is behaving in a browser to troubleshoot downloading issues in GiA.

Most commercial proxy/VPN providers target mainly the iPlayer browser API streams and possibly access on smart mobile devices... Successful working of GiA from overseas with them should never be regarded as a given... Something additional regarding ITVhub streams; If you use a full-fledged UK VPN, make sure with them they have the default RTMP port (1935) open in their VPN server config; else you'll be able to watch ITVhub in a browser, but not dump with GiA - check how your VPN is configured with this test. @skovatch: Is there a provision inside GiA's settings to change default RTMP port from 1935 to either 443 or 80 (these are always open in a VPN for HTTPS/HTTP URIs)?

@kw4c wrote:

And the BBC has specifically identified you are using get_iplayer to do the download over VPN

Inaccurate, also... GiP has an internal list of browser user agents, each time a connection is attempted with a BBC server, one of them is randomly chosen; the BBC cannot identify you're using get_iplayer, because it appears just as a random internet browser, not a downloading tool...

However, the beeb do check the IP address behind those GiP web requests, so as to impose their geo-filtering schemes; VPNs and similar geo-location circumvention means are identified (and hence blocked) in the manner detailed in the link earlier in this thread (i.e. an unjustifiably very large number of web requests all originating from the same IP address, the one belonging to a VPS/VPN server); once an IP has been flagged, it's child's play to identify it as a VPN, then blacklist it... Plus the fact many VPN services use IP pools from Data Centers already known to the iPlayer team; in fact, VPNs are being blocked even for people within the UK...

And geo-filtering in a browser works in several layers:

  1. Requirement of BBC ID login for a user inside the UK (iPlayer radio overseas and globally available video do not, yet, require an international BBC ID)
    1. IP address check and setting of a related browser cookie
    2. Mediaselector API URIs; these URIs provide access to stream data (GiP needs these to compose its downloading commands).
    3. Geo-fencing at the actual CDN server hosting the stream to be viewed in the client (browser).

1+2+3 could be regarded as soft "blocks", while 4 is a "hard" block; for the moment, and until May 2018, GiP (hence GiA) can do away with 1+2, so if your VPN fails is because of either 3/4 or both.

@skovatch

If you feel uncomfortable with the content I've posted, then chime in and I'll remove what you deem inappropriate... I am of the opinion though that once GiA users get the gist of it, the less likely this issue tracker is of being flooded in the future with additional "My VPN has stopped working" reports...

Kind regards

skovatch commented 6 years ago

This is fine for now... What we really need is some forum software for discussions.

I don't see anything for forcing the rtmpdump command to use a standard HTTPS port. I'd have to add it/test it.