Open damndingo opened 3 years ago
Can confirm this issue - I received exactly the same results when testing.
Same for me, from the UK, so no VPN issue. I tried the Jonathan Ross Show. Failed immediately.
The Python crypto support was changed in 1.2.10, and appears to be broken. If you see this before upgrading, don't upgrade and wait for a fix. If you have already upgraded, you can roll back to 1.21.9 and replace the bundled version of youtube-dl as describe in https://github.com/Ascoware/get-iplayer-automator/issues/331#issuecomment-832292012 . But the best course of action is probably to wait. This shouldn't affect BBC programmes.
thanks @sleaux-meaux !
I had already upgraded, but your fix worked perfectly!
It appears that the Python 3 version of PyCryptodome was included in GiA, but that isn't digestible to Python 2.
EDIT: I think a Python 2 version of PyCryptodome can be used with both Python 2 and 3.
Ignore my earlier suggestion to temporarily replace /Applications/Get iPlayer Automator.app/Contents/Resources/Crypto with the equivalent folder from 1.21.9. That is not a good idea if you are using Python 3 with GiA since the old PyCrypto package won't load and ITV downloads will be delegated to ffmpeg.
OK, already did that and downloaded some things. Are they compromised?
I do have Python 3 on my system but how do I know if GiA is using it (as opposed to v2).
Should I remove GiA and follow your first recommendation (install v.1.21.9 and replace YouTube.dl)?
Downloads won't be compromised. They either decrypt or they don't.
GiA 1.2.10 tries these locations for Python, in this order. Check versions with:
/usr/local/bin/python --version /usr/bin/python --version
GiA 1.2.9 and earlier don't check /usr/local/bin. With verbose logging, youtube-dl will also print a line that looks this:
[debug] Python version 3.9.4 (CPython) - macOS-10.15.7-x86_64-i386-64bit
If you replaced Crypto folder and it's working, it means Python 2 is being used, so no need to change. If something goes wrong, you can always reinstall GiA and start over.
thanks
for interest,
/usr/local/bin/python --version
gave me nothing
/usr/bin/python --version
gave me 2.7.16
python --version
gave me 2.7.16
python3 --version
gave me 3.9.0
youtube-dl only looks for "python", so "python3" would be ignored. For completeness, you should check full paths (/usr/local/bin/python3 and /usr/bin/python3) so you know what is where.
cheers.
it's in /usr/local/bin/python3
I am also having problems with ITV downloads, I was able to download 2 episodes of series 1 of Bancroft but the other 2 would not download and the same with episode 1 of series 2. Failed download solutions shows problem unknown, logs detail in attached file IPA Log 6th May 21.docx :
It appears that the Python 3 version of PyCryptodome was included in GiA, but that isn't digestible to Python 2.
EDIT: I think a Python 2 version of PyCryptodome can be used with both Python 2 and 3.
PyCryptodome is supposed to support Python 2.6, 2.7 and 3.4 and up, which is why I didn't expect a problem. I copied over what pip installed into my Python3 installation.
For those of you still having problems, can you post a verbose log? I need to see how youtube-dl is failing for you.
I can back out the crypto and path changes, but I'd like to understand a bit more why it's okay on my system and not others.
Error shown below, running under Python 2. There are separate PyCryptodome wheels installed for Python 2 and 3. The Python 3 wheel is built against Python 3 stable ABI, thus not compatible with Python 2, but the Python 2 wheel works with Python 3. Install PyCryptodome in Python 2 and copy its Crypto folder.
Downloading The Jonathan Ross Show: Series 17
INFO: Preparing Request for Auth Info
INFO: Requesting Metadata.
DEBUG: Metadata response status code: 200
INFO: Metadata processed.
DEBUG: youtube-dl args:["https://www.itv.com/hub/the-jonathan-ross-show/2a1166a0205", "-f", "mp4/best", "-o", "/Users/xxx/Movies/TV Shows/The Jonathan Ross Show - Series 17/The Jonathan Ross Show - Series 17.s17e04.Episode 4.%(ext)s", "--write-sub", "-k", "--verbose"]
DEBUG: youtube-dl environment: ["PATH": "/Applications/Get iPlayer Automator.app/Contents/MacOS:/Applications/Get iPlayer Automator.app/Contents/Resources/get_iplayer/utils/bin:/usr/local/bin:/usr/bin", "PYTHONPATH": "/Applications/Get iPlayer Automator.app/Contents/Resources"]
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/Applications/Get iPlayer Automator.app/Contents/Resources/youtube-dl/__main__.py", line 16, in <module>
File "/Applications/Get iPlayer Automator.app/Contents/Resources/youtube-dl/youtube_dl/__init__.py", line 15, in <module>
File "/Applications/Get iPlayer Automator.app/Contents/Resources/youtube-dl/youtube_dl/options.py", line 8, in <module>
File "/Applications/Get iPlayer Automator.app/Contents/Resources/youtube-dl/youtube_dl/downloader/__init__.py", line 5, in <module>
File "/Applications/Get iPlayer Automator.app/Contents/Resources/youtube-dl/youtube_dl/downloader/hls.py", line 6, in <module>
File "/Applications/Get iPlayer Automator.app/Contents/Resources/Crypto/Cipher/__init__.py", line 27, in <module>
from Crypto.Cipher._mode_ecb import _create_ecb_cipher
File "/Applications/Get iPlayer Automator.app/Contents/Resources/Crypto/Cipher/_mode_ecb.py", line 47, in <module>
"""
File "/Applications/Get iPlayer Automator.app/Contents/Resources/Crypto/Util/_raw_api.py", line 297, in load_pycryptodome_raw_lib
raise OSError("Cannot load native module '%s': %s" % (name, ", ".join(attempts)))
OSError: Cannot load native module 'Crypto.Cipher._raw_ecb': Trying '_raw_ecb.so': dlopen(/Applications/Get iPlayer Automator.app/Contents/Resources/Crypto/Util/../Cipher/_raw_ecb.so, 6): image not found, Trying '_raw_ecbmodule.so': dlopen(/Applications/Get iPlayer Automator.app/Contents/Resources/Crypto/Util/../Cipher/_raw_ecbmodule.so, 6): image not found
youtube-dl finished downloading
AppController: Downloads Finished
Correction: Don't just roll back to PyCrypto - it doesn't seem to work with Python 3. Forgot my own advice above. Would be OK if you return to forcing use of /usr/bin/python, though.
No go for me using Catalina. I was so pleased to see the prompt for the 21 update but alas it doesn't work. C'est la vie
Can I get a volunteer to try this test build? I think I have figured out how to include python in the app so GiA isn't dependent on a system-installed python.
If you can download ITV content (or you can't) with this build let me know. I'd especially like folks running earlier versions of macOS to try it, as I'm not sure the included python runs on older versions of macOS.
trying it now.
On May 6, 2021, at 4:49 PM, Scott Kovatch @.***> wrote:
Can I get a volunteer to try this test build? I think I have figured out how to include python in the app so GiA isn't dependent on a system-installed python.
Link to GDrive https://drive.google.com/file/d/1XZfsNbP1y5t77w7u0NeUmuI7H02ux7pD/view?usp=sharing β You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Ascoware/get-iplayer-automator/issues/339#issuecomment-833854886, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEDRDZQBPJPR6AZ3XX6BPSLTML6D5ANCNFSM44GMORZQ.
It failed for an ITV show..
Downloading In for a Penny: Series 3 INFO: Preparing Request for Auth Info INFO: Requesting Metadata. DEBUG: Metadata response status code: 200 INFO: Metadata processed. DEBUG: youtube-dl args:["/private/var/folders/7j/5bqkw1wn3kl_y3g7ym7zr9400000gn/T/AppTranslocation/C42905CC-BA09-41CC-9969-D8401C024110/d/Get iPlayer Automator.app/Contents/Resources/youtube-dl", "https://www.itv.com/hub/in-for-a-penny/2a5241a0019", "-f", "mp4/best", "-o", "/Users/ian02054/Movies/radio/In for a Penny/In for a Penny.s03e03.Episode 3.%(ext)s", "--verbose"] DEBUG: youtube-dl environment: ["PATH": "/private/var/folders/7j/5bqkw1wn3kl_y3g7ym7zr9400000gn/T/AppTranslocation/C42905CC-BA09-41CC-9969-D8401C024110/d/Get iPlayer Automator.app/Contents/Resources:/private/var/folders/7j/5bqkw1wn3kl_y3g7ym7zr9400000gn/T/AppTranslocation/C42905CC-BA09-41CC-9969-D8401C024110/d/Get iPlayer Automator.app/Contents/Resources/get_iplayer/utils/bin:/private/var/folders/7j/5bqkw1wn3kl_y3g7ym7zr9400000gn/T/AppTranslocation/C42905CC-BA09-41CC-9969-D8401C024110/d/Get iPlayer Automator.app/Contents/Resources/venv/bin"] youtube-dl finished downloading AppController: Downloads Finished
Was using the patched Sleaux-meaux version before, and that show downloaded ok with that. (I cleared the show from history before trying)
just failed with problem unknown
On May 6, 2021, at 4:49 PM, Scott Kovatch @.***> wrote:
Can I get a volunteer to try this test build? I think I have figured out how to include python in the app so GiA isn't dependent on a system-installed python.
Link to GDrive https://drive.google.com/file/d/1XZfsNbP1y5t77w7u0NeUmuI7H02ux7pD/view?usp=sharing β You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Ascoware/get-iplayer-automator/issues/339#issuecomment-833854886, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEDRDZQBPJPR6AZ3XX6BPSLTML6D5ANCNFSM44GMORZQ.
All I really want mostly is ITV news for my youtube channel. As I said I can't even watch via the kodi 'itv player' add on anymore :(
just failed with problem unknown
What version of macOS are you using?
URL: https://www.itv.com/hub/superstore/2a5460a0107
Runs on Catalina, but there is no crypto support bundled, so delegates to ffmpeg. Fails on Yosemite, with no clues in the log. I don't think you can just ship a virtualenv created with Homebrew Python. The included python binary references the Python framework in its Homebrew location, which won't exist for most people. I expect it only ran for me on Catalina because I have Homebrew Python installed.
Bah. I thought "python venv --copies" copies in everything you need for a python distribution, but I guess not.
High sierra 10.13.6
Tried others shows and got same issue..i knew that one had worked 5 mins before you sent the test...so wanted to try that.
On Thu, May 6, 2021, 17:19 Scott Kovatch @.***> wrote:
just failed with problem unknown
What version of macOS are you using?
β You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Ascoware/get-iplayer-automator/issues/339#issuecomment-833874402, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEDRDZVWVGX34NU5NBQ2U5TTMMBVXANCNFSM44GMORZQ .
OK, I give up for now. I've reverted out the crypto and path changes and pushed a new version.
URL: https://www.itv.com/hub/superstore/2a5460a0107
Runs on Catalina, but there is no crypto support bundled, so delegates to ffmpeg. Fails on Yosemite, with no clues in the log. I don't think you can just ship a virtualenv created with Homebrew Python. The included python binary references the Python framework in its Homebrew location, which won't exist for most people. I expect it only ran for me on Catalina because I have Homebrew Python installed.
It doesn't run on Catalina with me
OK, I give up for now. I've reverted out the crypto and path changes and pushed a new version.
Can confirm the rolled-back version is working perfectly on Big Sur now
As an alternative, I could supply youtube-dl with Python 3 bundled in (via PyInstaller).
That might actually work... I have the youtube-dl sources so I built it and then pyinstaller. I was able to get a runnable youtube-dl distribution. I don't have your patch, so I'd have to get that and rebuild it. I'll work with it some more after work today.
@skovatch wrote:
PyCryptodome is supposed to support Python 2.6, 2.7 and 3.4 and up, which is why I didn't expect a problem
FTR, this is only true up to pycryptodome==3.9.9
; BTW, the author only provides the following wheels:
pycryptodome-3.9.9-cp27-cp27m-macosx_10_6_intel.whl (14.1 MB)
pycryptodome-3.9.9-cp35-cp35m-macosx_10_6_intel.whl (13.3 MB)
pycryptodome-3.9.9-cp36-cp36m-macosx_10_6_intel.whl (13.3 MB)
pycryptodome-3.9.9-cp37-cp37m-macosx_10_6_intel.whl (14.1 MB)
pycryptodome-3.9.9-cp38-cp38-macosx_10_9_x86_64.whl (13.3 MB)
MacOS X 10.6 supported on Python 2.7.x/3.5.x/3.6.x/3.7.x; MacOS X 10.9 supported on Python 3.8.x ...
Current pycryptodome==3.10.1
dropped support for Python 2.6 and 3.4, so only 2.7.x and >=3.5.x are supported; official wheels include just
pycryptodome-3.10.1-cp27-cp27m-macosx_10_9_x86_64.whl (1.5 MB)
pycryptodome-3.10.1-cp35-abi3-macosx_10_9_x86_64.whl (1.5 MB)
i.e. MacOS X 10.6 support is dropped; the abi3 variety should be install-able on Python >= 3.5.0
The PyCryptodome module comprises C extensions (.so files - .pyd on Windows) that need to have been compiled by the same (or compatible) version of the C compiler used to compile the respective Python binaries - thus, interchange-abitlity between the cp27-cp27m
and cp35-abi3
flavours isn't a given... π
The dead (but not buried π ) PyCrypto module is only officially available as source, i.e. no wheels; it was tested against Python 2.1 through to 3.3; it, too, includes C extensions; in theory, it is possible to compile it from source (and install) on a Python 3 enabled Mac, but you should use the same compiler (OS X SDK?) your Python 3 version was built with... Of possible relevance: https://www.javaer101.com/en/article/14576333.html
I saw this error again this morning.
I had 1 successful ITV download and one failed (consistently). Same show. Log below...
Get iPlayer Automator 1.21.11 Initialized.
GetITVShows: ITV Cache Update Starting
<AppController: 0x7fb09273bf30>: Updating Programme Index Feeds...
/Applications/Get iPlayer Automator.app/Contents/Resources/get_iplayer/perl/bin/get_iplayer
-e86400
--type=tv,radio
--refresh
--nopurge
--profile-dir=/Users/jameshenderson/Library/Application Support/Get iPlayer Automator
.*
INFO: Indexing tv programmes (concurrent)
GetITVShows (Info): Episodes: 3550 Today Programmes: 351
GetITVShows: Update Finished
INFO: Added 9 tv programmes to cache
INFO: Indexing radio programmes (concurrent)
INFO: Added 4 radio programmes to cache
INFO: 22219 matching programmes
<AppController: 0x7fb09273bf30>: Index Updated.
/Applications/Get iPlayer Automator.app/Contents/Resources/get_iplayer/perl/bin/get_iplayer
--nocopyright
-e60480000000000000
--type=tv,itv,radio
--listformat
SearchResult|<pid>|<available>|<type>|<name>|<episode>|<channel>|<seriesnum>|<episodenum>|<desc>|<thumbnail>|<web>|<available>
--long
--nopurge
--search
superstore
--profile-dir=/Users/jameshenderson/Library/Application Support/Get iPlayer Automator
INFO: Loading proxy settings...
INFO: No proxy to load
INFO: Proxy load complete.
INFO: No proxy will be used
AppController: Starting Downloads
Downloading Show 1/1:
Downloading Superstore: Series 3
INFO: Preparing Request for Auth Info
INFO: Requesting Metadata.
DEBUG: Metadata response status code: 200
INFO: Metadata processed.
DEBUG: youtube-dl args:["https://www.itv.com/hub/superstore/2a5460a0042", "-f", "mp4/best", "-o", "/Users/jameshenderson/Downloads/downloads - GiA/Superstore - Series 3/Superstore - Series 3.s03e09.Golden Globes Party.%(ext)s", "--verbose"]
DEBUG: youtube-dl environment: ["PATH": "/Applications/Get iPlayer Automator.app/Contents/MacOS:/Applications/Get iPlayer Automator.app/Contents/Resources/get_iplayer/utils/bin:/usr/bin", "PYTHONPATH": "/Applications/Get iPlayer Automator.app/Contents/Resources"]
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://www.itv.com/hub/superstore/2a5460a0042', u'-f', u'mp4/best', u'-o', u'/Users/jameshenderson/Downloads/downloads - GiA/Superstore - Series 3/Superstore - Series 3.s03e09.Golden Globes Party.%(ext)s', u'--verbose']
[debug] Encodings: locale US-ASCII, fs utf-8, out None, pref US-ASCII
[debug] youtube-dl version 2021.04.26-gia_01
[debug] Python version 2.7.16 (CPython) - Darwin-20.5.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.3.2-tessus
[debug] Proxy map: {}
[debug] Using fake IP 25.139.171.87 (GB) as X-Forwarded-For.
[ITV] 2a5460a0042: Downloading webpage
[ITV] 2a5460a0042: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "/Applications/Get iPlayer Automator.app/Contents/Resources/youtube-dl/youtube_dl/extractor/common.py", line 634, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/Applications/Get iPlayer Automator.app/Contents/Resources/youtube-dl/youtube_dl/YoutubeDL.py", line 2279, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 435, in open
response = meth(req, response)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 548, in http_response
'http', request, response, code, msg, hdrs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 473, in error
return self._call_chain(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 556, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
youtube-dl finished downloading
AppController: Downloads Finished
You cannot download episodes that are not available to mobile devices, as is the case here. If you have a suitable phone, check the episode there first. If it isn't available on your phone, it's a good bet it won't be available to download.
OK, but does that make sense given I downloaded the episodes before and after (just checking)?
I've come to this party a little late, but my ITV downloads have also stopped working, but I haven't updated GIA, still running 1.21.2 on El Capitan. It was working fine for ITV last I checked, but I don't download from them that often. Has this issue somehow affected older un-updated versions too?
hi @pmhird I just downloaded Endeavour s08e02 just fine.
I am running GiA 1.21.12 on Big Sur.
I've now updated to 1.21.12 on El Capitan. BBC still working absolutely fine, but I can't get anything to succeed from ITV. With the latest update to 1.21.12 do I still need ot re-do the little Python 2.7 trick outlined in issue 277?
Can anyone help with this? It seems to be failing at the "Downloading m3u8 information" stage, suggesting it can't find a corresponding mobile format episode, but I checked and I can watch the episode (Hollington Drive S01E01) on both web browser and itvhub app. What is going on? It seems to affect all ITV shows for me, can't get anything to download at all. BBC both TV and radio are fine, but seem to take a lot longer to download after 1.21.12? I'm using El Capitan. Thanks in advance, here's the relevant bit of the log where the ITV downloads fail:
[ITV] 10a0552a0001: Downloading m3u8 information WARNING: Failed to download m3u8 information: <urlopen error [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590)> ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. Traceback (most recent call last): File "/Applications/Get iPlayer Automator-5.app/Contents/Resources/youtube-dl/youtube_dl/YoutubeDL.py", line 806, in wrapper return func(self, *args, **kwargs) File "/Applications/Get iPlayer Automator-5.app/Contents/Resources/youtube-dl/youtube_dl/YoutubeDL.py", line 827, in __extract_info ie_result = ie.extract(url) File "/Applications/Get iPlayer Automator-5.app/Contents/Resources/youtube-dl/youtube_dl/extractor/common.py", line 534, in extract ie_result = self._real_extract(url) File "/Applications/Get iPlayer Automator-5.app/Contents/Resources/youtube-dl/youtube_dl/extractor/itv.py", line 121, in _real_extract self._sort_formats(formats) File "/Applications/Get iPlayer Automator-5.app/Contents/Resources/youtube-dl/youtube_dl/extractor/common.py", line 1374, in _sort_formats raise ExtractorError('No video formats found') ExtractorError: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. youtube-dl finished downloading AppController: Downloads Finished
I donβt use El Capitan, but just to confirm, are you using a VPN?
Yes I am on a VPN, but the same VPN works fine for BBC and also for viewing ITV in a web browser or mobile app...
Hi, downloads from itv still stop almost instantly after starting and fail.
INFO: Loading proxy settings... INFO: No proxy to load INFO: Proxy load complete. INFO: No proxy will be used AppController: Starting Downloads
Downloading Show 1/1:
Downloading Superstore: Series 6 INFO: Preparing Request for Auth Info INFO: Metadata processed. youtube-dl finished downloading AppController: Downloads Finished