Proxy parameter is not working correctly inside of openload.py (PhantomJS)

xTended2600 commented 6 years ago

Please follow the guide below

You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your issue (like this: [x])
Use the Preview tab to see what your issue will actually look like

Make sure you are using the latest version: run `youtube-dl --version` and ensure your version is 2017.12.14. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

[x] I've verified and I assure that I'm running youtube-dl 2017.12.14

Before submitting an issue make sure you have:

[x] At least skimmed through the README, most notably the FAQ and BUGS sections
[x] Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

[x] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

Currently openload.py is using PhantomJS in order the obtain the final URL, but it is not passing correctly the parameter --proxy URL to PhantomJS. Output is: https://openload.co/stream/None?mime=true In the Proxy Server I could see only one https call to openload servers, missing the rest of them which are being originated in openload.py lines 228-231:

    p = subprocess.Popen([
        self.exe, '--ssl-protocol=any',
        self._TMP_FILES['script'].name
    ], stdout=subprocess.PIPE, stderr=subprocess.PIPE)

Here we can see that the only parameter used for PhantomJS is "--ssl-protocol=any". I have tested it in the latest version (2017.12.14) and the issue is still there.

yan12125 commented 6 years ago

Notices for those who are interesting:

PhantomJS is missing quite a few proxy features available in youtube-dl, so this feature will be rather limited. Missing functions include:
- Different proxies for http and https pages; PAC may solve it (https://github.com/ariya/phantomjs/issues/10834)
- Support for NO_PROXY (https://github.com/ariya/phantomjs/issues/12407)
- Support for socks4 (https://github.com/ariya/phantomjs/issues/13797)
--ignore-ssl-errors=true should be fed to PhantomJS if --no-check-certificate is fed to youtube-dl. This is useful for proxies like mitmproxy

xTended2600 commented 6 years ago

I agree with you that some additional parameters should be forwarded to PhantomJS.

Although, it is not necessary a different proxy for HTTP and HTTPS requests. A HTTP proxy server can retrieve SSL content. Basically, difference between a HTTP and HTTPS proxy sever lies in the communication (Plain HTTP or SSL) between the client and the proxy server.

Regarding to original issue, I made some changes locally in openload.py and utils.py, and now it is working fine, but this code is not enough clean to merge it into master.

Regards.

yan12125 commented 6 years ago

it is not necessary a different proxy for HTTP and HTTPS requests

Totally correct. I have never see an actual use case with different proxies; this is a design issue - what youtube-dl should pass to phantomjs if there were really different proxies? Report an error or pick one of them?

In the long term, I'm wondering if it's possible to replace PhantomJS with maintained projects like headless Chrome. For former is declared dead by former maintainers (https://github.com/ariya/phantomjs/issues/15105), and the latter supports proxies much better.

ytdl-org / youtube-dl