Problems proxying HTTPS through an HTTP proxy

[jason0x43]

Intern 4.x and 5-pre may have problems proxying HTTPS connections through an HTTP proxy. This axios PR suggests that some configuration may be required to achieve this.

Verify that HTTPS connections through HTTP proxies are failing, and if so, provide a way to tell Intern to tell Axios that a proxy supports HTTPS.

[jason0x43]

A fix for this should also be back ported to https://github.com/theintern/common.

[jason0x43]

@theintern/common@2.4.0-beta.0 has been published with a possible fix. To test it, npm install @theintern/common@2.4.0-beta.0 and manually remove node_modules/intern/node_modules/@theintern/common, then try running tests.

[jason0x43]

This is likely related to theintern/digdug#65

[amrtn]

Hi, here are detailed answers to the questions you posted on gitter

I was hoping the Axios proxy protocol flag would help out here. So, just to be sure I know what's going on...

• You're trying to make requests to https URLs through an http proxy

That's correct

• The proxy requires authentication
• Using @theintern/common@0.2.3 works through a local http proxy (and you're still making requests to https URLs)

Yes, common@0.2.3 works with the local http proxy and making requests to https URLS. This is my actual test:

Configure intern to use local Fiddler (http://localhost:8888 that needs no authentication)

"tunnelOptions": {
  "proxy": "http://localhost:8888"
}

Execute

$ rm -rf node_modules
$ npm install
$ npm ls @theintern/common
dojo-intern-example@0.2.0 D:\AMO\dev\learn\intern\intern-examples\dojo-example
`-- intern@4.8.0
  +-- @theintern/common@0.2.3 
  +-- @theintern/digdug@2.4.3
  | `-- @theintern/common@0.2.3
  `-- @theintern/leadfoot@2.3.0
    `-- @theintern/common@0.2.3

$ npm test

Intern performs this http request and executes the tests successfully.

curl -k -i --raw -o 0.dat "https://selenium-release.storage.googleapis.com/3.141/selenium-server-standalone-3.141.59.jar" -H "Accept: application/json, text/plain, */*" -H "User-Agent: axios/0.19.2" -H "host: selenium-release.storage.googleapis.com" -H "Connection: close"

• With 0.2.3, making an HTTPS request through the authenticating HTTP proxy gives you an error like Error: [GET https://selenium-release.storage.googleapis.com/3.5/selenium-server-standalone-3.5.2.jar] socket hang up

That's correct. This is my actual test:

Configure intern to use the external proxy

"tunnelOptions": {
  "proxy": "http://domain%5Cusername:password@proxy.zzz.com:3128"
}

Execute

$ rm -rf node_modules
$ npm install
$ npm ls @theintern/common
dojo-intern-example@0.2.0 D:\AMO\dev\learn\intern\intern-examples\dojo-example
`-- intern@4.8.0
  +-- @theintern/common@0.2.3 
  +-- @theintern/digdug@2.4.3
  | `-- @theintern/common@0.2.3
  `-- @theintern/leadfoot@2.3.0
    `-- @theintern/common@0.2.3

$ npm test
> dojo-intern-example@0.2.0 test D:\AMO\dev\learn\intern\intern-examples\dojo-example
> intern

Listening on localhost:9000 (ws 9001)
(ノಠ益ಠ)ノ彡┻━┻
Error: socket hang up
  at createHangUpError @ _http_client.js:323:15
  at Socket.socketOnEnd @ _http_client.js:426:23
  at Socket.emit @ events.js:203:15
  at endReadableNT @ _stream_readable.js:1145:12
  at process._tickCallback @ internal\process\next_tick.js:63:19

npm ERR! Test failed.  See above for more details.

• Using @theintern/common@0.2.4-beta.0doesn't work through the local http proxy or the external one
• With 0.2.4-beta.0, making an HTTPS request through the authenticating HTTP proxy gives you an error like Error: Client network socket disconnected before secure TLS connection was established

That's correct. This is my actual test:

Configure intern to use the external (authenticating) proxy

"tunnelOptions": {
  "proxy": "http://domain%5Cusername:password@proxy.zzz.com:3128"
}

Add "@theintern/common": "^0.2.4-beta.0", to the devDependencies in package.json

$ rm -rf node_modules
$ npm install
$ rm -rf node_modules\intern\node_modules\@theintern\common\
$ rm -rf node_modules\@theintern\digdug\node_modules\@theintern\common
$ rm -rf node_modules\@theintern\leadfoot\node_modules\@theintern\common
$ npm ls @theintern/common
dojo-intern-example@0.2.0 D:\AMO\dev\learn\intern\intern-examples\dojo-example
+-- @theintern/common@0.2.4-beta.0 
`-- intern@4.8.0
  +-- UNMET DEPENDENCY @theintern/common@0.2.3
  +-- @theintern/digdug@2.4.3
  | `-- UNMET DEPENDENCY @theintern/common@0.2.3
  `-- @theintern/leadfoot@2.3.0
    `-- UNMET DEPENDENCY @theintern/common@0.2.3

npm ERR! missing: @theintern/common@0.2.3, required by intern@4.8.0
npm ERR! missing: @theintern/common@0.2.3, required by @theintern/digdug@2.4.3
npm ERR! missing: @theintern/common@0.2.3, required by @theintern/leadfoot@2.3.0

$ npm test
> dojo-intern-example@0.2.0 test D:\AMO\dev\learn\intern\intern-examples\dojo-example
> intern

Listening on localhost:9000 (ws 9001)
(ノಠ益ಠ)ノ彡┻━┻
Error: Client network socket disconnected before secure TLS connection was established
  at TLSSocket.onConnectEnd @ _tls_wrap.js:1095:19
  at Object.onceWrapper @ events.js:286:20
  at TLSSocket.emit @ events.js:203:15
  at endReadableNT @ _stream_readable.js:1145:12
  at process._tickCallback @ internal\process\next_tick.js:63:19

npm ERR! Test failed.  See above for more details.

• With 0.2.4-beta.0 and a local non-autheticating HTTP proxy, making an HTTPS request gives you an error like Error: write EPROTO 13732:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:c:\ws\deps\openssl\openssl\ssl\record\ssl3_record.c:332:

That's correct. This is my actual test:

Configure intern to use the local proxy

"tunnelOptions": {
  "proxy": "http://localhost:8888"
}

Add "@theintern/common": "^0.2.4-beta.0", to the devDependencies in package.json

$ rm -rf node_modules
$ npm install
$ rm -rf node_modules\intern\node_modules\@theintern\common\
$ rm -rf node_modules\@theintern\digdug\node_modules\@theintern\common
$ rm -rf node_modules\@theintern\leadfoot\node_modules\@theintern\common
$ npm ls @theintern/common
dojo-intern-example@0.2.0 D:\AMO\dev\learn\intern\intern-examples\dojo-example
+-- @theintern/common@0.2.4-beta.0 
`-- intern@4.8.0
  +-- UNMET DEPENDENCY @theintern/common@0.2.3
  +-- @theintern/digdug@2.4.3
  | `-- UNMET DEPENDENCY @theintern/common@0.2.3
  `-- @theintern/leadfoot@2.3.0
    `-- UNMET DEPENDENCY @theintern/common@0.2.3

npm ERR! missing: @theintern/common@0.2.3, required by intern@4.8.0
npm ERR! missing: @theintern/common@0.2.3, required by @theintern/digdug@2.4.3
npm ERR! missing: @theintern/common@0.2.3, required by @theintern/leadfoot@2.3.0

$ npm test
> dojo-intern-example@0.2.0 test D:\AMO\dev\learn\intern\intern-examples\dojo-example
> intern

Listening on localhost:9000 (ws 9001)
(ノಠ益ಠ)ノ彡┻━┻
Error: write EPROTO 18592:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:c:\ws\deps\openssl\openssl\ssl\record\ssl3_record.c:332:

  at WriteWrap.afterWrite [as oncomplete] @ net.js:788:14

npm ERR! Test failed.  See above for more details.

No requests are logged in Fiddler.

[jason0x43]

Using Intern 4.8, common 0.2.3, and mitmproxy running on http://localhost:8080, test runs fail with

Error: Download server returned status code 400 for https://chromedriver.storage.googleapis.com/79.0.3945.36/chromedriver_mac64.zip
  @ node_modules/src/Tunnel.ts:400:21
  @ node_modules/@theintern/common/index.js:670:29
  at processTicksAndRejections @ internal/process/task_queues.js:97:5

The mitm log indicates a protocol error:

  -> HTTP protocol error in client request: Invalid request scheme: https

It looks like Axios may be making a request to https://<proxyhost> rather than http://proxyhost.

[jason0x43]

Or that could just be a problem with mitm proxy. Proxies are kind of a pain. 😄

[jason0x43]

Well, after making a small modification to mitmproxy (it's a bit picky about URL schemas) by commenting out this block, connections to https addresses go through when using @theintern/common@0.2.3.

[jason0x43]

That doesn't actually do much to solve the problem, but it suggests that the issue is with specific proxy configurations rather than proxy support being generally broken.