HTTP library does not yet support proxy servers

[ghost]

Emacs version: 28.1 OS version: macOS 12.4 ement.el version: latest

My code:

(ement-connect
  :user-id "@username:converser.eu"
  :password "<mypassword>"
  :uri-prefix "https://converser.eu/")

I got error in process sentinel: JSON readtable error: 72, how can I login on this homeserver?

[alphapapa]

That error suggests that the server is not responding with a JSON object, which suggests that that is not the correct URI prefix for the server.

Anyway, why do you say that it doesn't have a .well-known URI? When I request https://converser.eu/.well-known/matrix/client I get:

{
  "m.homeserver": {
    "base_url": "https://converser.eu"
  },
  "m.identity_server": {
    "base_url": "https://vector.im"
  }
}

And when I request https://converser.eu/_matrix/client/r0/login, I get:

{"flows":[{"type":"m.login.password"},{"type":"m.login.application_service"},{"type":"uk.half-shot.msc2778.login.application_service"}]}

And when I try to connect to it with ement-connect (using an invalid username and password), everything proceeds normally and I get the expected "invalid password" error.

So everything appears to be in order.

[ghost]

Anyway, why do you say that it doesn't have a .well-known URI?

Because it prompts Auto-discovery of server's well-known URI failed. Input server hostname, or leave blank to use server name:

which suggests that that is not the correct URI prefix for the server.

So, What is the correct URI prefix for this homeserver?

[alphapapa]

I don't know what the problem is. As I said, it works for me.

Try evaluating:

(plz 'get (concat "https://" "converser.eu" "/.well-known/matrix/client")
     :as 'response :then 'sync)

[ghost]

@alphapapa Reproduction steps:

1. M-x ement-connect
2. User-ID: @username:converser.eu
3. Password: <mypassword>
4. It prompts: Auto-discovery of server's well-known URI failed. Input server hostname, or leave blank to use server name:
5. If I press enter, I got an error:

   error in process sentinel: ement-api-error: Ement API error: "3: URL malformed. The syntax was not correct."
   error in process sentinel: Ement API error: "3: URL malformed. The syntax was not correct."

   If I type https://converser.eu，it appears error in process sentinel: JSON readtable error: 72.

[alphapapa]

Please evaluate the expression in my previous message and show me the result.

[ghost]

I tried other homeservers (matrix.org and kde.org) and got same errors and couldn't login. I have added some information at the beginning.

[alphapapa]

I've asked you twice to evaluate the expression in https://github.com/alphapapa/ement.el/issues/77#issuecomment-1152075554 and show me the result. If you won't do that, I can't help you.

[ghost]

@alphapapa Works as expected.

(plz-response 1.1 200 nil "HTTP/2 200 
server: nginx
date: Sat, 11 Jun 2022 16:34:22 GMT
content-type: application/json
last-modified: Thu, 06 Aug 2020 12:59:32 GMT
vary: Accept-Encoding
etag: W/\"5f2bfeb4-85\"
access-control-allow-origin: *
content-encoding: gzip

{
  \"m.homeserver\": {
    \"base_url\": \"https://converser.eu\"
  },
  \"m.identity_server\": {
    \"base_url\": \"https://vector.im\"
  }
}
")

[alphapapa]

Actually, that is not what is expected. The headers slot is nil, and the body slot contains the headers and body together. As well, the headers there show HTTP/2, while the version slot shows HTTP 1.1. So something is amiss. What version of Curl are you using?

[ghost]

macOS preinstalled curl

/usr/bin/curl -V
curl 7.79.1 (x86_64-apple-darwin21.0) libcurl/7.79.1 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.11 nghttp2/1.45.1
Release-Date: 2021-09-22
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL UnixSockets

curl installed from Homebrew

curl 7.83.1 (x86_64-apple-darwin21.3.0) libcurl/7.83.1 (SecureTransport) OpenSSL/1.1.1o zlib/1.2.11 brotli/1.0.9 zstd/1.5.2 libidn2/2.3.2 libssh2/1.10.0 nghttp2/1.47.0 librtmp/2.3 OpenLDAP/2.6.2
Release-Date: 2022-05-11
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL TLS-SRP UnixSockets zstd

I tried both versions (specified by PATH) with the same result.

[alphapapa]

Please try adding "--http1.1" to plz-curl-default-args, e.g. (cl-pushnew "--http1.1" plz-curl-default-args :test #'equal). That should prevent Curl from using HTTP/2 and it might solve the problem.

[ghost]

I added --http1.1 to the list, but the issue is still not solved. I don't think this issue is related to HTTP/2, because I had same result in Debian 11.

HTTP/2 200 
server: nginx
date: Tue, 14 Jun 2022 08:55:37 GMT
content-type: application/json
last-modified: Thu, 06 Aug 2020 12:59:32 GMT
vary: Accept-Encoding
etag: W/\"5f2bfeb4-85\"
access-control-allow-origin: *
content-encoding: gzip

{
  \"m.homeserver\": {
    \"base_url\": \"https://converser.eu\"
  },
  \"m.identity_server\": {
    \"base_url\": \"https://vector.im\"
  }
}
"

The curl version:

curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/1.1.1n zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.3.0) libssh2/1.9.0 nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.4.59
Release-Date: 2022-01-05
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd

Maybe this issue is related to the *_proxy env, because my network is behind a proxy.

I'll close this issue for now, because I found a workaround: Pantalaimon, it was introduced in the README.

[alphapapa]

I don't think this issue is related to HTTP/2, because I had same result in Debian 11.

Well, I don't know exactly what is that code block that followed, but it appears to have the same problem, that the whole response is in the body struct slot, which would indicate that plz is not parsing the HTTP response properly.

Maybe this issue is related to the *_proxy env, because my network is behind a proxy.

Yes, that seems likely to be related to the problem. Since HTTPS is being used, does that mean that your proxy is intercepting and rewriting HTTPS requests?

I'll close this issue for now, because I found a workaround: Pantalaimon, it was introduced in the README.

Okay then. I haven't used proxies in a long time, and I haven't tested plz or ement with them, so if you feel like helping to solve any problems with them, let me know.

[ghost]

Maybe this issue is related to the *_proxy env, because my network is behind a proxy.

Yes, that seems likely to be related to the problem. Since HTTPS is being used, does that mean that your proxy is intercepting and rewriting HTTPS requests?

I have confirmed that the problem only occurs when using a proxy, so I reopened the issue. My proxy does not require a self-signed SSL certificate to be installed, the answer is no.

This is the verbose output of curl:

$ curl -v https://converser.eu/.well-known/matrix/client
* Uses proxy env variable https_proxy == 'http://192.168.0.2:8080'
*   Trying 192.168.0.2:8080...
* Connected to 192.168.0.1 (192.168.0.2) port 8080 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to converser.eu:443
> CONNECT converser.eu:443 HTTP/1.1
> Host: converser.eu:443
> User-Agent: curl/7.79.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=matrix.6clones.net
*  start date: Jun  5 16:12:14 2022 GMT
*  expire date: Sep  3 16:12:13 2022 GMT
*  subjectAltName: host "converser.eu" matched cert's "converser.eu"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fb2a280b600)
> GET /.well-known/matrix/client HTTP/2
> Host: converser.eu
> user-agent: curl/7.79.1
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< server: nginx
< date: Sat, 18 Jun 2022 12:36:02 GMT
< content-type: application/json
< content-length: 133
< last-modified: Thu, 06 Aug 2020 12:59:32 GMT
< vary: Accept-Encoding
< etag: "5f2bfeb4-85"
< access-control-allow-origin: *
< accept-ranges: bytes
<
{
  "m.homeserver": {
    "base_url": "https://converser.eu"
  },
  "m.identity_server": {
    "base_url": "https://vector.im"
  }
}
* Connection #0 to host 196.168.0.2 left intact

[alphapapa]

@ghost Why did you close this issue? Does it not remain a problem?

Probably the issue is related to plz and the way it works with curl--maybe proxy support needs to be implemented there.

In any case, please don't close issues that remain unsolved.

...Oh, I see... ghost is just a placeholder. So the user that filed this issue apparently closed the issue and then deleted his GitHub account. sigh

[arstum]

@alphapapa Reproduction steps:

1. `M-x ement-connect`

2. User-ID: `@username:converser.eu`

3. Password: `<mypassword>`

4. It prompts: `Auto-discovery of server's well-known URI failed.  Input server hostname, or leave blank to use server name: `

5. If I press enter, I got an error:

error in process sentinel: ement-api-error: Ement API error: "3: URL malformed. The syntax was not correct."
error in process sentinel: Ement API error: "3: URL malformed. The syntax was not correct."

If I type https://converser.eu，it appears error in process sentinel: JSON readtable error: 72.

same situation as me

[alphapapa]

@arstum Are you also using a proxy like the reporter of this issue?

As I asked the reporter, please evaluate this expression and share the result:

(plz 'get (concat "https://" "converser.eu" "/.well-known/matrix/client")
     :as 'response :then 'sync)

[sawyerzheng]

@alphapapa Same problem, using latest commit: f721fe3fb408bc28a7dbcb296226d834fd2304e6

1. raw running

   (plz 'get (concat "https://" "converser.eu" "/.well-known/matrix/client")
   :as 'response :then 'sync)
   #s(plz-response 2 200 ((server . "nginx") (date . "Fri, 09 Dec 2022 01:06:04 GMT") (content-type . "application/json") (last-modified . "Thu, 06 Aug 2020 12:59:32 GMT") (vary . "Accept-Encoding") (etag . "W/\"5f2bfeb4-85\"") (access-control-allow-origin . "*") (content-encoding . "gzip")) "{
   \"m.homeserver\": {
     \"base_url\": \"https://converser.eu\"
   },
   \"m.identity_server\": {
     \"base_url\": \"https://vector.im\"
   }
   }
   ")

2. after add --http1.1 with: (cl-pushnew "--http1.1" plz-curl-default-args :test #'equal)

   (plz 'get (concat "https://" "converser.eu" "/.well-known/matrix/client")
   :as 'response :then 'sync)
   
   #s(plz-response 1.1 200 ((server . "nginx") (date . "Fri, 09 Dec 2022 01:58:02 GMT") (content-type . "application/json") (last-modified . "Thu, 06 Aug 2020 12:59:32 GMT") (transfer-encoding . "chunked") (connection . "keep-alive") (vary . "Accept-Encoding") (etag . "W/\"5f2bfeb4-85\"") (access-control-allow-origin . "*") (content-encoding . "gzip")) "{
   \"m.homeserver\": {
   \"base_url\": \"https://converser.eu\"
   },
   \"m.identity_server\": {
   \"base_url\": \"https://vector.im\"
   }
   }
   ")

3. after add --http1.1 and enable proxy:

   
   (plz 'get (concat "https://" "converser.eu" "/.well-known/matrix/client")
   :as 'response :then 'sync)
   #s(plz-response 1.1 200 nil "HTTP/1.1 200 OK
   Server: nginx
   Date: Fri, 09 Dec 2022 02:01:30 GMT
   Content-Type: application/json
   Last-Modified: Thu, 06 Aug 2020 12:59:32 GMT
   Transfer-Encoding: chunked
   Connection: keep-alive
   Vary: Accept-Encoding
   ETag: W/\"5f2bfeb4-85\"
   Access-Control-Allow-Origin: *
   Content-Encoding: gzip

{ \"m.homeserver\": { \"base_url\": \"https://converser.eu\" }, \"m.identity_server\": { \"base_url\": \"https://vector.im\" } } ")

[alphapapa]

This appears to be this issue in plz: https://github.com/alphapapa/plz.el/issues/2

[alphapapa]

Please see https://github.com/alphapapa/plz.el/commit/74536c5396abe6be1691193dc3c816a2a73d4655, which adds support for the proxy headers output by Curl. (Configuration is yet to be written, but you can add the argument to plz-curl-default-args manually while testing. Your input on that would be welcome on https://github.com/alphapapa/plz.el/issues/2.)

[alphapapa]

Since plz supports proxies now, I'm closing this.

As I said, configuration/customization is yet to be written, and I would welcome input from users who actually use proxy servers to help design that. Please chime in on https://github.com/alphapapa/plz.el/issues/37

[shymega]

Hey.

Just to clarify, by proxy, are we talking about E2EE proxies like Pantalaimon? I can provide the output you need, if that's the case. But seeing as Pantalaimon uses olm, which has been replaced, I need to hack on my own.

If we're not talking about E2EE proxies, I'll open a separate issue - I'm having a json-readtable error.

[alphapapa]

@shymega This issue is about users who need to use a mandatory, global HTTP proxy for everything; not Pantalaimon-style, MITM, E2EE proxies for Matrix clients.

If you're working on a replacement for Pantalaimon using vodozemac, that would be much appreciated! You can find interested users in the Pantalaimon-related issues here and in the #ement.el:matrix.org room.