Added example for not working REST API in the most recent containers

[egandro]

Hello again,

here an example of the no longer working REST API with the recent container updates:

• Using data from the offical demo: https://www.freeipa.org/page/Demo
• Curl Code taken from: https://vda.li/en/posts/2015/05/28/talking-to-freeipa-api-with-sessions/

export USER="admin"
export PASSWORD="Secret123"
export COOKIEJAR=/tmp/ipa-$$
export IPAHOSTNAME=ipa.demo1.freeipa.org

curl -v \
        -H referer:https://$IPAHOSTNAME/ipa \
        -H "Content-Type:application/x-www-form-urlencoded" \
        -H "Accept:text/plain"\
        -c $COOKIEJAR -b $COOKIEJAR \
        --data "user=$USER&password=$PASSWORD" \
        -X POST https://$IPAHOSTNAME/ipa/session/login_password

if [ ! -f $COOKIEJAR ]; then
    echo "ipa login failed - COOKIEJAR file not found!"
    exit -1
fi

Try this with an instance of the current container - and it fails. I am attaching a dump.

(please note two days ago this worked for about 15 test installations. after the container was updated it fails) - Of course the DNS and the A-Records are propperly configured (it used to work).

Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying xxx.xxx.xxx.xxx ...
* TCP_NODELAY set
* Connected to ipa.hostname-removed.removed (xxx.xxx.xxx.xxx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* NPN, negotiated HTTP2 (h2)
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Unknown (67):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=ipa.hostname-removed.removed
*  start date: Nov 10 21:50:33 2018 GMT
*  expire date: Feb  8 21:50:33 2019 GMT
*  subjectAltName: host "ipa.hostname-removed.removed" matched cert's "ipa.hostname-removed.removed"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x556e9461adc0)
> POST /ipa/session/login_password HTTP/1.1
> Host: ipa.hostname-removed.removed
> User-Agent: curl/7.52.1
> referer:https://ipa.hostname-removed.removed/ipa
> Content-Type:application/x-www-form-urlencoded
> Accept:text/plain
> Content-Length: 30
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* HTTP/2 stream 1 was not closed cleanly: REFUSED_STREAM (err 7)
* Curl_http_done: called premature == 1
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
curl: (92) HTTP/2 stream 1 was not closed cleanly: REFUSED_STREAM (err 7)
ipa login failed - COOKIEJAR file not found!

[egandro]

Ok edit:

This is semi solved :(

Apache is now (wtf.) running on http 2 calls.

You have to add "--http1.1" to curl.

No Idea if it's the container or freeipa.

Can you please document this?

[adelton]

Is the WebUI otherwise working properly? Is this fresh installation or upgrade from older container image? Which container image is this exactly? When you try your code against FreeIPA installed outside of container, does it work?

[adelton]

For the record, the script shown above, when I change the IPAHOSTNAME to the hostname of my Fedora 29-based FreeIPA server container and I add -k to the curl command, works just fine.

[egandro]

Is the WebUI otherwise working properly? Yes

Is this fresh installation or upgrade from older container image?

Yes fresh. I use the same script as in the last 2 months. I did >50 setups, proper SSL Certificates, proper DNS entries

Which container image is this exactly?

freeipa/freeipa-server:fedora-26

More recent Fedora (27,...) Dockered images have other issues (requiring IPv6) this is well documented and I will put a different bug report. But lets stick to this issue.

When you try your code against FreeIPA installed outside of container, does it work?

There is no "outside" world. I found out that it's a http2 call - please check the CURL dump:

"* Using HTTP2, server supports multi-use"

Try the script I provided for the official IPA Demo server. There is no HTTP2 protocol shift:

Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 52.57.162.88...
* TCP_NODELAY set
* Connected to ipa.demo1.freeipa.org (52.57.162.88) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=ipa.demo1.freeipa.org
*  start date: Oct 12 05:41:12 2018 GMT
*  expire date: Jan 10 05:41:12 2019 GMT
*  subjectAltName: host "ipa.demo1.freeipa.org" matched cert's "ipa.demo1.freeipa.org"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
> POST /ipa/session/login_password HTTP/1.1
> Host: ipa.demo1.freeipa.org
> User-Agent: curl/7.52.1
> referer:https://ipa.demo1.freeipa.org/ipa
> Content-Type:application/x-www-form-urlencoded
> Accept:text/plain
> Content-Length: 29
>
* upload completely sent off: 29 out of 29 bytes
< HTTP/1.1 200 Success
< Date: Mon, 26 Nov 2018 23:58:38 GMT
< Server: Apache/2.4.34 (Fedora) OpenSSL/1.1.1 mod_wsgi/4.6.4 Python/3.7 mod_auth_gssapi/1.6.1
* Added cookie ipa_session="MagBearerToken=4ttL%2bhrM2HbCEEZd3BFVK5CzOGHneTYNUz4WvVUBNRPWUuo%2bgDNIrR8RpdDBfO1geAc8We%2biv798OHzkB3Z%2f6Zb8PJK8U8I2CinWVqK6pF35ZCMTNwsGfgwpsMKg79E6TAtIyOgSR0lU94f7IR8torc%2fQU%2bzlFU1o7nkmZcRVIDdn9GnQ9FoFrE7r3lm7iWIMROqHywlrrWCfeD7rX0SpQ%3d%3d" for domain ipa.demo1.freeipa.org, path /ipa, expire 0
< Set-Cookie: ipa_session=MagBearerToken=4ttL%2bhrM2HbCEEZd3BFVK5CzOGHneTYNUz4WvVUBNRPWUuo%2bgDNIrR8RpdDBfO1geAc8We%2biv798OHzkB3Z%2f6Zb8PJK8U8I2CinWVqK6pF35ZCMTNwsGfgwpsMKg79E6TAtIyOgSR0lU94f7IR8torc%2fQU%2bzlFU1o7nkmZcRVIDdn9GnQ9FoFrE7r3lm7iWIMROqHywlrrWCfeD7rX0SpQ%3d%3d;path=/ipa;httponly;secure;
< X-Frame-Options: DENY
< Content-Security-Policy: frame-ancestors 'none'
< Vary: Accept-Encoding
< Cache-Control: no-cache
< Transfer-Encoding: chunked
< Content-Type: text/plain; charset=UTF-8
<
* Curl_http_done: called premature == 0
* Connection #0 to host ipa.demo1.freeipa.org left intact

This is a new behavior since the update on Saturday.

I can deal no with the issue - however - it lacks of some documentation that - for the REST API call for the dockered IPAs http 1.1 is required.

[egandro]

For the record, the script shown above, when I change the IPAHOSTNAME to the hostname of my Fedora 29-based FreeIPA server container and I add -k to the curl command, works just fine.

It's not a SSL Certificate issue. The certificates work fine - in curl - in browsers - in Windows - on Macs. It's just the forced HTTP/2. That's the only difference.

I guess that somebody added this to Apache to be this as default - that needs an active setup:

https://httpd.apache.org/docs/2.4/howto/http2.html

For some reasons FreeIPA Rest don't like it (not further digging into this).

[adelton]

Which container image is this exactly?

freeipa/freeipa-server:fedora-26

[...]

When you try your code against FreeIPA installed outside of container, does it work?

There is no "outside" world. I found out that it's a http2 call - please check the CURL dump:

Since your running theory is that something has changed in the freeipa/freeipa-server:fedora-26 image in the past couple of days, I'd assume that you'd cross-check this with FreeIPA installed on Fedora 26 on some virtual machine, outside of containers, to be sure that the change is really caused by the containerization and not by something else.

"* Using HTTP2, server supports multi-use"

Try the script I provided for the official IPA Demo server. There is no HTTP2 protocol shift:

With curl-7.59.0-7.fc28.x86_64, I don't see HTTP/2 used by ipa.demo1.freeipa.org, nor by any of the containers (:fedora-29, :fedora-26) that I start using docker run, with docker-1.13.1-61.git9cb56fd.fc28.x86_64.

The next question then would be -- how exactly do you run the containers? This looks like there is some HTTPS proxy in front of your containerized FreeIPA setup.

[egandro]

Since your running theory is that something has changed in the freeipa/freeipa-server:fedora-26 image in the past couple of days, I'd assume that you'd cross-check this with FreeIPA installed on Fedora 26 on some virtual machine, outside of containers, to be sure that the change is really caused by the containerization and not by something else.

I run this on a Cloud Server. I create the Cloud Server via docker-machine and kill it via docker machine. I didn't chance a thing :)

"* Using HTTP2, server supports multi-use" Try the script I provided for the official IPA Demo server. There is no HTTP2 protocol shift:

With curl-7.59.0-7.fc28.x86_64, I don't see HTTP/2 used by ipa.demo1.freeipa.org, nor by any of the containers (:fedora-29, :fedora-26) that I start using docker run, with docker-1.13.1-61.git9cb56fd.fc28.x86_64.

I am not aware it they run a Docker image of this site... nor if they updated to this container.

The next question then would be -- how exactly do you run the containers? This looks like there is some HTTPS proxy in front of your containerized FreeIPA setup.

Via nginx-proxy / nginx-letsencrypt-companion.

[adelton]

Is it possible then that the nginx-proxy / nginx-letsencrypt-companion is where the https connection terminates and thus it's this part that changed and that started to support HTTP/2?

[egandro]

Yeah you are right:

Check this: https://github.com/jwilder/nginx-proxy

"jwilder/nginx-proxy:alpine This image is based on the nginx:alpine image. Use this image to fully support HTTP/2 (including ALPN required by recent Chrome versions). A valid certificate is required as well (see eg. below "SSL Support using letsencrypt" for more info)."