malformed MIME header

[LarsBel]

I noticed after upgrading to the latest version that push isnt working anymore. I checked the database.json and I could see that last my phone date was 2 days before (when I updated the software) So I started clean and deleted all old files and build the lastest again..

When starting the service it now instantly fails with: xapsd[994]: 2023/03/24 10:24:30 Post "https://identity.apple.com/pushcert/caservice/new": net/http: HTTP/1.x transport connection broken: malformed MIME header line: 1;: mode=block

[freswa]

We can't do much about the issue with the Apple Server. Either they'll fix it soon or the service may vanish entirely. We'll never know for sure. Do you still have logs from the time before you deleted all old files? What commit do you mean by "latest version"?

[LarsBel]

I mean your latest commit https://github.com/freswa/dovecot-xaps-daemon/commit/4c6c9095db70e8f99a0ef6ca76fb3fbfc628fbe8

To me it seems that the daemon is not parsing the certificate correctly. I compiled the latest version and started it. I got tons of mails from apple that new certificates have been created, but the database.json is empty.

I found an old version of the binary in my backups, starting it, the certificate got parsed in database.json

[rsplaul]

Same here. An older version of xapsd stopped working a couple of month ago (for some unrelated reason I think), and I reconfigured everything just today, only to find out that I’m getting the same error message – and a bunch of success mails from Apple.

I think this has to do with the header that Apple’s server is generating:

$ curl -D - 'https://identity.apple.com/pushcert/caservice/renew'
HTTP/1.1 200
Server: Apple
Date: Fri, 24 Mar 2023 20:50:58 GMT
Content-Length: 0
Connection: keep-alive
1;: mode=block
max-age=31536000;: includeSubdomains
Strict-Transport-Security: max-age=31536000; includeSubdomains
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Host: identity.apple.com
X-Frame-Options: SAMEORIGIN

I’m not sure if the line 1;: mode=block is a valid HTTP header line at all.

[freultwah]

There are two malformed lines under 'Connection: keep-alive' that are doubled correctly immediately after:

Strict-Transport-Security: max-age=31536000; includeSubdomains X-XSS-Protection: 1; mode=block

[freswa]

For now I'd like to wait for Apple to fix these headers. We're not exclusively using this API endpoint so there will be more people running into it. I hope it's only a matter of days until Apple rolls out a fix. If anyone has problems with their database, please open a separate issue and provide (debug) logs.

[LarsBel]

Using your version from may 2022 works without problems. It gets the certificate.

[freswa]

Can you check 4d4ce185412298dffcefa8aa7016b4b330393bb6 which is one commit before the recent dependency update?

[rsplaul]

Can you check 4d4ce18 which is one commit before the recent dependency update?

At least for me this version doesn’t work either – same error.

[LarsBel]

This seems a go issue.. compile it with go-1.19.7, and it works go-1.20.2 dont

[FreelancerJay]

This seems a go issue.. compile it with go-1.19.7, and it works go-1.20.2 don't

That could be why I didn't run into this when I tried out 4c6c909 originally, and am still not having issues... I'm runing go-1.19

[freswa]

Upstream issue to work around the bug: https://github.com/golang/go/issues/21290

[Rjevski]

In the meantime, I came up with a workaround. The idea is to use mitmproxy to act as a reverse proxy that will strip the offending headers, and then edit the source code to hit this reverse proxy (most likely running on localhost, but doesn't technically have to?) instead.

First create the mitmproxy add-on script remove-malformed-headers.py that will strip out the offending headers:

class RemoveMalformedHeaders:
    def response(self, flow):
        headers_to_remove = [
            header for header in flow.response.headers.keys() if ';' in header
        ]

        for header in headers_to_remove:
            del flow.response.headers[header]

addons = [RemoveMalformedHeaders()]

First you should run mitmproxy with the following command:

mitmproxy --set validate_inbound_headers=false --mode reverse:https://identity.apple.com/ -s remove-malformed-headers.py

Now in the source code, edit pkg/apple_xserver_certs/http.go and replace https://identity.apple.com with http://localhost:8000 (or whatever your mitmproxy is listening on). Then rebuild, and run it as usual.

If all goes well, you should see a request to the Apple server in the mitmproxy console, and xapsd should be happy. Note that this is only needed when xapsd obtains or renews the certificate, so once that's done you can go back to using the stock, unmodified build and shut down mitmproxy.

[leonklingele]

I have created https://github.com/freswa/dovecot-xaps-daemon/pull/36 which adds code to ignore known, malformed HTTP response headers such as the ones returned from Apple.

Can you please verify if the change really works as intended? Unfortunately, I can't verify it myself at the moment.

[pelletierr]

I have created #36 which adds code to ignore known, malformed HTTP response headers such as the ones returned from Apple.

Can you please verify if the change really works as intended? Unfortunately, I can't verify it myself at the moment.

I just tried that version of http.go, but my servers fail to renew or get a new certificate.

The certificate gets created in the portal, but the process crashes.

Jun 25 17:45:57 ServerName xapsd[113517]: 2024/06/25 17:45:57 unexpected EOF Jun 25 17:45:57 ServerName systemd[1]: xapsd.service: Main process exited, code=exited, status=1/FAILURE Jun 25 17:45:57 ServerName systemd[1]: xapsd.service: Failed with result 'exit-code'.

[pelletierr]

I added output printing to see where the error was occurring.

It is this line (119) : respBody, err = ioutil.ReadAll(resp.Body)

There is data in respBody

[pelletierr]

I read that it could be due to gzip compression in the response. I tried disabling HTTP/2 and compression but that didn't work.

[pelletierr]

Found it!

Line buf, err := io.ReadAll(io.LimitReader(conn, 1 << 10))

It's too short, changed to buf, err := io.ReadAll(io.LimitReader(conn, 9223372036854775807)) and it works

[freultwah]

I wonder if anybody has an idea whether issued certificates can be revoked programmatically via API calls. The 'malformed MIME header line' issue that I did not notice on one machine has resulted in 69,876 new certificates having been generated over the course of two months last year, since 28 July up to 20 September, and now I am getting hundreds of thousands of e-mails from Apple regarding their upcoming expiry. I am on day three and it's almost 270,000 by now. The https://identity.apple.com/pushcert/ portal lists zero certs available to be revoked. Apple says they cannot help me at all. Wonder if there's a similar mechanism for revoking via, I don't know, https://identity.apple.com/pushcert/caservice/revoke or something.