api/v1/fleet/mdm/apple/enqueue fails to decode base64 encoded command if padding characters (=) are included

[zwinnerman-fleetdm]

Fleet version: (head to the "My account" page in the Fleet UI or run fleetctl --version)

Operating system: (e.g. macOS 11.2.3)

Web browser: (e.g. Chrome 88.0.4324)

---

🧑‍💻  Expected behavior

When I send a base64 encoded command to api/v1/fleet/mdm/apple/enqueue endpoint, I expect it to enqueue the command whether or not the base64 string has padding characters.

💥  Actual behavior

I receive a 500 error with the following response:

{
    "message": "illegal base64 data at input byte 187",
    "errors": [
        {
            "name": "base",
            "reason": "illegal base64 data at input byte 187"
        }
    ]
}

👣 Reproduction steps

More info

https://observability.kb.us-east-2.aws.elastic-cloud.com:9243/app/apm/services/harmonize-stg/transactions/view?kuery=http.response.status_code%20%3E%3D%20500&rangeFrom=now-15m&rangeTo=now&environment=ENVIRONMENT_ALL&serviceGroup=&comparisonEnabled=true&transactionName=POST%20%2Fapi%2Fv1%2Ffleet%2Fmdm%2Fapple%2Fenqueue&transactionType=request&offset=1d&latencyAggregationType=avg&score=0&timestamp=2023-04-27T17:34:12.636Z&transactionId=22ed4973aff60366&traceId=22ed4973aff60366760dddf7e6983066

level=error ts=2023-04-27T16:53:05.974667235Z component=http user=hmz-global-fleet-api-user@harmonize-dev.io method=POST uri=/api/v1/fleet/mdm/apple/enqueue took=2.114017ms err="decode base64 command: illegal base64 data at input byte 187"

Removing the padding characters successfully submits the string, but that is an invalid base64 string.

Additionally, the Configuration Profile Batch replacement API also accepts base64 encoded strings and seems to not have this problem.

[zwinnerman-fleetdm]

https://fleetdm.slack.com/archives/C04U3UWSVB7/p1682618305348269?thread_ts=1682617620.527039&cid=C04U3UWSVB7

[jrreed]

So I think there's two issues in here as we were trying to figure out how exactly to Base64 encode the command payload.

URL-Safe Base64 encoding: decode base64 command: illegal base64 data at input byte 187

We started by trying to Base64 encode the payload using the URL-safe standard, which is what produced the error reported above and we received the following response body:

{
    "message": "illegal base64 data at input byte 187",
    "errors": [
        {
            "name": "base",
            "reason": "illegal base64 data at input byte 187"
        }
    ]
}

It's probably worth calling this out in the API docs if it's not intended to be supported.

Padded Base64 encoding: illegal base64 data at input byte 819

We then tried to Base64 encode the payload using the non-URL-safe standard, which pads the output by default and this produced the following error in the response body:

{
    "message": "illegal base64 data at input byte 819",
    "errors": [
        {
            "name": "base",
            "reason": "illegal base64 data at input byte 819"
        }
    ]
}

The command value was 820 bytes long. The 819th character was not padding, and the 820th character was an = padding character.

Trimming the trailing = padding character off and resubmitting caused the request to succeed, but this also creates an invalid Base64 encoding result... 😕

Notes

We're using the standard Ruby Base64 encoding library:

require 'base64'
command = '<XML_COMMAND_STRING>'

#
# Produces `illegal base64 data at input byte 187`
#
invalid_api_request_data = Base64.urlsafe_encode64(command)
invalid_api_request_data = Base64.urlsafe_encode64(command, padding: false)

#
# Produces `illegal base64 data at input byte 819`
#
invalid_api_request_data = Base64.strict_encode64(command)
Base64.strict_decode64(invalid_api_request_data) == command # returns true ✅ 

#
# Produces an HTTP-2XX response from the server
#
valid_api_request_data = Base64.strict_encode64(command).chomp('=')
Base64.strict_decode64(valid_api_request_data) == command # raises an invalid base64 (ArgumentError) 💥 

[lukeheath]

@georgekarrv Moving this to the MDM board. Please make sure this gets addressed in this sprint.

[jrreed]

Hi @lukeheath and @zwinnerman-fleetdm!

I'm just now noticing that the way this is reported, it looks like you are expecting to return an HTTP-400 if the command has a base64 encoded string that has trailing = padding characters. Am I understanding that correctly?

So this change is just going to convert the HTTP-500 to and HTTP-400?

And actually, this is what we're seeing now when we submit a base64 encoded command that has trailing padding = characters

POST https://harmonize-stg.cloud.fleetdm.com/api/v1/fleet/mdm/apple/enqueue
Accept: "application/json"
User-Agent: "Faraday v1.10.3"
Authorization: "Bearer REDACTED"
Content-Type: "application/json"
{
  "device_ids":["A04F07D9-0AB0-5682-B99E-996F178A707E"],
  "command":"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KCjwhRE9DVFlQRSBwbGlzdCBQVUJMSUMgIi0vL0FwcGxlLy9EVEQgUExJU1QgMS4wLy9FTiIgImh0dHA6Ly93d3cuYXBwbGUuY29tL0RURHMvUHJvcGVydHlMaXN0LTEuMC5kdGQiPgoKPHBsaXN0IHZlcnNpb249IjEuMCI+CiAgPGRpY3Q+CiAgICA8a2V5PkNvbW1hbmQ8L2tleT4KICAgIDxkaWN0PgogICAgICA8a2V5PlJlcXVlc3RUeXBlPC9rZXk+CiAgICAgIDxzdHJpbmc+RGV2aWNlTG9jazwvc3RyaW5nPgogICAgICA8a2V5PlBJTjwva2V5PgogICAgICA8c3RyaW5nPjgyNzE3Nzwvc3RyaW5nPgogICAgPC9kaWN0PgogICAgPGtleT5Db21tYW5kVVVJRDwva2V5PgogICAgPHN0cmluZz44YjhlODRjYi1mOWM2LTRiMTEtYmU3YS0xMDE5YWQ4ZDU1NDk8L3N0cmluZz4KICA8L2RpY3Q+CjwvcGxpc3Q+Cg=="
}
Status 400
date: "Wed, 23 Aug 2023 22:00:33 GMT"
content-type: "application/json; charset=utf-8"
content-length: "143"
connection: "keep-alive"
{
  "message": "Validation Failed",
  "errors": [
    {
      "name": "command",
      "reason": "unable to decode base64 command"
    }
  ]
}

What is the issue with having = padding characters in a base64 encoded string? That looks like a valid base64 encoding to me, or am I missing something? https://stackoverflow.com/questions/4080988/why-does-base64-encoding-require-padding-if-the-input-length-is-not-divisible-by

Additionally, the Configuration Profile Batch replacement API also accepts base64 encoded strings and seems to not have this problem. So it seems like theres some inconsistency between these two.

Thanks!

[dherder]

@lukeheath I experienced this as well when executing base64 encoded commands. Essentially it seemed like the padding (==) at the end of the base64 encoded string was not something that is needed, which is a bit odd. I worked around this by manually stripping the padding. This seems to be the same issue?

[lukeheath]

@jrreed Thanks for following up! The infrastructure team initially reported this bug because it was triggering a 500. If it is really an invalid payload, we want to return a 4xx error. However, as you outlined, the more significant issue is that it's failing to decode base64 encoded commands with padding. I have updated the bug report to reflect the actual issue that you described. Thanks for including so much additional information - that's very helpful!

I'm sorry for the delay on this. I intended this to be prioritized months ago, but it was mislabeled and slipped off the radar. I'm escalating to the MDM team now. This should be a straightforward fix.

@dherder You're correct, it looks like the same problem you encountered.

[sabrinabuckets]

Able to successfully send a POST command to the enqueue endpoint, with a base64 encoded command that included trailing padding, received a 200 response.

All requests sent with invalid data or formatting triggered a 4xx response.

[fleet-release]

Padding characters fail, Base64 now prevails, Fleet's strength in details.