Create Server user-data-from-file results in JSON-Error

[sternze]

Hi!

I'm currently trying to create a Server via the following command (inside Powershell):

$SSH_KEY_NAME = "SSH-Key"
$SERVERNAME = "myserver"
$SERVERTYPE = "cpx41"
$IMAGE_ID = "123124124"  ## random Image ID for Github

hcloud server create --name $SERVERNAME --start-after-create=false --datacenter nbg1-dc3 --image $IMAGE_ID --type $SERVERTYPE --without-ipv6 --ssh-key $SSH_KEY_NAME --user-data-from-file .\cloud_config.yaml

The contents of cloud_config.yaml are:

#cloud-config
manage_etc_hosts: false

Currently, I'm receiving the error hcloud: NULL characters are not allowed (json_error)

Running the command without the user-data-from-file-Flag works without any problems.

Creating the server from a snapshot inside the Web-GUI and specifying this cloud-config information manually works also.

Is there anything wrong with my cloud-config-file?

Thanks, Daniel

[apricote]

Hey Daniel,

could you encode the file as base64 and comment the encoded string here? Copying just the text might introduce subtle differences when posted here, and we are interested in the actual bytes in the files to debug this.

[sternze]

Here we go. :)

PS C:\Users\user> [convert]::ToBase64String((Get-Content -path .\cloud_config.yaml -Encoding byte))
//4jAGMAbABvAHUAZAAtAGMAbwBuAGYAaQBnAAoAbQBhAG4AYQBnAGUAXwBlAHQAYwBfAGgAbwBzAHQAcwA6ACAAZgBhAGwAcwBlAAoA

[apricote]

tl;dr: It looks like your text file is encoded as UTF-16 Little Endian. Please try to encode with UTF-8 instead.

---

I was able to reproduce this locally using the base64 string:

$ HCLOUD_DEBUG=true hcloud server create --image ubuntu-22.04 --name test-user-data --type cax11 --location fsn1 --user-data-from-file <(echo "//4jAGMAbABvAHUAZAAtAGMAbwBuAGYAaQBnAAoAbQBhAG4AYQBnAGUAXwBlAHQAYwBfAGgAbwBzAHQAcwA6ACAAZgBhAGwAcwBlAAoA" | base64 -d)

[snip]
Body:
{"name":"test-primary-ip","server_type":45,"image":103908130,"location":"fsn1","user_data":"\ufffd\ufffd#\u0000c\u0000l\u0000o\u0000u\u0000d\u0000-\u0000c\u0000o\u0000n\u0000f\u0000i\u0000g\u0000\n\u0000m\u0000a\u0000n\u0000a\u0000g\u0000e\u0000_\u0000e\u0000t\u0000c\u0000_\u0000h\u0000o\u0000s\u0000t\u0000s\u0000:\u0000 \u0000f\u0000a\u0000l\u0000s\u0000e\u0000\n\u0000","start_after_create":false,"labels":{},"automount":false,"public_net":{"enable_ipv4":false,"enable_ipv6":true}}

[snip]
Response:
{
  "error": {
    "message": "NULL characters are not allowed",
    "code": "json_error",
    "details": null
  }
}

I took a deeper look into the bytes of the file, and after some googling and testing different encodings on https://www.base64decode.org/ I figured out that the source text is encoded as UTF-16 Little Endian. We can take a proper look at the bytes here:

$ echo "//4jAGMAbABvAHUAZAAtAGMAbwBuAGYAaQBnAAoAbQBhAG4AYQBnAGUAXwBlAHQAYwBfAGgAbwBzAHQAcwA6ACAAZgBhAGwAcwBlAAoA" |base64 -d | hexyl                                                            hexyl
┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐
│00000000│ ff fe 23 00 63 00 6c 00 ┊ 6f 00 75 00 64 00 2d 00 │××#⋄c⋄l⋄┊o⋄u⋄d⋄-⋄│
│00000010│ 63 00 6f 00 6e 00 66 00 ┊ 69 00 67 00 0a 00 6d 00 │c⋄o⋄n⋄f⋄┊i⋄g⋄_⋄m⋄│
│00000020│ 61 00 6e 00 61 00 67 00 ┊ 65 00 5f 00 65 00 74 00 │a⋄n⋄a⋄g⋄┊e⋄_⋄e⋄t⋄│
│00000030│ 63 00 5f 00 68 00 6f 00 ┊ 73 00 74 00 73 00 3a 00 │c⋄_⋄h⋄o⋄┊s⋄t⋄s⋄:⋄│
│00000040│ 20 00 66 00 61 00 6c 00 ┊ 73 00 65 00 0a 00       │ ⋄f⋄a⋄l⋄┊s⋄e⋄_⋄  │
└────────┴─────────────────────────┴─────────────────────────┴────────┴────────┘

When we load the user data in the CLI we naively assume that its UTF-8. This introduces NULL bytes (\u0000) between all of the characters. These NULL bytes then fail the JSON parser in our API.

[sternze]

Hi!

I can confirm, that this was the issue! With UTF8 everything works fine!

Thank you for the fast support!