add ssh key gives "Request failed with status code 400 " // "illegal base64 data at input byte 0"

[itfors]

When attempting to add an SSH key the web UI gives me a bright red banner stating "Request failed with status code 400".

Output of docker logsbelow.

Same error in both 2.8.90 and 2.8.91 (I haven't tried other versions).

Perusal of the docs and vigorous google-ing seem to indicate that various encryption keys should be generated with head -c32 /dev/urandom | base64, however issue comments here seem to argue against this?

In any case I've tried with both head -c32 /dev/urandom | base64 and (echo -n "$(pwgen -cns 32 1)")|base64 (the pwgen command creates random 32-letter strings - strictly alphanumeric, wrapping in echo -n gets rid of the trailing newline) - to no avail: whether or not the b64-encoded string is printable or not I still get the same error.

I've even tried leaving out the -----BEGIN OPENSSH PRIVATE KEY-----/-----END OPENSSH PRIVATE KEY----- lines and also tried concatenating the actual b64-data into a single string. No Joy.

As I'm not at all familiar with go I'm all out of ideas.

Here's the pwgen keys I've tried:

    "cookie_hash": "SXc5eWFhaTV5cXJ0bE5QVjNaQXo0RGs2T2VZU1YyRVE=",
    "cookie_encryption": "MnFZZnUwaldYQWRVbGxuSEo5T2JteTRTUEtrZHk4RXQ=",
    "access_key_encryption": "cW5kSElaSHNkQzlKVENBNERWTWJ6aURhdjZRTTExSEI=",

...and the output from docker logs -f:

Semaphore v2.8.91
Interface
Port :3000
Server is running
time="2023-07-12T12:12:19Z" level=error msg="illegal base64 data at input byte 0"
goroutine 8 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
        /usr/local/go/src/runtime/debug/stack.go:16 +0x19
github.com/ansible-semaphore/semaphore/api/helpers.WriteError({0x2298c58, 0xc00045c2a0}, {0x2294940?, 0x26a64a0?})
        /go/src/github.com/ansible-semaphore/semaphore/api/helpers/helpers.go:92 +0x1ae
github.com/ansible-semaphore/semaphore/api/projects.AddKey({0x2298c58, 0xc00045c2a0}, 0x22908e0?)
        /go/src/github.com/ansible-semaphore/semaphore/api/projects/keys.go:91 +0x66d
net/http.HandlerFunc.ServeHTTP(0xc0000bf2a8?, {0x2298c58?, 0xc00045c2a0?}, 0x8?)
        /usr/local/go/src/net/http/server.go:2109 +0x2f
github.com/ansible-semaphore/semaphore/api/projects.GetMustCanMiddlewareFor.func1.1({0x2298c58, 0xc00045c2a0}, 0xc0004d0c00)
        /go/src/github.com/ansible-semaphore/semaphore/api/projects/project.go:73 +0x31a
net/http.HandlerFunc.ServeHTTP(0xc0004d0c00?, {0x2298c58?, 0xc00045c2a0?}, 0xba02e0?)
        /usr/local/go/src/net/http/server.go:2109 +0x2f
github.com/ansible-semaphore/semaphore/api/projects.ProjectMiddleware.func1({0x2298c58, 0xc00045c2a0}, 0x0?)
        /go/src/github.com/ansible-semaphore/semaphore/api/projects/project.go:42 +0x3bb
net/http.HandlerFunc.ServeHTTP(0x2298c58?, {0x2298c58?, 0xc00045c2a0?}, 0x5a0000c000088000?)
        /usr/local/go/src/net/http/server.go:2109 +0x2f
github.com/ansible-semaphore/semaphore/api.authentication.func1({0x2298c58, 0xc00045c2a0}, 0xc000134b00?)
        /go/src/github.com/ansible-semaphore/semaphore/api/auth.go:111 +0x5d
net/http.HandlerFunc.ServeHTTP(0xb88ec0?, {0x2298c58?, 0xc00045c2a0?}, 0xc?)
        /usr/local/go/src/net/http/server.go:2109 +0x2f
github.com/ansible-semaphore/semaphore/api.JSONMiddleware.func1({0x2298c58, 0xc00045c2a0}, 0x13?)
        /go/src/github.com/ansible-semaphore/semaphore/api/router.go:35 +0xf4
net/http.HandlerFunc.ServeHTTP(0xc0000bf670?, {0x2298c58?, 0xc00045c2a0?}, 0xc0006da1b0?)
        /usr/local/go/src/net/http/server.go:2109 +0x2f
github.com/ansible-semaphore/semaphore/api.StoreMiddleware.func1.1()
        /go/src/github.com/ansible-semaphore/semaphore/api/router.go:26 +0x2e
github.com/ansible-semaphore/semaphore/db.StoreSession({0x22a1ee8, 0xc00011e6c0}, {0xc0006da1b0, 0x13}, 0xc0000bf6c8)
        /go/src/github.com/ansible-semaphore/semaphore/db/Store.go:336 +0x68
github.com/ansible-semaphore/semaphore/api.StoreMiddleware.func1({0x2298c58, 0xc00045c2a0}, 0xc0004d0c00)
        /go/src/github.com/ansible-semaphore/semaphore/api/router.go:25 +0xfc
net/http.HandlerFunc.ServeHTTP(0xc0004d0c00?, {0x2298c58?, 0xc00045c2a0?}, 0xb61da0?)
        /usr/local/go/src/net/http/server.go:2109 +0x2f
github.com/ansible-semaphore/semaphore/cli/cmd.runService.func1.1({0x2298c58, 0xc00045c2a0}, 0xc0000bf7a8?)
        /go/src/github.com/ansible-semaphore/semaphore/cli/cmd/root.go:67 +0x124
net/http.HandlerFunc.ServeHTTP(0x7f1df0dec7e8?, {0x2298c58?, 0xc00045c2a0?}, 0x100?)
        /usr/local/go/src/net/http/server.go:2109 +0x2f
github.com/gorilla/mux.CORSMethodMiddleware.func1.1({0x2298c58, 0xc00045c2a0}, 0xc0008bf020?)
        /go/src/github.com/ansible-semaphore/semaphore/vendor/github.com/gorilla/mux/middleware.go:51 +0xaa
net/http.HandlerFunc.ServeHTTP(0xc0004d0b00?, {0x2298c58?, 0xc00045c2a0?}, 0xbccd20?)
        /usr/local/go/src/net/http/server.go:2109 +0x2f
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0004e2000, {0x2298c58, 0xc00045c2a0}, 0xc0000d2500)
        /go/src/github.com/ansible-semaphore/semaphore/vendor/github.com/gorilla/mux/mux.go:212 +0x202
github.com/gorilla/handlers.ProxyHeaders.func1({0x2298c58, 0xc00045c2a0}, 0xc0000d2500)
        /go/src/github.com/ansible-semaphore/semaphore/vendor/github.com/gorilla/handlers/proxy_headers.go:59 +0x144
net/http.HandlerFunc.ServeHTTP(0x3b?, {0x2298c58?, 0xc00045c2a0?}, 0x72?)
        /usr/local/go/src/net/http/server.go:2109 +0x2f
github.com/ansible-semaphore/semaphore/cli/cmd.cropTrailingSlashMiddleware.func1({0x2298c58?, 0xc00045c2a0?}, 0xc0000d2500?)
        /go/src/github.com/ansible-semaphore/semaphore/cli/cmd/server.go:27 +0xf7
net/http.HandlerFunc.ServeHTTP(0x0?, {0x2298c58?, 0xc00045c2a0?}, 0x6ee7d4?)
        /usr/local/go/src/net/http/server.go:2109 +0x2f
net/http.serverHandler.ServeHTTP({0x2296a58?}, {0x2298c58, 0xc00045c2a0}, 0xc0000d2500)
        /usr/local/go/src/net/http/server.go:2947 +0x30c
net/http.(*conn).serve(0xc0001ea000, {0x2299578, 0xc000707f80})
        /usr/local/go/src/net/http/server.go:1991 +0x607
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:3102 +0x4db

[itfors]

I can add that this also occurs when using the API.

BTW: it took a fair bit of scrounging to figure out the correct (i hope) format for the request body.

In case anyone else is looking for the same info, it seems that the correct format is:

{
  "project_id": N,        # Integer, not string
  "name": "key name",
  "type": "ssh",
  "ssh": {
    "login": "username",  # optional, username to use with this key
    "private_key": "-----BEGIN OPENSSH PRIVATE KEY-----\nMUE8XJgFx21gv2RxyyQtC0VNb0IO0585O8+AbG3L8ERxXUiG99E+Zjzb2JcSL0saKZFVSefMt25k\nlGELt+B8hAGSARZSjY5gKKrnfBpkPcL8vMK+63PfcdLSPdSDiV84Y+d4s2lxX3LwMD/4lwP1jhzR\n5h6zol7qNiCCVwv4yr4wlEdnn7MgDsdbFwBnbdeNoIZC9lkMhJ4NujBxGbJqNt87ZWrJhNGyok7t\n2koByHcnRDxms+9pRTor8zpMyv4Cyzz6ckQ2AH3PC/dW2u3utFjtPK8w7Py8hTQfoPJDhknLJOve\nNYlz19yuBKTrkTxpPHl3EeOapclSiEC5/Vlf8q7W/7HEvTZiWOoy5fKcyWd2QFF2PTajxKJw6FUa\nN8/wt2iNGyB8+UYNoqeA/6dFFhL0OeoVZ+tEyYpAko99Bk47++qg4TbKF4qKcnKRp4ELfcCmy/O2\njbmoDOwnG6Gm7gddWrSME13iYW9UblI0g9KIXvnk9ffP44pjZVqQwtLpypkReXIgBq0ReXSULjuC\nyFsSzLXLYdDnkVKFBqxqDdW7I4GIGGuqTjl63jLhgisAy6tU+qHaixvUPk1/NQDVk0K+/i2WNp2Z\n95hzt9LRMmyA/NcaIMnoNe4IwXs3bQkiF6lHgDyOK01TU0Dkjn8vFNYhxUmi86o3hhVU0KppZt8T\n7Dzn5q19K2BHpuy7xWkgDo75vsG7qUgEfOlvFQoWbbfRBLllkoDmVMsUjpn38Ny5oC/pxq/VduTE\nvEgn6MIwWqXUmpxIY+Zh/8AXYJiRCcfJAQCH5/UJlmHXozvnevM2NDsWVcMG6DQL23GdEMxH1JkM\na9odK75zZsQG6CQ/8LeB+X2d45IlnSqfq2HKQmxyNXXLBSY+zDOYAC5TUyiQC03kjLlXPYkNyJlt\nzMqU6DPl/VK8kjn9hbMvE1JBu2fOyG6UyVT0/3LfSI91ukBxaKjj8yTvc8QtPubBHEMa+uhCL6Wf\nCasecvyE5VlRhGl7al0BY2f7XS+mjX61TLKj4ht3q6+MSmy54p4yoVzpM9nM9nSUqKN1TWlC7QMO\n1itnT786cH7fZHKwsjRR4NPHx7+4JeoDYQp0J+VB5/vxo9VpmC6DTS+rv9M0y/5IJGFQ1IeiV6r+\niWvWMISou3tIH0q2oGeAdqFK/MFHrhcNrr5tl+TKTPlkL87LKHs8aT9PmZJK6ylKie8VhaT/TFz2\nfMlgzJjvsuy9n8XDewZ6i9i8Cp11Il+AYcPOGlEB5MXdP7cHjsuncBHaC9zCubitKT26RnN+xd0r\ng1JyW59RrsmZ2lxQp6ePPwMOPlM++RAzv56HMlis2SVinGB2B+XTBcsIADG1eFeHDC1TAGL1Fir9\niL7diSj5KCi6+Nv5U0VLozvESUhy2e7PzWZODgs6qU1TiC0MliEHaUbtsLQw8aq9qJywomw9WhD9\nVXN+saEh+SpABQHIL+afUAC6YgEV+ne7S80q+DsCOGCbqipTyJZrC2VSnA5e1jLZf0kVJ5vb+xOC\nmecP4/IGuRPylzsT9MC7lu7rUp09io6fc/QlNJJHtYOwuDFjJf6H6+XzGQp/VU1XOBNdbRl0ou1m\nVTOiQMCniMSWRdvM244xwV6sR31LtbeenskkTQk091eCgEW2Wwjql+ytS6k/AKwmV0wZXxHjzWBd\nP2UgJ4U9ImbkRhoKyaSoWN1IqUgE2g==-----END OPENSSH PRIVATE KEY-----\n",
    "passphrase": "the passphrase for this key" # optional, passphrase for key
  }
}

[ansibleguy]

Greetings.

The only uses of base64 seem to be in here: https://github.com/ansible-semaphore/semaphore/blob/develop/db/AccessKey.go#L209 I'm not sure why the stacktrace does not show the helper/db functions as the error is 'found' here: https://github.com/ansible-semaphore/semaphore/blob/develop/api/projects/keys.go#L88

I guess you are using this docker image? How are you providing the value for 'access_key_encryption'? Inside a config file? Or via docker(-compose) environmental variables? If you are using any custom docker-compose config or dockerfile - could you provide it? Is the same problem occurring if you run a new/clean installation of semaphore?

- Greetings

[sergeifilippov]

I'm also having the same issue. As in, when adding a key via the GUI, API responds with error 400.

[itfors]

The access_key_encryption is specified in the compose-file as an env-var: - SEMAPHORE_ACCESS_KEY_ENCRYPTION="cW5kSElaSHNkQzlKVENBNERWTWJ6aURhdjZRTTExSEI=" (the same b64-string as above)

I've also edited the config/config.json to ensure that the 3 b64-strings mentioned above are present.

The full (redacted) compose-file is:

---
version: '3'

services:

  semaphore:
    container_name: semaphore
    hostname: semaphore
    image: semaphoreui/semaphore:latest
    ports:
      - 3022:3000
    environment:
      - SEMAPHORE_DB_USER=semaphore
      - SEMAPHORE_DB_PASS=k3enPuppy66
      - SEMAPHORE_DB_HOST=mariadb
      - SEMAPHORE_DB_PORT=3306
      - SEMAPHORE_DB_DIALECT=mysql
      - SEMAPHORE_DB=semaphore
      - SEMAPHORE_PLAYBOOK_PATH=/tmp/semaphore/
      - SEMAPHORE_ADMIN_PASSWORD=oliv3Button47
      - SEMAPHORE_ADMIN_NAME=admin
      - SEMAPHORE_ADMIN_EMAIL=admin@localhost
      - SEMAPHORE_ADMIN=admin
      - SEMAPHORE_ACCESS_KEY_ENCRYPTION="cW5kSElaSHNkQzlKVENBNERWTWJ6aURhdjZRTTExSEI="
      - SEMAPHORE_LDAP_ACTIVATED="yes"
      - SEMAPHORE_LDAP_HOST=my.ldap-AD.server
      - SEMAPHORE_LDAP_PORT="636"
      - SEMAPHORE_LDAP_NEEDTLS="yes"
      - SEMAPHORE_LDAP_DN_BIND="ldapauth@my.ad.domain"
      - SEMAPHORE_LDAP_PASSWORD="HumptyDumpty"
      - SEMAPHORE_LDAP_DN_SEARCH="DC=my,DC=ad,DC=domain"
      - SEMAPHORE_LDAP_SEARCH_FILTER="(rather convoluted, but functional, search filter)"
      - ANSIBLE_HOST_KEY_CHECKING=false
    volumes:
      - ./data/inventory/:/inventory:ro
      - ./data/authorized-keys/:/authorized-keys:ro
      - ./data/config/:/etc/semaphore:rw
    restart: unless-stopped
    depends_on:
      - mariadb

The phrasing of the error message seems to indicate that whatever is happening in keys.go#L88 is somehow not passing the unadultered contents of access_key_encryption ?

[kwanlowe]

Not sure if it's helpful to you, but I faced a similar issue that turned out to be some base64 conversion on my part. The docs do create a proper key with head -c32 /dev/urandom | base64. My mistake was in trying to use a manually generated password that didn't decode properly.

If you're using the Docker image, here is how I start it via Ansible: https://github.com/kwanlowe/digitalhermit_website/blob/master/linux/semaphore/playbooks/roles/semaphore/tasks/start-container.yaml

The above also wraps it in nginx for TLS.

[itfors]

Initially I did use the head -c32 /dev/urandom | base64 as suggested in the docs. When that failed I tried using the (current) random string of printable chars ("qndHIZHsdC9JTCA4DVMbziDav6QM11HB") which is reflected by the current bas64 string "cW5kSElaSHNkQzlKVENBNERWTWJ6aURhdjZRTTExSEI=".

It just struck me that maybe the - SEMAPHORE_ACCESS_KEY_ENCRYPTION="cW5kSElaSHNkQzlKVENBNERWTWJ6aURhdjZRTTExSEI=" might be the issue, so I tried removing the quotes and, hey presto!, it works.

It would seem that maybe the quotes are being passed as part of the value? (This would explain the issue with "illegal base64 data at input byte 0")

[kwanlowe]

Nice catch! I very likely made a similar error when moving from bash scripts to Ansible, which was what made it a bit frustrating because the manual process worked fine.

[sergeifilippov]

@itfors your suggestion fixed the issue for me 🥳🙏🏻

[ansibleguy]

So actually it's a problem with Semaphore lacking config-validation on service-startup.

[ruslan-mogilevskiy]

I've also stuck on this error with the latest Semaphore version (2.9.45). The problem was with my SEMAPHORE_ACCESS_KEY_ENCRYPTION (no '=' character on the end so the whole text wasn't a valid base64) and the introduced validation didn't catch this on start (the error appeared when I tried to add an ssh key).