Trouble using Vespa CLI with multinode-HA TLS guide

[kkraune]

I followed the steps in https://github.com/vespa-engine/sample-apps/tree/master/examples/operations/multinode-HA#secure-vespa-with-mutually-authenticated-tls

Everything works well, I can query using curl. Then tried to use Vespa CLI, based on https://docs.vespa.ai/en/reference/vespa-cli/vespa_auth_cert.html :

$ export VESPA_CLI_DATA_PLANE_CERT_FILE=pki/vespa/host.pem 
$ export VESPA_CLI_DATA_PLANE_KEY_FILE=pki/vespa/host.key 
$ vespa query -t https://localhost:8445 'select * from music where true'
Error: request failed: Get "https://localhost:8445/search/?timeout=10s&yql=select+%2A+from+music+where+true": tls: failed to verify certificate: x509: “Vespa Sample Apps” certificate is not standards compliant

(see the guide for port mappings)

I assume that https://github.com/vespa-engine/sample-apps/blob/master/examples/operations/multinode-HA/scripts/generate-cert-chains.sh generates non-compliant credentials, or something to fix in the CLI?

Testing this with vespa-8.221.29 and Vespa CLI version 8.238.22 compiled with go1.21.1 on darwin/amd64

[kkraune]

This works:

curl -s --key pki/client/client.key --cert pki/client/client.pem --cacert pki/vespa/ca-vespa.pem     --data-urlencode 'yql=select * from sources * where sddocname contains "music"'     https://localhost:8445/search/
{"root":{"id":"toplevel","relevance":1.0,"fields":{"totalCount":5},"coverage":{"coverage":100,"documents":5,"full":true,"nodes":2,"results":1,"resultsFull":1},"children":[{"id":"id:mynamespace:music::2","relevance":0.0,"source":"music","fields":{"sddocname":"music","documentid":"id:mynamespace:music::2","artist":"Rammstein","album":"Liebe ist für alle da","year":2009,"category_scores":{"type":"tensor<float>(cat{})","cells":{"pop":0.10000000149011612,"rock":1.0,"jazz":0.0}}}},{"id":"id:mynamespace:music::0","relevance":0.0,"source":"music","fields":{"sddocname":"music","documentid":"id:mynamespace:music::0","artist":"Coldplay","album":"A Head Full of Dreams","year":2015,"category_scores":{"type":"tensor<float>(cat{})","cells":{"pop":1.0,"rock":0.20000000298023224,"jazz":0.0}}}},{"id":"id:mynamespace:music::1","relevance":0.0,"source":"music","fields":{"sddocname":"music","documentid":"id:mynamespace:music::1","artist":"Metallica","album":"Hardwired...To Self-Destruct","year":2016,"category_scores":{"type":"tensor<float>(cat{})","cells":{"pop":0.0,"rock":1.0,"jazz":0.0}}}},{"id":"id:mynamespace:music::4","relevance":0.0,"source":"music","fields":{"sddocname":"music","documentid":"id:mynamespace:music::4","artist":"Billie Eilish","album":"When We All Fall Asleep, Where Do We Go?","year":2019,"category_scores":{"type":"tensor<float>(cat{})","cells":{"pop":1.0,"rock":0.0,"jazz":0.10000000149011612}}}},{"id":"id:mynamespace:music::3","relevance":0.0,"source":"music","fields":{"sddocname":"music","documentid":"id:mynamespace:music::3","artist":"Diana Krall","album":"Love Is Here To Stay","year":2018,"category_scores":{"type":"tensor<float>(cat{})","cells":{"pop":0.4000000059604645,"rock":0.0,"jazz":0.800000011920929}}}}]}}

[kkraune]

Also:

$ env | grep VESPA
VESPA_CLI_DATA_PLANE_CA_CERT_FILE=pki/vespa/ca-vespa.pem
VESPA_CLI_DATA_PLANE_CERT_FILE=pki/vespa/host.pem
VESPA_CLI_DATA_PLANE_KEY_FILE=pki/vespa/host.key

$ vespa query -t https://localhost:8445 'select * from music where true'
Error: request failed: Get "https://localhost:8445/search/?timeout=10s&yql=select+%2A+from+music+where+true": remote error: tls: bad certificate

[kkraune]

$ env | grep VESPA
VESPA_CLI_DATA_PLANE_CA_CERT_FILE=pki/vespa/ca-vespa.pem
VESPA_CLI_DATA_PLANE_CERT_FILE=pki/vespa/host.pem
VESPA_CLI_DATA_PLANE_KEY_FILE=pki/vespa/host.key

$ vespa query -v -t https://localhost:8445 'select * from music where true'
curl --key pki/vespa/host.key --cert pki/vespa/host.pem 'https://localhost:8445/search/?timeout=10s&yql=select+%2A+from+music+where+true'
Error: request failed: Get "https://localhost:8445/search/?timeout=10s&yql=select+%2A+from+music+where+true": remote error: tls: bad certificate

$ env | grep VESPA
VESPA_CLI_DATA_PLANE_CERT_FILE=pki/vespa/host.pem
VESPA_CLI_DATA_PLANE_KEY_FILE=pki/vespa/host.key

$ vespa query -v -t https://localhost:8445 'select * from music where true'
curl --key pki/vespa/host.key --cert pki/vespa/host.pem 'https://localhost:8445/search/?timeout=10s&yql=select+%2A+from+music+where+true'
Error: request failed: Get "https://localhost:8445/search/?timeout=10s&yql=select+%2A+from+music+where+true": tls: failed to verify certificate: x509: “Vespa Sample Apps” certificate is not standards compliant

the verbose printing of the equivalent curl command is identical, but does not look right - if VESPA_CLI_DATA_PLANE_CA_CERT_FILE is set, I assume the curl command should have --cacert pki/vespa/ca-vespa.pem ?

[kkraune]

Argh - I now see I used wrong files - let me test again

[kkraune]

$ export VESPA_CLI_DATA_PLANE_CERT_FILE=pki/client/client.pem
$ export VESPA_CLI_DATA_PLANE_KEY_FILE=pki/client/client.key

$ vespa query -v -t https://localhost:8445 'select * from music where true'
curl --key pki/client/client.key --cert pki/client/client.pem 'https://localhost:8445/search/?timeout=10s&yql=select+%2A+from+music+where+true'
Error: request failed: Get "https://localhost:8445/search/?timeout=10s&yql=select+%2A+from+music+where+true": tls: failed to verify certificate: x509: “Vespa Sample Apps” certificate is not standards compliant

Error still the same - but this works:

$ env | grep VESPA
VESPA_CLI_DATA_PLANE_CA_CERT_FILE=pki/vespa/ca-vespa.pem
VESPA_CLI_DATA_PLANE_CERT_FILE=pki/client/client.pem
VESPA_CLI_DATA_PLANE_KEY_FILE=pki/client/client.key
C02DR9A6MD6R:multinode-HA kraune$ vespa query -v -t https://localhost:8445 'select * from music where true' | head -5
curl --key pki/client/client.key --cert pki/client/client.pem 'https://localhost:8445/search/?timeout=10s&yql=select+%2A+from+music+where+true'
{
    "root": {
        "id": "toplevel",
        "relevance": 1.0,
        "fields": {

[kkraune]

Sorry for the confusion, I am trying to improve the guide - my summary:

If I fail to set VESPA_CLI_DATA_PLANE_CA_CERT_FILE the error message is tls: failed to verify certificate: x509: “Vespa Sample Apps” certificate is not standards compliant - It would help a lot if it instead said something like missing CA cert file

The -HA guide must be updated with query examples

[kkraune]

Added an example in https://github.com/vespa-engine/sample-apps/pull/1319 - We should point here from some TLS documentation, too.

@mpolden please look into the error message

[mpolden]

It's not possible to determine if the CA certificate is required to be set as this depends on which CA certificate the server-side is configured with, so claiming "missing CA cert file" would be wrong. I.e. choosing to have a self-signed CA certificate requires both the server and client to use the same CA, but the client cannot decide what is right on its own. In any case "failed to verify" is a pretty clear signal of mismatch between server and client.