Consul API Base64 Decode

[ldelossa]

If you have a question, please direct it to the consul mailing list if it hasn't been addressed in either the FAQ or in one of the Consul Guides.

When filing a bug, please include the following:

Description of the Issue (and unexpected/desired result)

Attempting to use the consul API to decode base64 value returned from the API. Getting bad results maybe doing this wrong.

Reproduction steps

Here is my function:

// DecodeValue takes a KV pair and returns the base64 decoded string
func DecodeValue(kv *consulapi.KVPair) (string, error) {
    // kv.Value is a base64 encoded byte array
    encodedBase64 := kv.Value
    buf := bytes.NewBuffer(encodedBase64)

    // create decoder
    decoder := base64.NewDecoder(base64.StdEncoding, buf)
    dbuf := make([]byte, base64.StdEncoding.DecodedLen(len(encodedBase64)))
    _, err := decoder.Read(dbuf)
    if err != nil && err != io.EOF {
        return "", err
    }

    return string(dbuf), nil
}

consul version for both Client and Server

Client: 1.0.7 Server: 1.0.7

[kasisnu]

Hi,

Could you provide some test values with which you're getting bad results?

The code looks like it should work fine - I've tested with some dummy values.

func TestDecodeValue(t *testing.T) {
    kv := consulapi.KVPair{
        Key:   "something",
        Value: []byte("c29tZXZhbHVl"), // "somevalue"
    }

    value, err := DecodeValue(&kv)
    if err != nil || value != "somevalue" {
        t.Errorf("Mismatched values, expected: %s got %s", "somevalue", value)
    }
}

[ldelossa]

So I was able to get the desired output with this function

func DecodeValue(kv *consulapi.KVPair) (string, error) {
    // kv.Value is a base64 encoded byte array
    encodedBase64 := kv.Value
    s := base64.StdEncoding.EncodeToString(encodedBase64)

    // decode string to bytes
    b, err := base64.StdEncoding.DecodeString(s)
    if err != nil {
        return "", err
    }

    return string(b), nil
}

I now get keys back that I care about. These are in json but reflect the keys and values I have at an arbitrary prefix.

{
  "spotify_client_id": "test-spotify-client-id",
  "feed_fm_client_id": "test-feedfm-client-id",
  "feed_fm_client_secret": "test-feedfm-client-secret",
  "pub_nub_subscribe_key": "test-pubnub-subscribe-key",
  "pub_nub_publish_key": "test-pubnub-publish-key-test5"
}

But using the original one I get:

2018/04/20 15:00:26 could not decode value for feedfmclientid: illegal base64 data at input byte 4

The error message after the colon is returned by the decode.Read() method.

[kasisnu]

Okay. So here's what I think is happening. In the first snippet, it is assumed that KVPair.Value contains base64 encoded data, which it doesn't. The second snippet, encodes the data first through base64.StdEncoding.EncodeToString(encodedBase64) and then decodes it, which works, of course.

I think in your case you don't have to base64 decode the KVPair.Value which contains unencoded data - https://godoc.org/github.com/hashicorp/consul/api#KVPair

Specifically, the part about encoding upon transport applies here I think,

    // Value is the value for the key. This can be any value, but it will be
    // base64 encoded upon transport.
    Value []byte

[ldelossa]

@kasisnu I do not believe that's the case.

When I assume the values are NOT base64 encoded, any base64 characters are not in the resulting strings. For example "pub_nub_publish_key": "test-pubnub-publish-key-test1" would result in the value "test-pubnub-publish-key-test".

Also why would the value sometimes be base64 encoded and sometimes not?

[pearkes]

Issues on GitHub for Consul are intended to be related to bugs or feature requests, so we recommend using our other community resources instead of asking here.

• FAQ (Frequently asked questions)
• Consul Guides
• Any other questions can be sent to the Consul mailing list

If you feel this is a bug, please open a new issue with the appropriate information.