map[interface{}]interface{} on a map[string]interface{}

[aanm]

Sorry to be persistent with this. I wasn't aware that yaml was able to have different types has keys than string. However, with the following example I see that newstruct is being considered has a map[interface{}]interface{}, is it a bug or is something I should expect?

type=string key=something type=string value=thing 
type=string key=newstruct type=map[interface {}]interface {} value=map[otherstruct:[map[foo-bar:12345]] bar:bar]

var data = `---
foo-map:
  something: "thing"
  newstruct:
    otherstruct:
      -
        foo-bar: 12345
    bar: "bar"`

type FooStruct struct {
    FooMap map[string]interface{} `yaml:"foo-map,omitempty"`
}

func main() {
    var foo FooStruct
    err := yaml.Unmarshal([]byte(data), &foo)
    if err != nil {
        fmt.Printf("error %+v", err)
    }
    for k, v := range foo.FooMap {
        fmt.Printf("type=%+v key=%+v type=%+v value=%+v\n", reflect.TypeOf(k), k, reflect.TypeOf(v), v)
    }
}

[danprince]

I'm having trouble with this too. Even if I unmarshal into a map[string]interface{} the nested maps are of type map[interface{}]interface{}, meaning I can't then marshal to JSON (which must have string keys).

Is it possible to analyse the type of the interface passed to unmarshal then use the same type for recursively unmarshalling all the way down?

[magnusbaeck]

Is it possible to analyse the type of the interface passed to unmarshal then use the same type for recursively unmarshalling all the way down?

I'm not sure that would work in all cases. I also had this problem and made a custom unmarshaller in github.com/elastic/beats:libbeat/common/mapstr.go that converts the keys of nested maps to strings so that the unmarshalled data can always be marshalled into JSON.

[sontags]

The approach of @magnusbaeck works like a charm! I changed unmarshalYAML a bit as well as cleanUpMapValue in order to fit my needs (eg. it has now the same signature as yaml.Unmarshal() and ints, bool et al are not quoted since this works fine with encoding/json)...

https://github.com/unprofession-al/gerty/blob/master/api/helpers.go

It would be nice to see this in go-yaml as "compatibility mode".

[maciejmrowiec]

+1

[ns-cweber]

+1

[ake-persson]

+1

[matiasinsaurralde]

+1

[ake-persson]

I modified the code from the suggestion above and it works nicely. Would be nicer not having to do it thought.

// Copyright (c) 2015-2016 Michael Persson
// Copyright (c) 2012–2015 Elasticsearch <http://www.elastic.co>
//
// Originally distributed as part of "beats" repository (https://github.com/elastic/beats).
// Modified specifically for "iodatafmt" package.
//
// Distributed underneath "Apache License, Version 2.0" which is compatible with the LICENSE for this package.

package yaml_mapstr

import (
    // Base packages.
    "fmt"

    // Third party packages.
    "gopkg.in/yaml.v2"
)

// Unmarshal YAML to map[string]interface{} instead of map[interface{}]interface{}.
func Unmarshal(in []byte, out interface{}) error {
    var res interface{}

    if err := yaml.Unmarshal(in, &res); err != nil {
        return err
    }
    *out.(*interface{}) = cleanupMapValue(res)

    return nil
}

// Marshal YAML wrapper function.
func Marshal(in interface{}) ([]byte, error) {
    return yaml.Marshal(in)
}

func cleanupInterfaceArray(in []interface{}) []interface{} {
    res := make([]interface{}, len(in))
    for i, v := range in {
        res[i] = cleanupMapValue(v)
    }
    return res
}

func cleanupInterfaceMap(in map[interface{}]interface{}) map[string]interface{} {
    res := make(map[string]interface{})
    for k, v := range in {
        res[fmt.Sprintf("%v", k)] = cleanupMapValue(v)
    }
    return res
}

func cleanupMapValue(v interface{}) interface{} {
    switch v := v.(type) {
    case []interface{}:
        return cleanupInterfaceArray(v)
    case map[interface{}]interface{}:
        return cleanupInterfaceMap(v)
    case string:
        return v
    default:
        return fmt.Sprintf("%v", v)
    }
}

[jveski]

+1

I'm currently maintaining a fork with a one line change to work around this issue. https://github.com/Navops/yaml/commit/85482c8b225b4cb87bcb8532d2bf81b72b2f4c3c

To be clear: I believe the library's design is correct given the way YAML handles boolean keys. However, I'd be willing to guess that a majority of implementations don't care about boolean keys and would rather have map keys default to strings.

Maybe exporting defaultMapType to allow implementations to mutate it (like my fork) is in order?

[rjeczalik]

Oh my, extremely sorry for this massive spam. Rebases ¯\(ツ)/¯

[niemeyer]

@rjeczalik No problem, but can it stop? :)

About the issue, I think the best approach is introducing an explicit Decoder type, as already planned/proposed/long overdue, and on that type have some sort of UseStringKeys or similar.

Will look into this.

[rjeczalik]

No problem, but can it stop? :)

I think it should stop now, apparently github also links on merges, which I can't really do much about. Lesson learned, I'll think twice before linking to external issue.

Once again, sorry!

[niemeyer]

No worries, it's indeed an obscure and unexpected consequence which I would easily get into as well.

[msvechla]

Any updates here? Exporting defaultMapType would still be an easy / nice fix.

[niemeyer]

We cannot change this in v2 since it's incompatible. But in v3 the default behavior will be of having map[string]interface{} decode further general maps into the same type.

[seagle0128]

+1

[zbindenren]

@niemeyer exporting defaultMapType as mentioned by @msvechla would not break the current v2 api, would it?

[mikefarah]

@superwhiskers hah I ended up doing a very similar thing independently ;)

I've raised a PR too https://github.com/go-yaml/yaml/pull/373

[mikefarah]

Took me a while to get imported forks working nicely, so to use the fork:

Replace imports of "gopkg.in/yaml.v2" with "gopkg.in/mikefarah/yaml.v2" then

go get gopkg.in/mikefarah/yaml.v2

[virtuald]

I'm sure v3 will eventually show up -- but until it does, this is clearly an issue that a lot of people want solved, so why not just push a fix for it? In particular, #385 addresses this issue cleanly.

[seagle0128]

Agree. This issue was reported 3 years ago, and the root cause is very clear. I don't understand why this issue wasn't fixed for so long time... I have to fork it and fix myself.

[sfwn]

just use gopkg.in/yaml.v3 .

[zbindenren]

This issue should be closed.

[BlizzTom]

This is still an issue in v3:

https://github.com/go-yaml/yaml/blob/v3/decode.go#L297

[leefernandes]

In v3 decoding any map[string]interface{} which contains a nested map[string]interface{} will result in a map[interface{}] interface{}

In our case yaml usage is expected to be compatible with json spec. It would be nice to configure the yaml decoder in such a way that map[string] is the expectation, and any keys not of type string throw an err or are typecast to string.

[PChou]

v3 try to introduce a solution, but still have problem as you said, I tried to fix this in v3: https://github.com/go-yaml/yaml/pull/507

In v3 decoding any map[string]interface{} which contains a nested map[string]interface{} will result in a map[interface{}] interface{}

In our case yaml usage is expected to be compatible with json spec. It would be nice to configure the yaml decoder in such a way that map[string] is the expectation, and any keys not of type string throw an err or are typecast to string.

[nervo]

I'd like to add that in v3 too, any map containing a yaml alias - although anchor is strictly string based - results in a map[interface{}]interface{}

foo:
    bar: baz

bar:
    baz: qux

-> both foo and bar are map[string]interface{} ✅

foo:
    &foo
    bar: baz

bar:
    <<: *foo
    baz: qux

-> foo is map[string]interface{} but bar is map[interface{}]interface{} 😢

[toywei]

a.yaml

a.go

package main

import (
    "fmt"
    "io/ioutil"

    "gopkg.in/yaml.v3"
)

func main() {
    var err error
    m := make(map[string]interface{})
    var bs []byte
    bs, err = ioutil.ReadFile("a.yaml")
    if err != nil {
        panic(err)
    }
    err = yaml.Unmarshal(bs, &m)
    if err != nil {
        panic(err)
    }
    f1v := m["f1"].(string)
    var f3v string
    var f4v []int
    var f5v []string
    var f6v map[string]interface{}

    for k, v := range m["f2"].(map[string]interface{}) {
        switch k {
        case "f3":
            f3v = v.(string)
        case "f4", "f5":
            l := v.([]interface{})
            if k == "f4" {
                for _, u := range l {
                    f4v = append(f4v, u.(int))
                }
            }
            if k == "f5" {
                for _, u := range l {
                    f5v = append(f5v, u.(string))
                }
            }
        case "f6":
            f6v = v.(map[string]interface{})
        default:
        }
    }

    fmt.Println("---->", f1v, f3v, f4v, f5v, f6v)
}

go run *go ----> hi世界 value [42 1024] [a b] map[f7:77 f8:ok]

just use gopkg.in/yaml.v3 .

[giulianopz]

@aanm, @niemeyer, please close this issue since it was fixed with v3 (as noted by some comments above as well):

package main

import (
    "fmt"
    "reflect"

    "gopkg.in/yaml.v3"
)

var data = `---
foo-map:
  something: "thing"
  newstruct:
    otherstruct:
      -
        foo-bar: 12345
    bar: "bar"`

type FooStruct struct {
    FooMap map[string]interface{} `yaml:"foo-map,omitempty"`
}

func main() {
    var foo FooStruct
    err := yaml.Unmarshal([]byte(data), &foo)
    if err != nil {
        fmt.Printf("error %+v", err)
    }
    for k, v := range foo.FooMap {
        fmt.Printf("type=%+v key=%+v type=%+v value=%+v\n", reflect.TypeOf(k), k, reflect.TypeOf(v), v)
    }
        // output:
        // type=string key=newstruct type=map[string]interface {} value=map[bar:bar otherstruct:[map[foo-bar:12345]]]
        // type=string key=something type=string value=thing
}