UnmarshalKey looses some data if a subitem is overridden

[SVilgelm]

Expected behavior (what you expected to happen): viper.UnmarshalKey should unmarshals whole structure, not overridden keys only

Actual behavior (what actually happened): if viper.Set has been used to override a value of a sub item, then viper.UnmarshalKey unmarshals only overridden keys

Repl.it link:

https://replit.com/@SVilgelm/viperUnmarshalKey-bug

Code reproducing the issue:

package main

import (
    "strings"

    "github.com/spf13/viper"
)

func main() {
    config := `
resources:
  baz:
    name: "Super bazzz"
    xyz: true
    test: 123
`
    viper.Debug()
    println("########################################################################")

    viper.SetConfigType("yaml")
    viper.ReadConfig(strings.NewReader(config))
    viper.Debug()
    println("########################################################################")

    res1 := struct {
        Name string `mapstructure:"name"`
        XYZ  bool   `mapstructure:"xyz"`
    }{}
    viper.UnmarshalKey("resources.baz", &res1)
    println("res1 Name:", res1.Name, "; XYZ:", res1.XYZ)
    println("########################################################################")

    viper.Set("resources.baz.name", "override name")
    viper.Debug()
    println("########################################################################")

    res2 := struct {
        Name string `mapstructure:"name"`
        XYZ  bool   `mapstructure:"xyz"`
    }{}
    viper.UnmarshalKey("resources.baz", &res2)
    println("res2 Name:", res2.Name, "; XYZ:", res2.XYZ)
    println("########################################################################")
}

Environment:

• Viper version:v1.7.1
• Config source: any
• File format: any

[github-actions[bot]]

👋 Thanks for reporting!

A maintainer will take a look at your issue shortly. 👀

In the meantime: We are working on Viper v2 and we would love to hear your thoughts about what you like or don't like about Viper, so we can improve or fix those issues.

⏰ If you have a couple minutes, please take some time and share your thoughts: https://forms.gle/R6faU74qPRPAzchZ9

📣 If you've already given us your feedback, you can still help by spreading the news, either by sharing the above link or telling people about this on Twitter:

https://twitter.com/sagikazarmark/status/1306904078967074816

Thank you! ❤️

[SVilgelm]

Actually the issue with viper.Get, it does not return the whole structure

[madsoigard]

any update on this?

[sagikazarmark]

The problem is that resources.baz exists as a key, so UnmashalKey won't look any further. I wonder what happens if you create a Resources struct and try to unmarshal resources instead. I think that should work.

[SVilgelm]

@sagikazarmark I have updated the repl, the problem still exists even if I unmarshal resources. I also updated th go.mod to use latest viper.

https://replit.com/@SVilgelm/viperUnmarshalKey-bug#main.go

Code:

package main

import (
    "strings"
  "fmt"

    "github.com/spf13/viper"
)

func main() {
    config := `
resources:
  baz:
    name: "Super bazzz"
    xyz: true
    test: 123
`
    viper.Debug()
    println("########################################################################")

    viper.SetConfigType("yaml")
    viper.ReadConfig(strings.NewReader(config))
    viper.Debug()
    println("########################################################################")

    res1 := struct {
        Name string `mapstructure:"name"`
        XYZ  bool   `mapstructure:"xyz"`
    }{}
    viper.UnmarshalKey("resources.baz", &res1)
    println("res1 Name:", res1.Name, "; XYZ:", res1.XYZ)
    println("########################################################################")

    viper.Set("resources.baz.name", "override name")
    viper.Debug()
    println("########################################################################")

    res2 := struct {
        Name string `mapstructure:"name"`
        XYZ  bool   `mapstructure:"xyz"`
    }{}
    viper.UnmarshalKey("resources.baz", &res2)
    println("res2 Name:", res2.Name, "; XYZ:", res2.XYZ)
    println("########################################################################")

  println("########################################################################")

  res3 := struct {
    Baz struct {
      Name string `mapstructure:"name"`
      XYZ  bool   `mapstructure:"xyz"`
    } `mapstructure:"baz"`
    }{}
    viper.UnmarshalKey("resources", &res3)
    println("res3 Name:", res3.Baz.Name, "; XYZ:", res3.Baz.XYZ)
    println("########################################################################")

  fmt.Printf("viper.Get: %+v\n", viper.Get("resources"))
  println("########################################################################")
}

Here is the output:

Aliases:
map[string]string{}
Override:
map[string]interface {}{}
PFlags:
map[string]viper.FlagValue{}
Env:
map[string]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{}
Defaults:
map[string]interface {}{}
########################################################################
Aliases:
map[string]string{}
Override:
map[string]interface {}{}
PFlags:
map[string]viper.FlagValue{}
Env:
map[string]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{"resources":map[string]interface {}{"baz":map[string]interface {}{"name":"Super bazzz", "test":123, "xyz":true
 go build
 ./main
Aliases:
map[string]string{}
Override:
map[string]interface {}{}
PFlags:
map[string]viper.FlagValue{}
Env:
map[string]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{}
Defaults:
map[string]interface {}{}
########################################################################
Aliases:
map[string]string{}
Override:
map[string]interface {}{}
PFlags:
map[string]viper.FlagValue{}
Env:
map[string]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{"resources":map[string]interface {}{"baz":map[string]interface {}{"name":"Super bazzz", "test":123, "xyz":true
 go build
 ./main
Aliases:
map[string]string{}
Override:
map[string]interface {}{}
PFlags:
map[string]viper.FlagValue{}
Env:
map[string]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{}
Defaults:
map[string]interface {}{}
########################################################################
Aliases:
map[string]string{}
Override:
map[string]interface {}{}
PFlags:
map[string]viper.FlagValue{}
Env:
map[string]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{"resources":map[string]interface {}{"baz":map[string]interface {}{"name":"Super bazzz", "test":123, "xyz":true
 go build
 ./main
Aliases:
map[string]string{}
Override:
map[string]interface {}{}
PFlags:
map[string]viper.FlagValue{}
Env:
map[string][]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{}
Defaults:
map[string]interface {}{}
########################################################################
Aliases:
map[string]string{}
Override:
map[string]interface {}{}
PFlags:
map[string]viper.FlagValue{}
Env:
map[string][]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{"resources":map[string]interface {}{"baz":map[string]interface {}{"name":"Super bazzz", "test":123, "xyz":true}}}
Defaults:
map[string]interface {}{}
########################################################################
res1 Name: Super bazzz ; XYZ: true
########################################################################
Aliases:
map[string]string{}
Override:
map[string]interface {}{"resources":map[string]interface {}{"baz":map[string]interface {}{"name":"override name"}}}
PFlags:
map[string]viper.FlagValue{}
Env:
map[string][]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{"resources":map[string]interface {}{"baz":map[string]interface {}{"name":"Super bazzz", "test":123, "xyz":true}}}
Defaults:
map[string]interface {}{}
########################################################################
res2 Name: override name ; XYZ: false
########################################################################
########################################################################
res3 Name: override name ; XYZ: false
########################################################################
viper.Get: map[baz:map[name:override name]]
########################################################################

[missing1984]

running into the same issue. any workaround?

[SVilgelm]

there is no workaround, the issue is that the viper has several internal storages. In order to fix the issue, the logic of Get should be changed to merge all possible combinations, like take the baz from the loaded config, then take the baz.name from the overrides and apply it and etc...

[kasparjarek]

Hi, since the release v1.18.0 the issue is also happening for the Unmarshal() function. I suspect that the issue was introduced by this pull request https://github.com/spf13/viper/pull/1429. FYI @krakowski @sagikazarmark

[hieuvubk]

any update on this?

[chris-dot-exe]

Unfortunately we run into this issue too. So I also would be interested on an update to this issue? Any solution in sight?

At the moment the only 'workaround' is to create a completely new viper instance and re-read the config after saving data which obviously isn't a good workaround nor a solution...