storage.conf oscillates between camel and lower case formats

[debarshiray]

/kind bug

When I remove the local configuration (rm -f ~/.config/containers/*.conf) and data (sudo rm -rf ~/.local/share/containers), and start from a clean state, podman creates a new storage.conf for me. The format is sometimes in camel case, and at other times in lower case.

Camel case:

RunRoot = "/run/user/1000"
GraphRoot = "/home/rishi/.local/share/containers/storage"
GraphDriverName = "overlay"
GraphDriverOptions = ["overlay.mount_program=/usr/bin/fuse-overlayfs"]

Lower case:

[storage]
  driver = "overlay"
  runroot = "/run/user/1000"
  graphroot = "/home/rishi/.local/share/containers/storage"
  [storage.options]
    mount_program = "/usr/bin/fuse-overlayfs"

I learnt in #podman that the latter is the correct format.

I haven't been able to figure out a reliable reproducer, but I have seen it fail with this error on some occasions when the camel case format was in play:

ERRO[0000] 'overlay' is not supported over extfs at "/home/rishi/.local/share/containers/storage/overlay"

Output of podman version:

Version:       1.0.0
Go Version:    go1.11.4
Git Commit:    "49780a1cf10d572edc4e1ea3b8a8429ce391d47d"
Built:         Mon Jan 14 21:38:17 2019
OS/Arch:       linux/amd64

Output of podman info:

host:
  BuildahVersion: 1.6-dev
  Conmon:
    package: podman-1.0.0-1.git82e8011.fc29.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: 49780a1cf10d572edc4e1ea3b8a8429ce391d47d'
  Distribution:
    distribution: fedora
    version: "29"
  MemFree: 1806233600
  MemTotal: 8127070208
  OCIRuntime:
    package: runc-1.0.0-66.dev.gitbbb17ef.fc29.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc6+dev
      commit: ead425507b6ba28278ef71ad06582df97f2d5b5f
      spec: 1.0.1-dev
  SwapFree: 4133482496
  SwapTotal: 4133482496
  arch: amd64
  cpus: 4
  hostname: kolache
  kernel: 4.19.13-300.fc29.x86_64
  os: linux
  rootless: true
  uptime: 2h 28m 31.86s (Approximately 0.08 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/rishi/.config/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /home/rishi/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
  ImageStore:
    number: 2
  RunRoot: /run/user/1000

[mheon]

@giuseppe PTAL

[giuseppe]

are you always using the same version of Podman? The old wrong format was generated by an older version and should not happen anymore.

[debarshiray]

Ok, that's a good question. I have been switching between Git master, 0.12.1.2, 1.0.0, and so on. I'll keep a closer eye on the version if it happens again.

Feel free to close this if you think it's fixed. Sorry for the noise.

[giuseppe]

no problem, it is good to report these issues so we can look at them. Yes this should be fixed upstream, and I think there was only one version of podman affected by the issue.

[debarshiray]

It might be that this isn't fully fixed, yet.

I just installed podman and buildah on a Fedora 28 machine which hadn't seen those tools yet.

$ rpm -q podman buildah
podman-1.0.0-1.git82e8011.fc28.x86_64
buildah-1.5-2.gite94b4f9.fc28.x86_64

Since fuse-overlayfs doesn't get pulled in automatically on Fedora 28, I got a storage.conf with the vfs driver, but the format was wrong:

RunRoot = "/run/user/1000"
GraphRoot = "/home/rishi/.local/share/containers/storage"
GraphDriverName = "vfs"

[debarshiray]

A user with podman-1.2.0-2.git3bd528e.fc30.x86_64 on Fedora Silverblue 30 just reported this in #silverblue. He had a storage.conf with camel case content and podman inspect fails with:

ERRO[0000] 'overlay' is not supported over extfs at "/var/home/lorenzodalrio/.local/share/containers/storage/overlay" 

And there's been a constant trickle of users in the wild who keep reporting this.

The full logs are:

$ toolbox -v create
toolbox: Fedora generational core is f30
toolbox: base image is fedora-toolbox:30
toolbox: customized user-specific image is fedora-toolbox-lorenzodalrio:30
toolbox: container is fedora-toolbox-lorenzodalrio:30
toolbox: checking if image fedora-toolbox-lorenzodalrio:30 already exists
ERRO[0000] 'overlay' is not supported over extfs at "/var/home/lorenzodalrio/.local/share/containers/storage/overlay" 
ERRO[0000] exit status 1                                
toolbox: looking for image localhost/fedora-toolbox:30
ERRO[0000] 'overlay' is not supported over extfs at "/var/home/lorenzodalrio/.local/share/containers/storage/overlay" 
Pulling docker://localhost/fedora-toolbox:30
ERRO[0000] exit status 1                                
toolbox: looking for image registry.fedoraproject.org/f30/fedora-toolbox:30
ERRO[0000] 'overlay' is not supported over extfs at "/var/home/lorenzodalrio/.local/share/containers/storage/overlay" 
Pulling docker://registry.fedoraproject.org/f30/fedora-toolbox:30
ERRO[0000] exit status 1                                
toolbox: failed to pull base image fedora-toolbox:30

Note that the eventual failure to pull is due to https://github.com/containers/buildah/issues/1504.

[debarshiray]

A user with podman-1.2.0-2.git3bd528e.fc30.x86_64 on Fedora Silverblue 30 just reported this in #silverblue. He had a storage.conf with camel case content and podman inspect

After some back and forth on IRC, it turned out that the user had been using Podman for a while using the same $HOME even though the rest of the installation was fresh. So the messed up storage.conf might have been due to an old Podman version.