search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
22.4k stars 2.31k forks source link

Every podman command returns "Error: scanning container state row: sql: Scan error on column index 1, name JSON: converting NULL to string is unsupported" #23156

Open AlphaJack opened 3 days ago

AlphaJack commented 3 days ago

Issue Description

I cannot start any network or container quadlet because of the error

Error: scanning container state row: sql: Scan error on column index 1, name "JSON": converting NULL to string is unsupported

Steps to reproduce the issue

Steps to reproduce the issue

  1. Have a working Podman setup
  2. Reboot
  3. Cannot restart any quadlet because Jul 01 17:31:46 P4 pod-network[8158]: Error: scanning container state row: sql: Scan error on column index 1, name "JSON": converting NULL to string is unsupported:
  4. podman container ps and other commands returns the same error

Describe the results you received

Describe the results you received

Jul 01 17:06:38 P4 systemd[564]: Starting Pod network...
░░ Subject: A start job for unit UNIT has begun execution
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit UNIT has begun execution.
░░ 
░░ The job identifier is 28.
Jul 01 17:08:48 P4 pod-network[660]: Error: scanning container state row: sql: Scan error on column index 1, name "JSON": converting NULL to string is unsupported
Jul 01 17:08:49 P4 systemd[564]: pod-network.service: Main process exited, code=exited, status=125/n/a
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ An ExecStart= process belonging to unit UNIT has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 125.
Jul 01 17:08:49 P4 systemd[564]: pod-network.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit UNIT has entered the 'failed' state with result 'exit-code'.
Jul 01 17:08:49 P4 systemd[564]: Failed to start Pod network.
░░ Subject: A start job for unit UNIT has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit UNIT has finished with a failure.
░░ 
░░ The job identifier is 28 and the job result is failed.

Describe the results you expected

Quadlets are activated as before

podman info output

~ Gasolio@P4|> podman container ps
Error: scanning container state row: sql: Scan error on column index 1, name "JSON": converting NULL to string is unsupported
~ Gasolio@P4|> podman info                  
Error: scanning container state row: sql: Scan error on column index 1, name "JSON": converting NULL to string is unsupported
~ Gasolio@P4|> podman version
Error: scanning container state row: sql: Scan error on column index 1, name "JSON": converting NULL to string is unsupported

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

sudo podman version works, and so do sudo podman info:

host:                                                                                                                                                                                                                                                                                                                                              
  arch: arm64                                                                                                                                                                                                                                                                                                                                      
  buildahVersion: 1.36.0                                                                                                                                                                                                                                                                                                                           
  cgroupControllers:                                                                                                                                                                                                                                                                                                                               
  - cpuset                                                                                                                                                                                                                                                                                                                                         
  - cpu                                                                                                                                                                                                                                                                                                                                            
  - io                                                                                                                                                                                                                                                                                                                                             
  - memory                                                                                                                                                                                                                                                                                                                                         
  - hugetlb                                                                                                                                                                                                                                                                                                                                        
  - pids                                                                                                                                                                                                                                                                                                                                           
  - rdma                                                                                                                                                                                                                                                                                                                                           
  cgroupManager: systemd                                                                                                                                                                                                                                                                                                                           
  cgroupVersion: v2                                                                                                                                                                                                                                                                                                                                
  conmon:                                                                                                                                                                                                                                                                                                                                          
    package: /usr/bin/conmon is owned by conmon 1:2.1.12-1                                                                                                                                                                                                                                                                                         
    path: /usr/bin/conmon                                                                                                                                                                                                                                                                                                                          
    version: 'conmon version 2.1.12, commit: e8896631295ccb0bfdda4284f1751be19b483264'                                                                                                                                                                                                                                                             
  cpuUtilization:                                                                                                                                                                                                                                                                                                                                  
    idlePercent: 96.93                                                                                                                                                                                                                                                                                                                             
    systemPercent: 2.18                                                                                                                                                                                                                                                                                                                            
    userPercent: 0.89                                                                                                                                                                                                                                                                                                                              
  cpus: 4                                                                                                                                                                                                                                                                                                                                          
  databaseBackend: sqlite                                                                                                                                                                                                                                                                                                                          
  distribution:                                                                                                                                                                                                                                                                                                                                    
    distribution: archarm                                                                                                                                                                                                                                                                                                                          
    version: unknown                                                                                                                                                                                                                                                                                                                               
  eventLogger: journald                                                                                                                                                                                                                                                                                                                            
  freeLocks: 2048                                                                                                                                                                                                                                                                                                                                  
  hostname: P4                                                                                                                                                                                                                                                                                                                                     
  idMappings:                                                                                                                                                                                                                                                                                                                                      
    gidmap: null                                                                                                                                                                                                                                                                                                                                   
    uidmap: null                                                                                                                                                                                                                                                                                                                                   
  kernel: 6.9.7-1-aarch64-ARCH                                                                                                                                                                                                                                                                                                                     
  linkmode: dynamic                                                                                                                                                                                                                                                                                                                                
  logDriver: journald                                                                                                                                                                                                                                                                                                                              
  memFree: 3171639296                                                                                                                                                                                                                                                                                                                              
  memTotal: 3908177920                                                                                                                                                                                                                                                                                                                             
  networkBackend: netavark                                                                                                                                                                                                                                                                                                                         
  networkBackendInfo:                                                                                                                                                                                                                                                                                                                              
    backend: netavark                                                                                                                                                                                                                                                                                                                              
    dns:                                                                                                                                                                                                                                                                                                                                           
      package: /usr/lib/podman/aardvark-dns is owned by aardvark-dns 1.11.0-1                                                                                                                                                                                                                                                                      
      path: /usr/lib/podman/aardvark-dns                                                                                                                                                                                                                                                                                                           
      version: aardvark-dns 1.11.0                                                                                                                                                                                                                                                                                                                 
    package: /usr/lib/podman/netavark is owned by netavark 1.11.0-2                                                                                                                                                                                                                                                                                
    path: /usr/lib/podman/netavark                                                  
    version: netavark 1.11.0                                                        
  ociRuntime:                                                                       
    name: crun                                                                      
    package: /usr/bin/crun is owned by crun 1.15-1
    path: /usr/bin/crun                                                             
    version: |-                                                                     
      crun version 1.15                                                             
      commit: e6eacaf4034e84185fd8780ac9262bbf57082278
      rundir: /run/user/1000/crun                                                   
      spec: 1.0.0                                                                   
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux                                                                         
  pasta:                                                                            
    executable: /usr/bin/pasta                                                      
    package: /usr/bin/pasta is owned by passt 2024_06_07.8a83b53-1
    version: |                                                                      
      pasta 2024_06_07.8a83b53                                                      
      Copyright Red Hat                                                             
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:                                                                     
    exists: true                                                                    
    path: /run/podman/podman.sock                                                   
  rootlessNetworkCmd: pasta                                                         
  security:                                                                         
    apparmorEnabled: false                                                          
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false                                                                 
    seccompEnabled: true                                                            
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false                                                           
  serviceIsRemote: false                                                            
  slirp4netns:                                                                      
    executable: ""                                                                  
    package: ""                                                                     
    version: ""                                                                     
  swapFree: 0                                                                       
  swapTotal: 0                                                                      
  uptime: 0h 36m 7.00s                                                              
  variant: v8
plugins:                                                                    
  authorization: null                                                               
  log:                                                                              
  - k8s-file                                                                        
  - none                                                                            
  - passthrough                                                                     
  - journald                                                                        
  network:                                                                          
  - bridge                                                                          
  - macvlan                                                                         
  - ipvlan                                                                          
  volume:                                                                           
  - local                                                                           
registries:                                                                         
  search:                                                                           
  - docker.io                                                                       
store:                                                                              
  configFile: /etc/containers/storage.conf
  containerStore:                                                                   
    number: 0                                                                       
    paused: 0                                                                       
    running: 0                                                                      
    stopped: 0                                                                      
  graphDriverName: overlay                                                          
  graphOptions:                                                                     
    overlay.mountopt: nodev                                                         
  graphRoot: /var/lib/containers/storage                                            
  graphRootAllocated: 65732751360                                                   
  graphRootUsed: 12476723200                                                        
  graphStatus:                                                                      
    Backing Filesystem: extfs                                                       
    Native Overlay Diff: "true"                                                     
    Supports d_type: "true"                                                         
    Supports shifting: "true"                                                       
    Supports volatile: "true"                                                       
    Using metacopy: "false"                                                         
  imageCopyTmpDir: /var/tmp                                                         
  imageStore:                                                                       
    number: 0                                                                       
  runRoot: /run/containers/storage                                                  
  transientStore: false                                                             
  volumePath: /var/lib/containers/storage/volumes
version:                                                                            
  APIVersion: 5.1.1                                                                 
  Built: 1717546137                                                                 
  BuiltTime: Wed Jun  5 02:08:57 2024                                               
  GitCommit: bda6eb03dcbcf12a5b7ae004c1240e38dd056d24-dirty
  GoVersion: go1.22.3                                                               
  Os: linux                                                                         
  OsArch: linux/arm64                                                               
  Version: 5.1.1
Luap99 commented 3 days ago

This sounds like some form of database corruption, either because podman wrote something invalid or because the actual sqlite db is corrupted.

cc @mheon

Luap99 commented 3 days ago

The JSON field is defined as JSON TEXT NOT NULL so why do we get NULL back from the db here? name "JSON": converting NULL to string is unsupported

Luap99 commented 3 days ago

@AlphaJack Can you run sqlite3 ~/.local/share/containers/storage/db.sql and then SELECT * FROM ContainerState; inside there to show us how the table looks there.

AlphaJack commented 3 days ago

@Luap99 

~ Gasolio@P4|> sqlite3 ~/.local/share/containers/storage/db.sql
SQLite version 3.46.0 2024-05-23 13:25:27
Enter ".help" for usage hints.

sqlite> SELECT * FROM ContainerState;
7dd71076b254bb2b7d7d45ff3b06c2f1ac1e8f5c3e1c0a1c342d670c5809fe83|21b209447f5eef5f8de6c095e7157b873e10ea7908f45eb6be368a2a3f3ad974||

sqlite> PRAGMA table_info(ContainerState);
0|ID|TEXT|1||1
1|State|INTEGER|1||0
2|ExitCode|INTEGER|0||0
3|JSON|TEXT|1||0

Removing the file seems to have fixed the issue for the network quadlet, but not for the containers:

Jul 01 18:26:45 P4 podman[25941]: 2024-07-01 18:26:45.82353107 +0200 CEST m=+0.084629881 image pull 8ef1bb9aad44f85399ca3bfbfb26f676e4299c76a424beb829d0288a88440042 ghcr.io/blakeblackshear/frigate:stable
Jul 01 18:26:46 P4 podman[25941]: 2024-07-01 18:26:46.035131519 +0200 CEST m=+0.296230311 container create ad2dcba28e63cf93bcd8f65d45d8d6bfd8e4904c058417f9562c5ad30c08d3b5 (image=ghcr.io/blakeblackshear/frigate:stable, name=frigate, PODMAN_S>
Jul 01 18:26:46 P4 conmon[26047]: conmon ad2dcba28e63cf93bcd8 <nwarn>: runtime stderr: access `/run/user/1000/crun/ad2dcba28e63cf93bcd8f65d45d8d6bfd8e4904c058417f9562c5ad30c08d3b5`: Permission denied
Jul 01 18:26:46 P4 conmon[26047]: conmon ad2dcba28e63cf93bcd8 <error>: Failed to create container: exit status 1
Jul 01 18:26:46 P4 frigate[26050]: cannot open run directory `/run/user/1000/crun`: Permission denied
Jul 01 18:26:46 P4 frigate[25941]: time="2024-07-01T18:26:46+02:00" level=error msg="Removing container ad2dcba28e63cf93bcd8f65d45d8d6bfd8e4904c058417f9562c5ad30c08d3b5 from runtime after creation failed"
Jul 01 18:26:47 P4 podman[25941]: 2024-07-01 18:26:47.567421118 +0200 CEST m=+1.828519891 container remove ad2dcba28e63cf93bcd8f65d45d8d6bfd8e4904c058417f9562c5ad30c08d3b5 (image=ghcr.io/blakeblackshear/frigate:stable, name=frigate, PODMAN_S>
Jul 01 18:26:47 P4 frigate[25941]: Error: crun: access `/run/user/1000/crun/ad2dcba28e63cf93bcd8f65d45d8d6bfd8e4904c058417f9562c5ad30c08d3b5`: Permission denied: OCI permission denied
Jul 01 18:26:47 P4 systemd[564]: frigate.service: Main process exited, code=exited, status=126/n/a
Luap99 commented 3 days ago

Thanks

7dd71076b254bb2b7d7d45ff3b06c2f1ac1e8f5c3e1c0a1c342d670c5809fe83|21b209447f5eef5f8de6c095e7157b873e10ea7908f45eb6be368a2a3f3ad974||

That does not look right, 21b209447f5eef5f8de6c095e7157b873e10ea7908f45eb6be368a2a3f3ad974 is clearly not an integer. I am not really familiar with sqlite so it is hard to tell what went wrong. Did you have a hard shutdown or powerloss maybe?

To fix your system the best chance is to remove or better rename it so we can have a peak at it if needed then the quadlets should regenerate the necessary containers in theory

Luap99 commented 3 days ago

Removing the file seems to have fixed the issue for the network quadlet, but not for the containers:

Maybe try to reboot again, there is some special refresh logic that must run after boot maybe the weird the state caused that to fail.

Luap99 commented 3 days ago

If you still have the broken file following the steps here (under 2) might be interesting: https://www.sqlite.org/recovery.html

AlphaJack commented 3 days ago

Now it works again, I run:

mv ~/.local/share/containers/storage/db.sql ~/.local/share/containers/storage/db.sql.old`
podman system reset
sudo reboot now

Would be still interesting to understand the cause and how a non-int value was saved to Sqlite

Luap99 commented 3 days ago

Well it is not like you can just like safe a invalid value. This really sounds like a proper database corruption hence why question if you had a hard shutdown or powerloss? Or maybe check your hard drive for disk errors etc... Of course it could also be a bug in podman or sqlite3 https://www.sqlite.org/howtocorrupt.html

Without having a reproducer it will be basically impossible to find out.

AlphaJack commented 2 days ago

Without having a reproducer it will be basically impossible to find out.

Then I think we can close the issue and leave the steps I mentioned as a workaround