Podman JSON format issues

[ajram23]

Issue Description

I am trying to copy a json file into a container using GitHub Actions, when I cat the json file after I do a copy (in Github action) I can see its formatted correctly but when the package is downloaded and I open the terminal and cat the file its missing double quotes for keys. My python fails with invalid JSON format.

Steps to reproduce the issue

Steps to reproduce the issue

1. Copy a JSON file in GitHub Actions, I am using a credentials.json in base64 in Github secrets
2. Cat the file after copy command in your docker file you will see the file is printed correctly
3. Once the image is installed, open podman desktop open terminal and cat the file, the json format will be messed up

Describe the results you received

JSON format is screwed up with double quotes for keys

Describe the results you expected

proper JSON format

podman info output

podman info
host:
  arch: arm64
  buildahVersion: 1.36.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 99.48
    systemPercent: 0.21
    userPercent: 0.31
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "40"
  eventLogger: journald
  freeLocks: 2045
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.8.11-300.fc40.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 72093696
  memTotal: 2044563456
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.11.0-1.20240531102943328308.main.4.g6838c50.fc40.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.0-dev
    package: netavark-1.11.0-1.20240606174759319307.main.8.gfebe31a.fc40.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.0-dev
  ociRuntime:
    name: crun
    package: crun-1.15-1.20240607090105650503.main.32.gea54402.fc40.aarch64
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: 7cfd0aeb40e4605b6b0ee0afd9cfca80f9c5f68a
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240510.g7288448-1.fc40.aarch64
    version: |
      pasta 0^20240510.g7288448-1.fc40.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 41h 13m 39.00s (Approximately 1.71 days)
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 6281281536
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 3
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 5.1.1
  Built: 1717459200
  BuiltTime: Mon Jun  3 20:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.3
  Os: linux
  OsArch: linux/arm64
  Version: 5.1.1

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

I am on Mac

Additional information

I have tried JQ as well, same results!

[ajram23]

Also I should mention I am copying the file to a shared volume. Not sure if it makes a difference.

[ajram23]

Looks like it did make a difference, copying the exact same file to the container (works) and the shared volume (doesnt work has screwed up the JSON formatting)

[Luap99]

Please provide an exact reproducer, what commands did you use? What file etc...

If you can only reproduce using Podman Desktop you have to file the issue there: https://github.com/containers/podman-desktop

[ajram23]

@Luap99 More details

Decode the base64 credentials and write to the shared volume

RUN echo "$BASE64_CREDENTIALS" | base64 --decode | jq '.' > /app/working-docs/config/credentials.json

Copy the same content to a non-shared volume for comparison

RUN echo "$BASE64_CREDENTIALS" | base64 --decode | jq '.' > /app/datasync/credentials.json

Verify content

RUN cat /app/working-docs/config/credentials.json RUN cat /app/datasync/credentials.json

Base64 Encoded Credentials (Dummy Data): bash BASE64_CREDENTIALS=$(echo -n '{ "installed": { "client_id": "dummy_client_id", "project_id": "dummy_project_id", "auth_uri": "https://dummy.auth.uri", "token_uri": "https://dummy.token.uri", "auth_provider_x509_cert_url": "https://dummy.cert.url", "client_secret": "dummy_client_secret", "redirect_uris": ["http://dummy.redirect.uri"] } }' | base64)

Expected Output: json { "installed": { "client_id": "dummy_client_id", "project_id": "dummy_project_id", "auth_uri": "https://dummy.auth.uri", "token_uri": "https://dummy.token.uri", "auth_provider_x509_cert_url": "https://dummy.cert.url", "client_secret": "dummy_client_secret", "redirect_uris": ["http://dummy.redirect.uri"] } } Actual Output in Shared Volume: json {installed:{client_id:dummy_client_id,project_id:dummy_project_id,auth_uri:https://dummy.auth.uri,token_uri:https://dummy.token.uri,auth_provider_x509_cert_url:https://dummy.cert.url,client_secret:dummy_client_secret,redirect_uris:[http://dummy.redirect.uri]}}

Actual Output in Non-Shared Volume: json { "installed": { "client_id": "dummy_client_id", "project_id": "dummy_project_id", "auth_uri": "https://dummy.auth.uri", "token_uri": "https://dummy.token.uri", "auth_provider_x509_cert_url": "https://dummy.cert.url", "client_secret": "dummy_client_secret", "redirect_uris": ["http://dummy.redirect.uri"] } } This is not a podman desktop issue as I can see the problem in podman cli and my python code which accesses the JSON and results in invalid JSON error.

[Luap99]

Please provide the full commands used and the Container/Dockerfile. It is hard for us to juts guess what you are doing based of this.

In general if you have a line like this

echo "$BASE64_CREDENTIALS" | base64 --decode | jq '.' > /app/datasync/credentials.json

then podman has really no way to magically remove quotes as it doesn't touch the output at all if it redirected via the shell.

I really don't understand why there would be a difference between volume or no volume.

[ajram23]

@Luap99 Neither do I understand why there is a difference. Just reporting what I am seeing. Here is the full Dockerfile # Base image
FROM python:3.11-slim

Set the working directory in the container

WORKDIR /app/datasync

Install jq

RUN apt-get update && apt-get install -y jq

RUN mkdir -p /app/working-docs RUN mkdir -p /app/working-docs/databases RUN mkdir -p /app/working-docs/config RUN mkdir -p /app/working-docs/input RUN mkdir -p /app/working-docs/tmp

Add the build argument for the base64-encoded credentials

ARG BASE64_CREDENTIALS

Copy the necessary files

Decode the base64 string and format it with jq

RUN echo "$BASE64_CREDENTIALS" | base64 --decode | jq '.' > /app/working-docs/config/credentials.json RUN echo "$BASE64_CREDENTIALS" | base64 --decode | jq '.' > /app/datasync/credentials.json

Verify the contents after decoding

RUN cat /app/working-docs/config/credentials.json RUN ls -la /app/working-docs/config

COPY .env_production /app/datasync/.env COPY datasync.py /app/datasync/ COPY utils.py /app/datasync/ COPY utils_google.py /app/datasync/ COPY config.py /app/datasync/ COPY requirements_datasync.txt /app/datasync/

Install dependencies

RUN pip install --no-cache-dir -r requirements_datasync.txt

Command to run the script

CMD ["python", "datasync.py", "service"]

[ajram23]

@Luap99 Hope you have everything you need. Please let me if you need anything else.

[ajram23]

@Luap99 Please, any updates on this thread?

[Luap99]

It would help if you could create a minimal reducers that I can run and please format your output (use markdown code block for the Dockerfile) and should the build command used. And I don't see here were you actually define BASE64_CREDENTIALS

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.