search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.04k stars 2.35k forks source link

[Bug]: podman build --secrets with environment variables does not work on MacOS #17382

Open csschwe opened 1 year ago

csschwe commented 1 year ago

Issue Description

Trying to pass secrets into podman build on MacOS does not function as expected

podman build --secret id=my_secret,env=MY_SECRET ls -l /run/secrets/my_secret

Steps to reproduce the issue

Steps to reproduce the issue

  1. export MY_SECRET="hello"
  2. create a Dockerfile 
    FROM alpine:3.17
RUN --mount=type=secret,id=my_secret ls -l /run/secrets/my_secret
  3. podman build --secret id=my_secret,env=MY_SECRET .

Describe the results you received

The secret file during the podman build is empty

Describe the results you expected

The secret file should contain the work hello

podman info output

$ podman info
host:
  arch: arm64
  buildahVersion: 1.28.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.5-1.fc37.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.5, commit: '
  cpuUtilization:
    idlePercent: 98.84
    systemPercent: 0.66
    userPercent: 0.5
  cpus: 2
  distribution:
    distribution: fedora
    variant: coreos
    version: "37"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 503
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.1.6-200.fc37.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 850440192
  memTotal: 4088811520
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.7.2-3.fc37.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.7.2
      commit: 0356bf4aff9a133d655dc13b1d9ac9424706cac4
      rundir: /run/user/503/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/503/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-8.fc37.aarch64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 63h 48m 5.00s (Approximately 2.62 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 4142288896
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 46
  runRoot: /run/user/503/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 1668178831
  BuiltTime: Fri Nov 11 09:00:31 2022
  GitCommit: ""
  GoVersion: go1.19.2
  Os: linux
  OsArch: linux/arm64
  Version: 4.3.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

MacOS

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

github-actions[bot] commented 1 year ago

A friendly reminder that this issue had no activity for 30 days.

rhatdan commented 1 year ago

@flouthoc PTAL

github-actions[bot] commented 1 year ago

A friendly reminder that this issue had no activity for 30 days.

PigeonF commented 1 year ago

This issue also happens on Windows.

Using the containerfile from the comment above

FROM alpine:3.17
RUN --mount=type=secret,id=my_secret ls -l /run/secrets/my_secret
C:\Users\pigeon\Desktop>set MY_SECRET="hello"

C:\Users\pigeon\Desktop>podman build --no-cache --secret id=my_secret,env=MY_SECRET .
STEP 1/2: FROM alpine:3.17
STEP 2/2: RUN --mount=type=secret,id=my_secret ls -l /run/secrets/my_secret
-r--------    1 root     root             0 Apr 12 07:18 /run/secrets/my_secret
COMMIT
--> Pushing cache []:f93e27daaf6f6ca142d2f2d32c9741a4f0839e802999ad2d446f9243f9b371f4
--> b0a20fd4166
b0a20fd416680777d16aa1ba3ff4b07bf04c4b7a40d6352e91f9b5690e94ad67
$ podman info
host:
  arch: amd64
  buildahVersion: 1.29.0
  cgroupControllers:
  - memory
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.5-1.fc36.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.5, commit: '
  cpuUtilization:
    idlePercent: 99.55
    systemPercent: 0.3
    userPercent: 0.14
  cpus: 24
  distribution:
    distribution: fedora
    variant: container
    version: "36"
  eventLogger: journald
  hostname: DESKTOP-FSV9JEV
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.90.1-microsoft-standard-WSL2
  linkmode: dynamic
  logDriver: journald
  memFree: 15860727808
  memTotal: 16602615808
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.1-1.fc36.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.1
      commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 4294967296
  swapTotal: 4294967296
  uptime: 0h 1m 46.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 13
    paused: 0
    running: 0
    stopped: 13
  graphDriverName: overlay
  graphRootAllocated: 1081101176832
  graphRootUsed: 14969270272
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 70
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 4.4.1
  Built: 1676629882
  BuiltTime: Fri Feb 17 11:31:22 2023
  GitCommit: ""
  GoVersion: go1.18.10
  Os: linux
  OsArch: linux/amd64
  Version: 4.4.1
rhatdan commented 1 year ago

@flouthoc Reminder?

flouthoc commented 1 year ago

This one is tricky , I think env variable are passed on to the machine only during init I am not sure if its possible to pass ENV variables again to the machine once it has already started.

rhatdan commented 1 year ago

Should we process the value on the client side? Basically change

--secret id=my_secret,env=MY_SECRET to --secret id=my_secret,env=MY_SECRET=$MY_SECRET Before sending it across?

rhatdan commented 1 year ago

@ashley-cui @flouthoc Thoughts

rhatdan commented 1 year ago

We could send os.environ across to the build side to be used by the processor? Does this work with Docker?

flouthoc commented 1 year ago

We could send os.environ across to the build side to be used by the processor? Does this work with Docker?

@rhatdan I like this idea this can work, we can create a new hidden field to transfer current ENV from client to server, and build can use this field for getting ENV details and use it while populating secret.

aaronmcohen commented 9 months ago

Any recommendations for a workaround? I am hitting this issue as well.

rhatdan commented 8 months ago

@flouthoc is not active maintainer of podman anymore, so we are a little short staffed right now. Interested in working on a PR?

flouthoc commented 8 months ago

We could send os.environ across to the build side to be used by the processor? Does this work with Docker?

@rhatdan I like this idea this can work, we can create a new hidden field to transfer current ENV from client to server, and build can use this field for getting ENV details and use it while populating secret.

I think this can still be done, is someone interested in taking this issue ? If not I'd like to take a stab at it in some time. Thanks.

flouthoc commented 8 months ago

@aaronmcohen I can implement this in few weeks as per my availability, but If you want I can help you implement it as well.

nightlark commented 5 months ago

Any updates on this issue, or pointers to where in the podman source code would be a good place to start looking to make changes to support this?

rhatdan commented 5 months ago

@nightlark interested in working on it?

nightlark commented 5 months ago

Sure, if I can figure out where in podman (and buildah?) changes need to be made.

filip-komarzyniec commented 2 months ago

Any progress on the issue? I've tried exporting the variable both locally (on mac) and in podman VM (in both .bashrc and _.bashprofile files).

podman info ```shell host: arch: arm64 buildahVersion: 1.36.0 cgroupControllers: - cpu - io - memory - pids cgroupManager: systemd cgroupVersion: v2 conmon: package: conmon-2.1.10-1.fc40.aarch64 path: /usr/bin/conmon version: 'conmon version 2.1.10, commit: ' cpuUtilization: idlePercent: 98.94 systemPercent: 0.13 userPercent: 0.93 cpus: 6 databaseBackend: sqlite distribution: distribution: fedora variant: coreos version: "40" eventLogger: journald freeLocks: 2046 hostname: localhost.localdomain idMappings: gidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 1000000 uidmap: - container_id: 0 host_id: 501 size: 1 - container_id: 1 host_id: 100000 size: 1000000 kernel: 6.8.11-300.fc40.aarch64 linkmode: dynamic logDriver: journald memFree: 153284608 memTotal: 2044387328 networkBackend: netavark networkBackendInfo: backend: netavark dns: package: aardvark-dns-1.11.0-1.20240531102943328308.main.4.g6838c50.fc40.aarch64 path: /usr/libexec/podman/aardvark-dns version: aardvark-dns 1.12.0-dev package: netavark-1.11.0-1.20240606174759319307.main.8.gfebe31a.fc40.aarch64 path: /usr/libexec/podman/netavark version: netavark 1.12.0-dev ociRuntime: name: crun package: crun-1.15-1.20240607090105650503.main.32.gea54402.fc40.aarch64 path: /usr/bin/crun version: |- crun version UNKNOWN commit: 7cfd0aeb40e4605b6b0ee0afd9cfca80f9c5f68a rundir: /run/user/501/crun spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL os: linux pasta: executable: /usr/bin/pasta package: passt-0^20240510.g7288448-1.fc40.aarch64 version: | pasta 0^20240510.g7288448-1.fc40.aarch64-pasta Copyright Red Hat GNU General Public License, version 2 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. remoteSocket: exists: true path: /run/user/501/podman/podman.sock rootlessNetworkCmd: pasta security: apparmorEnabled: false capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: true seccompEnabled: true seccompProfilePath: /usr/share/containers/seccomp.json selinuxEnabled: true serviceIsRemote: true slirp4netns: executable: /usr/bin/slirp4netns package: slirp4netns-1.2.2-2.fc40.aarch64 version: |- slirp4netns version 1.2.2 commit: 0ee2d87523e906518d34a6b423271e4826f71faf libslirp: 4.7.0 SLIRP_CONFIG_VERSION_MAX: 4 libseccomp: 2.5.3 swapFree: 0 swapTotal: 0 uptime: 25h 2m 27.00s (Approximately 1.04 days) variant: v8 plugins: authorization: null log: - k8s-file - none - passthrough - journald network: - bridge - macvlan - ipvlan volume: - local registries: search: - docker.io store: configFile: /var/home/core/.config/containers/storage.conf containerStore: number: 1 paused: 0 running: 1 stopped: 0 graphDriverName: overlay graphOptions: {} graphRoot: /var/home/core/.local/share/containers/storage graphRootAllocated: 106769133568 graphRootUsed: 12962926592 graphStatus: Backing Filesystem: xfs Native Overlay Diff: "true" Supports d_type: "true" Supports shifting: "false" Supports volatile: "true" Using metacopy: "false" imageCopyTmpDir: /var/tmp imageStore: number: 49 runRoot: /run/user/501/containers transientStore: false volumePath: /var/home/core/.local/share/containers/storage/volumes version: APIVersion: 5.1.1 Built: 1717459200 BuiltTime: Tue Jun 4 02:00:00 2024 GitCommit: "" GoVersion: go1.22.3 Os: linux OsArch: linux/arm64 Version: 5.1.1 ```
KarolGongola commented 2 days ago

I am not sure if this is the same issue, but if someone wants workaround, you can check this: https://github.com/containers/podman/issues/23815#issuecomment-2343865319