search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.71k stars 2.41k forks source link

Support native source folder for volume mount in remote model #8016

Closed jeffwubj closed 2 years ago

jeffwubj commented 4 years ago

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Description

Currently, looks like in podman's remote client case, bind volume will use source folders on podman server's host, could we support (or give some configurations to choose) using source folders on podman client's host?

podman client and podman server may run in different hosts, or one runs in the host, the other runs in a VM. Have the ability to mount folders on the client side might be able to open more use cases...

dmitris commented 2 years ago

I'm happy to test on my M1 anything you'd like.

+1

benben commented 2 years ago

Ben reporting with another M1 for testing duty.

afbjorklund commented 2 years ago

Docker Desktop is switching over to virtio-fs, from their previous four (or so) remote file systems on Mac.

https://www.docker.com/blog/file-sharing-with-docker-desktop/ (that is: https://virtio-fs.gitlab.io/)

The WSL2 VM is still using the same "9p" network protocol as virtfs (and also "minikube mount") is using.

https://devblogs.microsoft.com/commandline/whats-new-for-wsl-in-windows-10-version-1903/

jorhett commented 2 years ago

@afbjorklund I'm trying to parse this doc which appears to answer my question about how to build qemu... but it starts with an unqualified phrase

https://github.com/containers/podman/blob/main/docs/tutorials/mac_experimental.md

You must obtain a compressed tarball

From whom? Where?

Also after all that the example at the bottom doesn't use a mount. The example works from Homebrew without modification 🤷‍♂️

jorhett commented 2 years ago

As far as I know there is nothing pending in podman, but if you want to use "virtfs" you need a qemu version that supports it...

@afbjorklund I know this is all clear in your head, but I'm going to restate this for simplicity and you tell me if I'm right:

$ podman machine init --volume /Users --volume /Volumes

HOWEVER

So if we build QEMU according to those instructions and then build podman 4.0-rc3 we should have working Mac volumes mounting to the VM, which will then enable pods to mount them?

afbjorklund commented 2 years ago

@jorhett : thanks for the summary! That looks about right.

I'm hoping that there will be at least a brew version of qemu with 9p-darwin available, and that the resulting qemu-system-x86_64 and qemu-system-aarch64 can be bundled with Podman Desktop 4.0 so that the machine volumes will work also on Mac.

Then change would be something like this, for brew install --HEAD qemu:

--- Formula/qemu.rb.orig    2021-12-15 08:07:23.000000000 +0100
+++ Formula/qemu.rb 2022-01-30 10:38:21.992247586 +0100
@@ -4,7 +4,7 @@
   url "https://download.qemu.org/qemu-6.2.0.tar.xz"
   sha256 "68e15d8e45ac56326e0b9a4afa8b49a3dfe8aba3488221d098c84698bca65b45"
   license "GPL-2.0-only"
-  head "https://git.qemu.org/git/qemu.git", branch: "master"
+  head "https://gitlab.com/wwcohen/qemu.git", branch: "9p-darwin"

   bottle do
     sha256 arm64_monterey: "57ffc7f97cbd053121f7767f08c2386beac29ef8efa7f93398d347b24d979621"
@@ -59,6 +59,7 @@
       --enable-libssh
       --enable-slirp=system
       --enable-vde
+      --enable-virtfs
       --extra-cflags=-DNCURSES_WIDECHAR=1
       --disable-sdl
       --disable-gtk

Here is the compare against v6.2.0

Patch: qemu-6.2.0-9p-darwin.patch

So if we build QEMU according to those instructions and then build podman 4.0-rc3 we should have working Mac volumes mounting to the VM, which will then enable pods to mount them?

That is the idea. It will need some more testing and documentation, and hopefully the feature ("virtfs") will be included by default in future versions for say QEMU 7.0.0 or something like that ? The support in Podman should already be there in 4.0.0.

It is also possible to add more options (beyond "virtfs") for --volume-driver.

For instance: NFS, SMB

afbjorklund commented 2 years ago

I'm trying to parse this doc which appears to answer my question about how to build qemu...

The podman documentation has not updated beyond the man page, as far as I know ?

https://docs.podman.io/en/latest/markdown/podman-machine-init.1.html

In case you get stuck at details, the upstream docs are at: https://www.qemu.org/download/

https://github.com/containers/podman/blob/main/docs/tutorials/mac_experimental.md

It should be about adding a --enable-virtfs parameter to the ../configure invocation ?

The "You must obtain a compressed tarball" is outdated, afaik. They should all be in brew.

willcohen commented 2 years ago

There’s a few more changes we still need to make (https://github.com/NixOS/nixpkgs/pull/122420) but I am hoping that QEMU accepts the 9p changes for 7.0!

afbjorklund commented 2 years ago

The more I hear about the problems with remote filesystem by default, the more I believe the whole model is flawed...

Like in this article. https://medium.com/homullus/beating-some-performance-into-docker-for-mac-f5d1e732032c

This is why I earlier proposed to keep the files on the VM and mount them to the host, and it still seems like the best:

Whether it is vboxsf or 9p, or virtfs and even virtiofs, it seems like there is no comparison as to just using regular files ?

This Wizard of Oz model has all the problems of the nfs disks, except new and exiting bugs instead of the old ones.

ezeik1991 commented 2 years ago

So if we build QEMU according to those instructions and then build podman 4.0-rc3 we should have working Mac volumes mounting to the VM, which will then enable pods to mount them?

Hello! Were you able to build qemu following this instruction?

afbjorklund commented 2 years ago

Will try to make a custom brew repo with "qemu", for easier install on Darwin (and maybe also to fix the display on Linux, missing gtk)

EDIT: see https://github.com/afbjorklund/homebrew-core/tree/qemu-9p-darwin

matteosilv commented 2 years ago

As far as I know there is nothing pending in podman, but if you want to use "virtfs" you need a qemu version that supports it...

@afbjorklund I know this is all clear in your head, but I'm going to restate this for simplicity and you tell me if I'm right:

  • Version 4.0 (not yet released) includes podman machine init --volume which will mount a volume into the VM like this might to match Docker Desktop behavior:
$ podman machine init --volume /Users --volume /Volumes

HOWEVER

So if we build QEMU according to those instructions and then build podman 4.0-rc3 we should have working Mac volumes mounting to the VM, which will then enable pods to mount them?

Will we need to keep mounting as -v /Users:/mnt/Users or has this been addressed in some way?

afbjorklund commented 2 years ago

Will we need to keep mounting as -v /Users:/mnt/Users or has this been addressed in some way?

You should be able to mount it at /Users now, with the later versions of Fedora CoreOS (auto-updated)

matteosilv commented 2 years ago

Will we need to keep mounting as -v /Users:/mnt/Users or has this been addressed in some way?

You should be able to mount it at /Users now, with the later versions of Fedora CoreOS (auto-updated)

Thanks @afbjorklund!

I'm installing latest podman via brew HEAD formula: brew install --HEAD podman

Can't figure out the provided istruction to patch qemu. Can i patch the qemu version installed via brew? Guess i need to recompile it from scratch after applying the patch. Can't understand why some steps should be skipped for Intel Macs

georgettica commented 2 years ago

have you seen this? https://iongion.github.io/podman-desktop-companion/ seems to be a UI that might automatically merge the volume mounts

jimeh commented 2 years ago

Will try to make a custom brew repo with "qemu", for easier install on Darwin (and maybe also to fix the display on Linux, missing gtk)

This would be great @afbjorklund. I've tried various ways to compile a patched QEMU today without much luck. I juust end up with lots of errors from .h and .c files which is sadly outside my area of expertise.

afbjorklund commented 2 years ago

Will try to make a custom brew repo with "qemu", for easier install on Darwin (and maybe also to fix the display on Linux, missing gtk)

This would be great @afbjorklund. I've tried various ways to compile a patched QEMU today without much luck. I juust end up with lots of errors from .h and .c files which is sadly outside my area of expertise.

I know that the patches worked with qemu 6.0.0, but haven't tried the qemu 6.2.0 version myself yet.

Also don't understand why homebrew and linuxbrew have different versions, thought they had merged ?

EDIT: Never mind, was still on a branch

Unfortunately brew doesn't allow patches.

willcohen commented 2 years ago

FYI once my branch gets accepted into a submaintainer tree (meaning it's functionally settled down, even before mainline branch takes them in), I'll very likely set up a backport of the latest version of that patchset to 6.2, since I think nix is going to want this backported too in advance of the next upstream release.

(While the overall functionality isn't that different than the patchset you all are currently using, they did catch quite a few edge cases that could lead to crashes or unexpected behavior, so it'll still be better to use overall).

afbjorklund commented 2 years ago

Once you have a branch backported, it is easy to generate a patch "on the fly" from the GitHub repo

  patch do
    url "https://github.com/qemu/qemu/compare/v6.2.0...afbjorklund:9p-darwin-v6.2.0.patch"
    sha256 "54627d368b9332a64b105dcf3be6575eabdb33a67e5095dc20aa2d4f8c4fc3cb"
  end

Adding gtk support as well, since otherwise there is no QEMU GUI on Linux (the formula only has cocoa)

@@ -36,6 +41,7 @@ class Qemu < Formula

   on_linux do
     depends_on "gcc"
+    depends_on "gtk+3"
   end

   fails_with gcc: "5"
@@ -59,9 +65,9 @@ class Qemu < Formula
       --enable-libssh
       --enable-slirp=system
       --enable-vde
+      --enable-virtfs
       --extra-cflags=-DNCURSES_WIDECHAR=1
       --disable-sdl
-      --disable-gtk
     ]
     # Sharing Samba directories in QEMU requires the samba.org smbd which is
     # incompatible with the macOS-provided version. This will lead to
@@ -71,6 +77,7 @@ class Qemu < Formula
     args << "--smbd=#{HOMEBREW_PREFIX}/sbin/samba-dot-org-smbd"

     args << "--enable-cocoa" if OS.mac?
+    args << "--enable-gtk" if OS.linux?

     system "./configure", *args
     system "make", "V=1", "install"
afbjorklund commented 2 years ago

Needs some dependencies, too.

../meson.build:1401:6: ERROR: Problem encountered: virtio-9p (virtfs) requires libcap-ng-devel and libattr-devel

@@ -23,6 +28,8 @@ class Qemu < Formula
   depends_on "glib"
   depends_on "gnutls"
   depends_on "jpeg"
+  depends_on "libattr"
+  depends_on "libcap-ng"
   depends_on "libpng"
   depends_on "libslirp"
   depends_on "libssh"

Sadly libattr.rb is missing.

It was in attr.rb, naturally.

afbjorklund commented 2 years ago

Here is the whole "formula": https://github.com/afbjorklund/homebrew-core/blob/qemu-9p-darwin/Formula/qemu.rb

Tested it on Linux (had to add some quite unrelated things for that), so it seems like it didn't break too much there ?

brew edit qemu

brew install -s qemu

==> Summary
🍺  /home/linuxbrew/.linuxbrew/Cellar/qemu/6.2.0: 209 files, 952.3MB, built in 15 minutes 3 seconds

That's a crazy (30!!!) amount of platforms, right there.

As usual, it is plenty with two of them. (Ask Bob).

23M /home/linuxbrew/.linuxbrew/Cellar/qemu/6.2.0/bin/qemu-system-aarch64 19M /home/linuxbrew/.linuxbrew/Cellar/qemu/6.2.0/bin/qemu-system-x86_64

190M c501a3e60668cf80a44c4b7ae10f4c8780d448528bbd1c0ec754e5f72e89dea2--qemu--6.2.0.x86_64_linux.bottle.tar.gz

I would expect the sizes on macOS to be similar, I think that 1 GiB is somewhat excessive (but then again the OS is twice that)

1,5G fedora-coreos-35.20220131.2.0-qemu.x86_64.qcow2 639M fedora-coreos-35.20220131.2.0-qemu.x86_64.qcow2.xz

ghost commented 2 years ago

@matteosilv I've been watching the work @willcohen (and others) and short of it being abandoned (doubt it because docker-desktop licensing), I think it's a safe bet to assume this will work when the qemu release catches up. It just leaves a hole to be filled for end-to-end testing for now.

theherk commented 2 years ago

I'm trying to evaluate this volume improvement with v4.0.0-rc4. No success yet. I am using intel mac pro, and following these steps mostly based on this article.

  1. clone podman repository
  2. checkout v4.0.0-rc4
  3. make make podman-remote-darwin
  4. clone qeme
  5. checkout checkout v5.2.0
  6. apply patches
  7. configure using ../configure --target-list=x86_64-softmmu --disable-gnutls
  8. clone gvisor-tap-vsock
  9. build gvproxy
  10. install build qemu and copy podman and gvproxy to /usr/bin/local
  11. sign the binaries

Then, when initializing machine with podman machine init p4 --volume ~/someVol, I get the error:

qemu-system-x86_64: -virtfs local,path=/Users/h4s/someVol,mount_tag=vol0,security_model=mapped-xattr: There is no option group 'virtfs'
qemu-system-x86_64: -virtfs local,path=/Users/h4s/someVol,mount_tag=vol0,security_model=mapped-xattr: virtfs support is disabled

If I try to configure the qemu build with ../configure --target-list=x86_64-softmmu --enable-virtfs --disable-gnutls, I get the error:

ERROR: VirtFS is supported only on Linux

I'm surely missing something, but if anybody has ideas for what to try next, I'm happy to test them.

afbjorklund commented 2 years ago

I'm surely missing something, but if anybody has ideas for what to try next, I'm happy to test them.

You need to use a special branch, which has the needed patches (not yet merged to "master")

https://gitlab.com/wwcohen/qemu/-/tree/9p-darwin

theherk commented 2 years ago

You need to use a special branch, which has the needed patches (not yet merged to "master")

I see. Thank you. I pulled that branch and attempted build with ../configure --target-list=x86_64-softmmu --enable-hvf --enable-virtfs, but it fails with:

FAILED: libblockdev.fa.p/os-posix.c.o
...
../os-posix.c:348:10: warning: address of function 'pthread_fchdir_np' will always evaluate to 'true' [-Wpointer-bool-conversion]
    if (!pthread_fchdir_np) {
        ~^~~~~~~~~~~~~~~~~
../os-posix.c:348:10: note: prefix with the address-of operator to silence this warning
    if (!pthread_fchdir_np) {
         ^
         &
../os-posix.c:350:17: error: use of undeclared identifier 'ENOTSUPP'
        return -ENOTSUPP;
                ^
1 warning and 1 error generated.

I see some mail list messages regarding this, but none that gave me an indication how to get around it.

update: I got past the build error, by modifying the file it complained about to just return -1 in that case. Now the volume options work as expected. It seems something else is broken though with networking, but that will be another issue.

afbjorklund commented 2 years ago

I had some older branches for v6.0.0 - v6.2.0, the discussion is happening in the NixOS issue

viennaa commented 2 years ago

Here is the whole "formula": https://github.com/afbjorklund/homebrew-core/blob/qemu-9p-darwin/Formula/qemu.rb

Using this particular formula ends up couple of things it can't patch and therefore configure fails:

patching file hw/9pfs/9p-util-darwin.c
can't find file to patch at input line 818
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff --git a/hw/9pfs/9p-util-linux.c b/hw/9pfs/9p-util-linux.c
|index defa3a413075..3902378e627c 100644
|--- a/hw/9pfs/9p-util-linux.c
|+++ b/hw/9pfs/9p-util-linux.c
patching file hw/9pfs/9p-util-darwin.c
can't find file to patch at input line 960
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff --git a/hw/9pfs/9p-util-linux.c b/hw/9pfs/9p-util-linux.c
|index 3902378e627c..06399c59199c 100644
|--- a/hw/9pfs/9p-util-linux.c
|+++ b/hw/9pfs/9p-util-linux.c
--------------------------

Without the patch, just taking the branch it would work but still misses the virtfs option afterwards.

afbjorklund commented 2 years ago

It was supposed to be either/or, not both. (i.e. branch or patch)

philnalwalker commented 2 years ago

Overview

Here's how to get 9p working transparently with Podman on MacOS. This solution even changes podman-remote-darwin to map the mount path so it just works like with docker:

❯ podman machine init
Extracting compressed file
Image resized.
Machine init complete
To start your machine run:

    podman machine start

❯ podman machine start
Starting machine "podman-machine-default"
INFO[0000] waiting for clients...
INFO[0000] new connection from  to /var/folders/lm/46cdz9rj5k1438cr15zb86b40000gn/T/podman/qemu_podman-machine-default.sock
Waiting for VM ...
Machine "podman-machine-default" started successfully
❯ podman run -ti -v $HOME:/foobar docker.io/centos ls -als /foobar

Trying to pull docker.io/library/centos:latest...
Getting image source signatures
Copying blob sha256:a1d0c75327776413fa0db9ed3adcdbadedc95a662eb1d360dad82bb913f8a1d1
Copying blob sha256:a1d0c75327776413fa0db9ed3adcdbadedc95a662eb1d360dad82bb913f8a1d1
Copying config sha256:5d0da3dc976460b72c77d94c8a1ad043720b0416bfc16c52c45d4847e53fadb6
Writing manifest to image destination
Storing signatures
total 247136
     0 drwxr-xr-x. 163 nobody nobody        5216 Feb  9 18:40  .
     0 dr-xr-xr-x.   1 root   root            31 Feb  9 18:41  ..
     0 srwxr-xr-x.   1 nobody nobody           0 Feb  9 17:35  .Box_EngineServer
     0 srwxr-xr-x.   1 nobody nobody           0 Feb  9 17:35  .Box_UIServer
     4 -r--------.   1 nobody nobody           7 Oct  6  2020  .CFUserTextEncoding
    28 -rw-r--r--.   1 nobody nobody       24580 Feb  8 17:32  .DS_Store
...

We are rolling out this solution internally at Intuit and will follow up on this thread with our experiences using this solution. We would appreciate any folks willing to test this solution and provide feedback.

It would be really great if the Podman development team would like us to PR this work.

Performance

Preliminary results show it's at least 4x faster than SSHFS when copying the same 10gb file on the same exact Podman machine VM.

SSHFS:

[root@localhost core]#  time cp /var/vmhost/temp_10GB_file /var/tmp/otherfile

real    2m18.027s
user    0m0.000s
sys 0m7.640s

9p:

[root@localhost core]#  time cp /var/mnt/host/Users/pnalwalker/temp_10GB_file /var/tmp

real    0m27.957s
user    0m0.000s
sys 0m1.995s

Instructions

If the changes make into the homebrew-core and podman repos then this will just work and you won't need to do anything. For right now, it's still pretty easy to test:

  1. Remove QEMU home brew qemu installation:

brew uninstall --ignore-dependencies qemu

  1. Clone fork of homebrew-core:

git clone https://github.com/philnalwalker/homebrew-core && cd homebrew-core

  1. Check out the ‘qemu-9pfs’ branch:

git checkout qemu-9pfs

  1. Change to ‘Formula’ directory:

cd Formula

  1. Install QEMU home brew formula from source:

brew install --build-from-source ./qemu.rb

  1. Clone fork of Podman

git clone https://github.com/philnalwalker/podman

This assumes you have Golang setup on your Mac. Make sure to clone to your $GOPATH i.e. ~/golang/src/github.com/containers.

  1. Check out the ‘9pfs’ branch:

cd podman && git checkout 9pfs-v3.4

  1. Make the “podman-remote-darwin” target:

make podman-remote-darwin

  1. Copy Podman binary:

sudo cp bin/darwin/podman /usr/local/bin/podman

  1. Delete, recreate, and SSH into a new machine: 
podman machine init 
podman machine start
podman machine ssh
  1. Verify MacOS / is mounted to /var/mnt/host:

ls -als /var/mnt/host

  1. Verify you do not need to to prefix container mount operations with /var/mnt/host when using Podman Mac client:

podman run -ti -v $HOME:/foobar docker.io/centos ls -als /foobar

  1. (Optional) Enable rootful podman to fix issue seen with chown/permissions on volume mounts.

podman system connection default podman-machine-default-root

TODO

willcohen commented 2 years ago

@philnalwalker this is lovely. to add on to his point, if any of you encounter any issues with the functionality while testing, please let us know! he's running off of the latest version of the patch, and while it's passing the test suites within QEMU, any real-world testing beyond just performance numbers would help to shake out any remaining bugs (performance, xattrs, etc) before i submit a next and/or final version.

philnalwalker commented 2 years ago

@willcohen Thank you for your work on the 9p patch set! It's been working great for us so far! Will keep you posted on our results. Please let us know if there is anything we may do to help get this merged into upstream QEMU.

willcohen commented 2 years ago

I think literally just testing out this use case some more is all we need. The only reason I even volunteered to try to drag this not-written-by-me-but-in-progress-patch-set-from-years-ago over the finish line is because I'm a Mac user who wants to mount directories performantly on podman...

philnalwalker commented 2 years ago

@willcohen We have thousands of engineers at Intuit who will be using this solution and testing it out. I will keep you posted!

afbjorklund commented 2 years ago

Not sure about hardcoding a / volume like that, or dropping the drivers, but glad that it works.

podman machine init --volume /:/mnt/host --volume-driver virtfs

Note that sshfs is still a better option for a truly remote Podman server, like in the cloud or so.

"podman client and podman server may run in different hosts, or one runs in the host, the other runs in a VM."

viennaa commented 2 years ago

It was supposed to be either/or, not both. (i.e. branch or patch)

Tried both variations, however, patch is giving the error pasted above and branch only complains about missing linux

../meson.build:1396:6: ERROR: Problem encountered: virtio-9p (virtfs) requires Linux

I have commented out all bottle architectures but monterey, still the same error.

lehors commented 2 years ago

@philnalwalker Thank you for posting this solution! I just tried it and it seems to be working. :-)

A couple of comments though: Step 6 doesn't work for me. It tries to clone a repo, I get prompted for a github userid/passwd and then the clone fails with:

remote: Repository not found.
fatal: repository 'https://github.com/homebrew-qemu-9p/homebrew-formula/' not found
Error: Failure while executing; `git clone https://github.com/homebrew-qemu-9p/homebrew-formula /usr/local/Homebrew/Library/Taps/homebrew-qemu-9p/homebrew-formula --origin=origin --template=` exited with 128.

But it turns out that this step is actually not necessary, is it? Step 5 seems to do it all.

Step 8 the 'cd podman' command needs to be done before 'git checkout 9pfs' as in: cd podman && git checkout 9pfs

Step 11, the command should end with "podman machine ssh" rather than "podman ssh".

Dr0p42 commented 2 years ago

@philnalwalker Thank you for posting this solution! I just tried it and it seems to be working. :-)

A couple of comments though: Step 6 doesn't work for me. It tries to clone a repo, I get prompted for a github userid/passwd and then the clone fails with:

remote: Repository not found.
fatal: repository 'https://github.com/homebrew-qemu-9p/homebrew-formula/' not found
Error: Failure while executing; `git clone https://github.com/homebrew-qemu-9p/homebrew-formula /usr/local/Homebrew/Library/Taps/homebrew-qemu-9p/homebrew-formula --origin=origin --template=` exited with 128.

But it turns out that this step is actually not necessary, is it? Step 5 seems to do it all.

Step 8 the 'cd podman' command needs to be done before 'git checkout 9pfs' as in: cd podman && git checkout 9pfs

Step 11, the command should end with "podman machine ssh" rather than "podman ssh".

Hello @lehors, thank you for this feedback.

Are you running on apple M1? I tried on an M1 and got the following errors:

 ~/d/g/s/g/c/podman   9pfs …  bin/darwin/podman machine start                                                 21.8s  Thu Feb 10 14:59:55 2022
Starting machine "podman-machine-default"
INFO[0000] waiting for clients...
INFO[0000] new connection from  to /var/folders/ff/vv_v5cm11nq0hy5r131dp4zm0000gn/T/podman/qemu_podman-machine-default.sock
Waiting for VM ...
qemu-system-x86_64: invalid accelerator hvf
qemu-system-x86_64: falling back to tcg
qemu-system-x86_64: unable to find CPU model 'host'
Error: dial unix /var/folders/ff/vv_v5cm11nq0hy5r131dp4zm0000gn/T/podman/podman-machine-default_ready.sock: connect: connection refused
ERRO[0003] cannot receive packets from , disconnecting: cannot read size from socket: EOF
ERRO[0003] cannot read size from socket: EOF
philnalwalker commented 2 years ago

I tried on an M1 and got the following errors:


 ~/d/g/s/g/c/podman   9pfs …  bin/darwin/podman machine start                                                 21.8s  Thu Feb 10 14:59:55 2022

Starting machine "podman-machine-default"

INFO[0000] waiting for clients...

INFO[0000] new connection from  to /var/folders/ff/vv_v5cm11nq0hy5r131dp4zm0000gn/T/podman/qemu_podman-machine-default.sock

Waiting for VM ...

qemu-system-x86_64: invalid accelerator hvf

qemu-system-x86_64: falling back to tcg

qemu-system-x86_64: unable to find CPU model 'host'

Error: dial unix /var/folders/ff/vv_v5cm11nq0hy5r131dp4zm0000gn/T/podman/podman-machine-default_ready.sock: connect: connection refused

ERRO[0003] cannot receive packets from , disconnecting: cannot read size from socket: EOF

ERRO[0003] cannot read size from socket: EOF

I will test on a M1 to see if I can reproduce/debug this issue. Is the SSH_AUTH_SOCK environment variable set? (It shouldn't be.)

jeesmon commented 2 years ago

@philnalwalker Thank you for the detailed instructions. Wondering any idea on fixing permission errors when I volume mount a sub-folder from my home

podman run -ti -v $(pwd):/foobar docker.io/centos ls -als /foobar/
Error: statfs /var/mnt/host/Users/jjacob/Documents/workspace-git/podman-test: permission denied

I see the $HOME is owned by nobody inside the vm

podman run -ti -v $HOME/Documents:/foobar docker.io/centos ls -ld /foobar
drwx------. 20 nobody nobody 640 Feb 17  2021 /foobar

Thanks

UPDATE: Turned out to be because ~/Documents was created with 700 by default. chmod 755 ~/Documents fixed permission issue.

podman run --rm -ti -v $(pwd):/foobar docker.io/centos ls -als /foobar/
total 0
0 drwxr-xr-x. 3 nobody nobody 96 Feb 10 14:59 .
0 dr-xr-xr-x. 1 root   root   31 Feb 10 15:01 ..
0 -rw-r--r--. 1 nobody nobody  0 Feb 10 14:59 test
Dr0p42 commented 2 years ago

I tried on an M1 and got the following errors:

 ~/d/g/s/g/c/podman   9pfs …  bin/darwin/podman machine start                                                 21.8s  Thu Feb 10 14:59:55 2022

Starting machine "podman-machine-default"

INFO[0000] waiting for clients...

INFO[0000] new connection from  to /var/folders/ff/vv_v5cm11nq0hy5r131dp4zm0000gn/T/podman/qemu_podman-machine-default.sock

Waiting for VM ...

qemu-system-x86_64: invalid accelerator hvf

qemu-system-x86_64: falling back to tcg

qemu-system-x86_64: unable to find CPU model 'host'

Error: dial unix /var/folders/ff/vv_v5cm11nq0hy5r131dp4zm0000gn/T/podman/podman-machine-default_ready.sock: connect: connection refused

ERRO[0003] cannot receive packets from , disconnecting: cannot read size from socket: EOF

ERRO[0003] cannot read size from socket: EOF

I will test on a M1 to see if I can reproduce/debug this issue. Is the SSH_AUTH_SOCK environment variable set? (It shouldn't be.)

Yes SSH_AUTH_SOCK is defined in my env, I will retry without it and let you know.

lehors commented 2 years ago

Hello @lehors, thank you for this feedback.

Are you running on apple M1? I tried on an M1 and got the following errors:

No, X86.

Dr0p42 commented 2 years ago

@philnalwalker I tested again with SSH_AUTH_SOCK removed and I got the same error.

Something that may help is that I had to install this manually https://github.com/containers/gvisor-tap-vsock.

Without it I have this error

 ~/d/g/s/g/c/podman   9pfs …  ./bin/darwin/podman machine start
Starting machine "podman-machine-default"
Error: unable to start host networking: "could not find \"gvproxy\" in one of [/usr/local/opt/podman/libexec /opt/homebrew/bin /opt/homebrew/opt/podman/libexec /usr/local/bin /usr/local/libexec/podman /usr/local/lib/podman /usr/libexec/podman /usr/lib/podman].  To resolve this error, set the helper_binaries_dir key in the `[engine]` section of containers.conf to the directory containing your helper binaries."

Don't hesitate to let me know if I can be of any help.

lehors commented 2 years ago

@philnalwalker Have you tried using this with podman-compose? It doesn't work for me. 

~/Projects/github.com/containers/podman-compose/examples/busybox
$ podman-compose up
['podman', '--version', '']
using podman version: 4.0.0-dev
** excluding:  set()
['podman', 'network', 'exists', 'busybox_default']
['podman', 'network', 'create', '--label', 'io.podman.compose.project=busybox', '--label', 'com.docker.compose.project=busybox', 'busybox_default']
['podman', 'network', 'exists', 'busybox_default']
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 563, in assert_cnt_nets
    try: compose.podman.output([], "network", ["exists", net_name])
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 820, in output
    return subprocess.check_output(cmd_ls)
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['podman', 'network', 'exists', 'busybox_default']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/podman-compose", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 1775, in main
    podman_compose.run()
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 1024, in run
    cmd(self, args)
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 1248, in wrapped
    return func(*args, **kw)
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 1415, in compose_up
    podman_args = container_to_args(compose, cnt, detached=args.detach)
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 645, in container_to_args
    assert_cnt_nets(compose, cnt)
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 580, in assert_cnt_nets
    compose.podman.output([], "network", ["exists", net_name])
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 820, in output
    return subprocess.check_output(cmd_ls)
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['podman', 'network', 'exists', 'busybox_default']' returned non-zero exit status 1.
philnalwalker commented 2 years ago

@philnalwalker Have you tried using this with podman-compose? It doesn't work for me.

Will test. I suspect it's because I forked master which is Podman v4 development. I'll test, fork v3.4 stable, make the changes, and follow up to this thread.

jmileson commented 2 years ago

@philnalwalker Thank you for the detailed instructions. Wondering any idea on fixing permission errors when I volume mount a sub-folder from my home

podman run -ti -v $(pwd):/foobar docker.io/centos ls -als /foobar/
Error: statfs /var/mnt/host/Users/jjacob/Documents/workspace-git/podman-test: permission denied

I see the $HOME is owned by nobody inside the vm

podman run -ti -v $HOME/Documents:/foobar docker.io/centos ls -ld /foobar
drwx------. 20 nobody nobody 640 Feb 17  2021 /foobar

Thanks

UPDATE: Turned out to be because ~/Documents was created with 700 by default. chmod 755 ~/Documents fixed permission issue.

podman run --rm -ti -v $(pwd):/foobar docker.io/centos ls -als /foobar/
total 0
0 drwxr-xr-x. 3 nobody nobody 96 Feb 10 14:59 .
0 dr-xr-xr-x. 1 root   root   31 Feb 10 15:01 ..
0 -rw-r--r--. 1 nobody nobody  0 Feb 10 14:59 test

I needed to podman machine ssh and change the permissions of /var/mnt/host/Users/<me> to 0755 , then I was able to run the container with the mount.

jeesmon commented 2 years ago

@philnalwalker Another issue in exposing port to host. Could be related to 4.x branch. But just adding here:

podman run --rm -p 8080:80 nginx
ERRO[4254] net.Dial() = dial tcp 127.0.0.1:7777: connect: connection refused
ERRO[4254] net.Dial() = dial tcp 127.0.0.1:7777: connect: connection refused
Error: error preparing container f32633b6ee48f9669122ca540737587f079fc45c742062455c16c0e73c67b3f8 for attach: error configuring network namespace for container f32633b6ee48f9669122ca540737587f079fc45c742062455c16c0e73c67b3f8: error adding pod focused_davinci_focused_davinci to CNI network "podman": Post "http://host.crc.testing:7777/services/forwarder/expose": dial tcp 192.168.127.254:7777: connect: connection refused
podman network ls
NETWORK ID  NAME        DRIVER
Error: template: list:1:13: executing "list" at <.ID>: error calling ID: runtime error: slice bounds out of range [:12] with length 0
philnalwalker commented 2 years ago

@philnalwalker Another issue in exposing port to host. Could be related to 4.x branch. But just adding here:

podman run --rm -p 8080:80 nginx
ERRO[4254] net.Dial() = dial tcp 127.0.0.1:7777: connect: connection refused
ERRO[4254] net.Dial() = dial tcp 127.0.0.1:7777: connect: connection refused
Error: error preparing container f32633b6ee48f9669122ca540737587f079fc45c742062455c16c0e73c67b3f8 for attach: error configuring network namespace for container f32633b6ee48f9669122ca540737587f079fc45c742062455c16c0e73c67b3f8: error adding pod focused_davinci_focused_davinci to CNI network "podman": Post "http://host.crc.testing:7777/services/forwarder/expose": dial tcp 192.168.127.254:7777: connect: connection refused
podman network ls
NETWORK ID  NAME        DRIVER
Error: template: list:1:13: executing "list" at <.ID>: error calling ID: runtime error: slice bounds out of range [:12] with length 0

@jesmon Definitely seeing some weird behavior when using master with networking. This is most likely because it's pulling a v3.4 VM image when using the 4.0 RC client. I went ahead and created a branch based off of v3.4:

https://github.com/philnalwalker/podman/tree/9pfs-v3.4

Using this branch solves the networking issues we were seeing with podman play kube. Please test if you have a moment. Thank you!

philnalwalker commented 2 years ago

@philnalwalker Have you tried using this with podman-compose? It doesn't work for me.

~/Projects/github.com/containers/podman-compose/examples/busybox
$ podman-compose up
['podman', '--version', '']
using podman version: 4.0.0-dev
** excluding:  set()
['podman', 'network', 'exists', 'busybox_default']
['podman', 'network', 'create', '--label', 'io.podman.compose.project=busybox', '--label', 'com.docker.compose.project=busybox', 'busybox_default']
['podman', 'network', 'exists', 'busybox_default']
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 563, in assert_cnt_nets
    try: compose.podman.output([], "network", ["exists", net_name])
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 820, in output
    return subprocess.check_output(cmd_ls)
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['podman', 'network', 'exists', 'busybox_default']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/podman-compose", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 1775, in main
    podman_compose.run()
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 1024, in run
    cmd(self, args)
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 1248, in wrapped
    return func(*args, **kw)
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 1415, in compose_up
    podman_args = container_to_args(compose, cnt, detached=args.detach)
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 645, in container_to_args
    assert_cnt_nets(compose, cnt)
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 580, in assert_cnt_nets
    compose.podman.output([], "network", ["exists", net_name])
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 820, in output
    return subprocess.check_output(cmd_ls)
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['podman', 'network', 'exists', 'busybox_default']' returned non-zero exit status 1.

@lehors Please test with this branch when you have a moment. It should fix the networking issues you were encountering: https://github.com/philnalwalker/podman/tree/9pfs-v3.4

philnalwalker commented 2 years ago

/usr/local/opt/podman/libexec

@Dr0p42

Can you try uninstalling and reinstalling podman using brew, verifying it works, and then try running the compiled binary? Also, please try using this branch as it's based off of the stable v.34: https://github.com/philnalwalker/podman/tree/9pfs-v3.4 Thanks!

philnalwalker commented 2 years ago

@philnalwalker Have you tried using this with podman-compose? It doesn't work for me.

~/Projects/github.com/containers/podman-compose/examples/busybox
$ podman-compose up
['podman', '--version', '']
using podman version: 4.0.0-dev
** excluding:  set()
['podman', 'network', 'exists', 'busybox_default']
['podman', 'network', 'create', '--label', 'io.podman.compose.project=busybox', '--label', 'com.docker.compose.project=busybox', 'busybox_default']
['podman', 'network', 'exists', 'busybox_default']
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 563, in assert_cnt_nets
    try: compose.podman.output([], "network", ["exists", net_name])
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 820, in output
    return subprocess.check_output(cmd_ls)
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['podman', 'network', 'exists', 'busybox_default']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/podman-compose", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 1775, in main
    podman_compose.run()
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 1024, in run
    cmd(self, args)
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 1248, in wrapped
    return func(*args, **kw)
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 1415, in compose_up
    podman_args = container_to_args(compose, cnt, detached=args.detach)
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 645, in container_to_args
    assert_cnt_nets(compose, cnt)
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 580, in assert_cnt_nets
    compose.podman.output([], "network", ["exists", net_name])
  File "/usr/local/lib/python3.9/site-packages/podman_compose.py", line 820, in output
    return subprocess.check_output(cmd_ls)
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['podman', 'network', 'exists', 'busybox_default']' returned non-zero exit status 1.

Just tested podman-compose with https://github.com/philnalwalker/podman/tree/9pfs-v3.4 branch and it works for me. Can you give it a try?


❯ podman-compose up
['podman', '--version', '']
using podman version: 3.4.5-dev
** excluding:  set()
['podman', 'network', 'exists', 'pnalwalker_default']
['podman', 'network', 'create', '--label', 'io.podman.compose.project=pnalwalker', '--label', 'com.docker.compose.project=pnalwalker', 'pnalwalker_default']
['podman', 'network', 'exists', 'pnalwalker_default']
podman create --name=pnalwalker_config_1 --label io.podman.compose.config-hash=123 --label io.podman.compose.project=pnalwalker --label io.podman.compose.version=0.0.1 --label com.docker.compose.project=pnalwalker --label com.docker.compose.project.working_dir=/Users/pnalwalker --label com.docker.compose.project.config_files=compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=config --net pnalwalker_default --network-alias config docker.intuit.com/oicp/standard/java/amzn-corretto-jdk11:latest sleep infinity
Trying to pull .../java/amzn-corretto-jdk11:latest...
Getting image source signatures
Copying blob sha256:64bea62c21ee87ababb97806b6e41ea0b7740562ed772561ccbd8d00529ac21e
Copying blob sha256:8b8a142162d22658bdf0283afcd00a9dd216c6637943ffe5f2ba53c4e3da0bd9
Copying blob sha256:688a51069fb8bfe23d0aacce02c0c9065a366e190f733886f30ea519ba2dec87
Copying blob sha256:80d90a31b06816ad0bc77f82741c5e13cb79c8b2058cfae1eab47c81886e7a07
Copying blob sha256:875147590c4a88e6d961554fa433900e22d9435b27eb417bb77e0f96dc85ddbf
Copying blob sha256:86a27880e3b6bb5689e24b342a42d0b7e3631df4c32b744e8359b349775bdd1c
Copying blob sha256:86a27880e3b6bb5689e24b342a42d0b7e3631df4c32b744e8359b349775bdd1c
Copying blob sha256:91aa6c351133bf54b43239b3cd59070234f110469b327cb8246f67010f41d42f
Copying blob sha256:875147590c4a88e6d961554fa433900e22d9435b27eb417bb77e0f96dc85ddbf
Copying blob sha256:de2bea23523df887eb063a7f2fab8d9dfc28313aaf770227845075fdb8c95926
Copying blob sha256:8b8a142162d22658bdf0283afcd00a9dd216c6637943ffe5f2ba53c4e3da0bd9
Copying blob sha256:688a51069fb8bfe23d0aacce02c0c9065a366e190f733886f30ea519ba2dec87
Copying blob sha256:91aa6c351133bf54b43239b3cd59070234f110469b327cb8246f67010f41d42f
Copying blob sha256:b8cc81eceee2dea819bcc39ec8c83a2af2b0e98acd0fea3d6fab0e556dfe34db
Copying blob sha256:de2bea23523df887eb063a7f2fab8d9dfc28313aaf770227845075fdb8c95926
Copying blob sha256:bc2d55976ab60d909f83bdc0ba89d3333ba5359d7f292517d3eb38c5c940353e
Copying blob sha256:64bea62c21ee87ababb97806b6e41ea0b7740562ed772561ccbd8d00529ac21e
Copying blob sha256:1bc738bf2239d010ae17d8a0a800076b267a65f326a31b43f50251e3c2ed0edd
Copying blob sha256:80d90a31b06816ad0bc77f82741c5e13cb79c8b2058cfae1eab47c81886e7a07
Copying blob sha256:b8cc81eceee2dea819bcc39ec8c83a2af2b0e98acd0fea3d6fab0e556dfe34db
Copying blob sha256:bc2d55976ab60d909f83bdc0ba89d3333ba5359d7f292517d3eb38c5c940353e
Copying blob sha256:1bc738bf2239d010ae17d8a0a800076b267a65f326a31b43f50251e3c2ed0edd
Copying blob sha256:55fadf0826975fb62b3a29f6d7c73a09cff3bd0b8c9f55eacacd38199d3f8e5c
Copying blob sha256:55fadf0826975fb62b3a29f6d7c73a09cff3bd0b8c9f55eacacd38199d3f8e5c
Copying blob sha256:f33ff5caf9968fdfeeb4d90b2d0ca512922a58957a23b35d5c350802065ad93d
Copying blob sha256:37f13e66e8daec5bd539198cab4ad625fcd6bcf0deb8daff8bea53cda07afa03
Copying blob sha256:f33ff5caf9968fdfeeb4d90b2d0ca512922a58957a23b35d5c350802065ad93d
Copying blob sha256:37f13e66e8daec5bd539198cab4ad625fcd6bcf0deb8daff8bea53cda07afa03
Copying config sha256:4d38a01bc3e777d7878adbf1480ec12f03bd49977db877689b9dafe629003c0a
Writing manifest to image destination
Storing signatures
f3239b318dea4d5489899c3f2e0123b714c468471f0592e25609b161d57cb9aa
exit code: 0
podman start -a pnalwalker_config_1```