Run Docker-OSX with x11docker.

mviereck / x11docker

Run GUI applications and desktops in docker and podman containers. Focus on security.

MIT License

5.59k stars 375 forks source link

Run Docker-OSX with x11docker. #331

Closed hongyi-zhao closed 3 years ago

hongyi-zhao commented 3 years ago

I noticed the Docker-OSX project, and I've successfully run it as below on Ubuntu 20.04:

$ docker run -it \
    --device /dev/kvm \
    --device /dev/snd \
    -e RAM=4 \
    -p 50922:10022 \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    -e AUDIO_DRIVER=pa,server=unix:/tmp/pulseaudio.socket \
    -v "/run/user/$(id -u)/pulse/native:/tmp/pulseaudio.socket" \
    sickcodes/docker-osx:auto

In order to simplify the above workflow with x11docker, I must figure out the equivalent argument representations with x11docker. Any hints for this?

Regards, HY

mviereck commented 3 years ago

Did you already try? Where has been issues?

x11docker --hostdisplay --alsa --env RAM=4 --env AUDIO_DRIVER=pa,server=unix:/tmp/pulseaudio.socket \
  -- --device /dev/kvm -p 50922:10022 -- sickcodes/docker-osx:auto

You might also need --cap-default and maybe --user=root. The setup shares ALSA as well as pulseaudio sound. Normally only one of them should be needed.

As a core this should be enough:

x11docker -- --device /dev/kvm -p 50922:10022 -- sickcodes/docker-osx:auto

hongyi-zhao commented 3 years ago

Did you already try? Where has been issues?
x11docker --hostdisplay --alsa --env RAM=4 --env AUDIO_DRIVER=pa,server=unix:/tmp/pulseaudio.socket \
  -- --device /dev/kvm -p 50922:10022 -- sickcodes/docker-osx:auto
You might also need --cap-default and maybe --user=root.

Failed, see following:

$ x11docker --hostdisplay --alsa --cap-default --user=root --env RAM=4 --env AUDIO_DRIVER=pa,server=unix:/tmp/pulseaudio.socket  -- --device /dev/kvm -p 50922:10022 -- sickcodes/docker-osx:auto
x11docker WARNING: User werner is member of group docker.
  That allows unprivileged processes on host to gain root privileges.

x11docker WARNING: Clipboard isolation may fail.

x11docker WARNING: Option --cap-default disables security hardening
  for containers done by x11docker. Default docker capabilities are allowed.
  This is considered to be less secure.

x11docker note: Option --cap-default: Enabling option --newprivileges.
  You can avoid this with --newprivileges=no

x11docker WARNING: Option --hostdisplay provides only low container isolation!
  It is recommended to use another X server option like --nxagent or --xpra.

  To improve security with --hostdisplay x11docker uses untrusted cookies.
  This can lead to strange behaviour of some applications.

  If you encounter application errors, enable option --clipboard
  that disables security restrictions for --hostdisplay as a side effect.

x11docker note: Option --hostdisplay may fail with proprietary NVIDIA driver
  on host. In that case try other X server options like 
  --nxagent, --xpra or --xephyr.

x11docker WARNING: Found custom DOCKER_RUN_OPTIONS.
  x11docker will add them to 'docker run' command without
  a serious check for validity or security. Found options:
   '--device' '/dev/kvm' '-p' '50922:10022'

x11docker note: Option --user=root: Enabling option --sudouser.

x11docker WARNING: ALSA sound with option --alsa degrades container isolation.
  Shares device files in /dev/snd, container gains access to sound hardware.
  Container applications can catch audio output and microphone input.

x11docker note: It seems that pulseaudio is running on your host.
  Pulseaudio can interfere with ALSA sound (option --alsa).
  Host sound may not work while container is playing sound and vice versa.
  Alternative: with pulseaudio on host and in image, use option --pulseaudio.

x11docker WARNING: Option --sudouser severly reduces container security.
  Container gains additional capabilities to allow sudo and su.
  If an application breaks out of container, it can harm your system
  in many ways without you noticing. Password: x11docker

x11docker WARNING: Option --newprivileges=yes: x11docker does not set 
  docker run option --security-opt=no-new-privileges. 
  That degrades container security.
  However, this is still within a default docker setup.

x11docker WARNING: Sharing device file: /dev/snd

x11docker note: Found 'WORKDIR  "${TERMS_OF_USE}" = i_agree ' in image.  
  You can change it with option --workdir=DIR.

The setup shares ALSA as well as pulseaudio sound. Normally only one of them should be needed.

As a core this should be enough:
x11docker -- --device /dev/kvm -p 50922:10022 -- sickcodes/docker-osx:auto

Still failed as below:

$ x11docker -- --device /dev/kvm -p 50922:10022 -- sickcodes/docker-osx:auto
x11docker WARNING: User werner is member of group docker.
  That allows unprivileged processes on host to gain root privileges.

x11docker WARNING: You are running GNOME desktop in outdated version 
  GNOME Shell 3.36.4
  This might cause issues with host applications if using additional X servers.
  It is recommended to use another desktop environment or GNOME >= 3.38.
  Only otherwise discouraged option --hostdisplay might work as expected.

x11docker note: Using X server option --hostdisplay

x11docker WARNING: Clipboard isolation may fail.

x11docker note: To allow protection against X security leaks,
  please install 'xinit' and one or more of:
    xpra, Xephyr, nxagent, weston+Xwayland, kwin_wayland+Xwayland or Xnest,
  or run a second Xorg server with option --xorg.

x11docker WARNING: Option --hostdisplay provides only low container isolation!
  It is recommended to use another X server option like --nxagent or --xpra.

  To improve security with --hostdisplay x11docker uses untrusted cookies.
  This can lead to strange behaviour of some applications.

  If you encounter application errors, enable option --clipboard
  that disables security restrictions for --hostdisplay as a side effect.

x11docker note: Option --hostdisplay may fail with proprietary NVIDIA driver
  on host. In that case try other X server options like 
  --nxagent, --xpra or --xephyr.

x11docker WARNING: Found custom DOCKER_RUN_OPTIONS.
  x11docker will add them to 'docker run' command without
  a serious check for validity or security. Found options:
   '--device' '/dev/kvm' '-p' '50922:10022'

x11docker note: Found 'WORKDIR  "${TERMS_OF_USE}" = i_agree ' in image.  
  You can change it with option --workdir=DIR.

mviereck commented 3 years ago

There is no error message of the OSX image. So I only can guess.

x11docker shows several warnings and notes you should consider:

You are running GNOME desktop in outdated version GNOME Shell 3.36.4

If you encounter application errors, enable option --clipboard

Option --hostdisplay may fail with proprietary NVIDIA driver on host.

Found 'WORKDIR "${TERMS_OF_USE}" = i_agree ' in image.
You can change it with option --workdir=DIR.

hongyi-zhao commented 3 years ago

Still failed after making the following arguments' adjustment:

$ x11docker --workdir=$HOME --alsa --cap-default --user=root --clipboard --env RAM=4 --env AUDIO_DRIVER=pa,server=unix:/tmp/pulseaudio.socket   -- --device /dev/kvm -p 50922:10022 -- sickcodes/docker-osx:auto 
x11docker WARNING: User werner is member of group docker.
  That allows unprivileged processes on host to gain root privileges.

x11docker WARNING: You are running GNOME desktop in outdated version 
  GNOME Shell 3.36.4
  This might cause issues with host applications if using additional X servers.
  It is recommended to use another desktop environment or GNOME >= 3.38.
  Only otherwise discouraged option --hostdisplay might work as expected.

x11docker note: Using X server option --hostdisplay

x11docker WARNING: Option --clipboard: To allow clipboard sharing with
  option --hostdisplay, trusted cookies will be enabled.
  No protection against X security leaks is left!
  Consider to use another X server option.

x11docker WARNING: Option --cap-default disables security hardening
  for containers done by x11docker. Default docker capabilities are allowed.
  This is considered to be less secure.

x11docker note: Option --cap-default: Enabling option --newprivileges.
  You can avoid this with --newprivileges=no

x11docker note: Option --hostdisplay: To allow --hostdisplay with trusted cookies,
  x11docker must share host IPC namespace with container (option --hostipc)
  to allow shared memory for X extension MIT-SHM.

x11docker note: To allow protection against X security leaks,
  please install 'xinit' and one or more of:
    xpra, Xephyr, nxagent, weston+Xwayland, kwin_wayland+Xwayland or Xnest,
  or run a second Xorg server with option --xorg.

x11docker WARNING: Option --hostdisplay with trusted cookies provides
      QUITE BAD CONTAINER ISOLATION !
  Keylogging and controlling host applications is possible! 
  Clipboard sharing is enabled (option --cliboard).
  It is recommended to use another X server option like --nxagent or --xpra.

x11docker note: Option --hostdisplay may fail with proprietary NVIDIA driver
  on host. In that case try other X server options like 
  --nxagent, --xpra or --xephyr.

x11docker WARNING: Option --hostipc severely degrades 
  container isolation. IPC namespace remapping is disabled.

x11docker WARNING: Found custom DOCKER_RUN_OPTIONS.
  x11docker will add them to 'docker run' command without
  a serious check for validity or security. Found options:
   '--device' '/dev/kvm' '-p' '50922:10022'

x11docker note: Option --user=root: Enabling option --sudouser.

x11docker WARNING: ALSA sound with option --alsa degrades container isolation.
  Shares device files in /dev/snd, container gains access to sound hardware.
  Container applications can catch audio output and microphone input.

x11docker note: It seems that pulseaudio is running on your host.
  Pulseaudio can interfere with ALSA sound (option --alsa).
  Host sound may not work while container is playing sound and vice versa.
  Alternative: with pulseaudio on host and in image, use option --pulseaudio.

x11docker WARNING: Option --sudouser severly reduces container security.
  Container gains additional capabilities to allow sudo and su.
  If an application breaks out of container, it can harm your system
  in many ways without you noticing. Password: x11docker

x11docker WARNING: Option --newprivileges=yes: x11docker does not set 
  docker run option --security-opt=no-new-privileges. 
  That degrades container security.
  However, this is still within a default docker setup.

x11docker WARNING: Sharing device file: /dev/snd

mviereck commented 3 years ago

Try:

x11docker --hostdisplay --clipboard --user=root --cap-default  \
 --env RAM=4 --env AUDIO_DRIVER=pa,server=unix:/tmp/pulseaudio.socket \
 -- --device /dev/kvm --device /dev/snd -p 50922:10022  -v "/run/user/$(id -u)/pulse/native:/tmp/pulseaudio.socket" \ 
 -- sickcodes/docker-osx:auto

Two differences to your working example in the first post:

You are running with -ti. Is that on purpose? If yes, add -i or --interactive to x11docker.
x11docker changes WORKDIR to /tmp. The image contains a very odd setting:
```
Found 'WORKDIR "${TERMS_OF_USE}" = i_agree ' in image.
```
You could ask the developer to fix or explain this. That is obviously not a directory.

Which outcome do you expect? What do you get if running the working example?

hongyi-zhao commented 3 years ago

Try:

x11docker --hostdisplay --clipboard --user=root --cap-default  \
 --env RAM=4 --env AUDIO_DRIVER=pa,server=unix:/tmp/pulseaudio.socket \
 -- --device /dev/kvm --device /dev/snd -p 50922:10022  -v "/run/user/$(id -u)/pulse/native:/tmp/pulseaudio.socket" \ 
 -- sickcodes/docker-osx:auto

Still failed:

werner@X10DAi:~$ x11docker --hostdisplay --clipboard --user=root --cap-default  \
>  --env RAM=4 --env AUDIO_DRIVER=pa,server=unix:/tmp/pulseaudio.socket \
>  -- --device /dev/kvm --device /dev/snd -p 50922:10022  -v "/run/user/$(id -u)/pulse/native:/tmp/pulseaudio.socket" \
>  -- sickcodes/docker-osx:auto
x11docker WARNING: User werner is member of group docker.
  That allows unprivileged processes on host to gain root privileges.

x11docker WARNING: Option --clipboard: To allow clipboard sharing with
  option --hostdisplay, trusted cookies will be enabled.
  No protection against X security leaks is left!
  Consider to use another X server option.

x11docker WARNING: Option --cap-default disables security hardening
  for containers done by x11docker. Default docker capabilities are allowed.
  This is considered to be less secure.

x11docker note: Option --cap-default: Enabling option --newprivileges.
  You can avoid this with --newprivileges=no

x11docker note: Option --hostdisplay: To allow --hostdisplay with trusted cookies,
  x11docker must share host IPC namespace with container (option --hostipc)
  to allow shared memory for X extension MIT-SHM.

x11docker WARNING: Option --hostdisplay with trusted cookies provides
      QUITE BAD CONTAINER ISOLATION !
  Keylogging and controlling host applications is possible! 
  Clipboard sharing is enabled (option --cliboard).
  It is recommended to use another X server option like --nxagent or --xpra.

x11docker note: Option --hostdisplay may fail with proprietary NVIDIA driver
  on host. In that case try other X server options like 
  --nxagent, --xpra or --xephyr.

x11docker WARNING: Option --hostipc severely degrades 
  container isolation. IPC namespace remapping is disabled.

x11docker WARNING: Found custom DOCKER_RUN_OPTIONS.
  x11docker will add them to 'docker run' command without
  a serious check for validity or security. Found options:
   '--device' '/dev/kvm' '--device' '/dev/snd' '-p' '50922:10022' '-v' '/run/user/1000/pulse/native:/tmp/pulseaudio.socket'

x11docker note: Option --user=root: Enabling option --sudouser.

x11docker WARNING: Option --sudouser severly reduces container security.
  Container gains additional capabilities to allow sudo and su.
  If an application breaks out of container, it can harm your system
  in many ways without you noticing. Password: x11docker

x11docker WARNING: Option --newprivileges=yes: x11docker does not set 
  docker run option --security-opt=no-new-privileges. 
  That degrades container security.
  However, this is still within a default docker setup.

x11docker note: Found 'WORKDIR  "${TERMS_OF_USE}" = i_agree ' in image.  
  You can change it with option --workdir=DIR.

werner@X10DAi:~$

Two differences to your working example in the first post:
* You are running with `-ti`. Is that on purpose?

They are simply the options used by the official example and I haven't done any adaption on them.

If yes, add -i or --interactive to x11docker.
* x11docker changes `WORKDIR` to `/tmp`. The image contains a very odd setting:
Found 'WORKDIR "${TERMS_OF_USE}" = i_agree ' in image.
You could ask the developer to fix or explain this. That is obviously not a directory.

Which outcome do you expect? What do you get if running the working example?

It will let me enter the macOS Docker container as if it were a virtual machine.

mviereck commented 3 years ago

It will let me enter the macOS Docker container as if it were a virtual machine.

Do you get a window? A desktop? A terminal?

hongyi-zhao commented 3 years ago

See the following info printed on stdout:

werner@X10DAi:~$ docker run -it \
>     --device /dev/kvm \
>     --device /dev/snd \
>     -e RAM=4 \
>     -p 50922:10022 \
>     -v /tmp/.X11-unix:/tmp/.X11-unix \
>     -e "DISPLAY=${DISPLAY:-:0.0}" \
>     -e AUDIO_DRIVER=pa,server=unix:/tmp/pulseaudio.socket \
>     -v "/run/user/$(id -u)/pulse/native:/tmp/pulseaudio.socket" \
>     sickcodes/docker-osx:auto
By using this Dockerfile, you hereby agree that you are a security reseacher or developer and agree to use this Dockerfile to make the world a safer place. Examples include: making your apps safer, finding your mobile phone, compiling security products, etc. You understand that Docker-OSX is an Open Source project, which is released to the public under the GNU Pulic License version 3 and above. You acknowledge that the Open Source project is absolutely unaffiliated with any third party, in any form whatsoever. Any trademarks or intelectual property which happen to be mentioned anywhere in or around the project are owned by their respective owners. By using this Dockerfile, you agree to agree to the EULA of each piece of upstream or downstream software. The following code is released for the sole purpose of security research, under the GNU Public License version 3. If you are concerned about the licensing, please note that this project is not AGPL. A copy of the license is available online: https://github.com/sickcodes/Docker-OSX/blob/master/LICENSE. In order to use the following Dockerfile you must read and understand the terms. Once you have read the terms, use the -e TERMS_OF_USE=i_agree or -e TERMS_OF_USE=i_disagree
Disk is being copied between layers... Please wait a minute...
touch: cannot touch '': No such file or directory
chown: cannot access '': No such file or directory
  File: /home/arch/OSX-KVM/mac_hdd_ng.img
  Size: 19139461120 Blocks: 37381768   IO Block: 4096   regular file
Device: 50h/80d Inode: 49023176    Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/    arch)   Gid: ( 1000/    arch)
Access: 2021-03-13 00:37:20.368115876 +0000
Modify: 2021-03-13 00:37:20.368115876 +0000
Change: 2021-03-13 00:37:21.124085943 +0000
 Birth: 2021-03-13 00:31:17.302990747 +0000
Large image is being copied between layers, please wait a minute...
ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519 
nohup: appending output to 'nohup.out'
Booting Docker-OSX in the background. Please wait...
/usr/sbin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/arch/.ssh/id_docker_osx.pub"
ssh: connect to host 127.0.0.1 port 10022: Connection refused
Disk is being copied between layers. Repeating until able to copy SSH key into OSX...
/usr/sbin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/arch/.ssh/id_docker_osx.pub"
ssh: connect to host 127.0.0.1 port 10022: Connection refused
Disk is being copied between layers. Repeating until able to copy SSH key into OSX...
/usr/sbin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/arch/.ssh/id_docker_osx.pub"
ssh: connect to host 127.0.0.1 port 10022: Connection refused
Disk is being copied between layers. Repeating until able to copy SSH key into OSX...
/usr/sbin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/arch/.ssh/id_docker_osx.pub"
ssh: connect to host 127.0.0.1 port 10022: Connection refused
Disk is being copied between layers. Repeating until able to copy SSH key into OSX...
/usr/sbin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/arch/.ssh/id_docker_osx.pub"
ssh: connect to host 127.0.0.1 port 10022: Connection refused
Disk is being copied between layers. Repeating until able to copy SSH key into OSX...
/usr/sbin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/arch/.ssh/id_docker_osx.pub"
ssh: connect to host 127.0.0.1 port 10022: Connection refused
Disk is being copied between layers. Repeating until able to copy SSH key into OSX...
/usr/sbin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/arch/.ssh/id_docker_osx.pub"
ssh: connect to host 127.0.0.1 port 10022: Connection refused
Disk is being copied between layers. Repeating until able to copy SSH key into OSX...
/usr/sbin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/arch/.ssh/id_docker_osx.pub"
ssh: connect to host 127.0.0.1 port 10022: Connection refused
Disk is being copied between layers. Repeating until able to copy SSH key into OSX...
/usr/sbin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/arch/.ssh/id_docker_osx.pub"
ssh: connect to host 127.0.0.1 port 10022: Connection refused
Disk is being copied between layers. Repeating until able to copy SSH key into OSX...
/usr/sbin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/arch/.ssh/id_docker_osx.pub"
ssh: connect to host 127.0.0.1 port 10022: Connection refused
Disk is being copied between layers. Repeating until able to copy SSH key into OSX...
/usr/sbin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/arch/.ssh/id_docker_osx.pub"
ssh: connect to host 127.0.0.1 port 10022: Connection refused
Disk is being copied between layers. Repeating until able to copy SSH key into OSX...
/usr/sbin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/arch/.ssh/id_docker_osx.pub"
ssh: connect to host 127.0.0.1 port 10022: Connection refused
Disk is being copied between layers. Repeating until able to copy SSH key into OSX...
qemu-system-x86_64: warning: dbind: Couldn't connect to accessibility bus: Failed to connect to socket /tmp/dbus-IntytAUPF2: Connection refused
/usr/sbin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/arch/.ssh/id_docker_osx.pub"
ssh: connect to host 127.0.0.1 port 10022: Connection refused
Disk is being copied between layers. Repeating until able to copy SSH key into OSX...
/usr/sbin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/arch/.ssh/id_docker_osx.pub"
ssh: connect to host 127.0.0.1 port 10022: Connection refused
Disk is being copied between layers. Repeating until able to copy SSH key into OSX...
/usr/sbin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/arch/.ssh/id_docker_osx.pub"
ssh: connect to host 127.0.0.1 port 10022: Connection refused
Disk is being copied between layers. Repeating until able to copy SSH key into OSX...
QEMU 5.2.0 monitor - type 'help' for more information
(qemu) /usr/sbin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/arch/.ssh/id_docker_osx.pub"
qemu-system-x86_64: Slirp: Failed to send packet, ret: -1
qemu-system-x86_64: Slirp: Failed to send packet, ret: -1
qemu-system-x86_64: Slirp: Failed to send packet, ret: -1
qemu-system-x86_64: Slirp: Failed to send packet, ret: -1
Warning: Permanently added '[127.0.0.1]:10022' (ED25519) to the list of known hosts.
Host 127.0.0.1
    User user
    Port 10022
    IdentityFile ~/.ssh/id_docker_osx
    StrictHostKeyChecking no
    UserKnownHostsFile=/dev/null
Default username: user
Default password: alpine
Change it immediately using the command: passwd
Warning: Permanently added '[127.0.0.1]:10022' (ED25519) to the list of known hosts.
Last login: Sat Mar 13 00:38:25 2021
user@users-iMac-Pro ~ % usb_desc_get_descriptor: 2 unknown type 33 (len 10)
                                                                           usb_desc_get_descriptor: 1 unknown type 33 (len 10)

And the screenshot as shown below:

mviereck commented 3 years ago

I don't know why the container does not show any output at all, not even an error message. Two attempts:

The output shows:
```
Once you have read the terms, use the -e TERMS_OF_USE=i_agree or -e TERMS_OF_USE=i_disagree 
```
Add --env TERMS_OF_USE=i_agree. Maybe that helps.
- Run x11docker with --debug and compare the shown docker command with your working example.

After all, what advantage would you expect from using x11docker here instead of the working example?

hongyi-zhao commented 3 years ago

I don't know why the container does not show any output at all, not even an error message. Two attempts:

* The output shows:
  ```
  Once you have read the terms, use the -e TERMS_OF_USE=i_agree or -e TERMS_OF_USE=i_disagree 
  ```

  Add `--env TERMS_OF_USE=i_agree`. Maybe that helps.

* Run x11docker with `--debug` and compare the shown docker command with your working example.

This time, I tried with the following:

$ x11docker --debug --hostdisplay --clipboard --user=root --cap-default  \
 --env RAM=4 --env AUDIO_DRIVER=pa,server=unix:/tmp/pulseaudio.socket \
 --  --env TERMS_OF_USE=i_agree --device /dev/kvm --device /dev/snd -p 50922:10022  -v "/run/user/$(id -u)/pulse/native:/tmp/pulseaudio.socket" \
 -- sickcodes/docker-osx:auto

Though the run still failed, but courtesy of the --debug option, I see the following info about the real invoking on docker command:

DEBUGNOTE[22:01:14,972]: Docker command:
  docker run --tty --detach \
  --name x11docker_X1_sickcodes-docker-osx-auto_44072375976 \
  --user 0:0 \
  --userns host \
  --ipc host \
  --security-opt label=type:container_runtime_t \
  --volume '/usr/bin/docker-init':'/usr/local/bin/init':ro \
  --tmpfs /run --tmpfs /run/lock \
  --volume '/home/werner/.cache/x11docker/sickcodes-docker-osx-auto-44072375976/share':'/x11docker':rw \
  --volume '/tmp/.X11-unix/X1':'/X1':rw \
  --workdir '/tmp' \
  --entrypoint env \
  --env 'container=docker' \
  --env 'XAUTHORITY=/x11docker/Xauthority.client' \
  --env 'DISPLAY=:1' \
  --env 'USER=root' \
  --env 'AUDIO_DRIVER=pa,server=unix:/tmp/pulseaudio.socket' \
  --env 'RAM=4' \
   '--env' 'TERMS_OF_USE=i_agree' '--device' '/dev/kvm' '--device' '/dev/snd' '-p' '50922:10022' '-v' '/run/user/1000/pulse/native:/tmp/pulseaudio.socket' \
  -- sickcodes/docker-osx:auto /usr/local/bin/init -- /bin/sh - /x11docker/containerrc

After all, what advantage would you expect from using x11docker here instead of the working example?

Just out of curiosity. To be frank, for thus a case, maybe the VM based scheme is more preferable.

mviereck commented 3 years ago

Currently I have no further idea and not enough resources to test myself. You could ask the developer of docker-osx and show him the above docker run command. Maybe he has an idea what is gong wrong.

sickcodes commented 3 years ago

I don't know why the container does not show any output at all, not even an error message.

Two attempts:
The output shows:
Once you have read the terms, use the -e TERMS_OF_USE=i_agree or -e TERMS_OF_USE=i_disagree 
Add --env TERMS_OF_USE=i_agree. Maybe that helps.
Run x11docker with --debug and compare the shown docker command with your working example.

After all, what advantage would you expect from using x11docker here instead of the working example?

Hey all! Thanks for letting me know about this. That pause at the beginning of the container is the 20GB docker layer being duplicated. The second pause is waiting for SSH to activate (for the system to boot)

sickcodes commented 3 years ago

I would be happy to integrate with this project. If you want to test a faster one, you can use :latest and it will boot the recovery disk which does not duplicate large layers at runtime and is much smaller

mviereck commented 3 years ago

I would be happy to integrate with this project.

Thank you!

If you want to test a faster one, you can use :latest and it will boot the recovery disk which does not duplicate large layers at runtime and is much smaller

Unfortunately I cannot download a 20GB image, I have a very small internet bandwidth. That will hopefully change within a few month. So I cannot test myself.

Do you have an idea why no output at all is shown? x11docker prints all terminal messages of a container, so I assume it stops at an early stage, maybe waiting for something.

The second pause is waiting for SSH to activate (for the system to boot)

Do you run an init system in container?

If you look at this command, might any of the options be a problem?

DEBUGNOTE[22:01:14,972]: Docker command:
  docker run --tty --detach \
  --name x11docker_X1_sickcodes-docker-osx-auto_44072375976 \
  --user 0:0 \
  --userns host \
  --ipc host \
  --security-opt label=type:container_runtime_t \
  --volume '/usr/bin/docker-init':'/usr/local/bin/init':ro \
  --tmpfs /run --tmpfs /run/lock \
  --volume '/home/werner/.cache/x11docker/sickcodes-docker-osx-auto-44072375976/share':'/x11docker':rw \
  --volume '/tmp/.X11-unix/X1':'/X1':rw \
  --workdir '/tmp' \
  --entrypoint env \
  --env 'container=docker' \
  --env 'XAUTHORITY=/x11docker/Xauthority.client' \
  --env 'DISPLAY=:1' \
  --env 'USER=root' \
  --env 'AUDIO_DRIVER=pa,server=unix:/tmp/pulseaudio.socket' \
  --env 'RAM=4' \
   '--env' 'TERMS_OF_USE=i_agree' '--device' '/dev/kvm' '--device' '/dev/snd' '-p' '50922:10022' '-v' '/run/user/1000/pulse/native:/tmp/pulseaudio.socket' \
  -- sickcodes/docker-osx:auto /usr/local/bin/init -- /bin/sh - /x11docker/containerrc

Can you explain WORKDIR "${TERMS_OF_USE}" = i_agree ?

sickcodes commented 3 years ago

TERMS_OF_USE=i_agree is some boilerplate I added and ENV is already set to i_agree.

The actual pause is because Docker is copying the 20GB file to /var/lib/docker/overlay2/xxx/diff.

It takes a minute to do that, sometimes much longer depending on the disk.

Test it using the non-blocking layer version :latest as it's tiny

docker run --tty --detach \
  --name x11docker_X1_sickcodes-docker-osx-auto_44072375976 \
  --user 0:0 \
  --userns host \
  --ipc host \
  --security-opt label=type:container_runtime_t \
  --volume '/usr/bin/docker-init':'/usr/local/bin/init':ro \
  --tmpfs /run --tmpfs /run/lock \
  --volume '/home/werner/.cache/x11docker/sickcodes-docker-osx-auto-44072375976/share':'/x11docker':rw \
  --volume '/tmp/.X11-unix/X1':'/X1':rw \
  --workdir '/tmp' \
  --entrypoint env \
  --env 'container=docker' \
  --env 'XAUTHORITY=/x11docker/Xauthority.client' \
  --env 'DISPLAY=:1' \
  --env 'USER=root' \
  --env 'AUDIO_DRIVER=pa,server=unix:/tmp/pulseaudio.socket' \
  --env 'RAM=4' \
  --env 'TERMS_OF_USE=i_agree' \
  --share '/dev/kvm' \
  -- sickcodes/docker-osx:latest /usr/local/bin/init -- /bin/sh - /x11docker/containerrc

Then once it works, you will know :auto will work:

docker run --tty --detach \
  --name x11docker_X1_sickcodes-docker-osx-auto_44072375976 \
  --user 0:0 \
  --userns host \
  --ipc host \
  --security-opt label=type:container_runtime_t \
  --volume '/usr/bin/docker-init':'/usr/local/bin/init':ro \
  --tmpfs /run --tmpfs /run/lock \
  --volume '/home/werner/.cache/x11docker/sickcodes-docker-osx-auto-44072375976/share':'/x11docker':rw \
  --volume '/tmp/.X11-unix/X1':'/X1':rw \
  --workdir '/tmp' \
  --entrypoint env \
  --env 'container=docker' \
  --env 'XAUTHORITY=/x11docker/Xauthority.client' \
  --env 'DISPLAY=:1' \
  --env 'USER=root' \
  --env 'AUDIO_DRIVER=pa,server=unix:/tmp/pulseaudio.socket' \
  --env 'RAM=4' \
  --env 'TERMS_OF_USE=i_agree' \
  --share '/dev/kvm' \
  -- sickcodes/docker-osx:auto /usr/local/bin/init -- /bin/sh - /x11docker/containerrc

Use the container:10022 instead of localhost:10022

How do you open ports with x11Docker?

mviereck commented 3 years ago

Can you explain WORKDIR "${TERMS_OF_USE}" = i_agree ?

TERMS_OF_USE=i_agree is some boilerplate I added and ENV is already set to i_agree.

x11docker sets --workdir=/tmp. Is there some sort of directory, e.g. /i_agree, where x11docker should point to? I am confused about this variable in a WORKDIR template.

Test it using the non-blocking layer version :latest as it's tiny

I had a look at docker hub, 1.6 GB is still too much for my internet connection. I cannot run a test myself.

How do you open ports with x11Docker?

Nothing special, if the user specifies e.g. -p 50922:10022, it is just added to the docker command.

Maybe you could do a test run yourself? I'll take your example:

docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    sickcodes/docker-osx:latest

With x11docker the command would be:

x11docker --hostdisplay --clipboard --interactive --user=RETAIN --cap-default -- -p 50922:10022 -- sickcodes/docker-osx:latest

--interactive --user=RETAIN --cap-default might not be needed, just added them to be as close as possible to a native docker run.

hongyi-zhao commented 3 years ago

Failed as shown below:

werner@X10DAi:~$ x11docker --hostdisplay --clipboard --interactive --user=RETAIN --cap-default -- -p 50922:10022 -- sickcodes/docker-osx:latest
x11docker WARNING: User werner is member of group docker.
  That allows unprivileged processes on host to gain root privileges.

x11docker WARNING: Option --clipboard: To allow clipboard sharing with
  option --hostdisplay, trusted cookies will be enabled.
  No protection against X security leaks is left!
  Consider to use another X server option.

x11docker WARNING: Option --cap-default disables security hardening
  for containers done by x11docker. Default docker capabilities are allowed.
  This is considered to be less secure.

x11docker note: Option --cap-default: Enabling option --newprivileges.
  You can avoid this with --newprivileges=no

x11docker note: Option --hostdisplay: To allow --hostdisplay with trusted cookies,
  x11docker must share host IPC namespace with container (option --hostipc)
  to allow shared memory for X extension MIT-SHM.

x11docker WARNING: Option --hostdisplay with trusted cookies provides
      QUITE BAD CONTAINER ISOLATION !
  Keylogging and controlling host applications is possible! 
  Clipboard sharing is enabled (option --cliboard).
  It is recommended to use another X server option like --nxagent or --xpra.

x11docker note: Option --hostdisplay may fail with proprietary NVIDIA driver
  on host. In that case try other X server options like 
  --nxagent, --xpra or --xephyr.

x11docker WARNING: Option --hostipc severely degrades 
  container isolation. IPC namespace remapping is disabled.

x11docker WARNING: Found custom DOCKER_RUN_OPTIONS.
  x11docker will add them to 'docker run' command without
  a serious check for validity or security. Found options:
   '-p' '50922:10022'

x11docker WARNING: Option --newprivileges=yes: x11docker does not set 
  docker run option --security-opt=no-new-privileges. 
  That degrades container security.
  However, this is still within a default docker setup.

x11docker note: Found 'WORKDIR  "${NOPICKER}" == true ' in image.  
  You can change it with option --workdir=DIR.

cp: cannot stat '/root/.': Permission denied
usage: sudo -h | -K | -k | -V
usage: sudo -v [-AknS] [-g group] [-h host] [-p prompt] [-u user]
usage: sudo -l [-AknS] [-g group] [-h host] [-p prompt] [-U user] [-u user]
            [command]
usage: sudo [-AbEHknPS] [-C num] [-D directory] [-g group] [-h host] [-p
            prompt] [-R directory] [-T timeout] [-u user] [VAR=value] [-i|-s]
            [<command>]
usage: sudo -e [-AknS] [-C num] [-D directory] [-g group] [-h host] [-p prompt]
            [-R directory] [-T timeout] [-u user] file ...
chown: cannot access '/dev/kvm': No such file or directory
chown: cannot access '/dev/snd': No such file or directory
chown: cannot access '': No such file or directory
chown: cannot access '/env': No such file or directory
werner@X10DAi:~$

mviereck commented 3 years ago

At least some first error messages!

chown: cannot access '/dev/kvm': No such file or directory

Ups, I forgot to add /dev/kvm in my example. Please try:

x11docker --hostdisplay --clipboard --interactive --user=RETAIN --cap-default -- --device /dev/kvm -p 50922:10022 -- sickcodes/docker-osx:latest

sickcodes commented 3 years ago

Can you explain WORKDIR "${TERMS_OF_USE}" = i_agree ?

TERMS_OF_USE=i_agree is some boilerplate I added and ENV is already set to i_agree.

x11docker sets --workdir=/tmp. Is there some sort of directory, e.g. /i_agree, where x11docker should point to? I am confused about this variable in a WORKDIR template.

Test it using the non-blocking layer version :latest as it's tiny

I had a look at docker hub, 1.6 GB is still too much for my internet connection. I cannot run a test myself.

How do you open ports with x11Docker?

Nothing special, if the user specifies e.g. -p 50922:10022, it is just added to the docker command.

Maybe you could do a test run yourself?

I'll take your example:
docker run -it \

    --device /dev/kvm \

    -p 50922:10022 \

    -v /tmp/.X11-unix:/tmp/.X11-unix \

    -e "DISPLAY=${DISPLAY:-:0.0}" \

    sickcodes/docker-osx:latest
With x11docker the command would be:
x11docker --hostdisplay --clipboard --interactive --user=RETAIN --cap-default -- -p 50922:10022 -- sickcodes/docker-osx:latest
--interactive --user=RETAIN --cap-default might not be needed, just added them to be as close as possible to a native docker run.

The workdir is /home/arch/OSX-KVM

Let me test that out above.

A few months ago I had seen this project too and at the time we were using --privileged which did not appear to mesh with the security aspect of x11docker.

Learnt tons since beginning and the only things required now are /dev/kvm and /tmp/.X11-unix

The containers internet is running on the default docker bridge, I will give your above commands a try tonight

mviereck commented 3 years ago

The workdir is /home/arch/OSX-KVM

If that is essential, add --workdir /home/arch/OSX-KVM:

x11docker --hostdisplay --clipboard --workdir /home/arch/OSX-KVM --interactive --user=RETAIN --cap-default -- -p 50922:10022 -- sickcodes/docker-osx:latest

Learnt tons since beginning and the only things required now are /dev/kvm and /tmp/.X11-unix

I had a look at the Dockerfile and found that you run chown on /dev/kvm and /dev/snd. Instead of that I recommend to use --group-add kvm and --group-add audio.

hongyi-zhao commented 3 years ago

At least some first error messages!

chown: cannot access '/dev/kvm': No such file or directory

Ups, I forgot to add /dev/kvm in my example. Please try:
x11docker --hostdisplay --clipboard --interactive --user=RETAIN --cap-default -- --device /dev/kvm -p 50922:10022 -- sickcodes/docker-osx:latest

I added the option --device /dev/snd into your above suggested command, but still failed:

werner@X10DAi:~$ x11docker --hostdisplay --clipboard --interactive --user=RETAIN --cap-default -- --device /dev/kvm --device /dev/snd -p 50922:10022 -- sickcodes/docker-osx:latest
x11docker WARNING: User werner is member of group docker.
  That allows unprivileged processes on host to gain root privileges.

x11docker WARNING: Option --clipboard: To allow clipboard sharing with
  option --hostdisplay, trusted cookies will be enabled.
  No protection against X security leaks is left!
  Consider to use another X server option.

x11docker WARNING: Option --cap-default disables security hardening
  for containers done by x11docker. Default docker capabilities are allowed.
  This is considered to be less secure.

x11docker note: Option --cap-default: Enabling option --newprivileges.
  You can avoid this with --newprivileges=no

x11docker note: Option --hostdisplay: To allow --hostdisplay with trusted cookies,
  x11docker must share host IPC namespace with container (option --hostipc)
  to allow shared memory for X extension MIT-SHM.

x11docker WARNING: Option --hostdisplay with trusted cookies provides
      QUITE BAD CONTAINER ISOLATION !
  Keylogging and controlling host applications is possible! 
  Clipboard sharing is enabled (option --cliboard).
  It is recommended to use another X server option like --nxagent or --xpra.

x11docker note: Option --hostdisplay may fail with proprietary NVIDIA driver
  on host. In that case try other X server options like 
  --nxagent, --xpra or --xephyr.

x11docker WARNING: Option --hostipc severely degrades 
  container isolation. IPC namespace remapping is disabled.

x11docker WARNING: Found custom DOCKER_RUN_OPTIONS.
  x11docker will add them to 'docker run' command without
  a serious check for validity or security. Found options:
   '--device' '/dev/kvm' '--device' '/dev/snd' '-p' '50922:10022'

x11docker WARNING: Option --newprivileges=yes: x11docker does not set 
  docker run option --security-opt=no-new-privileges. 
  That degrades container security.
  However, this is still within a default docker setup.

x11docker note: Found 'WORKDIR  "${NOPICKER}" == true ' in image.  
  You can change it with option --workdir=DIR.

cp: cannot stat '/root/.': Permission denied
usage: sudo -h | -K | -k | -V
usage: sudo -v [-AknS] [-g group] [-h host] [-p prompt] [-u user]
usage: sudo -l [-AknS] [-g group] [-h host] [-p prompt] [-U user] [-u user]
            [command]
usage: sudo [-AbEHknPS] [-C num] [-D directory] [-g group] [-h host] [-p
            prompt] [-R directory] [-T timeout] [-u user] [VAR=value] [-i|-s]
            [<command>]
usage: sudo -e [-AknS] [-C num] [-D directory] [-g group] [-h host] [-p prompt]
            [-R directory] [-T timeout] [-u user] file ...
chown: cannot access '': No such file or directory
chown: cannot access '/env': No such file or directory
werner@X10DAi:~$

hongyi-zhao commented 3 years ago

The workdir is /home/arch/OSX-KVM

If that is essential, add --workdir /home/arch/OSX-KVM:
x11docker --hostdisplay --clipboard --workdir /home/arch/OSX-KVM --interactive --user=RETAIN --cap-default -- -p 50922:10022 -- sickcodes/docker-osx:latest
Learnt tons since beginning and the only things required now are /dev/kvm and /tmp/.X11-unix

I had a look at the Dockerfile and found that you run chown on /dev/kvm and /dev/snd. Instead of that I recommend to use --group-add kvm and --group-add audio.

Failed with the error below:

werner@X10DAi:~$ x11docker --hostdisplay --clipboard --workdir /home/arch/OSX-KVM --interactive --user=RETAIN --cap-default -- --group-add kvm --group-add audio -p 50922:10022 -- sickcodes/docker-osx:latest
x11docker WARNING: User werner is member of group docker.
  That allows unprivileged processes on host to gain root privileges.

x11docker WARNING: Option --clipboard: To allow clipboard sharing with
  option --hostdisplay, trusted cookies will be enabled.
  No protection against X security leaks is left!
  Consider to use another X server option.

x11docker WARNING: Option --cap-default disables security hardening
  for containers done by x11docker. Default docker capabilities are allowed.
  This is considered to be less secure.

x11docker note: Option --cap-default: Enabling option --newprivileges.
  You can avoid this with --newprivileges=no

x11docker note: Option --hostdisplay: To allow --hostdisplay with trusted cookies,
  x11docker must share host IPC namespace with container (option --hostipc)
  to allow shared memory for X extension MIT-SHM.

x11docker WARNING: Option --hostdisplay with trusted cookies provides
      QUITE BAD CONTAINER ISOLATION !
  Keylogging and controlling host applications is possible! 
  Clipboard sharing is enabled (option --cliboard).
  It is recommended to use another X server option like --nxagent or --xpra.

x11docker note: Option --hostdisplay may fail with proprietary NVIDIA driver
  on host. In that case try other X server options like 
  --nxagent, --xpra or --xephyr.

x11docker WARNING: Option --hostipc severely degrades 
  container isolation. IPC namespace remapping is disabled.

x11docker WARNING: Found custom DOCKER_RUN_OPTIONS.
  x11docker will add them to 'docker run' command without
  a serious check for validity or security. Found options:
   '--group-add' 'kvm' '--group-add' 'audio' '-p' '50922:10022'

x11docker note: Found option --group-add in custom DOCKER_RUN_OPTIONS.
  Please use x11docker option --group-add instead.

x11docker WARNING: Option --newprivileges=yes: x11docker does not set 
  docker run option --security-opt=no-new-privileges. 
  That degrades container security.
  However, this is still within a default docker setup.

cp: cannot stat '/root/.': Permission denied
usage: sudo -h | -K | -k | -V
usage: sudo -v [-AknS] [-g group] [-h host] [-p prompt] [-u user]
usage: sudo -l [-AknS] [-g group] [-h host] [-p prompt] [-U user] [-u user]
            [command]
usage: sudo [-AbEHknPS] [-C num] [-D directory] [-g group] [-h host] [-p
            prompt] [-R directory] [-T timeout] [-u user] [VAR=value] [-i|-s]
            [<command>]
usage: sudo -e [-AknS] [-C num] [-D directory] [-g group] [-h host] [-p prompt]
            [-R directory] [-T timeout] [-u user] file ...
chown: cannot access '/dev/kvm': No such file or directory
chown: cannot access '/dev/snd': No such file or directory
chown: cannot access '': No such file or directory
chown: cannot access '/env': No such file or directory
werner@X10DAi:~$

mviereck commented 3 years ago

The Dockerfile shows a complicated CMD instruction, maybe it is not translated well in x11docker. The logfile could show me that.

@hongyi-zhao Could you run the command again, terminate the container with exit and afterwards show me ~/.cache/x11docker/x11docker.log at www.pastebin.com?

x11docker --hostdisplay --clipboard --workdir /home/arch/OSX-KVM --interactive --user=RETAIN --cap-default -- -p 50922:10022 -- sickcodes/docker-osx:latest

mviereck commented 3 years ago

I've replaced the long CMD with a script in a custom Dockerfile. Could you build an image from this Dockerfile and test that instead?

FROM sickcodes/docker-osx:latest

RUN echo '#! /bin/bash \n\
 \n\
cd /home/arch/OSX-KVM \n\
 \n\
sudo touch /dev/kvm /dev/snd "${IMAGE_PATH}" "${BOOTDISK}" "${ENV}" \n\
sudo chown -R $(id -u):$(id -g) /dev/kvm /dev/snd "${IMAGE_PATH}" "${BOOTDISK}" "${ENV}" \n\
 \n\
[[ "${NOPICKER}" == true ]] && { \n\
  sed -i "/^.*InstallMedia.*/d" Launch.sh \n\
  export BOOTDISK="${BOOTDISK:=/home/arch/OSX-KVM/OpenCore-Catalina/OpenCore-nopicker.qcow2}" \n\
} || export BOOTDISK="${BOOTDISK:=/home/arch/OSX-KVM/OpenCore-Catalina/OpenCore.qcow2}" \n\
 \n\
[[ "${GENERATE_UNIQUE}" == true ]] && { \n\
  ./Docker-OSX/osx-serial-generator/generate-unique-machine-values.sh \\\n\
            --master-plist-url="${MASTER_PLIST_URL}" \\\n\
            --count 1 \\\n\
            --tsv ./serial.tsv \\\n\
            --bootdisks \\\n\
            --width "${WIDTH:-1920}" \\\n\
            --height "${HEIGHT:-1080}" \\\n\
            --output-bootdisk "${BOOTDISK:=/home/arch/OSX-KVM/OpenCore-Catalina/OpenCore.qcow2}" \\\n\
            --output-env "${ENV:=/env}" \n\
} \n\
 \n\
[[ "${GENERATE_SPECIFIC}" == true ]] && { \n\
  source "${ENV:=/env}" 2>/dev/null \n\
  ./Docker-OSX/osx-serial-generator/generate-specific-bootdisk.sh \\\n\
            --master-plist-url="${MASTER_PLIST_URL}" \\\n\
            --model "${DEVICE_MODEL}" \\\n\
            --serial "${SERIAL}" \\\n\
            --board-serial "${BOARD_SERIAL}" \\\n\
            --uuid "${UUID}" \\\n\
            --mac-address "${MAC_ADDRESS}" \\\n\
            --width "${WIDTH:-1920}" \\\n\
            --height "${HEIGHT:-1080}" \\\n\
            --output-bootdisk "${BOOTDISK:=/home/arch/OSX-KVM/OpenCore-Catalina/OpenCore.qcow2}" \n\
} \n\
 \n\
./enable-ssh.sh && envsubst < ./Launch.sh | bash \n\
' > /home/arch/OSX-KVM/CMD.sh && chmod +x /home/arch/OSX-KVM/CMD.sh

CMD /home/arch/OSX-KVM/CMD.sh

hongyi-zhao commented 3 years ago

The Dockerfile shows a complicated CMD instruction, maybe it is not translated well in x11docker. The logfile could show me that.

@hongyi-zhao Could you run the command again, terminate the container with exit

I can't execute the docker run command successfully till now. I mean I can't enter the container, so how can I terminate the container with exit?

and afterwards show me ~/.cache/x11docker/x11docker.log at www.pastebin.com?
x11docker --hostdisplay --clipboard --workdir /home/arch/OSX-KVM --interactive --user=RETAIN --cap-default -- -p 50922:10022 -- sickcodes/docker-osx:latest

Anyway, just FYI. I first deleted the old ~/.cache/x11docker/x11docker.log, and then upload the newly generated one by the above command here.

hongyi-zhao commented 3 years ago

I've replaced the long CMD with a script in a custom Dockerfile. Could you build an image from this Dockerfile and test that instead?

werner@X10DAi:~/docker-osx$ docker build --network host -t hongyizhao/docker-osx .
Sending build context to Docker daemon  3.584kB
Step 1/3 : FROM sickcodes/docker-osx:latest
 ---> 0a032ee76d58
Step 2/3 : RUN echo '#! /bin/bash \n \ncd /home/arch/OSX-KVM \n \nsudo touch /dev/kvm /dev/snd "${IMAGE_PATH}" "${BOOTDISK}" "${ENV}" \nsudo chown -R $(id -u):$(id -g) /dev/kvm /dev/snd "${IMAGE_PATH}" "${BOOTDISK}" "${ENV}" \n \n[[ "${NOPICKER}" == true ]] && { \n  sed -i "/^.*InstallMedia.*/d" Launch.sh \n  export BOOTDISK="${BOOTDISK:=/home/arch/OSX-KVM/OpenCore-Catalina/OpenCore-nopicker.qcow2}" \n} || export BOOTDISK="${BOOTDISK:=/home/arch/OSX-KVM/OpenCore-Catalina/OpenCore.qcow2}" \n \n[[ "${GENERATE_UNIQUE}" == true ]] && { \n  ./Docker-OSX/osx-serial-generator/generate-unique-machine-values.sh \\\n            --master-plist-url="${MASTER_PLIST_URL}" \\\n            --count 1 \\\n            --tsv ./serial.tsv \\\n            --bootdisks \\\n            --width "${WIDTH:-1920}" \\\n            --height "${HEIGHT:-1080}" \\\n            --output-bootdisk "${BOOTDISK:=/home/arch/OSX-KVM/OpenCore-Catalina/OpenCore.qcow2}" \\\n            --output-env "${ENV:=/env}" \n} \n \n[[ "${GENERATE_SPECIFIC}" == true ]] && { \n  source "${ENV:=/env}" 2>/dev/null \n  ./Docker-OSX/osx-serial-generator/generate-specific-bootdisk.sh \\\n            --master-plist-url="${MASTER_PLIST_URL}" \\\n            --model "${DEVICE_MODEL}" \\\n            --serial "${SERIAL}" \\\n            --board-serial "${BOARD_SERIAL}" \\\n            --uuid "${UUID}" \\\n            --mac-address "${MAC_ADDRESS}" \\\n            --width "${WIDTH:-1920}" \\\n            --height "${HEIGHT:-1080}" \\\n            --output-bootdisk "${BOOTDISK:=/home/arch/OSX-KVM/OpenCore-Catalina/OpenCore.qcow2}" \n} \n \n./enable-ssh.sh && envsubst < ./Launch.sh | bash \n' > /home/arch/OSX-KVM/CMD.sh && chmod +x /home/arch/OSX-KVM/CMD.sh
 ---> Running in 2bca79296c29
Removing intermediate container 2bca79296c29
 ---> 698759484a46
Step 3/3 : CMD /home/arch/OSX-KVM/CMD.sh
 ---> Running in ababd87ab8d7
Removing intermediate container ababd87ab8d7
 ---> e31622243a9a
Successfully built e31622243a9a
Successfully tagged hongyizhao/docker-osx:latest
werner@X10DAi:~/docker-osx$ x11docker --hostdisplay --clipboard --workdir /home/arch/OSX-KVM --interactive --user=RETAIN --cap-default -- -p 50922:10022 -- hongyizhao/docker-osx:latest
x11docker WARNING: User werner is member of group docker.
  That allows unprivileged processes on host to gain root privileges.

x11docker WARNING: Option --clipboard: To allow clipboard sharing with
  option --hostdisplay, trusted cookies will be enabled.
  No protection against X security leaks is left!
  Consider to use another X server option.

x11docker WARNING: Option --cap-default disables security hardening
  for containers done by x11docker. Default docker capabilities are allowed.
  This is considered to be less secure.

x11docker note: Option --cap-default: Enabling option --newprivileges.
  You can avoid this with --newprivileges=no

x11docker note: Option --hostdisplay: To allow --hostdisplay with trusted cookies,
  x11docker must share host IPC namespace with container (option --hostipc)
  to allow shared memory for X extension MIT-SHM.

x11docker WARNING: Option --hostdisplay with trusted cookies provides
      QUITE BAD CONTAINER ISOLATION !
  Keylogging and controlling host applications is possible! 
  Clipboard sharing is enabled (option --cliboard).
  It is recommended to use another X server option like --nxagent or --xpra.

x11docker note: Option --hostdisplay may fail with proprietary NVIDIA driver
  on host. In that case try other X server options like 
  --nxagent, --xpra or --xephyr.

x11docker WARNING: Option --hostipc severely degrades 
  container isolation. IPC namespace remapping is disabled.

x11docker WARNING: Found custom DOCKER_RUN_OPTIONS.
  x11docker will add them to 'docker run' command without
  a serious check for validity or security. Found options:
   '-p' '50922:10022'

x11docker WARNING: Option --newprivileges=yes: x11docker does not set 
  docker run option --security-opt=no-new-privileges. 
  That degrades container security.
  However, this is still within a default docker setup.

/bin/bash: \n \ncd /home/arch/OSX-KVM \n \nsudo touch /dev/kvm /dev/snd "${IMAGE_PATH}" "${BOOTDISK}" "${ENV}" \nsudo chown -R $(id -u):$(id -g) /dev/kvm /dev/snd "${IMAGE_PATH}" "${BOOTDISK}" "${ENV}" \n \n[[ "${NOPICKER}" == true ]] && { \n  sed -i "/: No such file or directory
werner@X10DAi:~/docker-osx$

mviereck commented 3 years ago

Thank you for the log file and the test build! The logfile gives me some hints what is going wrong. It seems x11docker gets some wrong results from docker inspect. Can you show me the output of docker inspect sickcodes/docker-osx:latest at pastebin.com?

I have fixed the Dockerfile, can you try again?

FROM sickcodes/docker-osx:latest

RUN echo -e '#! /bin/bash \n\
 \n\
cd /home/arch/OSX-KVM \n\
 \n\
sudo touch /dev/kvm /dev/snd "${IMAGE_PATH}" "${BOOTDISK}" "${ENV}" \n\
sudo chown -R $(id -u):$(id -g) /dev/kvm /dev/snd "${IMAGE_PATH}" "${BOOTDISK}" "${ENV}" \n\
 \n\
[[ "${NOPICKER}" == true ]] && { \n\
  sed -i "/^.*InstallMedia.*/d" Launch.sh \n\
  export BOOTDISK="${BOOTDISK:=/home/arch/OSX-KVM/OpenCore-Catalina/OpenCore-nopicker.qcow2}" \n\
} || export BOOTDISK="${BOOTDISK:=/home/arch/OSX-KVM/OpenCore-Catalina/OpenCore.qcow2}" \n\
 \n\
[[ "${GENERATE_UNIQUE}" == true ]] && { \n\
  ./Docker-OSX/osx-serial-generator/generate-unique-machine-values.sh \\\n\
            --master-plist-url="${MASTER_PLIST_URL}" \\\n\
            --count 1 \\\n\
            --tsv ./serial.tsv \\\n\
            --bootdisks \\\n\
            --width "${WIDTH:-1920}" \\\n\
            --height "${HEIGHT:-1080}" \\\n\
            --output-bootdisk "${BOOTDISK:=/home/arch/OSX-KVM/OpenCore-Catalina/OpenCore.qcow2}" \\\n\
            --output-env "${ENV:=/env}" \n\
} \n\
 \n\
[[ "${GENERATE_SPECIFIC}" == true ]] && { \n\
  source "${ENV:=/env}" 2>/dev/null \n\
  ./Docker-OSX/osx-serial-generator/generate-specific-bootdisk.sh \\\n\
            --master-plist-url="${MASTER_PLIST_URL}" \\\n\
            --model "${DEVICE_MODEL}" \\\n\
            --serial "${SERIAL}" \\\n\
            --board-serial "${BOARD_SERIAL}" \\\n\
            --uuid "${UUID}" \\\n\
            --mac-address "${MAC_ADDRESS}" \\\n\
            --width "${WIDTH:-1920}" \\\n\
            --height "${HEIGHT:-1080}" \\\n\
            --output-bootdisk "${BOOTDISK:=/home/arch/OSX-KVM/OpenCore-Catalina/OpenCore.qcow2}" \n\
} \n\
 \n\
./enable-ssh.sh && envsubst < ./Launch.sh | bash \n\
' > /home/arch/OSX-KVM/CMD.sh && chmod +x /home/arch/OSX-KVM/CMD.sh

CMD /home/arch/OSX-KVM/CMD.sh

Also worth a test: option --no-setup to avoid issues with docker inspect.

x11docker --no-setup --hostdisplay --clipboard --workdir /home/arch/OSX-KVM --interactive --user=RETAIN --cap-default -- -p 50922:10022 -- sickcodes/docker-osx:latest

hongyi-zhao commented 3 years ago

Can you show me the output of docker inspect sickcodes/docker-osx:latest at pastebin.com?

https://pastebin.com/QJjCHa2t

I have fixed the Dockerfile, can you try again?

See below.

Also worth a test: option --no-setup to avoid issues with docker inspect.

$ x11docker --no-setup --hostdisplay --clipboard --workdir /home/arch/OSX-KVM --interactive --user=RETAIN --cap-default -- -p 50922:10022 -- sickcodes/docker-osx:latest
x11docker note: Option --no-setup: experimental option.

x11docker WARNING: User werner is member of group docker.
  That allows unprivileged processes on host to gain root privileges.

x11docker WARNING: Option --clipboard: To allow clipboard sharing with
  option --hostdisplay, trusted cookies will be enabled.
  No protection against X security leaks is left!
  Consider to use another X server option.

x11docker WARNING: Option --cap-default disables security hardening
  for containers done by x11docker. Default docker capabilities are allowed.
  This is considered to be less secure.

x11docker note: Option --cap-default: Enabling option --newprivileges.
  You can avoid this with --newprivileges=no

x11docker note: Option --hostdisplay: To allow --hostdisplay with trusted cookies,
  x11docker must share host IPC namespace with container (option --hostipc)
  to allow shared memory for X extension MIT-SHM.

x11docker WARNING: Option --hostdisplay with trusted cookies provides
      QUITE BAD CONTAINER ISOLATION !
  Keylogging and controlling host applications is possible! 
  Clipboard sharing is enabled (option --cliboard).
  It is recommended to use another X server option like --nxagent or --xpra.

x11docker note: Option --hostdisplay may fail with proprietary NVIDIA driver
  on host. In that case try other X server options like 
  --nxagent, --xpra or --xephyr.

x11docker WARNING: Option --hostipc severely degrades 
  container isolation. IPC namespace remapping is disabled.

x11docker WARNING: Found custom DOCKER_RUN_OPTIONS.
  x11docker will add them to 'docker run' command without
  a serious check for validity or security. Found options:
   '-p' '50922:10022'

x11docker WARNING: Option --newprivileges=yes: x11docker does not set 
  docker run option --security-opt=no-new-privileges. 
  That degrades container security.
  However, this is still within a default docker setup.

touch: cannot touch '': No such file or directory
chown: cannot access '': No such file or directory
ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519 
nohup: appending output to 'nohup.out'
qemu-system-x86_64: warning: dbind: Couldn't connect to accessibility bus: Failed to connect to socket /tmp/dbus-zQgs5HL23I: Connection refused
kvm version too old
qemu-system-x86_64: failed to initialize kvm: Inappropriate ioctl for device
qemu-system-x86_64: falling back to tcg
QEMU 5.2.0 monitor - type 'help' for more information
(qemu) qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.avx [bit 28]
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.80000007H:EDX.invtsc [bit 8]
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.avx [bit 28]
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.80000007H:EDX.invtsc [bit 8]
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.avx [bit 28]
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.80000007H:EDX.invtsc [bit 8]
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.avx [bit 28]
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.80000007H:EDX.invtsc [bit 8]
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5233:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM default
alsa: Could not initialize DAC
alsa: Failed to open `default':
alsa: Reason: No such file or directory
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5233:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM default
alsa: Could not initialize DAC
alsa: Failed to open `default':
alsa: Reason: No such file or directory
audio: Failed to create voice `dac'
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5233:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM default
alsa: Could not initialize ADC
alsa: Failed to open `default':
alsa: Reason: No such file or directory
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5233:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM default
alsa: Could not initialize ADC
alsa: Failed to open `default':
alsa: Reason: No such file or directory
audio: Failed to create voice `adc'

But qemu encountered a problem at a certain step and stopped there, as shown in the screenshot below.

mviereck commented 3 years ago

Finally I found a quite unexpected bug in x11docker. It was a parsing error of docker inspect output caused by the chars [ and ] in the image command of docker-osx. That also explains the odd WORKDIR result before.

I've uploaded a fix. Please update and try again to run the original image, hopefully it works now.

x11docker --hostdisplay --clipboard --workdir /home/arch/OSX-KVM --interactive --user=RETAIN --cap-default \
 -- --device /dev/kvm -p 50922:10022 -- sickcodes/docker-osx:latest

If that works, we can try to simplify the command.

hongyi-zhao commented 3 years ago

Great, see below:

$ x11docker --hostdisplay --clipboard --workdir /home/arch/OSX-KVM --interactive --user=RETAIN --cap-default \
>  -- --device /dev/kvm -p 50922:10022 -- sickcodes/docker-osx:latest
x11docker WARNING: User werner is member of group docker.
  That allows unprivileged processes on host to gain root privileges.

x11docker WARNING: Option --clipboard: To allow clipboard sharing with
  option --hostdisplay, trusted cookies will be enabled.
  No protection against X security leaks is left!
  Consider to use another X server option.

x11docker WARNING: Option --cap-default disables security hardening
  for containers done by x11docker. Default docker capabilities are allowed.
  This is considered to be less secure.

x11docker note: Option --cap-default: Enabling option --newprivileges.
  You can avoid this with --newprivileges=no

x11docker note: Option --hostdisplay: To allow --hostdisplay with trusted cookies,
  x11docker must share host IPC namespace with container (option --hostipc)
  to allow shared memory for X extension MIT-SHM.

x11docker WARNING: Option --hostdisplay with trusted cookies provides
      QUITE BAD CONTAINER ISOLATION !
  Keylogging and controlling host applications is possible! 
  Clipboard sharing is enabled (option --cliboard).
  It is recommended to use another X server option like --nxagent or --xpra.

x11docker note: Option --hostdisplay may fail with proprietary NVIDIA driver
  on host. In that case try other X server options like 
  --nxagent, --xpra or --xephyr.

x11docker WARNING: Option --hostipc severely degrades 
  container isolation. IPC namespace remapping is disabled.

x11docker WARNING: Found custom DOCKER_RUN_OPTIONS.
  x11docker will add them to 'docker run' command without
  a serious check for validity or security. Found options:
   '--device' '/dev/kvm' '-p' '50922:10022'

x11docker WARNING: Option --newprivileges=yes: x11docker does not set 
  docker run option --security-opt=no-new-privileges. 
  That degrades container security.
  However, this is still within a default docker setup.

usage: sudo -h | -K | -k | -V
usage: sudo -v [-AknS] [-g group] [-h host] [-p prompt] [-u user]
usage: sudo -l [-AknS] [-g group] [-h host] [-p prompt] [-U user] [-u user]
            [command]
usage: sudo [-AbEHknPS] [-C num] [-D directory] [-g group] [-h host] [-p
            prompt] [-R directory] [-T timeout] [-u user] [VAR=value] [-i|-s]
            [<command>]
usage: sudo -e [-AknS] [-C num] [-D directory] [-g group] [-h host] [-p prompt]
            [-R directory] [-T timeout] [-u user] file ...
chown: cannot access '/dev/snd': No such file or directory
chown: cannot access '/env': No such file or directory
ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519 
nohup: appending output to 'nohup.out'
qemu-system-x86_64: warning: dbind: Couldn't connect to accessibility bus: Failed to connect to socket /tmp/dbus-EtA6dp9qWF: Connection refused
QEMU 5.2.0 monitor - type 'help' for more information
(qemu) ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5233:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM default
alsa: Could not initialize DAC
alsa: Failed to open `default':
alsa: Reason: No such file or directory
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5233:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM default
alsa: Could not initialize DAC
alsa: Failed to open `default':
alsa: Reason: No such file or directory
audio: Failed to create voice `dac'
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5233:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM default
alsa: Could not initialize ADC
alsa: Failed to open `default':
alsa: Reason: No such file or directory
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4745:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5233:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM default
alsa: Could not initialize ADC
alsa: Failed to open `default':
alsa: Reason: No such file or directory
audio: Failed to create voice `adc'

mviereck commented 3 years ago

Great!

We can try to simplify the command now. This one should just work

x11docker  --user=RETAIN --share /dev/kvm --cap-default -- -p 50922:10022 -- sickcodes/docker-osx:latest

@sickcodes How important is user arch? Would it work with another user, too? Maybe we could omit --user=RETAIN.

Maybe this one works, too:

x11docker --user=RETAIN --share /dev/kvm --group-add kvm --  -p 50922:10022 -- sickcodes/docker-osx:latest

Without --cap-default sudo won't work. But access to /dev/kvm is granted with --group-add kvm.

@sickcodes You are running sudo chown -R $(id -u):$(id -g) /dev/kvm /dev/snd. That is not a good practice and changes the ownership on host, too. Instead I recommend to add user arch to group kvm. This group at least exists on debian:

$ ls -l /dev/kvm
crw-rw----+ 1 root kvm 10, 232 17. Mär 08:58 /dev/kvm
$ getent group kvm
kvm:x:127:

This can be done with --group-add kvm in the docker command or already in the image generation for user arch. However, a group id can be different across different linux systems, so --group-add kvm is more reliable. Also it has to be checked if group kvm exists on all important linux distributions. The same goes for /dev/snd: Just use group audio with --group-add audio.

Maybe this works although sudo will fail and user arch is replaced with the host user:

x11docker --share /dev/kvm --group-add kvm --alsa -- -p 50922:10022 -- sickcodes/docker-osx

hongyi-zhao commented 3 years ago

Great. All your above x11docker commands works.

mviereck commented 3 years ago

Great. All your above x11docker commands works.

Thank you for testing! If sound is not needed and the port is also not needed, the command can be reduced to:

x11docker --share /dev/kvm --group-add kvm  sickcodes/docker-osx

mviereck commented 3 years ago

How to fix this problem?

It seems that docker-osx tries to start sshd with sudo and fails because x11docker disallowed this. As a workaround --user=RETAIN --cap-default should fix that.

@sickcodes Would you consider to run ssh without sudo as an unprivileged user? This thread shows setup examples: https://serverfault.com/questions/344295/is-it-possible-to-run-sshd-as-a-normal-user

hongyi-zhao commented 3 years ago

I deleted my previous comment and retry with your suggested options, but still failed as below:

werner@X10DAi:~$ x11docker --share /dev/kvm --group-add kvm --user=RETAIN --cap-default --alsa -- -p 50922:10022 -- sickcodes/docker-osx:auto
x11docker WARNING: User werner is member of group docker.
  That allows unprivileged processes on host to gain root privileges.

x11docker WARNING: You are running GNOME desktop in outdated version 
  GNOME Shell 3.36.4
  This might cause issues with host applications if using additional X servers.
  It is recommended to use another desktop environment or GNOME >= 3.38.
  Only otherwise discouraged option --hostdisplay might work as expected.

x11docker note: Using X server option --hostdisplay

x11docker WARNING: Clipboard isolation may fail.

x11docker WARNING: Option --cap-default disables security hardening
  for containers done by x11docker. Default docker capabilities are allowed.
  This is considered to be less secure.

x11docker note: Option --cap-default: Enabling option --newprivileges.
  You can avoid this with --newprivileges=no

x11docker note: To allow protection against X security leaks,
  please install 'xinit' and one or more of:
    xpra, Xephyr, nxagent, weston+Xwayland, kwin_wayland+Xwayland or Xnest,
  or run a second Xorg server with option --xorg.

x11docker WARNING: Option --hostdisplay provides only low container isolation!
  It is recommended to use another X server option like --nxagent or --xpra.

  To improve security with --hostdisplay x11docker uses untrusted cookies.
  This can lead to strange behaviour of some applications.

  If you encounter application errors, enable option --clipboard
  that disables security restrictions for --hostdisplay as a side effect.

x11docker note: Option --hostdisplay may fail with proprietary NVIDIA driver
  on host. In that case try other X server options like 
  --nxagent, --xpra or --xephyr.

x11docker WARNING: Found custom DOCKER_RUN_OPTIONS.
  x11docker will add them to 'docker run' command without
  a serious check for validity or security. Found options:
   '-p' '50922:10022'

x11docker WARNING: ALSA sound with option --alsa degrades container isolation.
  Shares device files in /dev/snd, container gains access to sound hardware.
  Container applications can catch audio output and microphone input.

x11docker note: It seems that pulseaudio is running on your host.
  Pulseaudio can interfere with ALSA sound (option --alsa).
  Host sound may not work while container is playing sound and vice versa.
  Alternative: with pulseaudio on host and in image, use option --pulseaudio.

x11docker WARNING: Option --newprivileges=yes: x11docker does not set 
  docker run option --security-opt=no-new-privileges. 
  That degrades container security.
  However, this is still within a default docker setup.

x11docker WARNING: Sharing device file: /dev/kvm

x11docker WARNING: Sharing device file: /dev/snd

x11docker note: Found 'WORKDIR /home/arch/OSX-KVM' in image.  
  You can change it with option --workdir=DIR.

Disk is being copied between layers... Please wait a minute...
  File: /home/arch/OSX-KVM/mac_hdd_ng.img
  Size: 19139461120 Blocks: 37381768   IO Block: 4096   regular file
Device: 39h/57d Inode: 49023176    Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/    arch)   Gid: ( 1000/    arch)
Access: 2021-03-17 20:43:06.458018470 +0800
Modify: 2021-03-17 20:43:06.458018470 +0800
Change: 2021-03-17 20:43:06.782027999 +0800
 Birth: 2021-03-17 20:40:03.584324250 +0800
Large image is being copied between layers, please wait a minute...
ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519 
Saving key "/tmp/.ssh/id_docker_osx" failed: No such file or directory
Booting Docker-OSX in the background. Please wait...
grep: /tmp/.ssh/config: No such file or directory
tee: /tmp/.ssh/config: No such file or directory
Host 127.0.0.1
tee: /tmp/.ssh/config: No such file or directory
    User user
tee: /tmp/.ssh/config: No such file or directory
    Port 10022
tee: /tmp/.ssh/config: No such file or directory
    IdentityFile ~/.ssh/id_docker_osx
tee: /tmp/.ssh/config: No such file or directory
    StrictHostKeyChecking no
tee: /tmp/.ssh/config: No such file or directory
    UserKnownHostsFile=/dev/null

mviereck commented 3 years ago

@hongyi-zhao Can you show me the logfile at www.pastebin.com? It seems that x11docker sets the workdir to /tmp instead of /home/arch/OSX-KVM.

This can likely be fixed with --workdir=/home/arch/OSX-KVM. But show me the logfile without this, please.

hongyi-zhao commented 3 years ago

This can likely be fixed with --workdir=/home/arch/OSX-KVM. But show me the logfile without this, please.

https://pastebin.com/eFhiATgq

mviereck commented 3 years ago

I found in the log that x11docker sets HOME=/tmp if using --user=RETAIN. This is fixed now, please update. I am not sure if that caused the issue here, but maybe.

hongyi-zhao commented 3 years ago

This time it runs into the masOS desktop:

$ x11docker --share /dev/kvm --group-add kvm --user=RETAIN --cap-default --alsa -- -p 50922:10022 -- sickcodes/docker-osx:auto
x11docker WARNING: User werner is member of group docker.
  That allows unprivileged processes on host to gain root privileges.

x11docker WARNING: You are running GNOME desktop in outdated version 
  GNOME Shell 3.36.4
  This might cause issues with host applications if using additional X servers.
  It is recommended to use another desktop environment or GNOME >= 3.38.
  Only otherwise discouraged option --hostdisplay might work as expected.

x11docker note: Using X server option --hostdisplay

x11docker WARNING: Clipboard isolation may fail.

x11docker WARNING: Option --cap-default disables security hardening
  for containers done by x11docker. Default docker capabilities are allowed.
  This is considered to be less secure.

x11docker note: Option --cap-default: Enabling option --newprivileges.
  You can avoid this with --newprivileges=no

x11docker note: To allow protection against X security leaks,
  please install 'xinit' and one or more of:
    xpra, Xephyr, nxagent, weston+Xwayland, kwin_wayland+Xwayland or Xnest,
  or run a second Xorg server with option --xorg.

x11docker WARNING: Option --hostdisplay provides only low container isolation!
  It is recommended to use another X server option like --nxagent or --xpra.

  To improve security with --hostdisplay x11docker uses untrusted cookies.
  This can lead to strange behaviour of some applications.

  If you encounter application errors, enable option --clipboard
  that disables security restrictions for --hostdisplay as a side effect.

x11docker note: Option --hostdisplay may fail with proprietary NVIDIA driver
  on host. In that case try other X server options like 
  --nxagent, --xpra or --xephyr.

x11docker WARNING: Found custom DOCKER_RUN_OPTIONS.
  x11docker will add them to 'docker run' command without
  a serious check for validity or security. Found options:
   '-p' '50922:10022'

x11docker WARNING: ALSA sound with option --alsa degrades container isolation.
  Shares device files in /dev/snd, container gains access to sound hardware.
  Container applications can catch audio output and microphone input.

x11docker note: It seems that pulseaudio is running on your host.
  Pulseaudio can interfere with ALSA sound (option --alsa).
  Host sound may not work while container is playing sound and vice versa.
  Alternative: with pulseaudio on host and in image, use option --pulseaudio.

x11docker WARNING: Option --newprivileges=yes: x11docker does not set 
  docker run option --security-opt=no-new-privileges. 
  That degrades container security.
  However, this is still within a default docker setup.

x11docker WARNING: Sharing device file: /dev/kvm

x11docker WARNING: Sharing device file: /dev/snd

x11docker note: Found 'WORKDIR /home/arch/OSX-KVM' in image.  
  You can change it with option --workdir=DIR.

Disk is being copied between layers... Please wait a minute...
  File: /home/arch/OSX-KVM/mac_hdd_ng.img
  Size: 19139461120 Blocks: 37381776   IO Block: 4096   regular file
Device: 3eh/62d Inode: 49023176    Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/    arch)   Gid: ( 1000/    arch)
Access: 2021-03-17 21:57:34.480748123 +0800
Modify: 2021-03-17 21:57:34.480748123 +0800
Change: 2021-03-17 21:57:34.672758711 +0800
 Birth: 2021-03-17 21:54:37.198428706 +0800
Large image is being copied between layers, please wait a minute...
ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519 
Booting Docker-OSX in the background. Please wait...
/usr/sbin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/arch/.ssh/id_docker_osx.pub"
ssh: connect to host 127.0.0.1 port 10022: Connection refused
Disk is being copied between layers. Repeating until able to copy SSH key into OSX...
qemu-system-x86_64: warning: dbind: Couldn't connect to accessibility bus: Failed to connect to socket /tmp/dbus-WH2Cg0QlLP: Connection refused
QEMU 5.2.0 monitor - type 'help' for more information
(qemu) /usr/sbin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/arch/.ssh/id_docker_osx.pub"
qemu-system-x86_64: Slirp: Failed to send packet, ret: -1
qemu-system-x86_64: Slirp: Failed to send packet, ret: -1
qemu-system-x86_64: Slirp: Failed to send packet, ret: -1
qemu-system-x86_64: Slirp: Failed to send packet, ret: -1
Warning: Permanently added '[127.0.0.1]:10022' (ED25519) to the list of known hosts.
Host 127.0.0.1
    User user
    Port 10022
    IdentityFile ~/.ssh/id_docker_osx
    StrictHostKeyChecking no
    UserKnownHostsFile=/dev/null
Default username: user
Default password: alpine
Change it immediately using the command: passwd
Warning: Permanently added '[127.0.0.1]:10022' (ED25519) to the list of known hosts.
Last login: Wed Mar 17 13:58:14 2021
user@users-iMac-Pro ~ % usb_desc_get_descriptor: 2 unknown type 33 (len 10)
usb_desc_get_descriptor: 1 unknown type 33 (len 10)

However, the following problems still exist:

The screen is too large by default and I can't see the bottom toolbar.
There's no sound at all.
The starting time is longer than in VMware. In addition, this method also has the characteristics of slow and easy to lose response, which disqualified the practicality of this method.

mviereck commented 3 years ago

The screen is too large by default and I can't see the bottom toolbar.
N sound at all. The starting time is longer than in VMware. In addition, this method also has the characteristics of slow and easy to lose response, which disqualified the practicality of this method.

These issues are specific to docker-osx, not caused by x11docker. Though, this can be done:

Sound: Maybe you have to specify a sound card. Compare x11docker --help:

     --alsa [=ALSA_CARD]  Sound with ALSA. You can define a desired sound card
                       with ALSA_CARD. List of available sound cards: aplay -l

Screen size: It might help to run with --nxagent --wm --size=1200x800. docker-osx might generate a smaller VNC window. You can enlarge the nxagent desktop window afterwards.

So far, the underlying issues of this ticket are solved.

@sickcodes Some improvements could be done within docker-osx: Especially of interest is to allow an entirely unprivileged container user. Can be tested with docker run --cap-drop=all --security-opt=no-new-privileges [...]. Steps needed:

Avoid sudo
--group-add kvm --group-add audio instead of chown /dev/kvm /dev/snd.
Set up an unprivileged sshd.

sickcodes commented 3 years ago

@hongyi-zhao You can change the screen resolution using this: https://github.com/sickcodes/Docker-OSX#change-resolution-docker-osx---change-resolution-opencore-osx-kvm

Thanks @mviereck, I will test those additional notes you've just mentioned!

Although the entire project is hacky, reducing the security issues is a priority.

If someone was to break out of the guest, they would also have to break out of the container, but I set the user up with NOPASSWD anyway.

Will give it a run this weekend using x11Docker anyway

hongyi-zhao commented 3 years ago

--alsa [=ALSA_CARD] Sound with ALSA. You can define a desired sound card with ALSA_CARD. List of available sound cards: aplay -l

For my case, a USB sound card as shown below is attached to the host:

$  aplay -l
**** List of PLAYBACK Hardware Devices ****
card 0: PCH [HDA Intel PCH], device 0: ALC888-VD Analog [ALC888-VD Analog]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 0: PCH [HDA Intel PCH], device 3: ALC888-VD Digital [ALC888-VD Digital]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 1: Device [USB Audio Device], device 0: USB Audio [USB Audio]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 2: NVidia [HDA NVidia], device 3: HDMI 0 [HDMI 0]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 2: NVidia [HDA NVidia], device 7: HDMI 1 [HDMI 1]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 2: NVidia [HDA NVidia], device 8: HDMI 2 [HDMI 2]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
card 2: NVidia [HDA NVidia], device 9: HDMI 3 [HDMI 3]
  Subdevices: 1/1
  Subdevice #0: subdevice #0

It might help to run with --nxagent --wm --size=1200x800.

Hence, I tried with the following but failed to start:

werner@X10DAi:~$ x11docker --share /dev/kvm --group-add kvm --user=RETAIN --cap-default --alsa 1 --nxagent --wm --size=1200x800 -- -p 50922:10022 -- sickcodes/docker-osx:auto
x11docker WARNING: User werner is member of group docker.
  That allows unprivileged processes on host to gain root privileges.

x11docker WARNING: You are running GNOME desktop in outdated version 
  GNOME Shell 3.36.4
  This might cause issues with host applications if using additional X servers.
  It is recommended to use another desktop environment or GNOME >= 3.38.
  Only otherwise discouraged option --hostdisplay might work as expected.

x11docker note: --nxagent: nxagent not found.
  You can look for the package name of this command at: 
 https://github.com/mviereck/x11docker/wiki/dependencies#table-of-all-packages

x11docker note: Using X server option --xephyr

x11docker WARNING: Option --cap-default disables security hardening
  for containers done by x11docker. Default docker capabilities are allowed.
  This is considered to be less secure.

x11docker note: Option --cap-default: Enabling option --newprivileges.
  You can avoid this with --newprivileges=no

x11docker WARNING: ALSA sound with option --alsa degrades container isolation.
  Shares device files in /dev/snd, container gains access to sound hardware.
  Container applications can catch audio output and microphone input.

x11docker note: It seems that pulseaudio is running on your host.
  Pulseaudio can interfere with ALSA sound (option --alsa).
  Host sound may not work while container is playing sound and vice versa.
  Alternative: with pulseaudio on host and in image, use option --pulseaudio.

x11docker WARNING: Option --newprivileges=yes: x11docker does not set 
  docker run option --security-opt=no-new-privileges. 
  That degrades container security.
  However, this is still within a default docker setup.

x11docker WARNING: Sharing device file: /dev/kvm

x11docker WARNING: Sharing device file: /dev/snd

x11docker note: Option --wm: Did not find window manager image 
      x11docker/openbox 
  to provide a containerized window manager. Please run: 
      docker pull x11docker/openbox 
  If you want to use a host window manager instead and avoid this warning, 
  use option                         --wm=host  or  --wm=COMMAND 
  or provide a local image with e.g. --wm=x11docker/fvwm 
  To run without a window manager:   --wm=none  or  --desktop 
  Fallback: Will try to run a host window manager: mutter

x11docker note: Option --wm: Starting host window manager: mutter

/x11docker/cmdrc: line 51: -p: not found
werner@X10DAi:~$

hongyi-zhao commented 3 years ago

@sickcodes You can change the screen resolution using this: https://github.com/sickcodes/Docker-OSX#change-resolution-docker-osx---change-resolution-opencore-osx-kvm

I've tried with the following settings, but none of them can do the trick:

$ x11docker --share /dev/kvm --group-add kvm --user=RETAIN --cap-default -- -e WIDTH=1920 -e HEIGHT=1080  -p 50922:10022 -- sickcodes/docker-osx:auto
$ x11docker --share /dev/kvm --group-add kvm --user=RETAIN --cap-default -- -e WIDTH=1280 -e HEIGHT=768  -p 50922:10022 -- sickcodes/docker-osx:auto
$ x11docker --share /dev/kvm --group-add kvm --user=RETAIN --cap-default -- -e WIDTH=2560 -e HEIGHT=1600  -p 50922:10022 -- sickcodes/docker-osx:auto

hongyi-zhao commented 3 years ago

@mviereck @sickcodes From my point of view, I don’t think that using docker to run/encapsulate a qemu/kvm based macOS is more advantageous than in virtual machines, say, pve/VMware. To be simple, the former involves two-layer virtualization technologies: docker and qemu/kvm, while the latter only involving one type of them. So, it's obvious that the latter should be more efficient and robust for a real production scenario. Even only for a testing case, the former method is still inefficient and cumbersome.

sickcodes commented 3 years ago

For CI/CD Docker has known benefits but I use OSX both inside Docker and outside Docker.

I have added more runtime features in Docker OSX now that I use that more than straight system qemu or virt-manager.

All 3 are good tho!

hongyi-zhao commented 3 years ago

All 3 are good tho!

Which 3?

mviereck commented 3 years ago

From my point of view, I don’t think that using docker to run/encapsulate a qemu/kvm based macOS is more advantageous than in virtual machines, say, pve/VMware.

I see two points that mainly influence the speed:

Display: VNC is slow by design because it sends screen captures over TCP. Some VNC servers/clients are more effective than others in (de)compressing data, but this is always a bottleneck. If macOS would use X, much faster setups would be possible.
CPU: Virtualization can use a virtual CPU or somehow/partially allow access to the physical CPU. A virtual CPU is always slower. I don't have enough insight what is possible here.

To be simple, the former involves two-layer virtualization technologies: docker and qemu/kvm, while the latter only involving one type of them.

The docker layer barely affects performance. After all, docker is just a frontend for kernel namespaces and applications in a container are about as fast as on host.

Hence, I tried with the following but failed to start: werner@X10DAi:~$ x11docker --share /dev/kvm --group-add kvm --user=RETAIN --cap-default --alsa 1 --nxagent --wm --size=1200x800 -- -p 50922:10022 -- sickcodes/docker-osx:auto

The correct syntax is --alsa=1 instead of --alsa 1.

hongyi-zhao commented 3 years ago

The correct syntax is --alsa=1 instead of --alsa 1 .

Failed as below:

werner@X10DAi:~$ x11docker --share /dev/kvm --group-add kvm --user=RETAIN --cap-default --alsa=1 --nxagent --wm --size=1200x800 -- -p 50922:10022 -- sickcodes/docker-osx:auto
x11docker WARNING: User werner is member of group docker.
  That allows unprivileged processes on host to gain root privileges.

x11docker WARNING: You are running GNOME desktop in outdated version 
  GNOME Shell 3.36.4
  This might cause issues with host applications if using additional X servers.
  It is recommended to use another desktop environment or GNOME >= 3.38.
  Only otherwise discouraged option --hostdisplay might work as expected.

x11docker note: --nxagent: nxagent not found.
  You can look for the package name of this command at: 
 https://github.com/mviereck/x11docker/wiki/dependencies#table-of-all-packages

x11docker note: Using X server option --xephyr

x11docker WARNING: Option --cap-default disables security hardening
  for containers done by x11docker. Default docker capabilities are allowed.
  This is considered to be less secure.

x11docker note: Option --cap-default: Enabling option --newprivileges.
  You can avoid this with --newprivileges=no

x11docker WARNING: Found custom DOCKER_RUN_OPTIONS.
  x11docker will add them to 'docker run' command without
  a serious check for validity or security. Found options:
   '-p' '50922:10022'

x11docker WARNING: ALSA sound with option --alsa degrades container isolation.
  Shares device files in /dev/snd, container gains access to sound hardware.
  Container applications can catch audio output and microphone input.

x11docker note: It seems that pulseaudio is running on your host.
  Pulseaudio can interfere with ALSA sound (option --alsa).
  Host sound may not work while container is playing sound and vice versa.
  Alternative: with pulseaudio on host and in image, use option --pulseaudio.

x11docker WARNING: Option --newprivileges=yes: x11docker does not set 
  docker run option --security-opt=no-new-privileges. 
  That degrades container security.
  However, this is still within a default docker setup.

x11docker WARNING: Sharing device file: /dev/kvm

x11docker WARNING: Sharing device file: /dev/snd

x11docker note: Found 'WORKDIR /home/arch/OSX-KVM' in image.  
  You can change it with option --workdir=DIR.

x11docker note: Option --wm: Did not find window manager image 
      x11docker/openbox 
  to provide a containerized window manager. Please run: 
      docker pull x11docker/openbox 
  If you want to use a host window manager instead and avoid this warning, 
  use option                         --wm=host  or  --wm=COMMAND 
  or provide a local image with e.g. --wm=x11docker/fvwm 
  To run without a window manager:   --wm=none  or  --desktop 
  Fallback: Will try to run a host window manager: mutter

x11docker note: Option --wm: Starting host window manager: mutter

docker: Error response from daemon: driver failed programming external connectivity on endpoint x11docker_X113_sickcodes-docker-osx-auto_63257611989 (01cf57c5081a2b001ac8bf702939425910cde4040474f46a081c203405e2ed39): Error starting userland proxy: listen tcp4 0.0.0.0:50922: bind: address already in use.

x11docker ERROR: dockerrc(): Did not receive PID of PID1 in container. 
  Maybe the container immediately stopped for unknown reasons. 
  Just in case, check if host and image architecture are compatible: 
  Host architecture: amd64 (x86_64), image architecture: amd64. 
  Output of "docker ps | grep x11docker": 

  Content of container log: 
WARNING: No swap limit support 
WARNING: No blkio weight support 
WARNING: No blkio weight_device support 
[] 
Error: No such object: x11docker/openbox 
Error response from daemon: Container c99e820dd36db24a0c15a7a4b177b4d9bf7ed5736026bb685f82eaa877d81ef7 is not running

  Type 'x11docker --help' for usage information
  Debug options: '--verbose' (full log) or '--debug' (log excerpt).
  Logfile will be: /home/werner/.cache/x11docker/x11docker.log
  Please report issues at https://github.com/mviereck/x11docker

werner@X10DAi:~$

mviereck commented 3 years ago

docker: Error response from daemon: driver failed programming external connectivity on endpoint x11docker_X113_sickcodes-docker-osx-auto_63257611989 (01cf57c5081a2b001ac8bf702939425910cde4040474f46a081c203405e2ed39): Error starting userland proxy: listen tcp4 0.0.0.0:50922: bind: address already in use.

It seems you have two containers running using port 50922. For the screen size, rather try the proposal of @sickcodes . Setting WIDTH and HEIGHT is not enough, look at the documentation.

hongyi-zhao commented 3 years ago

This problem is caused by the confliction with a previously started but still running docker container. Now it will be trapped into the dead loop with the following info:

$ x11docker --share /dev/kvm --group-add kvm --user=RETAIN --cap-default --alsa=1 -- -p 50922:10022 -- sickcodes/docker-osx:auto
[...]
Disk is being copied between layers. Repeating until able to copy SSH key into OSX...
/usr/sbin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/arch/.ssh/id_docker_osx.pub"
ssh: connect to host 127.0.0.1 port 10022: Connection refused

For my case, this message persists for > 30 mins and still flashes forever on the screen.