search

sickcodes / Docker-OSX

Run macOS VM in a Docker! Run near native OSX-KVM in Docker! X11 Forwarding! CI/CD for OS X Security Research! Docker mac Containers.
https://hub.docker.com/r/sickcodes/docker-osx
GNU General Public License v3.0
36.13k stars 1.78k forks source link

Big Sur & Monterey not booting fully on WSL2 #503

Open SakulFlee opened 2 years ago

SakulFlee commented 2 years ago

Hello everyone :) I am having trouble running macOS Monterey (and Big Sur) on WSL2 (W11). The other images (latest, naked, Catalina, High Sierra and Mojave) do work flawlessly.

Big Sur

Big Sur boots and after a few crashes/reboots the installer opens. However, after a few minutes in the installer QEMU reboots and has to restart multiple times again until the installer is shown. Can't install.

Command used (just like in the README.md):

docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    sickcodes/docker-osx:big-sur

Monterey

Monterey doesn't even get to the Installer stage. It either reboots multiple times or is stuck with console output.

Command used (just like in the README.md):

docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    -e GENERATE_UNIQUE=true \
    -e MASTER_PLIST_URL='https://raw.githubusercontent.com/sickcodes/osx-serial-generator/master/config-custom.plist' \
    sickcodes/docker-osx:monterey

Additional Information

If there is a way for me to attach macOS logs and/or other information is required please let me know!
I've tried all images on WSL-Ubuntu and WSL-OpenSUSE.
Apart from some missing packages on the OpenSUSE side both show the same behaviour.

I am not sure if this is a Windows/WSL related issue?
I am running WIndows 11; Version 22H2 (OS Build 22610.1) and WSL is the following:

> wsl --version
WSL version: 0.58.3.0
Kernel version: 5.10.102.1
WSLg version: 1.0.33
MSRDC version: 1.2.2924
Direct3D version: 1.601.0
Windows version: 10.0.22610.1

OS related issued, please help us identify the issue by posting the output of this

uname -a \
> ; echo "${DISPLAY}" \
> ; echo 1 | sudo tee /sys/module/kvm/parameters/ignore_msrs \
> ; grep NAME /etc/os-release \
 df -h > ; df -h . \
> ; qemu-system-x86_64 --version \
> ; libvirtd --version \
> ; free -mh \
 nproc \> ; nproc \
> ; egrep -c '(svm|vmx)' /proc/cpuinfo \
/dev/k> ; ls -lha /dev/kvm \
 /tmp/> ; ls -lha /tmp/.X11-unix/ \
> ; ps aux | grep dockerd \
> ; docker ps | grep osx \
grep "d> ; grep "docker\|kvm\|virt" /etc/group
Linux Citra 5.10.102.1-microsoft-standard-WSL2 #1 SMP Wed Mar 2 00:30:59 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
:0
1
NAME="openSUSE Tumbleweed"
PRETTY_NAME="openSUSE Tumbleweed"
CPE_NAME="cpe:/o:opensuse:tumbleweed:20220415"
Filesystem      Size  Used Avail Use% Mounted on
drvfs           931G  801G  131G  86% /mnt/wsl/docker-desktop-bind-mounts/openSUSE-Tumbleweed/47339e9566e9267e46c034b3e3415ae191e4952ba50ab1b15912a5385d3a376c
QEMU emulator version 6.2.0 (openSUSE Tumbleweed)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers       
libvirtd (libvirt) 8.4.0
               total        used        free      shared  buff/cache   available
Mem:            15Gi       4.1Gi       9.4Gi        22Mi       2.0Gi        11Gi
Swap:          4.0Gi          0B       4.0Gi
12
24
crw-rw-rw- 1 root root 10, 232 Jun  6 19:34 /dev/kvm
total 0
drwxrwxrwx 2 root      root       60 Jun  6 19:34 .
drwxrwxrwt 6 root      root      260 Jun  6 19:34 ..
srwxrwxrwx 1 sakul6499 sakul6499   0 Jun  6 19:34 X0
sakul64+   642  0.0  0.0   6624  2244 pts/3    S+   23:00   0:00 grep --color=auto dockerd
6e6c20a7c998   sickcodes/docker-osx:naked   "/bin/bash -c 'sudo …"   2 hours ago   Up 2 hours   0.0.0.0:50922->10022/tcp   focused_wilson
kvm:x:36:qemu,sakul6499
docker:x:1001:sakul6499
libvirt:x:108:sakul6499

Thank you very much! I am open for further testing this.

EddieV2 commented 1 year ago

I noticed the same rebooting behavior with the Monterey image. I can't attest to the other images working, but it seemed like it tried to install and then crashed due to a memory issue perhaps?

SakulFlee commented 1 year ago

Possibly, I don't know how to further debug this. Increasing the RAM amount doesn't help.

mateo-m commented 1 year ago

Same issue here, W11 & WSL2.

LiarOnce commented 1 year ago

Same issue here, I've tried allocating more memory and CPU, but it doesn't work:

docker run -it \
    --device /dev/kvm \
    -p 60000:10022 \
    -v /tmp/.X11-unix:/tmp/.X11-unix \
    -e "DISPLAY=${DISPLAY:-:0.0}" \
    -e GENERATE_UNIQUE=true \
    -e MASTER_PLIST_URL='https://raw.githubusercontent.com/sickcodes/osx-serial-generator/master/config-custom.plist' \
    -e RAM=16 \
    -e EXTRA='-smp 8,sockets=4,cores=2' \
    sickcodes/docker-osx:monterey
MikeCoder96 commented 1 year ago

KVM on WSL is different from KVM on Linux native, in fact, KVM, use a Hyper-V Enlightened VMCS technology, here you will find information about Said that, I assume that some CPU call are not able to be "translate" from Hyper-V engine and everything fall to break. Suggest: Swap to Linux or use VMWare/VirtualBox

SakulFlee commented 1 year ago

If it's some virtualization issue inside WSL, wouldn't that mean QEMU itself shouldn't work?

I can boot up and fully install both macOS versions using QEMU inside WSL. It just hangs eventually (think it's some sleep power issue). But with Docker-OSX it's not even fully booting up.

MikeCoder96 commented 1 year ago

@Sakul6499 Not fully clear how Docker on Windows work but seem to create a layer of communication and a new instance of WSL and this slow down the performace. Can you write specs of your pc and how the virtualization work with QEMU (not near native I suppose)

SakulFlee commented 1 year ago

@MikeCoder96 CPU: Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz RAM: 32GB DDR4 NVMe: 1TB GPU: NVIDIA GeForce RTX 3060 (6GB)

Windows Version: 22H2 (OS Build 22621.382) > wsl --version

WSL version: 0.65.3.0
Kernel version: 5.15.57.1
WSLg version: 1.0.41
MSRDC version: 1.2.3213
Direct3D version: 1.601.0
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22621.382

Docker version: 20.10.17, build 100c701

As of how the QEMU part works: Simply follow the instructions of OSX-KVM. Didn't change much, but added a USB Keyboard, Mouse and Touchpad to get all functionality working. If I am not mistaken, this repository is based/is using OSX-KVM, right? Meaning, it may be an issue with having Docker as an additional layer as it's working fine in WSL.

MikeCoder96 commented 1 year ago

@MikeCoder96 CPU: Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz RAM: 32GB DDR4 NVMe: 1TB GPU: NVIDIA GeForce RTX 3060 (6GB)

Windows Version: 22H2 (OS Build 22621.382) > wsl --version

WSL version: 0.65.3.0
Kernel version: 5.15.57.1
WSLg version: 1.0.41
MSRDC version: 1.2.3213
Direct3D version: 1.601.0
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22621.382

Docker version: 20.10.17, build 100c701

As of how the QEMU part works: Simply follow the instructions of OSX-KVM. Didn't change much, but added a USB Keyboard, Mouse and Touchpad to get all functionality working. If I am not mistaken, this repository is based/is using OSX-KVM, right? Meaning, it may be an issue with having Docker as an additional layer as it's working fine in WSL.

Have you changed something in config.plist or in wslconfig? The installer reboot everytime and I can't figure out why...

SakulFlee commented 1 year ago

@MikeCoder96 all I did was this: WSL2 Config:

[wsl2]
memory=22G                     # More than 50% of system memory
processors=12                    # Use all available CPU Nodes
nestedVirtualization=true    # Should be default? But just in case.

Docker Engine configuration: (Not that it matters for running QEMU under WSL2)

{
  "builder": {
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "experimental": true,
  "features": {
    "buildkit": true
  }
}

Although sometimes macOS fails to boot in QEMU. Restarting/Resetting help though. Furthermore, making your own OpenCore boot image seems to help with those lockups during boot completely. ( I followed this guide )

MikeCoder96 commented 1 year ago

@Sakul6499 we have similar hardware "i have only 16gb ram and rtx 2070". Can u give to me the plist file? I've already tried to make one by myself but same problem and probably i miss something

SakulFlee commented 1 year ago

@MikeCoder96 Pretty sure it's the original config from OSX-KVM, but this one is working with my setup: https://gist.github.com/Sakul6499/a285bad5fbf4cb070521838656937be8. Can't run a diff right now, so unsure if it's the same or not.