sickcodes / Docker-OSX

Run macOS VM in a Docker! Run near native OSX-KVM in Docker! X11 Forwarding! CI/CD for OS X Security Research! Docker mac Containers.
https://hub.docker.com/r/sickcodes/docker-osx
GNU General Public License v3.0
36.11k stars 1.78k forks source link

Can't contact recovery server from Docker-OSX during installation in a proxied environment #413

Open OverthrowTheGoauld opened 2 years ago

OverthrowTheGoauld commented 2 years ago

Hello:

Before I post the problem, here are the outputs you have required.

$ uname -a ; echo "${DISPLAY}" ; echo 1 | sudo tee /sys/module/kvm/parameters/ignore_msrs ; grep NAME /etc/os-release ; df -h . ; qemu-system-x86_64 --version ; libvirtd --version ; free -mh ; nproc ; egrep -c '(svm|vmx)' /proc/cpuinfo ; ls -lha /dev/kvm ; ls -lha /tmp/.X11-unix/ ; ps aux | grep dockerd ; docker ps | grep osx ; grep "docker\|kvm\|virt" /etc/group
Linux vulcan 5.15.0-2-amd64 #1 SMP Debian 5.15.5-2 (2021-12-18) x86_64 GNU/Linux
:0
1
PRETTY_NAME="Debian GNU/Linux bookworm/sid"
NAME="Debian GNU/Linux"
Filesystem                                    Size  Used Avail Use% Mounted on
/dev/mapper/XYZ-lv_home  719G  454G  229G  67% /home
QEMU emulator version 6.1.0 (Debian 1:6.1+dfsg-8+b2)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
bash: libvirtd: command not found
               total        used        free      shared  buff/cache   available
Mem:            31Gi       2.9Gi        24Gi       187Mi       4.3Gi        27Gi
Swap:             0B          0B          0B
8
16
crw-rw----+ 1 root kvm 10, 232 Jan  1 11:13 /dev/kvm
total 8.0K
drwxrwxrwt  2 root root 4.0K Jan  1 11:13 .
drwxrwxrwt 18 root root 4.0K Jan  1 11:18 ..
srwxrwxrwx  1 root root    0 Jan  1 11:13 X0
root        1627  0.0  0.2 1680496 83248 ?       Ssl  11:13   0:00 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
jon      13307  0.0  0.0   6368  2296 pts/1    S+   11:18   0:00 grep --color=auto dockerd
kvm:x:106:jon
docker:x:998:jon
libvirt:x:134:jon
libvirt-qemu:x:64055:libvirt-qemu

Now, the output of installation (I deleted all images and started with a clean slate):

$ docker run --env http_proxy="http://10.10.78.61:3128" --env https_proxy="http://10.10.78.61:3128" -it --device /dev/kvm -p 50922:10022 -v /tmp/.X11-unix:/tmp/.X11-unix -e "DISPLAY=${DISPLAY:-:0.0}" sickcodes/docker-osx:big-sur
Unable to find image 'sickcodes/docker-osx:big-sur' locally
big-sur: Pulling from sickcodes/docker-osx
16bd9c7b02cb: Pull complete
e568f2eaead7: Pull complete
9bf33efced35: Pull complete
0398701f6a91: Pull complete
7e7ba1a887a2: Pull complete
6f01c30060d4: Pull complete
bc67fa3d9f4f: Pull complete
9c05d603f7ec: Pull complete
26e4dc5aa5ef: Pull complete
fecde0ad628c: Pull complete
7613912f3a02: Pull complete
c750272df401: Pull complete
e9351b2b9fae: Pull complete
9d3a5216e542: Pull complete
b97d0404193f: Pull complete
31a48571ca33: Pull complete
425a085b2c57: Pull complete
ffd7e21f11ee: Pull complete
5c7ccda5a216: Pull complete
519f358aabd4: Pull complete
Digest: sha256:225cc52bc384deb0069f29afb95cd3201f9586ae9749304980ee187e66d772f3
Status: Downloaded newer image for sickcodes/docker-osx:big-sur
ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519
nohup: appending output to 'nohup.out'
++ id -u
++ id -g
+ sudo chown 1000:1000 /dev/kvm
++ id -u
++ id -g
+ sudo chown -R 1000:1000 /dev/snd
+ [[ 3 = max ]]
+ [[ 3 = half ]]
++ id -u
++ id -g
+ sudo chown -R 1000:1000 /dev/snd
+ exec qemu-system-x86_64 -m 3000 -cpu Penryn,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check, -machine q35,accel=kvm:tcg -smp 4,cores=4 -usb -device usb-kbd -device usb-tablet -device 'isa-applesmc,osk=ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc' -drive if=pflash,format=raw,readonly=on,file=/home/arch/OSX-KVM/OVMF_CODE.fd -drive if=pflash,format=raw,file=/home/arch/OSX-KVM/OVMF_VARS-1024x768.fd -smbios type=2 -audiodev alsa,id=hda -device ich9-intel-hda -device hda-duplex,audiodev=hda -device ich9-ahci,id=sata -drive id=OpenCoreBoot,if=none,snapshot=on,format=qcow2,file=/home/arch/OSX-KVM/OpenCore/OpenCore.qcow2 -device ide-hd,bus=sata.2,drive=OpenCoreBoot -device ide-hd,bus=sata.3,drive=InstallMedia -drive id=InstallMedia,if=none,file=/home/arch/OSX-KVM/BaseSystem.img,format=qcow2 -drive id=MacHDD,if=none,file=/home/arch/OSX-KVM/mac_hdd_ng.img,format=qcow2 -device ide-hd,bus=sata.4,drive=MacHDD -netdev user,id=net0,hostfwd=tcp::10022-:22,hostfwd=tcp::5900-:5900, -device vmxnet3,netdev=net0,id=net0,mac=52:54:00:09:49:17 -monitor stdio -boot menu=on -vga vmware
qemu-system-x86_64: warning: dbind: Couldn't connect to accessibility bus: Failed to connect to socket /tmp/dbus-WmFBAKq6wu: No such file or directory
QEMU 6.1.0 monitor - type 'help' for more information
(qemu) ALSA lib confmisc.c:855:(parse_card) cannot find card '0'
ALSA lib conf.c:5111:(_snd_config_evaluate) function snd_func_card_inum returned error: No such file or directory
ALSA lib confmisc.c:422:(snd_func_concat) error evaluating strings
ALSA lib conf.c:5111:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1334:(snd_func_refer) error evaluating name
ALSA lib conf.c:5111:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5599:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM default
alsa: Could not initialize DAC
alsa: Failed to open `default':
alsa: Reason: No such file or directory
ALSA lib confmisc.c:855:(parse_card) cannot find card '0'
ALSA lib conf.c:5111:(_snd_config_evaluate) function snd_func_card_inum returned error: No such file or directory
ALSA lib confmisc.c:422:(snd_func_concat) error evaluating strings
ALSA lib conf.c:5111:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1334:(snd_func_refer) error evaluating name
ALSA lib conf.c:5111:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5599:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM default
alsa: Could not initialize DAC
alsa: Failed to open `default':
alsa: Reason: No such file or directory
audio: Failed to create voice `dac'
ALSA lib confmisc.c:855:(parse_card) cannot find card '0'
ALSA lib conf.c:5111:(_snd_config_evaluate) function snd_func_card_inum returned error: No such file or directory
ALSA lib confmisc.c:422:(snd_func_concat) error evaluating strings
ALSA lib conf.c:5111:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1334:(snd_func_refer) error evaluating name
ALSA lib conf.c:5111:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5599:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM default
alsa: Could not initialize ADC
alsa: Failed to open `default':
alsa: Reason: No such file or directory
ALSA lib confmisc.c:855:(parse_card) cannot find card '0'
ALSA lib conf.c:5111:(_snd_config_evaluate) function snd_func_card_inum returned error: No such file or directory
ALSA lib confmisc.c:422:(snd_func_concat) error evaluating strings
ALSA lib conf.c:5111:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1334:(snd_func_refer) error evaluating name
ALSA lib conf.c:5111:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5599:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2660:(snd_pcm_open_noupdate) Unknown PCM default
alsa: Could not initialize ADC
alsa: Failed to open `default':
alsa: Reason: No such file or directory
audio: Failed to create voice `adc'

When this boots up, I get an initial 5 second delay (some wake-failure), and then the boot resumes.

As per your instructions, I ran Disk Utility, and converted the 278+GB QEMU harddisk space (where does this number come from? - I do not have a 278 GB free partition, or even 278 GB free space in /var/lib) to APFS/GUID with the name Mac OSX HD. I leave the other 400MB QEMU partition and the mac OS base system alone.

Next, I click on "Reinstall mac OS Big Sur". The machine waits a while before presenting me with the mac OS Big Sur (To set up the installation of mac OS Big Sur, click Continue) screen.

After I click Continue, I get "The recovery server could not be contacted." after a short while.

Then I open up Terminal from the top menu.

Examining the network leads to: (can't copy paste)

ifconfig -a

lo0: ...
gif0: ...
stf0:...
UHC93:...
UHC61:...
UHC29:...
EHC253:...
en0: flags 8863...
options=424...
ether 52:54:00:09:49:17
inet6 ...
inet6 ...
inet6 ...
inet 10.0.2.15 netmask 0xffffff00 broadcast 10.0.2.255
nd6 options=201...
media: autoselect ...
status: active

(I assume that 10.0.2.15 is the IP address provided by the fake DHCP server in the arch-based docker container inside which you are running OSX.)

The network on the host (while the container is running) is:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 18:66:da:2a:51:97 brd ff:ff:ff:ff:ff:ff
    inet 10.224.33.2/24 brd 10.224.33.255 scope global noprefixroute enp0s31f6
       valid_lft forever preferred_lft forever
    inet6 fe80::9f1e:1d4f:d16d:a6ba/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:f2:7c:9e:2c brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:f2ff:fe7c:9e2c/64 scope link
       valid_lft forever preferred_lft forever
5: veth5ec357b@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
    link/ether 0a:b5:ca:1d:ed:12 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::8b5:caff:fe1d:ed12/64 scope link
       valid_lft forever preferred_lft forever

I next tried a fix suggested in (https://gist.github.com/levlaz/16b63384bd5e1bee3593be0d91aedbd7):

-bash-3.2# sntp -sS time.apple.com
sntp: Exchange failed: Timeout
... repeats 4 times ...
sntp: Clock select failed
-bash-3.2# sntp -sS <fqdn of my local time server, non-proxied>
+1.079295 +/- 0.065041 <fqdn of local NTP time server> 10.7.172.184 (probably the IP address of the local time server)

After quitting Terminal, I retry the above reinstall ... steps. Same result.

The above strongly suggests that my problem is not the time offset, but an actual inability to contact non-proxied addresses. This sounds like a proxy problem (see below).

To check whether it was the authentication in the proxy playing a role, I supplied URLs: http://username:password@10.10.78.61:3128 in the --env options above (is the .json file (see below) overriding what I supply on command line?).

(The password contains an underscore, just in case that matters.)

The same happens again.

When I add --dns 10.10.1.2 (taken from /etc/resolv.conf on the host), the same happens again. This is not a DNS issue as supplying the IP address of a public time server outside of the proxied environment (time-a-g.nist.gov: 129.6.15.28) results in the same problem as the Exchange problem above.

So, this is not a DNS issue.

Finally, I added the username and password to the config.json (see below) file and tried this process again. No joy.

I have tried to Reset NVRAM on the boot screen as well, since that often impacts things with OSX. No effect.

So, whatever network service (since that is what I suspect) is directly related to the fact that mac OS recover server needs to communicate over a non-proxiable port (such as whois, etc.). Can't I download the BaseImage.dmg file from somewhere? I have downloaded the InstallAssistant.pkg file for Big Sur (and tried to follow the instructions at https://davejansen.com/install-macos-11-big-sur-in-a-vm-qemu-kvm/), but that cannot work since the xar utility needed to extract this is not available at the link therein.

However, puzzlingly, Docker is able to connect (? - the 301 error does not look promising) to net for the nginx container (so the problem above is probably not directly related to the fact I have a proxied setup - see details below):

$ docker run --rm nginx sh -c "curl -I google.com"  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   219    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
HTTP/1.0 301 Moved Permanently
Location: http://www.google.com/
Content-Type: text/html; charset=UTF-8
BFCache-Opt-In: unload
Date: Mon, 13 Dec 2021 06:58:16 GMT
Expires: Wed, 12 Jan 2022 06:58:16 GMT
Cache-Control: public, max-age=2592000
Server: gws
Content-Length: 219
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
Age: 1640245
Warning: 113 localhost (squid/3.1.19) This cache hit is still fresh and more than 1 day old
X-Cache: HIT from localhost
X-Cache-Lookup: HIT from localhost:3128
Via: 1.0 localhost (squid/3.1.19)
Connection: keep-alive
$ docker run --rm nginx sh -c "curl -I apple.com"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   304    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
HTTP/1.0 301 Moved Permanently
Date: Sat, 01 Jan 2022 06:36:30 GMT
Server: ATS/9.0.3
Cache-Control: no-store
Location: https://www.apple.com/
Content-Type: text/html
Content-Language: en
CDNUUID: 05d7ba78-5a0c-43d8-b257-01dcdc312c51-7120726169
X-Cache: none
Content-Length: 304
X-Cache: MISS from localhost
X-Cache-Lookup: MISS from localhost:3128
Via: http/1.1 sgsin8-edge-bx-001.ts.apple.com (ApacheTrafficServer/9.0.3), 1.0 localhost (squid/3.1.19)
Connection: keep-alive

I have setup proxy the best I can:

$ docker info | grep -i proxy
 HTTP Proxy: http://10.10.78.61:3128
 HTTPS Proxy: http://10.10.78.61:3128
 No Proxy: localhost,127.0.0.1

Some of the environment variables:

declare -x ftp_proxy="ftp://10.10.78.61:3128/"
declare -x http_proxy="http://10.10.78.61:3128/"
declare -x https_proxy="https://10.10.78.61:3128/"

Contents of config.json (under the user jon):

$ cat ~/.docker/config.json
{
 "proxies":
 {
   "default":
   {
     "httpProxy": "http://10.10.78.61:3128",
     "httpsProxy": "http://10.10.78.61:3128",
     "noProxy": "localhost,127.0.0.1,127.0.0.0/8"
   }
 }
}

So, I am forced to conclude that the network issue (if it is a network issue) is unique to the Docker-OSX container. Despite the ability to download images over proxy, I have a nagging suspicion that this is due to the nature of the proxy I have to work with.

One possible lacuna in my setup:

$ sudo apt install qemu qemu-kvm libvirt-clients libvirt-daemon-system bridge-utils virt-manager libguestfs-tools
[sudo] password for jon:
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'qemu-system-x86' instead of 'qemu-kvm'
bridge-utils is already the newest version (1.7-1).
libvirt-clients is already the newest version (7.10.0-2).
libvirt-clients set to manually installed.
libvirt-daemon-system is already the newest version (7.10.0-2).
libvirt-daemon-system set to manually installed.
qemu-system-x86 is already the newest version (1:6.1+dfsg-8+b2).
virt-manager is already the newest version (1:3.2.0-3).
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 libguestfs-tools : Depends: libguestfs0 (= 1:1.46.2-3) but it is not going to be installed
                    Depends: guestfs-tools but it is not installable
                    Depends: guestmount but it is not going to be installed
                    Depends: guestfish but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

I have already invested a non-trivial amount of time in this, and believe that the likelihood of me finding a solution to the issue above on my own is low enough for me at this time to request your assistance.

If you need further information, please ask.

sickcodes commented 2 years ago

Xar is available: https://github.com/sickcodes/aur/tree/master/xar

Fork: https://github.com/tpoechtrager/xar

OverthrowTheGoauld commented 2 years ago

Thanks.

This is not available on Debian. Is there any source code available?

On Thu, Jan 13, 2022 at 2:36 PM sickcodes @.***> wrote:

Xar is available: https://github.com/sickcodes/aur/tree/master/xar

Fork: https://github.com/tpoechtrager/xar

— Reply to this email directly, view it on GitHub https://github.com/sickcodes/Docker-OSX/issues/413#issuecomment-1011934366, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXDSH2TTLKQXDWMEDFVXVY3UV2IXPANCNFSM5LCHZ5LA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you authored the thread.Message ID: @.***>

sickcodes commented 2 years ago
git clone https://aur.archlinux.org/xar.git
cd xar
makedeb
dpkg -i xar_1.6.1-5_amd64.deb

Here you go: https://github.com/sickcodes/aur/raw/master/xar/xar_1.6.1-5_amd64.deb

OverthrowTheGoauld commented 2 years ago

Sorry, have been traveling for a few weeks. Lost track of this.

Can you suggest why networking does not appear to work at all?

rodrigofbm commented 1 year ago

I'm facing the "The recovery server could not be contacted." message too on Fedora 36