Errors when using TART_EXECUTOR_HOST_DIR

[waddles]

This may not be the right place to report this issue as I think it may be in the virtualization framework itself but turning the flag off serves as a workaround.

I have a large repo (about 4GB, ~30000 files) and when I have the TART_EXECUTOR_HOST_DIR=true flag set, I get errors which look to be due to a saturation of the VirtIO disk interface.

The errors happen when

• cloning the repository

  Getting source from Git repository
  Fetching changes with git depth set to 50...
  Initialized empty Git repository in /Volumes/My Shared Files/hostdir/path/to/repo/.git/
  Created fresh repository.
  Checking out feb79015 as detached HEAD (ref is refs/merge-requests/9667/head)...
  error: unable to create file app/path/to/file: No such file or directory
  fatal: cannot create directory at 'app/path/to/directory': No such file or directory
  2023/05/31 14:20:03 Process exited with status 128
  Uploading artifacts for failed job

• unpacking cache.zip (from cache dir mounted from host into temp dir mounted from host)

  Restoring cache
  Checking cache for 0-2769b0d987c57097719305e6ec99532b0b41ab84-9-non_protected...
  Runtime platform                                    arch=arm64 os=darwin pid=919 revision=456e3482 version=15.10.0
  No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
  FATAL: lchmod /Volumes/My Shared Files/hostdir/path/to/repo/file.txt: no such file or directory 
  Failed to extract cache

• later in the build when installing ruby gems

  Errno::ENOENT: No such file or directory @ apply2files - /Volumes/My Shared Files/hostdir/path/to/repo/vendor/bundle/ruby/3.2.0/gems/ffi-1.15.5/ext/ffi_c/Type.c
  /Users/admin/.asdf/installs/ruby/3.2.1/lib/ruby/3.2.0/fileutils.rb:2222:in `chmod'
  ...
  /Volumes/My Shared Files/hostdir/path/to/repo/vendor/bundle/ruby/3.2.0/gems/bundler-2.3.10/lib/bundler/rubygems_gem_installer.rb:26:in `install'

[edigaryev]

Which macOS version and Tart VM image are you using? Tart's directory mounting feature requires both host and guest to run at least macOS 13 (Ventura).

[rigellin73]

Hello! I think, I'm having similar issue. I wasn't sure it is executor problem too, but it occurs only when using TART_EXECUTOR_HOST_DIR=true I'm using macOS Ventura for both host and guest My error happens at the start of Unity project build (Unity tries to create package cache):

An error occurred while resolving packages:
One or more packages could not be added to the local file system:
     com.unity.addressables: ENOENT: no such file or directory, chmod '/Volumes/My Shared Files/hostdir/project/Library/PackageCache/.tmp-1382-3txiATFvKXMw/copy/Tests/Editor/CustomTestSchema.cs.meta'
    com.unity.cinemachine: ENOENT: no such file or directory, chmod '/Volumes/My Shared Files/hostdir/project/Library/PackageCache/.tmp-1382-iR0pNEwDge07/copy/Samples~/Cinemachine Example Scenes/Scenes/CameraMagnets/CameraMagnetTargetController.cs'
     com.unity.collab-proxy: ENOENT: no such file or directory, open '/Volumes/My Shared Files/hostdir/project/Library/PackageCache/.tmp-1382-tmjjpXy0eqjP/copy/Editor/PlasticSCM/Views/DownloadPlasticExeWindow.cs'

[waddles]

As per #21, Ventura 13.3. Strangely I have not had this problem at all with cachedir mounted from host, but then cache only reads a single file at the start of the job and writes it again at the end - it's not doing thousands of iops like hostdir does.

[edigaryev]

@waddles, @rigellin73 could you provide a .gitlab-ci.yml or a link to the repository that could help reproducing this issue?

[waddles]

Unfortunately I don't have a public repo to test this with. I think the difference of using a temporary dir mounted from the host is fairly negligible though. I'm trying to create a script that will exhibit the behaviour but while testing I timed this test which is creating 10000 random CSV files, about 2MB each on a recent M2 with 2TB SSD.

Results

• 334 seconds on VM bounded disk
• 330 seconds on host mounted disk

  gitlab@mac-mini-gitlab-runner-4 ~ % daemonize $(brew --prefix)/bin/tart run vm --no-graphics --dir=hostdir:~/builds

  admin@admins-Virtual-Machine ~ % pwd
  /Users/admin
  admin@admins-Virtual-Machine ~ % time python3 /Volumes/My\ Shared\ Files/hostdir/create_files.py
  python3 /Volumes/My\ Shared\ Files/hostdir/create_files.py  334.25s user 5.06s system 99% cpu 5:39.57 total
  admin@admins-Virtual-Machine ~ % cd /Volumes/My\ Shared\ Files/hostdir
  admin@admins-Virtual-Machine hostdir % time python3 create_files.py
  python3 create_files.py  330.34s user 6.46s system 97% cpu 5:44.79 total

[fkorotkov]

@waddles we've got a similar report and it seems making sure that both guest and host run macOS Ventura 13.3+ helped. Which version of macOS you have on your host and for the guests?

[waddles]

Confirming, we saw this with Ventura 13.3 host and guests. I was not able to create a test scenario that could repeatedly trigger it though.

[fkorotkov]

Seems there is a bug in virtiofs support in macOS. I reported it to Apple via FB12594177. I thought https://github.com/cirruslabs/tart/pull/555 fixed the issue but unfortunelty it did not.

[fkorotkov]

Closing and tracking the issue in https://github.com/cirruslabs/tart/issues/567 for visibility