nixos/test-driver: "bare" driver crashes when $out env var is not a writable directory

Synthetica9 commented 2 years ago

Describe the bug

A clear and concise description of what the bug is.

Steps To Reproduce

Steps to reproduce the behavior:

nix-shell -p hello
`nix-build . -A "nixosTests.sway.driver"
./result/bin/nixos-test-driver

Expected behavior

Driver runs normally, placing outputs in current directory.

Driver output:

machine: waiting for file ‘/tmp/test-x11-exit-ok‘
(finished: waiting for file ‘/tmp/test-x11-exit-ok‘, in 1.12 seconds)
machine: must succeed: cat /tmp/test-x11.out
(finished: must succeed: cat /tmp/test-x11.out, in 0.03 seconds)
GL_VERSION: 4.5 (Compatibility Profile) Mesa 21.3.2
GL_RENDERER: llvmpipe (LLVM 13.0.0, 128 bits)
GL_VENDOR: Mesa/X.org

machine: making screenshot /nix/store/99k3djmxmgynmsdcpd1qfy953x07k5pp-shell/alacritty_glinfo.png
machine: sending monitor command: screendump /nix/store/99k3djmxmgynmsdcpd1qfy953x07k5pp-shell/alacritty_glinfo.png.ppm
machine: waiting for monitor prompt
(finished: waiting for monitor prompt, in 0.00 seconds)
(finished: sending monitor command: screendump /nix/store/99k3djmxmgynmsdcpd1qfy953x07k5pp-shell/alacritty_glinfo.png.ppm, in 0.00 seconds)
/nix/store/iqprjr5k5385bhf1dzj07zwd5p43py1n-bash-5.1-p12/bin/sh: line 1: /nix/store/99k3djmxmgynmsdcpd1qfy953x07k5pp-shell/alacritty_glinfo.png: No such file or directory
cleanup
kill machine (pid 592567)
machine # qemu-system-x86_64: terminating on signal 15 from pid 592565 (/nix/store/rppr9s436950i1dlzknbmz40m2xqqnxc-python3-3.9.9/bin/python3.9)
(finished: cleanup, in 0.02 seconds)
Traceback (most recent call last):
  File "/nix/store/zmyxgfrr2lhfcgczljwrh8h50jhs06lx-nixos-test-driver-1.0/bin/.nixos-test-driver-wrapped", line 9, in <module>
    sys.exit(main())
  File "/nix/store/zmyxgfrr2lhfcgczljwrh8h50jhs06lx-nixos-test-driver-1.0/lib/python3.9/site-packages/test_driver/__init__.py", line 86, in main
    driver.run_tests()
  File "/nix/store/zmyxgfrr2lhfcgczljwrh8h50jhs06lx-nixos-test-driver-1.0/lib/python3.9/site-packages/test_driver/driver.py", line 115, in run_tests
    self.test_script()
  File "/nix/store/zmyxgfrr2lhfcgczljwrh8h50jhs06lx-nixos-test-driver-1.0/lib/python3.9/site-packages/test_driver/driver.py", line 111, in test_script
    exec(self.tests, symbols, None)
  File "<string>", line 19, in <module>
  File "/nix/store/zmyxgfrr2lhfcgczljwrh8h50jhs06lx-nixos-test-driver-1.0/lib/python3.9/site-packages/test_driver/machine.py", line 712, in screenshot
    os.unlink(tmp)
FileNotFoundError: [Errno 2] No such file or directory: '/nix/store/99k3djmxmgynmsdcpd1qfy953x07k5pp-shell/alacritty_glinfo.png.ppm'
kill vlan (pid 592566)

Notify maintainers

... Now that we're packaging test-driver as a proper package we should probably also include a meta section

Anyways: @marijanp @Artturin @cole-h

Synthetica9 commented 2 years ago

Seems to be because nix-shell sets $out which is implicitly picked up by the driver...

marijanp commented 2 years ago

@Synthetica9 I used the nixos/tests/yabar.nix test as a base for reproducing this and added a copy_from_vm call, similar to the test you wrote in your pull-request. I couldn't reproduce the error. Now I'm going to checkout your branch and run your test...

EDIT: Now I can reproduce, I missed that you built the driver and then you ran it. When you just run nix-build . -A nixosTests.test everything works fine in a nix-shell. Now the problem doesn't only occur inside a nix-shell, but also outside of it, where the out environment-variable may not even be set. As a solution we could get rid of using the out environment-variable and introduce a command-line parameter --output-dir. When the test-driver is called in runTests we could pass in --output-dir $out

NixOS / nixpkgs