search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
120.66k stars 17.33k forks source link

x/build: LUCI support for plan9 #62025

Open heschi opened 10 months ago

heschi commented 10 months ago

In https://github.com/golang/go/issues/61671#issuecomment-1661755492, @millerresearch reported that the plan9 builders don't support python3. The LUCI swarming bot requires python3, which means that we aren't going to be able to test plan9 once we finish our migration. We can probably keep the old infrastructure going through the 1.22 cycle, but after that we'll likely have to declare the ports as broken in 1.23.

I have no idea how much work it would be to port python3 to plan9. I imagine that if it were easy to do it'd have been done already...? The LUCI team has long-term plans to port the bot to Go but it won't happen on our timeline. (For the record, even with Python supported, the LUCI project has a number of other programs that we'd really like to have working. Fortunately, those are already written in Go and porting them should be pretty easy. Worst case we may be able to work around them.)

Alternatively, maybe it's possible to run it virtualized somehow, say with the bot running under linux, then booting a qemu VM to do its work? The plan9 builders are already kinda slow, though.

cc @golang/plan9 @golang/release

millerresearch commented 10 months ago

According to the person who did the python2.7 Plan 9 port, python3 has more dependencies on GCC or Clang (both unavailable on Plan 9), so getting that working would not be trivial.

As for virtualisation, I can only speak for plan9-arm, maybe @0intro can comment about other platforms.

The plan9-arm builders are real hardware (currently a cluster of Raspbery Pi 4 boards); to my knowledge Plan 9 hasn't been ported to qemu for ARM. In any case, I don't think testing Go in a virtual machine is the same as testing on real hardware. The most interesting bugs are usually the ones that depend on timing or caches or other nuances which are likely to be simplified away by emulation.

Do the tasks run by the LUCI bot need to interact with each other (apart from through the shared file system)? If not, it should be feasible to add a linux machine to the plan9-arn cluster to run LUCI bots, which would do a remote execution of tasks on a Plan 9 machine (sharing the bot's file sysem) in place of local execution as a subprocess.

How is testing going to be done for things like android and iOS, which presumably aren't able to run a python3 bot?

bcmills commented 10 months ago

How is testing going to be done for things like android and iOS, which presumably aren't able to run a python3 bot?

I would expect that the android OS can run a python3 executable directly, although I would also expect that the LUCI bot will run on a Linux host machine and send commands to an emulator to run the tests. (That more closely mimics what a typical Android developer would experience using Go anyway.)

The ios builder currently uses a virtualized environment provided by Corellium, which allows it to exec programs that would normally be disallowed by iOS, presumably including python3.

Adding a Linux machine that sends remote commands to a Plan9 instance sounds like a reasonable architecture to me.

heschi commented 10 months ago

Yeah, that sounds viable if you're willing to maintain the extra hop. We'll have to add some kind of interceptor logic to our build script, but that doesn't seem prohibitive.

0intro commented 8 months ago

I think the easiest way would indeed be to execute the commands in a remote Plan 9 instance, running in either a virtual machine (386 and amd64) or on real hardware (arm).

@heschi Could you indicate where the build scripts are located? I'd like to take a look and see how this could be achieved.

I think the idea would be to execute the commands remotely (think ssh glenda@plan9 ls /), then retrieve the output.

millerresearch commented 4 months ago

Gentle reminder -- we'd like to proceed with moving Plan 9 builders to LUCI, but we're somewhat in the dark. I've found https://chromium.googlesource.com/infra/luci/luci-py -- is this the relevant source or does golang have its own fork? The documentation in that repository refers to "device bots", which is I think what we need, because the Plan 9 tests will run on devices or qemu instances separate from the (linux) platform where the swarming bot runs. It would be helpful to see documentation on setting up a device bot, and -- as @0intro asked a few months ago -- to see examples of how existing build scripts work. I'm guessing android will be the most similar. Also in #63599 a problem launching the swarming bot was reported; we are stalled waiting for a response to that too.

dmitshur commented 4 weeks ago

Thank you for working on this and your patience. I'd like to try to help move this forward. Apologies for not commenting here sooner.

We don't quite have an ideal existing builder that we can point to as the canonical example of how a builder like this can be implemented. Our work in progress on the Android emulator builder (#61097) and iOS Simulator (#66360) is the closest, in that both of those systems have distinct host and target OSes. An advantage they have is the host OS runs the emulator/simulator locally, whereas here it's likely networking to a Plan 9 system will be needed.

For both iOS and Android builders, we're relying the the approach of having a go_{goos}_{goarch}_exec wrapper in $PATH and setting GOOS + GOARCH to the target OS/arch as well as GOHOSTOS + GOHOSTARCH to the host OS/arch in the environment. (See "If the -exec flag is not given, ..." at https://pkg.go.dev/cmd/go#hdr-Compile_and_run_Go_program.) Provided the go_…_exec wrapper is doing its job as intended, at that point there are no further modifications needed to make a builder work. The builder runs the equivalent of make.bash followed by go tool dist test -json (or go test -json for testing golang.org/x repos) and ends up testing the target OS/arch.

In the case of mobile builders, their go_…_exec scripts invoke appropriate emulators/simulators, but in this case your script would be either SSHing into a Plan 9 machine—or something equivalent like Plan 9 in qemu—in order to execute GOOS=plan9 test binaries. The scripts may also need to copy testdata that tests need. I recently saw @FiloSottile using this approach.

To make progress on all this, I suggest splitting the work into two fairly orthogonal halves:

  1. Get the host machine connected and running. It won't be testing the Plan 9 port yet, but it will make sure everything needed for a functional LUCI builder is in place.
  2. Create a go_plan9_{goarch}_exec script that executes Plan 9 binaries by any means that are convenient. (E.g., such that GOOS=plan9 GOARCH=amd64 go test -short bytes can work in your local testing.)

Once those two pieces are working independently, there's a clear path forward to connect them together to get a LUCI builder that provides initial signal on Plan 9 test execution.

For 1, I think we can track the next steps in any of issues #63599, #63600, or #63601. For 2, maybe this issue is good to use to track progress.

Thanks again.

0intro commented 3 weeks ago

Thank you for the instructions.

The go_$GOOS_$GOARCH_exec script is an interesting approach. The difference, from what I had in mind initially, is that the tests will be cross compiled on Linux, then executed on Plan 9, instead of being built on Plan 9 directly.

I think we should be good for 1. See #63599. The other architectures will follow.

For 2, I've started to experiment the go_$GOOS_$GOARCH_exec script approach, and it looks promising so far.


##### Test execution environment.
$ GOOS=plan9 GOARCH=386 go tool dist test

ssh -p 2222 glenda@localhost /tmp/main
# GOARCH: 386
# CPU: QEMU Virtual CPU version 2.5+
# GOOS: plan9
# OS Version: 2000

##### Testing packages.
ok      archive/tar 0.297s
ok      archive/zip 0.250s
ok      bufio   0.174s
ok      bytes   0.272s
ok      cmp 0.145s
ok      compress/bzip2  0.179s
ssh -p 2222 glenda@localhost /tmp/flate.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestWriterPersistentWriteError (0.00s)
    deflate_test.go:724: ReadFile: open ../../testdata/Isaac.Newton-Opticks.txt: '../../testdata' file does not exist
FAIL
FAIL    compress/flate  0.549s
ok      compress/gzip   1.068s
ssh -p 2222 glenda@localhost /tmp/lzw.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestWriter (0.00s)
    writer_test.go:30: ../testdata/gettysburg.txt (order=0 litWidth=7): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:30: ../testdata/gettysburg.txt (order=0 litWidth=8): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:30: ../testdata/gettysburg.txt (order=1 litWidth=7): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:30: ../testdata/gettysburg.txt (order=1 litWidth=8): open ../testdata/gettysburg.txt: '../testdata' file does not exist
FAIL
FAIL    compress/lzw    0.139s
ssh -p 2222 glenda@localhost /tmp/zlib.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestWriterBig (0.00s)
    writer_test.go:32: ../testdata/gettysburg.txt (level=-1, dict=""): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:32: ../testdata/gettysburg.txt (level=0, dict=""): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:32: ../testdata/gettysburg.txt (level=-2, dict=""): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:32: ../testdata/gettysburg.txt (level=1, dict=""): open ../testdata/gettysburg.txt: '../testdata' file does not exist
--- FAIL: TestWriterDict (0.00s)
    writer_test.go:32: ../testdata/gettysburg.txt (level=-1, dict="0123456789."): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:32: ../testdata/gettysburg.txt (level=0, dict="0123456789."): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:32: ../testdata/gettysburg.txt (level=-2, dict="0123456789."): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:32: ../testdata/gettysburg.txt (level=1, dict="0123456789."): open ../testdata/gettysburg.txt: '../testdata' file does not exist
--- FAIL: TestWriterReset (0.00s)
    writer_test.go:99: ../testdata/gettysburg.txt (level=0): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:99: ../testdata/gettysburg.txt (level=-1): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:99: ../testdata/gettysburg.txt (level=-2): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:99: ../testdata/gettysburg.txt (level=0): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:99: ../testdata/gettysburg.txt (level=-1): open ../testdata/gettysburg.txt: '../testdata' file does not exist
    writer_test.go:99: ../testdata/gettysburg.txt (level=-2): open ../testdata/gettysburg.txt: '../testdata' file does not exist
FAIL
FAIL    compress/zlib   0.143s
ok      container/heap  0.154s
ok      container/list  0.142s
ok      container/ring  0.135s
ok      context 0.349s
ok      crypto  0.146s
ok      crypto/aes  0.152s
ok      crypto/cipher   1.374s
ok      crypto/des  0.346s
ok      crypto/dsa  0.191s
ok      crypto/ecdh 0.310s
ok      crypto/ecdsa    0.265s
ok      crypto/ed25519  0.312s
ok      crypto/elliptic 0.175s
ok      crypto/hmac 0.141s
ok      crypto/internal/alias   0.136s
ok      crypto/internal/bigmod  0.166s
ok      crypto/internal/boring  0.142s
ok      crypto/internal/boring/bcache   0.236s
ok      crypto/internal/edwards25519    0.460s
ok      crypto/internal/edwards25519/field  0.170s
ok      crypto/internal/mlkem768    0.291s
ok      crypto/internal/nistec  0.759s
ok      crypto/internal/nistec/fiat 0.135s [no tests to run]
ok      crypto/md5  0.143s
ok      crypto/rand 0.148s
ok      crypto/rc4  0.158s
ok      crypto/rsa  0.809s
ok      crypto/sha1 0.138s
ok      crypto/sha256   0.140s
ok      crypto/sha512   0.141s
ok      crypto/subtle   0.259s
ok      crypto/tls  12.282s
ok      crypto/x509 1.567s
ok      database/sql    0.772s
ok      database/sql/driver 0.136s
ok      debug/buildinfo 0.153s
ok      debug/dwarf 0.218s
ok      debug/elf   0.254s
ok      debug/gosym 0.159s
ok      debug/macho 0.162s
ok      debug/pe    0.222s
ok      debug/plan9obj  0.145s
ok      embed   0.148s [no tests to run]
ok      embed/internal/embedtest    0.184s
ok      encoding/ascii85    0.140s
ok      encoding/asn1   0.146s
ok      encoding/base32 0.154s
ok      encoding/base64 0.154s
ok      encoding/binary 0.142s
ok      encoding/csv    0.161s
ok      encoding/gob    1.773s
ok      encoding/hex    0.139s
ok      encoding/json   0.357s
ok      encoding/pem    0.463s
ok      encoding/xml    0.171s
ok      errors  0.144s
ok      expvar  0.156s
ok      flag    0.183s
ok      fmt 0.199s
ok      go/ast  0.162s
ssh -p 2222 glenda@localhost /tmp/build.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestLocalDirectory (0.00s)
    build_test.go:145: no buildable Go source files in /tmp
--- FAIL: TestImportCmd (0.00s)
    build_test.go:432: go/build: go list cmd/internal/objfile: fork/exec /usr/local/go/bin/go: '/usr/local' does not exist

--- FAIL: TestFindImports (0.00s)
    deps_test.go:861: open /usr/local/go/src/go/build: '/usr/local' does not exist
FAIL
FAIL    go/build    1.174s
ok      go/build/constraint 0.138s
ok      go/constant 0.144s
ok      go/doc  0.314s
ok      go/doc/comment  0.649s
ssh -p 2222 glenda@localhost /tmp/format.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestNode (0.00s)
    format_test.go:43: open format_test.go: 'format_test.go' file does not exist
--- FAIL: TestSource (0.00s)
    format_test.go:101: open format_test.go: 'format_test.go' file does not exist
FAIL
FAIL    go/format   0.144s
ok      go/importer 0.164s
ok      go/internal/gccgoimporter   0.196s
ok      go/internal/gcimporter  0.189s
ssh -p 2222 glenda@localhost /tmp/srcimporter.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestImportedTypes (0.01s)
    srcimporter_test.go:126: go/build: go list flag: fork/exec /usr/local/go/bin/go: '/usr/local' does not exist

    srcimporter_test.go:126: go/build: go list io: fork/exec /usr/local/go/bin/go: '/usr/local' does not exist

    srcimporter_test.go:126: go/build: go list io: fork/exec /usr/local/go/bin/go: '/usr/local' does not exist

    srcimporter_test.go:126: go/build: go list math: fork/exec /usr/local/go/bin/go: '/usr/local' does not exist

    srcimporter_test.go:126: go/build: go list math: fork/exec /usr/local/go/bin/go: '/usr/local' does not exist

    srcimporter_test.go:126: go/build: go list math/big: fork/exec /usr/local/go/bin/go: '/usr/local' does not exist

    srcimporter_test.go:126: go/build: go list golang.org/x/text/unicode/norm: fork/exec /usr/local/go/bin/go: '/usr/local' does not exist

--- FAIL: TestReimport (0.00s)
    srcimporter_test.go:193: got go/build: go list math: fork/exec /usr/local/go/bin/go: '/usr/local' does not exist

        ; want reimport error
--- FAIL: TestIssue20855 (0.00s)
    srcimporter_test.go:204: got unexpected or no error: go/build: go list go/internal/srcimporter/testdata/issue20855: fork/exec /usr/local/go/bin/go: '/usr/local' does not exist

--- FAIL: TestIssue24392 (0.00s)
    srcimporter_test.go:220: go/build: go list go/internal/srcimporter/testdata/issue24392: fork/exec /usr/local/go/bin/go: '/usr/local' does not exist

FAIL
FAIL    go/internal/srcimporter 0.185s
ssh -p 2222 glenda@localhost /tmp/parser.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
panic: open ../printer/nodes.go: '../printer' file does not exist

goroutine 1 [running]:
go/parser.readFile(...)
    /usr/local/go/src/go/parser/performance_test.go:18
go/parser.init()
    /usr/local/go/src/go/parser/performance_test.go:13 +0x329
FAIL    go/parser   0.215s
ssh -p 2222 glenda@localhost /tmp/printer.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestWriteErrors (0.00s)
panic: open printer.go: 'printer.go' file does not exist [recovered]
    panic: open printer.go: 'printer.go' file does not exist

goroutine 21 [running]:
testing.tRunner.func1.2({0x16d240, 0x8717d0})
    /usr/local/go/src/testing/testing.go:1632 +0x282
testing.tRunner.func1()
    /usr/local/go/src/testing/testing.go:1635 +0x3fc
panic({0x16d240, 0x8717d0})
    /usr/local/go/src/runtime/panic.go:759 +0xf8
go/printer.TestWriteErrors(0x898c08)
    /usr/local/go/src/go/printer/printer_test.go:652 +0x231
testing.tRunner(0x898c08, 0x19126c)
    /usr/local/go/src/testing/testing.go:1690 +0x118
created by testing.(*T).Run in goroutine 1
    /usr/local/go/src/testing/testing.go:1743 +0x3d0
FAIL    go/printer  0.223s
ok      go/scanner  0.140s
ok      go/token    0.156s
ssh -p 2222 glenda@localhost /tmp/types.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestGenerate (0.00s)
    generate_test.go:59: open /usr/local/go/src/cmd/compile/internal/types2/selection.go: '/usr/local' does not exist
2024/05/17 12:09:33 main:2:8: could not import fmt (can't find import: "fmt": cannot find package "fmt" in any of:
    /usr/local/go/src/fmt (from $GOROOT)
    /usr/glenda/go/src/fmt (from $GOPATH))
FAIL    go/types    0.254s
ok      go/version  0.139s
ok      hash    0.141s
ok      hash/adler32    0.141s
ok      hash/crc32  0.140s
ok      hash/crc64  0.140s
ok      hash/fnv    0.143s
ok      hash/maphash    0.167s
ok      html    0.150s
ok      html/template   0.226s
ok      image   0.213s
ok      image/color 0.150s
ssh -p 2222 glenda@localhost /tmp/draw.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestPaletted (0.00s)
    draw_test.go:724: open: open ../testdata/video-001.png: '../testdata' file does not exist
FAIL
FAIL    image/draw  0.184s
ssh -p 2222 glenda@localhost /tmp/gif.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestWriter (0.00s)
    writer_test.go:91: ../testdata/video-001.png open ../testdata/video-001.png: '../testdata' file does not exist
    writer_test.go:91: ../testdata/video-001.gif open ../testdata/video-001.gif: '../testdata' file does not exist
    writer_test.go:91: ../testdata/video-001.interlaced.gif open ../testdata/video-001.interlaced.gif: '../testdata' file does not exist
--- FAIL: TestSubImage (0.00s)
    writer_test.go:121: readImg: open ../testdata/video-001.gif: '../testdata' file does not exist
--- FAIL: TestEncodeAllGo1Dot4 (0.00s)
    writer_test.go:184: ../testdata/video-001.gif open ../testdata/video-001.gif: '../testdata' file does not exist
--- FAIL: TestEncodeAllGo1Dot5 (0.00s)
    writer_test.go:184: ../testdata/video-001.gif open ../testdata/video-001.gif: '../testdata' file does not exist
--- FAIL: TestEncodeAllGo1Dot5GlobalColorModel (0.00s)
    writer_test.go:184: ../testdata/video-001.gif open ../testdata/video-001.gif: '../testdata' file does not exist
--- FAIL: TestEncodeWrappedImage (0.00s)
    writer_test.go:603: readImg: open ../testdata/video-001.gif: '../testdata' file does not exist
FAIL
FAIL    image/gif   0.182s
ssh -p 2222 glenda@localhost /tmp/jpeg.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestDecodeProgressive (0.00s)
    reader_test.go:41: ../testdata/video-001.jpeg: open ../testdata/video-001.jpeg: '../testdata' file does not exist
    reader_test.go:41: ../testdata/video-001.q50.410.jpeg: open ../testdata/video-001.q50.410.jpeg: '../testdata' file does not exist
    reader_test.go:41: ../testdata/video-001.q50.411.jpeg: open ../testdata/video-001.q50.411.jpeg: '../testdata' file does not exist
    reader_test.go:41: ../testdata/video-001.q50.420.jpeg: open ../testdata/video-001.q50.420.jpeg: '../testdata' file does not exist
    reader_test.go:41: ../testdata/video-001.q50.422.jpeg: open ../testdata/video-001.q50.422.jpeg: '../testdata' file does not exist
    reader_test.go:41: ../testdata/video-001.q50.440.jpeg: open ../testdata/video-001.q50.440.jpeg: '../testdata' file does not exist
    reader_test.go:41: ../testdata/video-001.q50.444.jpeg: open ../testdata/video-001.q50.444.jpeg: '../testdata' file does not exist
    reader_test.go:41: ../testdata/video-005.gray.q50.jpeg: open ../testdata/video-005.gray.q50.jpeg: '../testdata' file does not exist
    reader_test.go:41: ../testdata/video-005.gray.q50.2x2.jpeg: open ../testdata/video-005.gray.q50.2x2.jpeg: '../testdata' file does not exist
    reader_test.go:41: ../testdata/video-001.separate.dc.progression.jpeg: open ../testdata/video-001.separate.dc.progression.jpeg: '../testdata' file does not exist
--- FAIL: TestDecodeEOF (0.00s)
    reader_test.go:123: open ../testdata/video-001.jpeg: '../testdata' file does not exist
--- FAIL: TestTruncatedSOSDataDoesntPanic (0.00s)
    reader_test.go:194: open ../testdata/video-005.gray.q50.jpeg: '../testdata' file does not exist
--- FAIL: TestIssue56724 (0.00s)
    reader_test.go:496: open ../testdata/video-001.jpeg: '../testdata' file does not exist
--- FAIL: TestBadRestartMarker (0.00s)
    reader_test.go:510: open ../testdata/video-001.restart2.jpeg: '../testdata' file does not exist
--- FAIL: TestWriter (0.00s)
    writer_test.go:136: ../testdata/video-001.png open ../testdata/video-001.png: '../testdata' file does not exist
    writer_test.go:136: ../testdata/video-001.png open ../testdata/video-001.png: '../testdata' file does not exist
    writer_test.go:136: ../testdata/video-001.png open ../testdata/video-001.png: '../testdata' file does not exist
    writer_test.go:136: ../testdata/video-001.png open ../testdata/video-001.png: '../testdata' file does not exist
    writer_test.go:136: ../testdata/video-001.png open ../testdata/video-001.png: '../testdata' file does not exist
    writer_test.go:136: ../testdata/video-001.png open ../testdata/video-001.png: '../testdata' file does not exist
FAIL
FAIL    image/jpeg  0.197s
ok      image/png   0.402s
ok      index/suffixarray   0.235s
ok      internal/abi    0.154s
ok      internal/buildcfg   0.140s
ok      internal/chacha8rand    0.143s
ok      internal/concurrent 0.166s
ok      internal/coverage/cformat   0.140s
ok      internal/coverage/cmerge    0.142s
ok      internal/coverage/pods  0.147s
ok      internal/coverage/slicereader   0.151s
ok      internal/coverage/slicewriter   0.138s
ok      internal/coverage/test  0.156s
ok      internal/cpu    0.141s
ok      internal/dag    0.141s
ok      internal/diff   0.165s
ok      internal/fmtsort    0.139s
ok      internal/fuzz   0.154s
ok      internal/godebug    0.144s
ok      internal/godebugs   0.149s
ok      internal/gover  0.151s
ok      internal/itoa   0.139s
ok      internal/platform   0.182s
ok      internal/poll   0.166s
ok      internal/profile    0.145s
ok      internal/reflectlite    0.152s
ok      internal/runtime/atomic 0.162s
ok      internal/saferio    0.154s
ok      internal/singleflight   0.190s
ok      internal/sysinfo    0.139s
ok      internal/testenv    0.144s
ssh -p 2222 glenda@localhost /tmp/trace.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestSummarizeGoroutinesTrace (0.00s)
    summary_test.go:269: malformed test v2/testdata/tests/go122-gc-stress.test: bad trace file: failed to read test file for v2/testdata/tests/go122-gc-stress.test: open v2/testdata/tests/go122-gc-stress.test: 'v2' file does not exist
--- FAIL: TestSummarizeGoroutinesRegionsTrace (0.00s)
    summary_test.go:269: malformed test v2/testdata/tests/go122-annotations.test: bad trace file: failed to read test file for v2/testdata/tests/go122-annotations.test: open v2/testdata/tests/go122-annotations.test: 'v2' file does not exist
--- FAIL: TestSummarizeTasksTrace (0.00s)
    summary_test.go:269: malformed test v2/testdata/tests/go122-annotations-stress.test: bad trace file: failed to read test file for v2/testdata/tests/go122-annotations-stress.test: open v2/testdata/tests/go122-annotations-stress.test: 'v2' file does not exist
--- FAIL: TestRelatedGoroutinesV2Trace (0.00s)
    summary_test.go:393: malformed test v2/testdata/tests/go122-gc-stress.test: bad trace file: failed to read test file for v2/testdata/tests/go122-gc-stress.test: open v2/testdata/tests/go122-gc-stress.test: 'v2' file does not exist
FAIL
FAIL    internal/trace  0.223s
ssh -p 2222 glenda@localhost /tmp/trace.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestOldtrace (0.00s)
    oldtrace_test.go:87: didn't see expected test case user_task_region_1_21_good
FAIL
FAIL    internal/trace/v2   0.286s
ok      internal/trace/v2/internal/oldtrace 0.232s
ssh -p 2222 glenda@localhost /tmp/errors.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestErrorCodeStyle (0.00s)
    codes_test.go:151: open codes.go: 'codes.go' file does not exist
FAIL
FAIL    internal/types/errors   0.159s
ok      internal/unsafeheader   0.142s
ok      internal/weak   0.142s
ok      internal/xcoff  0.194s
ssh -p 2222 glenda@localhost /tmp/zstd.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestAlloc (0.00s)
    zstd_test.go:165: open ../../testdata/Isaac.Newton-Opticks.txt: '../../testdata' file does not exist
FAIL
FAIL    internal/zstd   0.179s
ok      io  0.317s
ssh -p 2222 glenda@localhost /tmp/fs.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestGlob (0.02s)
    glob_test.go:35: Glob(`glob.go`) = []string(nil) want glob.go
    glob_test.go:35: Glob(`gl?b.go`) = []string(nil) want glob.go
    glob_test.go:35: Glob(`gl\ob.go`) = []string(nil) want glob.go
    glob_test.go:35: Glob(`*`) = []string{"fs.test", "xxx"} want glob.go
    glob_test.go:35: Glob(`*/glob.go`) = []string(nil) want fs/glob.go
FAIL
FAIL    io/fs   0.389s
ssh -p 2222 glenda@localhost /tmp/ioutil.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestReadFile (0.00s)
    ioutil_test.go:36: ReadFile ioutil_test.go: open ioutil_test.go: 'ioutil_test.go' file does not exist
--- FAIL: TestReadOnlyWriteFile (0.00s)
    ioutil_test.go:94: Expected an error when writing to read-only file /tmp/TestReadOnlyWriteFile1731551095/blurp.txt
--- FAIL: TestReadDir (0.00s)
    ioutil_test.go:129: ReadDir ..: io_test.go file not found
FAIL
FAIL    io/ioutil   1.018s
ok      iter    0.141s
ok      log 0.157s
ok      log/slog    0.191s
ok      log/slog/internal/benchmarks    0.165s
ok      log/slog/internal/buffer    0.144s
ok      maps    0.152s
ok      math    0.156s
ok      math/big    0.879s
ok      math/bits   0.151s
ok      math/cmplx  0.144s
ok      math/rand   0.212s
ok      math/rand/v2    0.306s
ok      mime    0.164s
ok      mime/multipart  0.703s
ok      mime/quotedprintable    0.160s
ssh -p 2222 glenda@localhost /tmp/net.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestSendfile (0.01s)
    sendfile_test.go:137: received 0 bytes; expected 567198
    sendfile_test.go:141: retrieved data hash did not match
    sendfile_test.go:145: open ../testdata/Isaac.Newton-Opticks.txt: '../testdata' file does not exist
--- FAIL: TestSendfileParts (0.02s)
    sendfile_test.go:198: unexpected server reply "", want "Produced "
    sendfile_test.go:202: open ../testdata/Isaac.Newton-Opticks.txt: '../testdata' file does not exist
--- FAIL: TestSendfileSeeked (0.02s)
    sendfile_test.go:258: Got 0 bytes; want 10240
    sendfile_test.go:262: open ../testdata/Isaac.Newton-Opticks.txt: '../testdata' file does not exist
--- FAIL: TestSendfileOnWriteTimeoutExceeded (0.02s)
    sendfile_test.go:427: open ../testdata/Isaac.Newton-Opticks.txt: '../testdata' file does not exist
FAIL
FAIL    net 17.559s
ssh -p 2222 glenda@localhost /tmp/http.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestEmptyDirOpenCWD (0.00s)
    fs_test.go:492: open of fs_test.go: open fs_test.go: 'fs_test.go' file does not exist
http.test 33984: warning: process exceeds 100 file descriptors
--- FAIL: TestServeFileFromCWD (0.00s)
    --- FAIL: TestServeFileFromCWD/h1 (0.05s)
        fs_test.go:556: expected 200 OK, got 404 Not Found
    --- FAIL: TestServeFileFromCWD/h2 (0.08s)
        fs_test.go:556: expected 200 OK, got 404 Not Found
FAIL
FAIL    net/http    144.934s
ok      net/http/cgi    0.523s
ok      net/http/cookiejar  0.408s
ok      net/http/fcgi   0.377s
ok      net/http/httptest   3.828s
ok      net/http/httptrace  0.154s
ok      net/http/httputil   14.883s
ok      net/http/internal   0.261s
ok      net/http/internal/ascii 0.141s
ok      net/http/pprof  4.366s
ok      net/internal/cgotest    0.139s
ok      net/mail    0.150s
ok      net/netip   0.174s
ok      net/rpc 1.715s
ok      net/rpc/jsonrpc 0.177s
ok      net/smtp    1.955s
ok      net/textproto   0.157s
ok      net/url 0.157s
ssh -p 2222 glenda@localhost /tmp/os.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestRemoveAllButReadOnlyAndPathError (0.00s)
    removeall_test.go:332: RemoveAll succeeded unexpectedly
--- FAIL: TestReadDir (0.00s)
    read_test.go:133: ReadDir .: read_test.go file not found
--- FAIL: TestReadOnlyWriteFile (0.00s)
    read_test.go:96: Expected an error when writing to read-only file /tmp/TestReadOnlyWriteFile1135810301/blurp.txt
--- FAIL: TestReadFile (0.00s)
    read_test.go:37: ReadFile read_test.go: open read_test.go: 'read_test.go' file does not exist
--- FAIL: TestFileReadDir (0.00s)
    --- FAIL: TestFileReadDir/. (0.00s)
        os_test.go:469: could not find dir_unix.go
        os_test.go:469: could not find env.go
        os_test.go:469: could not find error.go
        os_test.go:469: could not find file.go
        os_test.go:469: could not find os_test.go
        os_test.go:469: could not find types.go
        os_test.go:469: could not find stat_darwin.go
        os_test.go:469: could not find stat_linux.go
--- FAIL: TestFileReaddir (0.00s)
    --- FAIL: TestFileReaddir/. (0.00s)
        os_test.go:414: could not find dir_unix.go
        os_test.go:414: could not find env.go
        os_test.go:414: could not find error.go
        os_test.go:414: could not find file.go
        os_test.go:414: could not find os_test.go
        os_test.go:414: could not find types.go
        os_test.go:414: could not find stat_darwin.go
        os_test.go:414: could not find stat_linux.go
--- FAIL: TestFileReaddirnames (0.00s)
    --- FAIL: TestFileReaddirnames/. (0.00s)
        os_test.go:377: could not find dir_unix.go
        os_test.go:377: could not find env.go
        os_test.go:377: could not find error.go
        os_test.go:377: could not find file.go
        os_test.go:377: could not find os_test.go
        os_test.go:377: could not find types.go
        os_test.go:377: could not find stat_darwin.go
        os_test.go:377: could not find stat_linux.go
FAIL
FAIL    os  1.327s
ssh -p 2222 glenda@localhost /tmp/exec.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestCatGoodAndBadFile (0.01s)
    exec_test.go:430: expected test code; got "Error: open exec_test.go: 'exec_test.go' file does not exist\n" (len 61)
FAIL
FAIL    os/exec 0.878s
ok      os/exec/internal/fdtest 0.151s
ok      os/signal   0.256s
ok      os/user 0.146s
ok      path    0.147s
ssh -p 2222 glenda@localhost /tmp/filepath.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestGlob (0.03s)
    match_test.go:133: Glob(`match.go`) = []string(nil) want match.go
    match_test.go:133: Glob(`mat?h.go`) = []string(nil) want match.go
    match_test.go:133: Glob(`*`) = []string{"filepath.test", "xxx"} want match.go
    match_test.go:133: Glob(`../*/match.go`) = []string(nil) want ../filepath/match.go
--- FAIL: TestNonWindowsGlobEscape (0.00s)
    match_test.go:371: Glob(`\match.go`) = [] want [match.go]
--- FAIL: TestBug3486 (0.00s)
    path_test.go:1735: stat /usr/local/go/src/unicode: '/usr/local' does not exist
FAIL
FAIL    path/filepath   0.217s
ok      plugin  0.143s
ok      reflect 0.299s
ok      regexp  0.422s
ok      regexp/syntax   0.603s
ssh -p 2222 glenda@localhost /tmp/runtime.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestTracebackSystem (0.04s)
    traceback_system_test.go:114: stderr:
        Callers: [0x3b18ff 0x3b1896 0x3b184b 0x3b6efa 0x3b6ef5 0x3b6ef4 0x3b6ef3 0x3b6ef2 0x82ad1]
        panic: open /usr/local/go/src/runtime/traceback_system_test.go: '/usr/local' does not exist

        [...]
FAIL    runtime 8.720s
ok      runtime/coverage    0.167s
ok      runtime/debug   0.214s
ok      runtime/internal/math   0.151s
ok      runtime/internal/sys    0.144s
ok      runtime/internal/wasitest   0.161s
ssh -p 2222 glenda@localhost /tmp/metrics.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestDocs (0.00s)
    description_test.go:94: open doc.go: 'doc.go' file does not exist
FAIL
FAIL    runtime/metrics 0.155s
ok      runtime/pprof   9.515s
ok      runtime/trace   0.267s
ok      slices  0.181s
ok      sort    0.175s
ok      strconv 0.284s
ok      strings 0.203s
ssh -p 2222 glenda@localhost /tmp/sync.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: ExampleOnceValues (0.02s)
got:
Reading file once
error: open example_test.go: 'example_test.go' file does not exist
error: open example_test.go: 'example_test.go' file does not exist
error: open example_test.go: 'example_test.go' file does not exist
error: open example_test.go: 'example_test.go' file does not exist
error: open example_test.go: 'example_test.go' file does not exist
error: open example_test.go: 'example_test.go' file does not exist
error: open example_test.go: 'example_test.go' file does not exist
error: open example_test.go: 'example_test.go' file does not exist
error: open example_test.go: 'example_test.go' file does not exist
error: open example_test.go: 'example_test.go' file does not exist
want:
Reading file once
FAIL
FAIL    sync    1.284s
ok      sync/atomic 1.428s
ok      syscall 0.141s
ok      testing 1.020s
ok      testing/fstest  0.147s
ok      testing/iotest  0.148s
ok      testing/quick   0.164s
ok      testing/slogtest    0.153s
ok      text/scanner    0.149s
ok      text/tabwriter  0.143s
ok      text/template   0.198s
ok      text/template/parse 0.153s
ssh -p 2222 glenda@localhost /tmp/time.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
panic: cannot load America/Los_Angeles for testing: '../../lib/time' does not exist; you may want to use -tags=timetzdata

goroutine 1 [running]:
time.initTestingZone()
    /usr/local/go/src/time/internal_test.go:21 +0x12f
sync.(*Once).doSlow(0x413e50, 0x294318)
    /usr/local/go/src/sync/once.go:74 +0xa0
sync.(*Once).Do(0x413e50, 0x294318)
    /usr/local/go/src/sync/once.go:65 +0x3e
time.ForceUSPacificForTesting(...)
    /usr/local/go/src/time/export_test.go:18
time.init.0()
    /usr/local/go/src/time/internal_test.go:9 +0x7b
FAIL    time    0.206s
ok      unicode 0.153s
ok      unicode/utf16   0.150s
ok      unicode/utf8    0.150s
ok      unique  0.187s
ok      cmd/addr2line   0.170s
ok      cmd/api 0.235s
ssh -p 2222 glenda@localhost /tmp/asm.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
testdata/386.s:6: #include: open ../../../../runtime/textflag.h: '../../../../runtime' file does not exist
FAIL    cmd/asm/internal/asm    0.247s
ok      cmd/asm/internal/lex    0.153s
ok      cmd/cgo/internal/swig   0.172s
ok      cmd/cgo/internal/test   0.144s
ok      cmd/cgo/internal/testcshared    0.204s
ok      cmd/cgo/internal/testerrors 0.184s
ok      cmd/cgo/internal/testfortran    0.167s
ok      cmd/cgo/internal/testgodefs 0.165s
ok      cmd/cgo/internal/testlife   0.163s
ok      cmd/cgo/internal/testnocgo  0.149s
ok      cmd/cgo/internal/testplugin 0.295s
ok      cmd/cgo/internal/testsanitizers 0.202s [no tests to run]
ok      cmd/cgo/internal/testshared 0.313s
ok      cmd/cgo/internal/testso 0.175s
ok      cmd/cgo/internal/teststdio  0.170s
ok      cmd/cgo/internal/testtls    0.150s
ok      cmd/compile/internal/abt    0.153s
ok      cmd/compile/internal/amd64  0.224s
ok      cmd/compile/internal/base   0.156s
ok      cmd/compile/internal/compare    0.179s
ok      cmd/compile/internal/devirtualize   0.186s
ok      cmd/compile/internal/dwarfgen   0.223s
ok      cmd/compile/internal/importer   0.221s
ok      cmd/compile/internal/inline/inlheur 0.211s
ok      cmd/compile/internal/ir 0.181s
ok      cmd/compile/internal/liveness   0.222s
ok      cmd/compile/internal/logopt 0.159s
ok      cmd/compile/internal/loopvar    0.219s
ok      cmd/compile/internal/noder  0.231s
ok      cmd/compile/internal/reflectdata    0.189s [no tests to run]
ok      cmd/compile/internal/ssa    0.380s
ssh -p 2222 glenda@localhost /tmp/syntax.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestParse (0.00s)
    parser_test.go:30: open parser.go: 'parser.go' file does not exist
--- FAIL: TestVerify (0.00s)
    parser_test.go:34: open parser.go: 'parser.go' file does not exist
FAIL
FAIL    cmd/compile/internal/syntax 0.192s
ok      cmd/compile/internal/test   0.385s
ok      cmd/compile/internal/typecheck  0.182s
ok      cmd/compile/internal/types  0.161s
ssh -p 2222 glenda@localhost /tmp/types2.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
2024/05/17 12:13:54 main:2:8: could not import fmt (can't find import: "fmt": cannot find package "fmt" in any of:
    /usr/local/go/src/fmt (from $GOROOT)
    /usr/glenda/go/src/fmt (from $GOPATH))
FAIL    cmd/compile/internal/types2 0.298s
ok      cmd/covdata 0.178s
ok      cmd/cover   0.237s
ok      cmd/dist    0.151s
ok      cmd/distpack    0.186s
ssh -p 2222 glenda@localhost /tmp/doc.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
FAIL    cmd/doc 0.204s
ok      cmd/fix 0.348s
ok      cmd/go  1.659s
ok      cmd/go/internal/auth    0.157s
ok      cmd/go/internal/cache   0.637s
ok      cmd/go/internal/cfg 0.146s [no tests to run]
ok      cmd/go/internal/fsys    0.255s
ok      cmd/go/internal/generate    0.218s
ok      cmd/go/internal/gover   0.149s
ok      cmd/go/internal/imports 0.213s
ok      cmd/go/internal/load    0.191s
ok      cmd/go/internal/lockedfile  0.327s
ok      cmd/go/internal/modfetch    0.258s
ok      cmd/go/internal/modfetch/codehost   0.252s
ok      cmd/go/internal/modfetch/zip_sum_test   0.199s
ssh -p 2222 glenda@localhost /tmp/modindex.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestIndex (0.00s)
    --- FAIL: TestIndex/crypto (0.00s)
        index_test.go:29: cannot find package in:
                /usr/local/go/src/crypto
    --- FAIL: TestIndex/encoding (0.00s)
        index_test.go:29: cannot find package in:
                /usr/local/go/src/encoding
    --- FAIL: TestIndex/unsafe (0.00s)
        index_test.go:29: cannot find package in:
                /usr/local/go/src/unsafe
    --- FAIL: TestIndex/encoding/json (0.00s)
        index_test.go:29: cannot find package in:
                /usr/local/go/src/encoding/json
    --- FAIL: TestIndex/runtime (0.00s)
        index_test.go:29: cannot find package in:
                /usr/local/go/src/runtime
    --- FAIL: TestIndex/net (0.00s)
        index_test.go:29: cannot find package in:
                /usr/local/go/src/net
    --- FAIL: TestIndex/all (0.00s)
        index_test.go:29: cannot find package in:
                /usr/local/go/src/crypto
FAIL
FAIL    cmd/go/internal/modindex    0.178s
ok      cmd/go/internal/modload 0.261s
ok      cmd/go/internal/mvs 0.164s
ok      cmd/go/internal/par 0.154s
ok      cmd/go/internal/str 0.150s
ok      cmd/go/internal/test    0.240s
ok      cmd/go/internal/toolchain   0.222s
ok      cmd/go/internal/vcs 0.197s
ok      cmd/go/internal/vcweb   0.396s
ssh -p 2222 glenda@localhost /tmp/vcstest.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
2024/05/17 12:14:05 serving /
2024/05/17 12:14:05 serving /
--- FAIL: TestScripts (0.06s)
    vcstest_test.go:168: stat /testdata/vcstest: '/testdata' file does not exist
FAIL
FAIL    cmd/go/internal/vcweb/vcstest   0.350s
ok      cmd/go/internal/web 0.181s
ok      cmd/go/internal/work    0.219s
ssh -p 2222 glenda@localhost /tmp/gofmt.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestRewrite (0.03s)
    --- FAIL: TestRewrite/gofmt.go (0.00s)
        gofmt_test.go:62: stat gofmt.go: 'gofmt.go' file does not exist
    --- FAIL: TestRewrite/gofmt_test.go (0.00s)
        gofmt_test.go:62: stat gofmt_test.go: 'gofmt_test.go' file does not exist
FAIL
FAIL    cmd/gofmt   0.258s
ok      cmd/internal/archive    0.172s
ok      cmd/internal/bootstrap_test 0.149s
ok      cmd/internal/buildid    0.271s
ok      cmd/internal/cov    0.160s
ssh -p 2222 glenda@localhost /tmp/dwarf.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestPutVarAbbrevGenerator (0.00s)
    putvarabbrevgen_test.go:85: open ./dwarf.go: './dwarf.go' file does not exist
FAIL
FAIL    cmd/internal/dwarf  0.153s
ok      cmd/internal/edit   0.153s
ok      cmd/internal/goobj  0.147s
ok      cmd/internal/moddeps    0.155s
ok      cmd/internal/notsha256  0.521s
ok      cmd/internal/obj    1.069s
ok      cmd/internal/obj/loong64    0.148s
ok      cmd/internal/obj/ppc64  0.157s
ok      cmd/internal/obj/riscv  0.175s
ok      cmd/internal/obj/s390x  0.149s
ok      cmd/internal/obj/x86    0.161s
ok      cmd/internal/objabi 0.154s
ok      cmd/internal/pgo    0.175s
ok      cmd/internal/pkgpath    0.176s
ok      cmd/internal/pkgpattern 0.152s
ok      cmd/internal/quoted 0.150s
ok      cmd/internal/src    0.153s
ok      cmd/internal/test2json  0.253s
ok      cmd/link    0.264s
ok      cmd/link/internal/benchmark 0.152s
ok      cmd/link/internal/ld    0.263s
ok      cmd/link/internal/loader    0.159s
ssh -p 2222 glenda@localhost /tmp/nm.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestNonGoExecs (0.04s)
    nm_test.go:80: go tool nm /usr/local/go/src/debug/elf/testdata/gcc-386-freebsd-exec: exit status: 'nm.test 38329: 1'
        open /usr/local/go/src/debug/elf/testdata/gcc-386-freebsd-exec: '/usr/local' does not exist
    nm_test.go:80: go tool nm /usr/local/go/src/debug/elf/testdata/gcc-amd64-linux-exec: exit status: 'nm.test 38332: 1'
        open /usr/local/go/src/debug/elf/testdata/gcc-amd64-linux-exec: '/usr/local' does not exist
    nm_test.go:70: obscuretestdata.DecodeToTempFile(/usr/local/go/src/debug/macho/testdata/gcc-386-darwin-exec.base64): open /usr/local/go/src/debug/macho/testdata/gcc-386-darwin-exec.base64: '/usr/local' does not exist
    nm_test.go:70: obscuretestdata.DecodeToTempFile(/usr/local/go/src/debug/macho/testdata/gcc-amd64-darwin-exec.base64): open /usr/local/go/src/debug/macho/testdata/gcc-amd64-darwin-exec.base64: '/usr/local' does not exist
    nm_test.go:80: go tool nm /usr/local/go/src/debug/pe/testdata/gcc-386-mingw-exec: exit status: 'nm.test 38335: 1'
        open /usr/local/go/src/debug/pe/testdata/gcc-386-mingw-exec: '/usr/local' does not exist
    nm_test.go:80: go tool nm /usr/local/go/src/debug/plan9obj/testdata/amd64-plan9-exec: exit status: 'nm.test 38338: 1'
        open /usr/local/go/src/debug/plan9obj/testdata/amd64-plan9-exec: '/usr/local' does not exist
    nm_test.go:80: go tool nm /usr/local/go/src/debug/plan9obj/testdata/386-plan9-exec: exit status: 'nm.test 38341: 1'
        open /usr/local/go/src/debug/plan9obj/testdata/386-plan9-exec: '/usr/local' does not exist
    nm_test.go:80: go tool nm /usr/local/go/src/internal/xcoff/testdata/gcc-ppc64-aix-dwarf2-exec: exit status: 'nm.test 38344: 1'
        open /usr/local/go/src/internal/xcoff/testdata/gcc-ppc64-aix-dwarf2-exec: '/usr/local' does not exist
FAIL
FAIL    cmd/nm  0.244s
ok      cmd/objdump 0.206s
ok      cmd/pack    0.177s
ok      cmd/pprof   0.255s
ok      cmd/relnote 0.154s
ok      cmd/trace   0.223s
ok      cmd/trace/v2    0.221s
ok      cmd/vet 0.303s
FAIL

One of the problems I have is that the test data file or directory location is not always deterministic, so we don't really know what should be copied on the remote host to be able to run the test.

0intro commented 3 weeks ago

I've realized that the go_$GOOS_$GOARCH_exec script is executed from each of the test directory, so we can retrieve the current working directory. This solves the test data issue.

Now, most of the tests are passing successfully. There are two remaining issues.

  1. The go/types and cmd/compile/internal/types2 tests expect the Go toolchain binaries to be present on the machine were the tests are executed.
ssh -p 2222 glenda@localhost /tmp/types.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
2024/05/21 16:45:42 main:2:8: could not import fmt (can't find import: "fmt": fork/exec /tmp/gotest/bin/go: '/tmp/gotest/bin/go' does not exist)
FAIL    go/types    0.485s
ssh -p 2222 glenda@localhost /tmp/types2.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
2024/05/21 16:50:43 main:2:8: could not import fmt (can't find import: "fmt": fork/exec /tmp/gotest/bin/go: '/tmp/gotest/bin/go' does not exist)
FAIL    cmd/compile/internal/types2 0.422s
  1. The TestTracebackSystem test expects to access the sources files from the absolute path on the machine on which the Go toolchain has been built.
ssh -p 2222 glenda@localhost /tmp/runtime.test -test.paniconexit0 -test.count 1 -test.timeout 9m0s -test.short true
--- FAIL: TestTracebackSystem (0.04s)
    traceback_system_test.go:114: stderr:
        Callers: [0x3b2bcf 0x3b2b66 0x3b2b1b 0x3b81ca 0x3b81c5 0x3b81c4 0x3b81c3 0x3b81c2 0x83931]
        panic: open /usr/local/go/src/runtime/traceback_system_test.go: '/usr/local' does not exist
0intro commented 2 weeks ago

Hi @dmitshur,

@millerresearch and I finally succeeded to run all tests on a remote Plan 9 machine from Linux, using a go_$GOOS_$GOARCH_exec script.

What is the next step to receive builds on the LUCI bot, for example plan9-386?

##### Test execution environment.
# GOARCH: 386
# CPU: QEMU Virtual CPU version 2.5+
# GOOS: plan9
# OS Version: 2000

##### Testing packages.
ok      archive/tar 26.857s
ok      archive/zip 19.933s
ok      bufio   19.461s
ok      bytes   20.347s
ok      cmp 14.258s
ok      compress/bzip2  11.098s
ok      compress/flate  21.189s
ok      compress/gzip   35.042s
ok      compress/lzw    13.900s
ok      compress/zlib   17.329s
ok      container/heap  14.175s
ok      container/list  14.406s
ok      container/ring  14.266s
ok      context 20.261s
ok      crypto  18.324s
ok      crypto/aes  13.502s
ok      crypto/cipher   16.156s
ok      crypto/des  13.194s
ok      crypto/dsa  14.896s
ok      crypto/ecdh 21.032s
ok      crypto/ecdsa    25.033s
ok      crypto/ed25519  17.302s
ok      crypto/elliptic 18.358s
ok      crypto/hmac 14.029s
ok      crypto/internal/alias   12.607s
ok      crypto/internal/bigmod  16.517s
ok      crypto/internal/boring  12.705s
ok      crypto/internal/boring/bcache   15.177s
ok      crypto/internal/edwards25519    17.482s
ok      crypto/internal/edwards25519/field  14.955s
ok      crypto/internal/hpke    14.798s
ok      crypto/internal/mlkem768    17.030s
ok      crypto/internal/nistec  20.657s
ok      crypto/internal/nistec/fiat 12.109s [no tests to run]
ok      crypto/md5  16.397s
ok      crypto/rand 15.468s
ok      crypto/rc4  12.910s
ok      crypto/rsa  23.734s
ok      crypto/sha1 15.148s
ok      crypto/sha256   15.467s
ok      crypto/sha512   14.653s
ok      crypto/subtle   16.188s
ok      crypto/tls  57.737s
ok      crypto/x509 48.663s
ok      database/sql    31.000s
ok      database/sql/driver 13.304s
ok      debug/buildinfo 18.417s
ok      debug/dwarf 17.190s
ok      debug/elf   20.040s
ok      debug/gosym 16.197s
ok      debug/macho 13.657s
ok      debug/pe    21.261s
ok      debug/plan9obj  13.122s
ok      embed   15.152s [no tests to run]
ok      embed/internal/embedtest    16.462s
ok      encoding/ascii85    13.215s
ok      encoding/asn1   18.056s
ok      encoding/base32 16.513s
ok      encoding/base64 15.210s
ok      encoding/binary 17.202s
ok      encoding/csv    18.412s
ok      encoding/gob    28.222s
ok      encoding/hex    14.567s
ok      encoding/json   43.914s
ok      encoding/pem    21.207s
ok      encoding/xml    22.529s
ok      errors  15.858s
ok      expvar  20.351s
ok      flag    19.004s
ok      fmt 20.885s
ok      go/ast  20.500s
ok      go/build    87.102s
ok      go/build/constraint 13.869s
ok      go/constant 17.510s
ok      go/doc  15.574s
ok      go/doc/comment  18.992s
ok      go/format   19.578s
ok      go/importer 19.980s
ok      go/internal/gccgoimporter   21.706s
ok      go/internal/gcimporter  22.117s
ok      go/internal/srcimporter 181.146s
ok      go/parser   12.415s
ok      go/printer  19.353s
ok      go/scanner  15.300s
ok      go/token    20.722s
ok      go/types    56.506s
ok      go/version  12.694s
ok      hash    15.305s
ok      hash/adler32    13.213s
ok      hash/crc32  15.411s
ok      hash/crc64  13.802s
ok      hash/fnv    12.913s
ok      hash/maphash    13.851s
ok      html    16.912s
ok      html/template   36.005s
ok      image   16.728s
ok      image/color 13.912s
ok      image/draw  18.167s
ok      image/gif   19.915s
ok      image/jpeg  19.859s
ok      image/png   21.120s
ok      index/suffixarray   17.556s
ok      internal/abi    15.114s
ok      internal/buildcfg   13.453s
ok      internal/chacha8rand    12.694s
ok      internal/concurrent 13.962s
ok      internal/coverage/cfile 14.778s
ok      internal/coverage/cformat   14.075s
ok      internal/coverage/cmerge    12.867s
ok      internal/coverage/pods  14.925s
ok      internal/coverage/slicereader   12.454s
ok      internal/coverage/slicewriter   12.961s
ok      internal/coverage/test  15.255s
ok      internal/cpu    13.447s
ok      internal/dag    13.625s
ok      internal/diff   13.215s
ok      internal/fmtsort    13.953s
ok      internal/fuzz   19.854s
ok      internal/godebug    14.606s
ok      internal/godebugs   14.715s
ok      internal/gover  12.764s
ok      internal/itoa   12.560s
ok      internal/platform   17.359s
ok      internal/poll   14.742s
ok      internal/profile    13.003s
ok      internal/reflectlite    16.793s
ok      internal/runtime/atomic 15.138s
ok      internal/saferio    14.977s
ok      internal/singleflight   12.708s
ok      internal/sysinfo    13.120s
ok      internal/testenv    14.693s
ok      internal/trace  22.580s
ok      internal/trace/internal/oldtrace    16.058s
ok      internal/types/errors   20.680s
ok      internal/unsafeheader   12.805s
ok      internal/weak   14.160s
ok      internal/xcoff  13.609s
ok      internal/zstd   19.823s
ok      io  18.744s
ok      io/fs   16.100s
ok      io/ioutil   17.810s
ok      iter    13.175s
ok      log 16.713s
ok      log/slog    31.815s
ok      log/slog/internal/benchmarks    14.257s
ok      log/slog/internal/buffer    13.013s
ok      maps    14.617s
ok      math    16.266s
ok      math/big    37.726s
ok      math/bits   15.721s
ok      math/cmplx  15.236s
ok      math/rand   23.625s
ok      math/rand/v2    24.381s
ok      mime    16.407s
ok      mime/multipart  22.123s
ok      mime/quotedprintable    17.781s
ok      net 52.135s
ok      net/http    234.664s
ok      net/http/cgi    36.081s
ok      net/http/cookiejar  30.626s
ok      net/http/fcgi   20.307s
ok      net/http/httptest   44.136s
ok      net/http/httptrace  17.389s
ok      net/http/httputil   53.158s
ok      net/http/internal   16.070s
ok      net/http/internal/ascii 12.700s
ok      net/http/pprof  43.825s
ok      net/internal/cgotest    12.708s
ok      net/mail    17.862s
ok      net/netip   24.957s
ok      net/rpc 41.945s
ok      net/rpc/jsonrpc 28.593s
ok      net/smtp    31.805s
ok      net/textproto   16.843s
ok      net/url 21.740s
ok      os  28.525s
ok      os/exec 31.105s
ok      os/exec/internal/fdtest 12.691s
ok      os/signal   13.719s
ok      os/user 13.514s
ok      path    14.604s
ok      path/filepath   18.894s
ok      plugin  12.705s
ok      reflect 39.806s
ok      regexp  21.934s
ok      regexp/syntax   17.658s
ok      runtime 89.929s
ok      runtime/debug   20.890s
ok      runtime/internal/math   12.494s
ok      runtime/internal/sys    12.716s
ok      runtime/internal/wasitest   15.041s
ok      runtime/metrics 16.052s
ok      runtime/pprof   51.264s
ok      runtime/trace   21.357s
ok      slices  19.777s
ok      sort    16.257s
ok      strconv 18.838s
ok      strings 20.799s
ok      sync    22.014s
ok      sync/atomic 19.580s
ok      syscall 13.355s
ok      testing 25.946s
ok      testing/fstest  16.897s
ok      testing/iotest  14.854s
ok      testing/quick   15.453s
ok      testing/slogtest    17.590s
ok      text/scanner    15.594s
ok      text/tabwriter  14.823s
ok      text/template   27.435s
ok      text/template/parse 16.385s
ok      time    57.464s
ok      unicode 15.071s
ok      unicode/utf16   13.459s
ok      unicode/utf8    14.663s
ok      unique  17.722s
ok      cmd/addr2line   24.305s
ok      cmd/api 20.799s
ok      cmd/asm/internal/asm    20.286s
ok      cmd/asm/internal/lex    15.089s
ok      cmd/cgo/internal/swig   13.855s
ok      cmd/cgo/internal/test   12.590s
ok      cmd/cgo/internal/testcshared    16.106s
ok      cmd/cgo/internal/testerrors 15.592s
ok      cmd/cgo/internal/testfortran    13.631s
ok      cmd/cgo/internal/testgodefs 14.858s
ok      cmd/cgo/internal/testlife   12.583s
ok      cmd/cgo/internal/testnocgo  12.653s
ok      cmd/cgo/internal/testplugin 14.110s
ok      cmd/cgo/internal/testsanitizers 12.312s [no tests to run]
ok      cmd/cgo/internal/testshared 17.599s
ok      cmd/cgo/internal/testso 15.370s
ok      cmd/cgo/internal/teststdio  12.510s
ok      cmd/cgo/internal/testtls    13.061s
ok      cmd/compile/internal/abt    13.440s
ok      cmd/compile/internal/amd64  29.045s
ok      cmd/compile/internal/base   19.046s
ok      cmd/compile/internal/compare    24.562s
ok      cmd/compile/internal/devirtualize   25.150s
ok      cmd/compile/internal/dwarfgen   34.248s
ok      cmd/compile/internal/importer   29.641s
ok      cmd/compile/internal/inline/inlheur 28.211s
ok      cmd/compile/internal/ir 23.560s
ok      cmd/compile/internal/liveness   27.871s
ok      cmd/compile/internal/logopt 16.100s
ok      cmd/compile/internal/loopvar    24.047s
ok      cmd/compile/internal/noder  30.532s
ok      cmd/compile/internal/rangefunc  23.316s
ok      cmd/compile/internal/reflectdata    22.811s [no tests to run]
ok      cmd/compile/internal/ssa    63.622s
ok      cmd/compile/internal/syntax 20.536s
ok      cmd/compile/internal/test   44.259s
ok      cmd/compile/internal/typecheck  24.457s
ok      cmd/compile/internal/types  19.342s
ok      cmd/compile/internal/types2 52.947s
ok      cmd/covdata 20.150s
ok      cmd/cover   27.545s
ok      cmd/dist    16.587s
ok      cmd/distpack    16.949s
ok      cmd/doc 36.613s
ok      cmd/fix 27.678s
ok      cmd/go  43.206s
ok      cmd/go/internal/auth    16.043s
ok      cmd/go/internal/cache   20.150s
ok      cmd/go/internal/cfg 14.356s [no tests to run]
ok      cmd/go/internal/fsys    20.581s
ok      cmd/go/internal/generate    32.038s
ok      cmd/go/internal/gover   15.568s
ok      cmd/go/internal/imports 16.262s
ok      cmd/go/internal/load    26.671s
ok      cmd/go/internal/lockedfile  17.793s
ok      cmd/go/internal/modfetch    37.304s
ok      cmd/go/internal/modfetch/codehost   35.629s
ok      cmd/go/internal/modfetch/zip_sum_test   26.861s
ok      cmd/go/internal/modindex    71.032s
ok      cmd/go/internal/modload 38.336s
ok      cmd/go/internal/mvs 16.667s
ok      cmd/go/internal/par 13.364s
ok      cmd/go/internal/str 14.154s
ok      cmd/go/internal/test    34.910s
ok      cmd/go/internal/toolchain   31.300s
ok      cmd/go/internal/vcs 27.064s
ok      cmd/go/internal/vcweb   32.042s
ok      cmd/go/internal/vcweb/vcstest   33.413s
ok      cmd/go/internal/web 22.206s
ok      cmd/go/internal/work    33.102s
ok      cmd/gofmt   18.753s
ok      cmd/internal/archive    13.542s
ok      cmd/internal/bootstrap_test 13.520s
ok      cmd/internal/buildid    16.776s
ok      cmd/internal/cov    14.618s
ok      cmd/internal/dwarf  15.905s
ok      cmd/internal/edit   12.749s
ok      cmd/internal/goobj  14.402s
ok      cmd/internal/moddeps    17.277s
ok      cmd/internal/notsha256  14.388s
ok      cmd/internal/obj    16.857s
ok      cmd/internal/obj/loong64    15.035s
ok      cmd/internal/obj/ppc64  18.101s
ok      cmd/internal/obj/riscv  16.338s
ok      cmd/internal/obj/s390x  14.308s
ok      cmd/internal/obj/x86    19.433s
ok      cmd/internal/objabi 14.487s
ok      cmd/internal/pgo    15.714s
ok      cmd/internal/pkgpath    14.516s
ok      cmd/internal/pkgpattern 13.804s
ok      cmd/internal/quoted 12.900s
ok      cmd/internal/src    13.351s
ok      cmd/internal/test2json  13.244s
ok      cmd/link    30.890s
ok      cmd/link/internal/benchmark 14.212s
ok      cmd/link/internal/ld    29.405s
ok      cmd/link/internal/loader    16.953s
ok      cmd/nm  33.944s
ok      cmd/objdump 28.518s
ok      cmd/pack    23.089s
ok      cmd/pprof   36.629s
ok      cmd/relnote 18.513s
ok      cmd/trace   23.768s
ok      cmd/vet 34.453s

##### os/user with tag osusergo
ok      os/user 13.522s

##### hash/maphash purego implementation
ok      hash/maphash    14.188s

##### crypto with tag purego
ok      crypto  0.072s
ok      crypto/aes  0.073s
ok      crypto/cipher   0.073s
ok      crypto/des  0.072s
ok      crypto/dsa  0.072s
ok      crypto/ecdh 0.072s
ok      crypto/ecdsa    0.072s
ok      crypto/ed25519  0.072s
ok      crypto/elliptic 0.075s
ok      crypto/hmac 0.071s
ok      crypto/internal/alias   0.073s
ok      crypto/internal/bigmod  0.072s
ok      crypto/internal/boring  0.072s
?       crypto/internal/boring/bbig [no test files]
ok      crypto/internal/boring/bcache   0.072s
?       crypto/internal/boring/sig  [no test files]
ok      crypto/internal/edwards25519    0.071s
ok      crypto/internal/edwards25519/field  0.072s
ok      crypto/internal/hpke    0.073s
ok      crypto/internal/mlkem768    0.071s
ok      crypto/internal/nistec  0.072s
ok      crypto/internal/nistec/fiat 0.072s
?       crypto/internal/randutil    [no test files]
ok      crypto/md5  0.073s
ok      crypto/rand 0.071s
ok      crypto/rc4  0.182s
ok      crypto/rsa  0.072s
ok      crypto/sha1 0.072s
ok      crypto/sha256   0.072s
ok      crypto/sha512   0.078s
ok      crypto/subtle   0.072s
ok      crypto/tls  0.073s
ok      crypto/x509 0.072s
?       crypto/x509/pkix    [no test files]

##### GOEXPERIMENT=rangefunc
ok      iter    13.064s
ok      slices  19.868s
ok      maps    14.614s

##### Testing without libgcc.
ok      net 52.157s
ok      os/user 15.028s

##### sync -cpu=10
ok      sync    22.105s

##### GOMAXPROCS=2 runtime -cpu=1 -quick
ok      runtime 64.892s

##### GOMAXPROCS=2 runtime -cpu=2 -quick
ok      runtime 64.991s

##### GOMAXPROCS=2 runtime -cpu=4 -quick
ok      runtime 91.586s

ALL TESTS PASSED
dmitshur commented 2 weeks ago

That's great to hear, thanks!

One idea I wanted to briefly mention about testdata: I wonder if could work well to rely on //go:embed testdata directives in _test.go files to embed testdata directory content into test binaries. It can be constrained to Plan 9 needs in _plan9_test.go files, or applied more generally if it's a pattern we're okay with using more broadly.

For next steps, using the current plan9-386 LUCI builder as the example, we'll need to update the builder definition at https://ci.chromium.org/ui/p/golang/builders/luci.golang.ci/gotip-plan9-386 to match your builder. If you follow the "Machine Pool" link there now, it filters by cipd_platform:plan9-386 pool:luci.golang.shared-workers (you'll need to login to view the page content) which currently matches no bots. Updating cipd_platform to linux-amd64 (since your builder is a Linux AMD64 host) matches the plan9-386 builder and more, see here. We'll need to think the dimensions to use long term, but maybe narrowing down by bot id can work initially:

https://chromium-swarm.appspot.com/botlist?f=cipd_platform%3Alinux-amd64&f=pool%3Aluci.golang.shared-workers&f=id%3Aplan9-386

Sent CL 588756.

gopherbot commented 2 weeks ago

Change https://go.dev/cl/588756 mentions this issue: main.star: use id dimension to select Plan 9 builders

dmitshur commented 2 weeks ago

@0intro https://chromium-swarm.appspot.com/task?id=69d9f835adb76f10 was a recent build after the aforementioned CL:

15819 2024-05-28 21:41:44.550 E: Failed to run run_isolated
Traceback (most recent call last):
  File "/home/swarming/.swarming/swarming_bot.1.zip/__main__.py", line 324, in main
    return fn(args)
           ^^^^^^^^
  File "/home/swarming/.swarming/swarming_bot.1.zip/__main__.py", line 142, in CMDrun_isolated
    import run_isolated
  File "/home/swarming/.swarming/swarming_bot.1.zip/client/run_isolated.py", line 53, in <module>
    import distutils
ModuleNotFoundError: No module named 'distutils'
millerresearch commented 2 weeks ago

One idea I wanted to briefly mention about testdata: I wonder if could work well to rely on //go:embed testdata directives in _test.go files to embed testdata directory content into test binaries. It can be constrained to Plan 9 needs in _plan9_test.go files, or applied more generally if it's a pattern we're okay with using more broadly.

Our plan is to have one filesystem shared via 9p between the swarming-bot proxy and the plan9 test [virtual] machine, so the source and test files will be visible with the same names (presumably under /home/swarming/.swarming/c). Will these mean it's not necessary to embed testdata?

millerresearch commented 2 weeks ago

A plan9-arm builder is now running. Would you like to try sending it a task?

This builder has python 3.11 so I think the missing distutils problem will not occur here. It seems python 3.12 has removed the distutils module from the standard library?

millerresearch commented 2 weeks ago

The first task seems to have got as far as building go for linux-amd64 and plan9-arm, then failed in the "update prebuilt go / cas archive" stage:

[E2024-05-29T15:05:21.028203+01:00 29195 0 annotate.go:273] original error: retry budget exhausted (6 attempts): context deadline exceeded

goroutine 1:
#0 go.chromium.org/luci/client/cmd/cas/casimpl/archive.go:169 - casimpl.(*archiveRun).doArchive()
  reason: failed to call UploadIfMissing

#1 go.chromium.org/luci/client/cmd/cas/casimpl/archive.go:221 - casimpl.(*archiveRun).Run()
#2 github.com/maruel/subcommands@v1.1.1/subcommands.go:395 - subcommands.Run()
#3 cas/main.go:95 - main.main()
#4 runtime/proc.go:271 - runtime.main()
#5 runtime/asm_amd64.s:1695 - runtime.goexit()
cas: failed to call UploadIfMissing: retry budget exhausted (6 attempts): context deadline exceeded
0intro commented 2 weeks ago

I've just installed the Python distutils package.

One idea I wanted to briefly mention about testdata: I wonder if could work well to rely on //go:embed testdata directives in _test.go files to embed testdata directory content into test binaries. It can be constrained to Plan 9 needs in _plan9_test.go files, or applied more generally if it's a pattern we're okay with using more broadly.

Our plan is to have one filesystem shared via 9p between the swarming-bot proxy and the plan9 test [virtual] machine, so the source and test files will be visible with the same names (presumably under /home/swarming/.swarming/c). Will these mean it's not necessary to embed testdata?

That said, if we had the possibility to run Go tests without access to the full sources tree, it would be great.

dmitshur commented 2 weeks ago

Sharing a filesystem seems good if it works well. Embedding is just an alternative path. It would make the test binaries larger and may require them to write out the embedded data to a temporary directory, so the main advantage is that it doesn't require copying files via a separate mechanism, and that it might make it easier to avoid copying unnecessary files.

That "update prebuilt go / cas archive" build step ran for 6.2 minutes, which is quite long to upload a pre-built Go toolchain. Is the uplink on that machine expected to be slow? Let's also wait and see if the 6.2 minute time keeps happening more than just once.

millerresearch commented 2 weeks ago

That "update prebuilt go / cas archive" build step ran for 6.2 minutes, which is quite long to upload a pre-built Go toolchain. Is the uplink on that machine expected to be slow?

The link is nominally 20Mbit/sec upstream.

dmitshur commented 2 weeks ago

In b8746572283039962513, the 'upload prebuilt go' step took 54 seconds. A pre-built toolchain is around 150 MB compressed, so that seems to line up with 20 Mbit/s.

The build ran for 57 minutes. 27020 tests passed, 74 failed. It's great to see this progress. It seems we should expect these builders to be slow—please feel free add them here so that they get increased timeouts.

millerresearch commented 2 weeks ago

Today's tests have been failing because I restarted the bot and accidentally didn't have go_plan9_arm_exec in the PATH, so it was trying to run arm test binaries on amd64. Fixed now.

With the old build infrastructure we could use x/build/cmd/retrybuilds to re-run a failed test. Does LUCI have a way for us to do that?

dmitshur commented 2 weeks ago

We haven't updated retrybuilds for LUCI, but may do so in the future. So far it hasn't been a high priority because it's generally okay to wait for future commits to come in and be built. In LUCI UI, on the Infra tab there is a "Retry Build" action available (for authorized users) that acts similarly, though it causes a new build to be scheduled without making the old one unavailable.

0intro commented 1 week ago

Is there a way with the Swarming Bot to execute a script before or after running a task? It could be helpful, for example, to reboot the remote Plan 9 VM after running tests.