Closed tklauser closed 4 years ago
See the link here:
https://paste.sr.ht/~sircmpwn/3a76f2ee0b914eccf2cc70fc6ca6b7a75de547a3
I'm on a real riscv64 board, HiFive Unleashed. Kernel is 4.20-rc4 plus this patch:
And this config:
https://paste.sr.ht/~sircmpwn/88c5f57715b904896c5cbe57c5d6674f738d0e51
Ah, real hardware. Let me lift the exact CAS code from go (it does some extra looping in asm) and give you a new thing to test. Also, can you make threads like 100 and add one or two zeroes to the iterations? Real hardware may have to be beaten harder.
Can you try the other 2 tests as well that I linked earlier?
@marcopeereboom On Qemu 4.0.0:
Test1:
# ./a.out
in thread 10
in thread 11
in thread 17
in thread 12
in thread 19
in thread 14
in thread 24
in thread 21
in thread 15
in thread 16
in thread 18
in thread 13
in thread 20
in thread 27
in thread 25
in thread 22
in thread 23
in thread 29
in thread 28
in thread 26
womp womp 1999992 2000000
Test2:
# ./a.out
ok 400000 400000
Test3:
# ./a.out
ok 400000 400000
@marcopeereboom On Qemu 4.0.0:
Test1:
# ./a.out in thread 10 in thread 11 in thread 17 in thread 12 in thread 19 in thread 14 in thread 24 in thread 21 in thread 15 in thread 16 in thread 18 in thread 13 in thread 20 in thread 27 in thread 25 in thread 22 in thread 23 in thread 29 in thread 28 in thread 26 womp womp 1999992 2000000
Test2:
# ./a.out ok 400000 400000
Test3:
# ./a.out ok 400000 400000
Do run it several times or even in a tight loop. Also, play with the #defines a bit.
Also, can you make threads like 100 and add one or two zeroes to the iterations?
I gave you 50 threads and an extra zero, this hardware isn't the fastest and I am not made of cycles :wink:
ok 50000000 50000000
https://gist.github.com/marcopeereboom/1e40d4baffdcc9a2066310d770f5ac12
https://gist.github.com/marcopeereboom/4357c59b57dc998a58d37817d59f99b0
Your includes are missing the headers you're trying to include here
Would you like shell access to this machine?
@marcopeereboom ssh -p 24169 marco@home.drewdevault.com
Would you like shell access to this machine?
Yes, please!
Mail me marco @ peereboom.us
and I'll reply with an ssh pubkey?
Mail me marco @ peereboom.us and I'll reply with an ssh pubkey?
Try connecting -- maybe he already got your pubkeys through GitHub.
@marcopeereboom Yep, lots of womp womps with more threads in Qemu 4.0.0 on Test2:
womp womp 2251433 2500000
womp womp 2357934 2500000
womp womp 1890639 2500000
womp womp 1926171 2500000
womp womp 1886507 2500000
womp womp 1948392 2500000
womp womp 2338358 2500000
womp womp 2363039 2500000
womp womp 2066714 2500000
womp womp 2315551 2500000
womp womp 1704307 2500000
womp womp 1901324 2500000
womp womp 1656480 2500000
womp womp 2085709 2500000
womp womp 2257562 2500000
womp womp 2347022 2500000
womp womp 2187506 2500000
womp womp 2172819 2500000
womp womp 1957879 2500000
Also fails with test 3 with 40 threads.
All goes fine in my real boards, both x64 and ARM.
Try connecting -- maybe he already got your pubkeys through GitHub.
Yep - you should be all set @marcopeereboom
I've successfully built gojsontoyaml project for testing Go 1.12 on Risc-V Qemu. On some actions like go get .
I had to run with GOGC=off or I would get errors (listed below).
[root@fedora-riscv gojsontoyaml]# go version
go version go1.12 linux/riscv64
[root@fedora-riscv gojsontoyaml]#
[root@fedora-riscv gojsontoyaml]# which go
/root/riscv-go/bin/go
If I just do go get .
, I get the following error:
But disabling GC:
[root@fedora-riscv gojsontoyaml]# GOGC=off go get .
go: finding github.com/ghodss/yaml v1.0.0
go: finding gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
go: downloading github.com/ghodss/yaml v1.0.0
go: extracting github.com/ghodss/yaml v1.0.0
go: downloading gopkg.in/yaml.v2 v2.1.1
go: extracting gopkg.in/yaml.v2 v2.1.1
[root@fedora-riscv gojsontoyaml]# go build .
[root@fedora-riscv gojsontoyaml]# ls
gojsontoyaml go.mod Gopkg.lock Gopkg.toml go.sum LICENSE main.go README.md vendor
And it runs fine:
[root@fedora-riscv gojsontoyaml]# echo "{a: 1, b: {b1: 1, b2: 2}}" | ./gojsontoyaml
a: 1
b:
b1: 1
b2: 2
Try connecting -- maybe he already got your pubkeys through GitHub.
Yep - you should be all set @marcopeereboom
Still asks for a password.
I am using this key:
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL1c5HWmHG/Ne4tBuVb8POnlAbm/gZKD6lb7GPS0hMAi
Try again now.
Still asks for a password. I added an ssh config to explicitly only use that identity.
Are you on irc or keybase so that we don't pollute this issue with this anymore?
Oh, sorry, my sshd doesn't have ed25519 support. Do you have an RSA key? I'm ddevault on freenode.
CC @palmer-dabbelt @jim-wilson @alistair23
If there are issues with QEMU, kernel, glibc or toolchain these people (i.e. maintainers) are the best persons help out.
Yeah, let me know if you see an issue with RISC-V QEMU and I can investigate.
@alistair23 try running the following under RISC-V QEMU:
https://gist.github.com/4a6f656c/8433032a3f70893a278259f8108aad90
https://gist.github.com/4a6f656c/d883091f5ca811822720213be343a75a
They may need to be run a few times, but seem to eventually fail under qemu, yet I'm told they work correctly under real hardware.
I tried this, and wasn't able to reproduce the problem. I tried hardware, system qemu, and user qemu, and they all worked. But maybe you have a different version of something, or used different compiler options, etc.
@jim-wilson did you use bbl or opensbi?
Can you also paste the versions of various things you tried?
I used a Fedora Core 30 system running on hardware to compile the two testcases. I tried the binaries on hardware and they worked. I tried them with both user and system qemu, using both a riscv-qemu and an upstream qemu.org qemu, and it worked for me in all four cases.
FC30 uses bbl.
@jim-wilson have you tried increasing the TCOUNT
variable for more threads? Sometimes on Qemu with 4 threads it work but with something like 40 threads it eats up data. On hardware it should work perfectly.
Nothing in this bug report mentions TCOUNT. I did try increasing it to 40, and now it is failing using system qemu with both an old riscv-qemu and a top of tree qemu.org. It still works on my HiFive Unleashed board though. And it works with a user mode qemu, both old and new, though this is using an older glibc.
So maybe a linux kernel bug or a system qemu bug, and I'm not an expert with either one of those.
@marcopeereboom mentioned up there to play with threads and iterations, that's what TCOUNT does. It's apparently a Qemu bug. I'm just clearing some stuff up and will post this into Qemu mailing list so @alistair23 and @palmer-dabbelt can help checking this out.
Yeah we are really leaning toward a qemu-system bug. I'll try to write up an issue but they use a silly bug tracker.
@marcopeereboom, Palmer pointed out to write to the mailing list at qemu-riscv@nongnu.org (https://lists.nongnu.org/mailman/listinfo/qemu-riscv) describing the bug.
We seem to have liftoff 🚀Couple of failing tests but nothing of real interest and it seems relatively simple to fix.
Latest commits seem to fix the majority of the problems and I am running relatively large go programs in qemu-system-riscv.
Can you guys test and provide some feedback?
I still see some problems related to GC, now on the Unleashed board. I rebuilt my bootstrap on my Mac, transferred it to the board and checked-out the latest tree.
I tried to build go-jsonnet and got this error:
root@buildroot:~/go-jsonnet# git clone https://github.com/google/go-jsonnet --depth=1
root@buildroot:~/go-jsonnet# cd go-jsonnet
root@buildroot:~/go-jsonnet# go build ./cmd/jsonnet/
go: finding github.com/mattn/go-isatty v0.0.7
go: finding github.com/fatih/color v1.7.0
go: finding github.com/sergi/go-diff v1.0.0
go: finding github.com/stretchr/testify v1.3.0
go: finding github.com/mattn/go-colorable v0.1.1
go: finding github.com/mattn/go-isatty v0.0.5
go: finding golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223
go: finding github.com/pmezard/go-difflib v1.0.0
go: finding github.com/stretchr/objx v0.1.0
go: finding github.com/davecgh/go-spew v1.1.0
go: downloading github.com/fatih/color v1.7.0
go: extracting github.com/fatih/color v1.7.0
go: downloading github.com/mattn/go-isatty v0.0.7
go: downloading github.com/mattn/go-colorable v0.1.1
go: extracting github.com/mattn/go-isatty v0.0.7
go: downloading golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223
go: extracting github.com/mattn/go-colorable v0.1.1
go: extracting golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223
# golang.org/x/sys/unix
../go/pkg/mod/golang.org/x/sys@v0.0.0-20190222072716-a9d3bda3a223/unix/syscall_linux_gc.go:10:6: missing function body
../go/pkg/mod/golang.org/x/sys@v0.0.0-20190222072716-a9d3bda3a223/unix/syscall_linux_gc.go:14:6: missing function body
../go/pkg/mod/golang.org/x/sys@v0.0.0-20190222072716-a9d3bda3a223/unix/syscall_unix_gc.go:12:6: missing function body
../go/pkg/mod/golang.org/x/sys@v0.0.0-20190222072716-a9d3bda3a223/unix/syscall_unix_gc.go:13:6: missing function body
../go/pkg/mod/golang.org/x/sys@v0.0.0-20190222072716-a9d3bda3a223/unix/syscall_unix_gc.go:14:6: missing function body
../go/pkg/mod/golang.org/x/sys@v0.0.0-20190222072716-a9d3bda3a223/unix/syscall_unix_gc.go:15:6: missing function body
root@buildroot:~/go-jsonnet#
root@buildroot:~/go-jsonnet# uname -a
Linux buildroot 4.15.0-00044-g2b0aa1d #1 SMP Tue Mar 20 12:18:35 PDT 2018 riscv64 GNU/Linux
root@buildroot:~/go-jsonnet#
root@buildroot:~/go-jsonnet# go version
go version go1.12 linux/riscv64
Same error happened building echo.
I'm using a Debian rootfs. Go was built with:
root@buildroot:~/riscv-go# export GOROOT_BOOTSTRAP=/root/go-linux-riscv64-bootstrap
root@buildroot:~/riscv-go# export PATH="$(pwd)/misc/riscv:$(pwd)/bin:$PATH"
root@buildroot:~/riscv-go# cd src/
root@buildroot:~/riscv-go/src# ./make.bash
Building Go cmd/dist using /root/go-linux-riscv64-bootstrap.
Building Go toolchain1 using /root/go-linux-riscv64-bootstrap.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for linux/riscv64.
---
Installed Go for linux/riscv64 in /root/riscv-go
Installed commands in /root/riscv-go/bin
So that error is outside of go. I saw it on bolt db. That tree is not locked so we'll try to get that updated sooner rather than later.
Quick status update.
There are a couple of compiler/SSA issues that still need to be tracked down and maybe some additional polish but that is about it. We should be pretty healthy here in the next few weeks and therefore the second the tree opens we can create a pull request.
What this code does NOT do is 32 bit riscv. The code differences should be pretty trivial but someone has to do the work. So if you want to help and care about 32 bit, that is the next task at hand.
Quick status update.
I'm very excited about this work. Thanks for doing it.
There are a couple of compiler/SSA issues that still need to be tracked down and maybe some additional polish but that is about it. We should be pretty healthy here in the next few weeks and therefore the second the tree opens we can create a pull request.
Please do create a pull request as soon as the port is ready so it can be reviewed and ready to merge when the tree opens. Review of a new port takes considerable time and will probably result in changes that have to be integrated before merging.
# golang.org/x/sys/unix ../go/pkg/mod/golang.org/x/sys@v0.0.0-20190222072716-a9d3bda3a223/unix/syscall_linux_gc.go:10:6: missing function body ../go/pkg/mod/golang.org/x/sys@v0.0.0-20190222072716-a9d3bda3a223/unix/syscall_linux_gc.go:14:6: missing function body ../go/pkg/mod/golang.org/x/sys@v0.0.0-20190222072716-a9d3bda3a223/unix/syscall_unix_gc.go:12:6: missing function body ../go/pkg/mod/golang.org/x/sys@v0.0.0-20190222072716-a9d3bda3a223/unix/syscall_unix_gc.go:13:6: missing function body ../go/pkg/mod/golang.org/x/sys@v0.0.0-20190222072716-a9d3bda3a223/unix/syscall_unix_gc.go:14:6: missing function body ../go/pkg/mod/golang.org/x/sys@v0.0.0-20190222072716-a9d3bda3a223/unix/syscall_unix_gc.go:15:6: missing function body```
This is due to missing assembly syscall wrappers for linux/riscv64
in x/sys/unix
. We didn't add these yet because the assembly syntax for riscv64
is not yet considered final, please see the discussion on https://golang.org/cl/157901. I abandoned the CL for the time being, but once the riscv64
port is submitted for review and the assembly syntax deemed stable we can resurrect said CL.
That repo is open so if you feel inclined :-)
I sent a report to the QEMU list. http://lists.nongnu.org/archive/html/qemu-riscv/2019-05/msg00021.html
@tklauser I see the assembly syscalls on https://github.com/4a6f656c/riscv-go/blob/riscvdev/src/cmd/vendor/golang.org/x/sys/unix/asm_linux_riscv64.s and https://github.com/4a6f656c/riscv-go/blob/riscvdev/src/syscall/asm_linux_riscv64.s. Is there anything else missing?
That code uses mod.
@marcopeereboom What do you mean by "mod"?
@carlosedp The code that you are building, that fails, is using modules, which means that it is downloading the golang.org/x/sys/unix package to build from the Internet or from some module cache. It is not using your riscv64-specific variant of golang.org/x/sys/unix.
I've build the most recent x/sys
package with Risc-V ASM instructions, replaced it on my repo and it successfully tested and build the application I was seeing the GC problems previously.
The merge is on https://github.com/carlosedp/sys/tree/riscv64
I've cloned it into my $GOPATH/src/golang.org/x/sys
and copied it over to '$GOPATH/pkg/mod/golang.org/x/sys....' where go modules are downloaded now.
$ GOOS=linux GOARCH=riscv64 go test
PASS
ok golang.org/x/sys/unix 0.658s
An application like Go-Jsonnet that was previously failing, now builds and runs:
$ go build ./cmd/jsonnet/
$ cd cmd/jsonnet
$ go test
PASS
ok github.com/google/go-jsonnet/cmd/jsonnet 0.015s
$ ./jsonnet -e "local a = [1,2]; {a: a}"
{
"a": [
1,
2
]
}
This is on the Unleashed with these specs:
carlosedp@HiFiveU:~/work $ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
carlosedp@HiFiveU:~/work $ uname -a
Linux HiFiveU 4.19.0-sifive-1+ #9 SMP Thu May 16 12:46:59 -03 2019 riscv64 GNU/Linux
Change https://golang.org/cl/177799 mentions this issue: unix: provide linux/riscv64 assembly for syscalls
I've successfully built Prometheus on Unleashed.
Had to add support to x/net module. Already submitted to Gerrit on: https://go-review.googlesource.com/c/net/+/177997
The Prometheus change is on my repo (https://github.com/carlosedp/prometheus/tree/riscv64). I also had to link to the RiscV x/sys package.
Other than that, it built fine.
All tests should now pass.
Change https://golang.org/cl/177997 mentions this issue: ipv4, ipv6, internal/socket: add riscv64 support
I'm still seeing failed tests on runtime
. cmd/go
is failing with timeout, even increasing it, it fails. Details as seen below:
Test run:
cmd/go test:
@carlosedp all of the standard library tests pass here (there are around three in test/
that are still failing), however you'll probably need to bump up GO_TEST_TIMEOUT_SCALE
, particularly if you're running on a slow I/O device:
GOROOT_BOOTSTRAP=... GO_TEST_TIMEOUT_SCALE=5 ./all.bash
Thanks for the tip on the variable @4a6f656c . After I adjusted it and fixed my NFS filesystem server (had lock errors), it ran 100% (with the exceptions on test/
like you mentioned.
One question though, how hard it to implement the required functions for CGO on src/runtime/cgo
? A build I tried complained about not finding _cgo_sys_thread_start
and it's implemented on the ARCH specific files. Any tips?
One question though, how hard it to implement the required functions for CGO on src/runtime/cgo? A build I tried complained about not finding _cgo_sys_thread_start and it's implemented on the ARCH specific files. Any tips?
You can probably just copy runtime/cgo/gcc_linux_amd64.c to runtime/gcc_linux_riscv64.c. Take a look at it, but I don't see any obvious reason that wouldn't work. (Yes, this should be consolidated somewhat.)
The more interesting issue is runtime/cgo/asm_riscv64.s. That too should be pretty simple, but it will have to be written. See the existing files.
I've just pushed up a branch that is merged to Go tip at 2d7cb295 and contains a variety of fixes. All tests now pass:
$ GOROOT_BOOTSTRAP=~/riscv-go2 GO_TEST_TIMEOUT_SCALE=5 ./all.bash
Building Go cmd/dist using /home/joel/riscv-go2.
Building Go toolchain1 using /home/joel/riscv-go2.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for linux/riscv64.
##### Testing packages.
...
ALL TESTS PASSED
---
Installed Go for linux/riscv64 in /home/joel/riscv-go
Installed commands in /home/joel/riscv-go/bin
Awesome @4a6f656c , I'm working on the CGO support. I believe I'm close.
Can anyone recommend Hardware ? Sorry to ask in this thread but we are building golang based systems for Government, and the RISC-V ISA is perfect because its secure, UNLIKE intel, arm etc. We want to do some prototype testing and hook it up to CI process with other embedded boards.
Its fantastic btw that the golang team are supporting RISC-V
This issue serves to track the port to the RISC-V architecture. There is an out-of-tree port at https://github.com/riscv/riscv-go based on Go 1.8 which - according to riscv/riscv-go#19 - is no longer maintained and would need quite some work to be updated to the current Go tip.
Also see https://golang.org/cl/106256#message-2d9a5c5b89ad55b8b7999f794983f993649232c8 and https://groups.google.com/forum/#!searchin/golang-dev/RISC%7Csort:date/golang-dev/VpsyGdi-sQQ/FMu6IB_2CwAJ where @josharian summarized the current state of the existing port.
The
GOARCH
valuesriscv
andriscv64
were reserved in https://golang.org/cl/106256. These values are already used by gccgo. Additional changes were made todebug/elf
(https://golang.org/cl/107339),cmd/cgo
(https://golang.org/cl/110066),cmd/dist
andcmd/types
(https://golang.org/cl/118618) in order be able to generate type definition files in thex/sys/unix
package (https://golang.org/cl/133735)./cc @bradfitz @ianlancetaylor @josharian