Support GOARCH=mips64

[clktmr]

Hi there, I started to play around a bit and am at a point where emgo generates an ELF file for MIPS. My ultimate goal would be running Go on a Nintendo64. Am I right with the assumption that I need to implement:

• tasker_noos_mips.s
• rt0_noos_mips.s
• a board configuration

and I am basically ready to go? I only have basic understanding of writing assembly and having a hard time figuring out how to implement the tasker routines. Can you give me a starting point? Can I find MIPS assembly that does something similar somewhere?

[michalderkacz]

Hello Timur!

Sorry for the long time without any response from me. I was involved in a completely different project and now returned to Embedded Go.

I'm excited that someone has come that far. I need some time to review you code. Fill free to ask here any questions related to your work.

[clktmr]

Hi Michal! Thanks for taking a look at this! Still working on this. It's usable but needs cleanup. Some code is N64 specific. But you probably already noticed, that most of this is from your riscv implementation.

Hopefully I get some time to spend in this in the holidays. In general, would you want to merge mips support?

[michalderkacz]

I'd love to see nnos/mips64 in Embedded Go. Maybe I would wonder if we already had support for many different architectures but at the current stage of this project, a third one, although a bit exotic, should probably improve many things as the noos/riscv64 did.

I've just skimmed through your commits, so I don't have the full picture. I've just learned a bit about Nintendo 64 architecture. It's a bit ancient and you probably need to extend the GOMIPS environment variable to support it without sacrifice the linux/mips64 target.

One of the main assumptions is that the embedded branch should work as much well as the genuine Go for any supported GOOS/GOARCH so it must pass all tests on Linux running on qemu-mips.

Do you have any experiences with the currently supported MCUs and development boards? I mean the ability to test changes made to the common code.

[embeddedgo]

Well, I fell down the N64 rabbit hole!

Just bought the 1996 console, with the expansion pak and one original game. Plus, the Super 64 cartridge with the compact flash full of games headed my way...

[clktmr]

:laughing: It's a rabbit hole indeed. Not sure if you can run ROMs generated from this branch without modifications. I had some code that was specific to the EverDrive64 USB, but now it should correctly probe for available hardware. Let me know if you want help in generating or running ROMs. I recommend Ares emulator for testing and also the N64brew Discord is very active.

Regarding the other supported MCUs; I don't have any of these but might just get a kendryte for testing common code. I have some general feedback and recommendations that I would like to bring up.

[michalderkacz]

For now, I'm just studying the N64 architecture and some concepts related to it, communication over pad ports, etc. When the Super 64 cartridge arrives, I will be able to run n64 / z64 binaries. I plan to disassemble it and check if there are any remnants of the USB port that the ED64 PLUS had (though the FT245R was unpopulated).

What surprised me is the lack of any game that uses a light gun. I'm curious if the architecture allows to write a code that will support the old-style light guns, based on the CRT vertical and horizontal refresh. It would be great to make a light-gun hardware (or adapt an easily accessible one) and write a simple light-gun game, especially in Go :)

The initial challenge may be to implement a simple light pen hardware (fotodiode/fototransistor and simple circuit that generates proper pulses over the pad port) and try to detect its position on the screen. ​I'm curious if this is possible.

If I may suggest an MCU for testing the code, the Teensy 4.1 will be the best one. I'm slowly adding support for more it's peripherals. For now GPIO, DMA, USB and UART are ready to use. And now I'm working on SPI. By the way, when SPI will be ready, the Teensy + FTDI EVE display combo will be similar in some way to the N64 arch (separate display processor, display lists, etc.)

Kendryte is a bit of an exotic board, although it has its advantages, it also has many drawbacks, especially when it comes to documentation and debugging capabilities (maybe something has improved recently).

[michalderkacz]

What I can try to do for your n64 port is to try to add support for GOOUT=n64 / z64 to the emgo tool. Just need to learn something about these formats. The last version of emgo got native support for bin and hex without using objdump.

[clktmr]

I rebased this on the wip branch for reviewing, but I think we should wait until wip is merged. I will then rebase again for the merge.

What I have currently on my mind:

• I have moved all n64 specific code to target_noos_n64.*. These files have a dependency to the n64 module, which provides target specific implementations. I'm not sure if this is a good approach, but it works. Alternatively we could probably provide default implementations for the mips64 arch and overrride them with the go:linkname pragme (not tested). That would hide the dependency somewhat and remove "n64" completely from the runtime, but also disable compiler type checking.
• Nested interrupts should now work. But I probably need to write some targeted test for this. What I'm not sure about are the interrupt priorities, which don't exist in mips. I will just allow every interrupt to preempt another interrupt, as long as it is not already pending. Does that make sense?
• Is it considered allowed to use FPU in interrupthandlers by the user? I assumed no and hence I never save fprs.
• I didn't implement some syscalls. For example cachemaint or setprivlevel. Currently all code runs in kernel mode, with exception level the only exception. Do the other targets make use of user mode? Why are these syscalls needed?
• I haven't protected the signal stack, but intend to do so if I find some more time to spend on this.
• I also haven't used the GOMIPS variable to define the ISA level. However this changes (the first few commits) don't break MIPS for other targets, it just makes is slighty slower by removing some instructios that aren't in MIPS-III. There is an ongoing issue at Go to allow specifying instruction set via GOMIPS, so I would rather wait on that: https://github.com/golang/go/issues/60072

[michalderkacz]

I rebased this on the wip branch for reviewing, but I think we should wait until wip is merged. I will then rebase again for the merge.

I've finished preparing master-embedded and release-branch.go1.22-embedded two days ago. For now wip and 1.22-embedded are identical, that is git diff release-branch.go1.22-embedded wip returns nothing.

Wip is a good place for testing and reviewing things so for now i'll merge it here and will try use it to play with my N64.

* I have moved all n64 specific code to target_noos_n64.*. These files have a dependency to the n64 module, which provides target specific implementations. I'm not sure if this is a good approach, but it works. Alternatively we could probably provide default implementations for the mips64 arch and overrride them with the go:linkname pragme (not tested). That would hide the dependency somewhat and remove "n64" completely from the runtime, but also disable compiler type checking.

I'm very busy until Sunday but I'll try to look at it in a free time, probably not until Sunday.

* Nested interrupts should now work. But I probably need to write some targeted test for this. What I'm not sure about are the interrupt priorities, which don't exist in mips. I will just allow every interrupt to preempt another interrupt, as long as it is not already pending. Does that make sense?

I don't known anything about the MIPS interrupts. Is there any standard interrupt controller for it or the N64 specific one?

I'll tell you what it's like with the RISCV+CLINT+PLIC combo because its probably most similar to MIPS.

In contrast to the ARMv7-M, the RISCV+CLINT and probably MIPS too enters the handler with interrupts disabled. There are some default priorities in the CLINT but they are relevant only when two or more interrupts/exceptions will occur at exactly the same time.

So at the beginning of the main handler your goal is to as fast as possible enable interrupts again to minimize the latency for the higher priority interrupts. In case of RISCV I implemented my own priority levels different from the CLINT ones by simply disabling the the same and lower priority exceptions/interrupts (lower in my point of view) before enabling interrupts again. Of course, you should do all bookkeeping required to enable interrupts safely, so the higher priority interrupt can safely enter its handler.

If an interrupt is the RISCV external interrupt I rely on the priorities configured by the user in the PLIC. User does it using the rtos.IRQ.Enable function. Generally the rtos.IRQ type is for user interrupts only, typically external interrupts. You shouldn't expose here the interrupts/exceptions used only by runtime (e.g. syscall).

When it comes to testing, try find a way to rise a higher priority interrupt in the handler of the lower priority one and vice versa. In case of a typical MCU it's easy, because you can easily connect a GPIO pin to some interrupt input. Sometimes there may be a pure software way to do it. Here is how I did it for K210.

* Is it considered allowed to use FPU in interrupthandlers by the user? I assumed no and hence I never save fprs.

Yes, It is. User can use any Go in the interrupt handler, provided it doesn't allocate memory.

Perhaps what confused you was the fact that the tasker itself must not use FPU. It's required to allow cheap context switching, e.g. you may check is it required to save/restore FPU context during switch.

* I didn't implement some syscalls. For example cachemaint or setprivlevel. Currently all code runs in kernel mode, with exception level the only exception. Do the other targets make use of user mode? Why are these syscalls needed?

Both noos/thumb and noos/riscv64 run user code in the user mode. It makes clear demarcation line between the user and the handler code. It's also required to use MPU (memory protection unit) in the ARMv7-M. But I can't give a strict justification for requiring this.

You probably should implement cache maintenance operations if you use DMA. Without it you can't be sure that all data are in RAM before starting a DMA transaction or you read the content of cache instead of the RAM modified by DMA.

* I haven't protected the signal stack, but intend to do so if I find some more time to spend on this.

Ok. It's not required so much because of quite large RAM in N64. I wrote you some nonsense about using user/handler mode to protect the handler stack in the handler mode. Sometimes I write faster than I think. Forget about them. The TLB is probably the way to go.

* I also haven't used the GOMIPS variable to define the instruction set. However this changes (the first few commits) don't break MIPS for other targets, it just makes is slighty slower by removing some instructios that aren't in MIPS-III. There is an ongoing issue at Go to allow specifying instruction set via GOMIPS, so I would rather wait on that: [cmd/go: add GOMIPS32, GOMIPS64 ISA levels (iii, r1, r2, r5, r6) golang/go#60072](https://github.com/golang/go/issues/60072)

Ok. Did you run ./all.bash on any MIPS64 linux system?

[clktmr]

In case of RISCV I implemented my own priority levels different from the CLINT ones by simply disabling the the same and lower priority exceptions/interrupts (lower in my point of view) before enabling interrupts again.

In MIPS (on the VR4300) there are two software interrupts, which can be triggered by writing to a register. I use one of them to signal newwork(), and consider the other one unused and not exposed to the user. There is one timer interrupt, also unused as of now and considered not exposed to the user.

Then there are the external interrupts, which I do consider exposed to the user. Only these support nesting. Currently I will only mask all pending interrupts, then reenable them. I didn't see rtos.IRQ.Enable. From what I read I need to implement sysirqctl for mips64 to support priorities correctly.

Yes, It is. User can use any Go in the interrupt handler, provided it doesn't allocate memory.

Then I should also add a call to savefprs in the externalInterruptHandler. It's already implemented, should be a quick fix.

You probably should implement cache maintenance operations if you use DMA. Without it you can't be sure that all data are in RAM before starting a DMA transaction or you read the content of cache instead of the RAM modified by DMA.

I was very careful doing the caching correctly on the n64. But since I run in kernel mode all the time, I currently just have these functions which will call the cache ops directly. I should probably move this code to syscachemaint.

Ok. Did you run ./all.bash on any MIPS64 linux system?

No :smirk: Unfortunately I probably won't have the time to do so.

[embeddedgo]

No 😏 Unfortunately I probably won't have the time to do so.

For riscv64 I use qemu-system-riscv64 with small linux image. Now the same qemu-system-riscv64 is also used by noostest target.

Try to install linux on the qemu-system-mips64 virtual machine. Below as a reference my script that starts my riscv64 one:

qemu-system-riscv64 -nographic -machine virt -smp 2 -m 2G \
 -kernel fw_jump.elf \
 -device loader,file=u-boot.bin,addr=0x80200000 \
 -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-device,rng=rng0 \
 -append "console=ttyS0 rw root=/dev/vda1" \
 -device virtio-blk-device,drive=hd0 -drive file=rootfs.img,format=raw,id=hd0 \
 -device virtio-net-device,netdev=usernet -netdev user,id=usernet,hostfwd=tcp::22222-:22

If started you can ssh localhost:22222 to it.

I think its crucial for your mips64 port to test it against the full set of standard tests (all.bash/run.bash) after any change to the mips code.

You can try qemu-mips64 and run test on the linux/amd64 system provided the binfmt_misc is configured. It works for linux/thumb somehow and is more convenient for working on a single test.

[clktmr]

Ok, I will take a look at it. But there were good news in the meantime: I got feedback from one of the Go mips contributors. All of the revert commits can be removed. It was my fault by not initializing src/internal/cpu/ correctly. I will push the patch later.

[embeddedgo]

In MIPS (on the VR4300) there are two software interrupts, which can be triggered by writing to a register. I use one of them to signal newwork(), and consider the other one unused and not exposed to the user. There is one timer interrupt, also unused as of now and considered not exposed to the user.

CLINT does almost the same for RISCV. Its timer is well designed and very useful.

Then there are the external interrupts, which I do consider exposed to the user. Only these support nesting. Currently I will only mask all pending interrupts, then reenable them. I didn't see rtos.IRQ.Enable. From what I read I need to implement sysirqctl for mips64 to support priorities correctly.

Slow interrupts like some syscalls definitely should have the lowest priority and be preemptable. The tasker code should be considered very slow. Consider a lightpen/lightgun that uses IRQ to report seeing an electron beam. Its ISR should as fast as possible read the current vertical and horizontal position from DAC. The time for one pixel on the horizontal scan is 63.6μs / 320 = 0.2 μs or even less because apart from the visible horizontal pixels the line contains also sync and back/front porch. If this ISR sometime must wait for the context switch in the tasker or some slow system call the user experience will be bad.

[clktmr]

Slow interrupts like some syscalls definitely should have the lowest priority and be preemptable.

The terminology is a bit different in MIPS. Syscalls come as exceptions, not as interrupts and they aren't disabled at the beginning of the exceptionHandler. And my wording was bad: Nesting for fast syscalls is also impemented (not optional I think).

Everything (external and software interrupts, exceptions) is preemptable as soon as possible. What was broken before was preemption done by the external interrupt and as a workaround I would have them always masked and only shortly enabled at safe points. This workaround is now gone and latencies should be as short as possible.

To get a feeling of what's implemented, I recommend reading the comments in tasker_noos_mips64.s. I rewrote most of them during my review, to refresh my understanding of the code.

[michalderkacz]

The terminology is a bit different in MIPS. Syscalls come as exceptions, not as interrupts and they aren't disabled at the beginning of the exceptionHandler. And my wording was bad: Nesting for fast syscalls is also impemented (not optional I think).

Yes. The terminology...

RISCV has traps.

Trap can be an exception or an interrupt.

Exceptions are exceptional (syscall, illegal instruction, breakpoint, addres misalignment, etc.). There is no way to mask them.

Interrupts can be internal or external. Every hart (core) has its mask register to mask any kind of interrupt.

Internal interrupts may come from CLINT: software generated, timer generated.

External interrupts come from PLIC. PLIC can handle (mask, prioritize) multiple exception sources and direct them to the one or more harts (cores).

This CLINT+PLIC combo is only one of many possibilities.

Cortex-M has only exceptions. Some exceptions are called interrupts. There is no clear distinction between internal and external interrupts.

Maybe we need some unified terminology for this trap/exception/interrupt zoo.

Everything (external and software interrupts, exceptions) is preemptable as soon as possible. What was broken before was preemption done by the external interrupt and as a workaround I would have them always masked and only shortly enabled at safe points. This workaround is now gone and latencies should be as short as possible.

To get a feeling of what's implemented, I recommend reading the comments in tasker_noos_mips64.s. I rewrote most of them during my review, to refresh my understanding of the code.

I read it quickly and it refreshed my mind too. It seems to be based on the riscv64 code. The R26 and R27 are used for bootstrapping. It makes thing easier and faster than the RISCV mscratch register that allows you to free only one GPR for bootstrapping.

What I understood is that now all three supported architectures (mips64, riscv64 and thumb) allow nesting interrupts and preempt exceptions like syscalls (fast and slow). One thing that should be done for mips64 is to implement priorities for external interrupts and allow user to configure them using the rtos package.