Full Go "net" package port, WIP

scottfeldman commented 1 month ago

[This is a resurrection of #4187, which I accidentally closed by deleting the fork it was based on.]

This PR is WIP to port the full Go "net" package to TinyGo.

With this PR, I can compile and link a simple example:

package main

import (
        "fmt"
        "net"
)

func main() {
        conn, err := net.Dial("tcp", "localhost")
        if err != nil {
                fmt.Println(err)
        }
        conn.Close()
}

tinygo build -target nano-rp2040 -tags netgo main.go

Notes:

This PR uses Go's full "net", "net/http", and "crypto/tls" packages, unmodified.
Need to compile with -tags netgo to pick Go's DNS resolver.
We no longer need the repo/submodule tinygo-org/net.
None of the network drivers are hooked up; we'll need to redefine the network driver interfaces at the syscall level.
I've copied in the stock Go src/syscall/ files for linux,arm. This defines all of the syscall types and data structures for linux,arm. We can copy in other arch files as needed. I'm only testing with nano-rp2040.
There are new files in sys/syscall:
- src/syscall/env_tinygo.go
- src/syscall/exec_tinygo.go // can go away if issue # 1 is addressed
- src/syscall/netlink_tinygo.go // can go away if issue # 1 is addressed
- src/syscall/syscall_tinygo.go
syscall_tinygo.go has the stubbed out syscalls. These all println("not implemented") and return EOPNOTSUPP.

The idea is this: with the netdev work in the last release, we truncated the "net", "net/http", and "crypto/tls" packages and inserted our own stubs to call into the network wifi drivers. With this PR, we use the full Go packages, but this time the insertion point is at the syscall level. Syscall is now where we define the interfaces to the network stack, and the network stack calls into the network drivers. It makes sense...if we were a full OS, syscall is where we'd have our OS-specific code.

Issues:

Issue # 1

Import cycle with "sync" package. Unfortunately, syscall package imports "sync" in some places, which causes an import cycle:

package tinygo.org/x/drivers/examples/net/http-get
       imports bytes
       imports io
       imports sync
       imports internal/task
       imports runtime/interrupt
       imports device/arm
       imports syscall
       imports sync: import cycle not allowed
(See src/syscall/netlink_linux.go for an example).

I've worked around this issue by stubbing out any imports of "sync", but that's not a workable solution in the long term. We'll need "sync" in the implementation of the stubbed out syscalls for the network stack and drivers. So we'll need to revisit "sync" so as to not import "syscall". Is it possible?

leongross commented 1 month ago

@scottfeldman could you maybe elaborate on the process of how the board-specific drivers are abstracted here? As my understanding of tinygos network driver model goes, drivers are represented as the a combination of Netdever and Netlinker interfaces. But I cannot find any reference to that in the inserted syscall code, so how are the bare metal board specifics picked up here?

And to enable network support for linux platforms, couldn't the functions in syscall_tinygo be linked against the according and already existing interfaces provided bey the unix package?

scottfeldman commented 1 month ago

@scottfeldman could you maybe elaborate on the process of how the board-specific drivers are abstracted here? As my understanding of tinygos network driver model goes, drivers are represented as the a combination of Netdever and Netlinker interfaces. But I cannot find any reference to that in the inserted syscall code, so how are the bare metal board specifics picked up here?

@leongross your're right, the netdev/netlink interfaces are the current TinyGo driver model for embedded net devices. In this PR, these interfaces will be replaced with a TBD interface at the syscall level, since all the "net" package calls ultimately resolve into syscalls. The next step in this PR is to discover which syscalls are needed by "net" (and crypto/tls), and let those define the interface to the device drivers. The goal is to support both raw-MAC devices (i.e. Pico-W) as well as devices with an embedded stack (i.e. wifinina, rtl8720n). Some ASCII pics:

raw-MAC stack:

your app
"net" package
-------------           <-- syscall interface TBD
network stack
device driver
-------------           <-- hw interface
device

embedded stack:

your app
"net" package
-------------           <-- syscall interface TBD
device driver
-------------           <-- embedded fw interface
embedded network stack
-------------           <-- hw interface
device

And to enable network support for linux platforms, couldn't the functions in syscall_tinygo be linked against the according and already existing interfaces provided bey the unix package?

Enabling full OS support for "net" wasn't the goal of this PR, but it seems we could make it work by linking in the OS syscalls when compiling against a full OS, bypassing the driver interface mentioned above. I suspect the work to do this is around loader/goroot.go and some built tag magic.