search

LekKit / RVVM

The RISC-V Virtual Machine
GNU General Public License v3.0
935 stars 65 forks source link

tap_user usability tracking #87

Open fish4terrisa-MSDSM opened 1 year ago

fish4terrisa-MSDSM commented 1 year ago

( Sorry for my poor english) The QEMU has a feature to bind the guest`s port to the host\s port.Will RVVM support it later?It\s impossible to support x11 or SDL on android,so the vnc is the only choice.It`s important to make it possible to connect to the guest using the network port.

LekKit commented 1 year ago

Yeah it will be there in future. The networking is WIP as you can see from README, my attention is pretty scrambled over multiple things. Can we convert this issue into tap_user usability tracking issue?

P.S. Is there a way to draw a raw framebuffer on Android using native GUI APIs? I am going to add JNI support and then we can have native Android RVVM app without VNC shenanigans. A while ago I expected SDL to be usable on Android and only later realized it isn't...

LekKit commented 1 year ago

Current tap_user issues/TODOs:

LekKit commented 1 year ago

Maybe it is worth to add VNC support into RVVM itself. There is such feature in QEMU (It acts like a VNC server), but I am unaware how complicated that might be (At least we have a nice networking lib).

Doing that via guest means is arguably not an ideal solution (We don't see the boot process, requires user intervention, what about non-Linux guests?), and we need a proper Android port anyways. So I guess JNI + android GUI is the best long term solution for this usecase.

fish4terrisa-MSDSM commented 1 year ago

Yeah it will be there in future. The networking is WIP as you can see from README, my attention is pretty scrambled over multiple things. Can we convert this issue into tap_user usability tracking issue?

P.S. Is there a way to draw a raw framebuffer on Android using native GUI APIs? I am going to add JNI support and then we can have native Android RVVM app without VNC shenanigans. A while ago I expected SDL to be usable on Android and only later realized it isn't...

Ok i will change the issue name.

P.S. I dont know much about the android api .(in fact,my project heavily depends on alpine-term.I dont found any android app can draw a raw fb,even the termux-x11 or xserver xsdl dont use it).Besides, adding gui support is not so important i think .Many people dont need the gui support on android,they may prefer terminal .And if the port forwarding and the gpu support is done ,the user can start a vnc by themselves.A unused vnc causes profermance decrease. If you want to build the own RVVM app ,editing on my project may be a good idea ,just delete the descriptions of archlinux and you will get one .(My project is still undone ,it will take me about 2 weeks to fully move from QEMU to RVVM .And if its done ,i will notice you in the discussions area)

fish4terrisa-MSDSM commented 1 year ago

Maybe it is worth to add VNC support into RVVM itself. There is such feature in QEMU (It acts like a VNC server), but I am unaware how complicated that might be (At least we have a nice networking lib).

Doing that via guest means is arguably not an ideal solution (We don't see the boot process, requires user intervention, what about non-Linux guests?), and we need a proper Android port anyways. So I guess JNI + android GUI is the best long term solution for this usecase.

Thats true ,I prefer the vnc method ,its also useful on other platforms.The Android GUI method is a bit complicated (termux-x11 might be a exmaple , it has developed for more than 3y but still have large amounts of bugs) ,but adding a vnc viewer in the app might be a good idea as the user can connect to the machine without using other apps.

fish4terrisa-MSDSM commented 1 year ago

A good vnc viewer project is avnc, its license is GPL3.You can insert some parts of it into the app.

LekKit commented 1 year ago

Commit 833a16e solves all previous TCP traffic issues and running out of UDP ports. It also includes groundwork for TCP NAT (host->vm connections) but the way to lookup socket via 2-tuple is still in the works.

Please report if you still have any TCP stability issues.

LekKit commented 1 year ago

Figured there is an issue with keepalive ACKs not being properly handled. Fixed locally, will be upstreamed soon. (This only matters if you have completely idle connections that live for hours)

LekKit commented 1 year ago

TCP port forwarding works. However, there is an issue when connecting directly to 127.0.0.1 from the host, the guest gets a connection from 127.0.0.1 as well and gets confused, so the connection is dropped. I don't know how to solve that correctly yet. Will be upstreamed soon.

image

fish4terrisa-MSDSM commented 1 year ago

A bug occured on NetBSD:

[root@archlinux ~]# echo $LANG
C.UTF-8
[root@archlinux ~]# curl www.google.com
WARN: net_poll_add() failed!
curl: (1) Received HTTP/0.9 when not allowed
[root@archlinux ~]# curl https://www.google.com
curl: (35) OpenSSL/3.0.8: error:0A0000C6:SSL routines::packet length too long
[root@archlinux ~]#

and when run curl www.google.com --http0.9,it will output the index.html of google,but the connection won`t close before timeout,so i think both the openssl error and the curl error is because that tcp connections cannot close accurately on NetBSD.The ping runs correctly ,so icmp have no problems. When runs curl ip.sb --http0.9 to get the ip , it outputed

[root@archlinux ~]# curl ip.sb
curl: (1) Received HTTP/0.9 when not allowed
[root@archlinux ~]# curl ip.sb --http0.9
TTP/1.1 200 OK
Date: Tue, 09 May 2023 01:38:16 GMT
Content-Type: text/plain
Content-Length: 13
Connection: keep-alive
Cache-Control: no-cache
CF-Cache-Status: DYNAMIC
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=cgybpWVU5jI3iZjC8NPmm7lMwYhsUfoaNkW7zU5VFKGN%2BxbuCpdO9E1Z%2Fl%2BKK3Rz3gAVbsJx9Pm9q9B9LGOwd0Lrt4dE5hLwJK1y4pfyf3X3ut8XYFoV"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 7c4638d53bc13087-SEA
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

205.166.94.4

The \'H\' of \"HTTP/1.1\" disappeared ,so both the length of the packet and the head of the packets might be wrong(?) Output of ifconfig

[root@archlinux ~]# ifconfig
enp0s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.100  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::8c2e:4fff:fed9:45f  prefixlen 64  scopeid 0x20<link>
        ether 8e:2e:4f:d9:04:5f  txqueuelen 1000  (Ethernet)
        RX packets 45  bytes 10639 (10.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 79  bytes 6345 (6.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The dhcp just worked well.I haven`t tested udp connections yet,and I also don`t know much about how to test it. p.s. : now the VMA worked well on sdf.Great!

fish4terrisa-MSDSM commented 1 year ago

A bug occured on NetBSD:

[root@archlinux ~]# echo $LANG
C.UTF-8
[root@archlinux ~]# curl www.google.com
WARN: net_poll_add() failed!
curl: (1) Received HTTP/0.9 when not allowed
[root@archlinux ~]# curl https://www.google.com
curl: (35) OpenSSL/3.0.8: error:0A0000C6:SSL routines::packet length too long
[root@archlinux ~]#

and when run curl www.google.com --http0.9,it will output the index.html of google,but the connection wont close before timeout,so i think both the openssl error and the curl error is because that tcp connections cannot close accurately on NetBSD.The ping runs correctly ,so icmp have no problems. When runscurl ip.sb --http0.9` to get the ip , it outputed

[root@archlinux ~]# curl ip.sb
curl: (1) Received HTTP/0.9 when not allowed
[root@archlinux ~]# curl ip.sb --http0.9
TTP/1.1 200 OK
Date: Tue, 09 May 2023 01:38:16 GMT
Content-Type: text/plain
Content-Length: 13
Connection: keep-alive
Cache-Control: no-cache
CF-Cache-Status: DYNAMIC
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=cgybpWVU5jI3iZjC8NPmm7lMwYhsUfoaNkW7zU5VFKGN%2BxbuCpdO9E1Z%2Fl%2BKK3Rz3gAVbsJx9Pm9q9B9LGOwd0Lrt4dE5hLwJK1y4pfyf3X3ut8XYFoV"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 7c4638d53bc13087-SEA
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

205.166.94.4

The 'H' of "HTTP/1.1" disappeared ,so both the length of the packet and the head of the packets might be wrong(?) Output of ifconfig

[root@archlinux ~]# ifconfig
enp0s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.100  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::8c2e:4fff:fed9:45f  prefixlen 64  scopeid 0x20<link>
        ether 8e:2e:4f:d9:04:5f  txqueuelen 1000  (Ethernet)
        RX packets 45  bytes 10639 (10.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 79  bytes 6345 (6.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The dhcp just worked well.I havent tested udp connections yet,and I also dont know much about how to test it. p.s. : now the VMA worked well on sdf.Great!

I just tried curl --output ip.sb.out ip.sb --http0.9 -m 20,but the output is:

 /root__curl --output ip.sb.out ip.sb --http0.9 -m 20      05/09/2023 09:47:39 AM
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   602    0   602    0     0     30      0 --:--:--  0:00:20 --:--:--     0
curl: (28) Operation timed out after 20018 milliseconds with 602 bytes received
/root__ls                                                 05/09/2023 09:48:07 AM
_───_───────────_──────_───────_──────────_
│ # │   name    │ type │ size  │ modified │
├───┼───────────┼──────┼───────┼──────────┤
│ 0 │ ip.sb.out │ file │ 602 B │ now      │
_───┴───────────┴──────┴───────┴──────────_
/root__cat ./ip.sb.out                                    05/09/2023 09:48:09 AM
TTP/1.1 200 OK
Date: Tue, 09 May 2023 01:47:50 GMT
Content-Type: text/plain
Content-Length: 13
Connection: keep-alive
Cache-Control: no-cache
CF-Cache-Status: DYNAMIC
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=%2B9y26TUxB6H%2FDgXjYX%2Fqi1JhdVY9i3aeAMhRmG6SPz8mgvaDPSuw8pk0cJ3tpiAhzive8aTSb3E7kP1BNUmkLCEFn9ZwlvFTcqNG%2BKEi511bq5ceN2Y8"}],"group":"cf-nel","max_age":604800}NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 7c4646d5be7608a1-SEA
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

205.166.94.4

It seems that curl can`t detect the end of the connection.

LekKit commented 1 year ago

The 'H' of "HTTP/1.1" disappeared

Ideally I'd need a full tcpdump during this curl invocation, thanks. Will look what it might be in the meantime.

WARN: net_poll_add() failed!

Oh, this is unexpected. Usually this indicates host resource exhaustion and only seen to happen on Windows with >1024 open connections. It is unexpected from kqueue to act like this, and this may also be the reason of connections not being closed. Can you try commenting out this line and re-testing TCP? https://github.com/LekKit/RVVM/blob/226330d096d8bcf1c38337e230b7eab306c4944f/src/networking.c#L60

LekKit commented 1 year ago

Weird... kqueue udata is not a void* in NetBSD??? image

fish4terrisa-MSDSM commented 1 year ago

The 'H' of "HTTP/1.1" disappeared

Ideally I'd need a full tcpdump during this curl invocation, thanks. Will look what it might be in the meantime.

WARN: net_poll_add() failed!

Oh, this is unexpected. Usually this indicates host resource exhaustion and only seen to happen on Windows with >1024 open connections. It is unexpected from kqueue to act like this, and this may also be the reason of connections not being closed. Can you try commenting out this line and re-testing TCP?

https://github.com/LekKit/RVVM/blob/226330d096d8bcf1c38337e230b7eab306c4944f/src/networking.c#L60

Commented . It worked!!! Both bug is fixed.Great.Maybe a port of NetBSD can be added.

fish4terrisa-MSDSM commented 1 year ago

Yes...I got that warning when compling RVVM on sdf.

LekKit commented 1 year ago

Yes...I got that warning when compling RVVM on sdf.

This basically means NetBSD violates kqueue API, and either needs workarounds or will have to use slower select implementation which is already used on Windows and Haiku as a safe generic implementation. Commenting out KQUEUE_NET_IMPL disables kqueue.

LekKit commented 1 year ago

Fixed kqueue issues on NetBSD in 2aba896. Report if something isn't right still.

LekKit commented 1 year ago

Fixed another (rare) kqueue bug, perhaps it hasn't bothered anyone yet but there it is

LekKit commented 1 year ago

Weird warning in lynx browser. May be related to FIN/RST flags handling.

image

fish4terrisa-MSDSM commented 1 year ago

Now RVVM seems not support multi task downloading (I dont know the right way to describe that),will RVVM support it? Screenshot_20230617_112723 (As it showed , the other download task is rather slow except one)

LekKit commented 1 year ago

Now RVVM seems not support multi task downloading

I am not sure at all that it's on RVVM side. I occasionally see it's balancing multiple downloads properly, sometimes not, and I have no idea what could be going wrong (I am just pumping data from a socket into ethernet frames). It might be just that the download speed is peaking on one download, and other ones actually are received at that speed on the host.

However, running iperf in clean conditions over local network seems to distribute bandwidth evenly: image

I will examine if, for example it accidentally prioritizes one socket over others. You could also try to comment out this and re-test: https://github.com/LekKit/RVVM/blob/5ab80661ac89f69b905f06fdd6c6e9955bf21a1d/src/networking.c#L54

LekKit commented 1 year ago

Running with epoll on Illumos. Apparently it balances the downloads fine, and it still peaks at around 4.5 MB/s downlink - which makes me believe this is actually peak of the Arch RISC-V mirrors capabilities (per IP?). So far I am almost convinced this is related environment (mirrors/host) rather than RVVM.

image

fish4terrisa-MSDSM commented 10 months ago

TCP port forwarding works. However, there is an issue when connecting directly to 127.0.0.1 from the host, the guest gets a connection from 127.0.0.1 as well and gets confused, so the connection is dropped. I don't know how to solve that correctly yet. Will be upstreamed soon.

image

Is this problem fixed now? And, there isn't an option matched with port forward, maybe an option like rvvm -hostfwd ${GUEST_PORT}:${HOST_PORT} can be added? That may enable me to mount the guest fs to the host(or a 9pfs like qemu might be a better choice, but currently RVVM didn't support it) And, although nearly nobody need udp port forwarding, -tcpfwd and -udpfwd might be a better choice?

LekKit commented 10 months ago

I've almost finished a fix for localhost connections via port forwarding. Also working on distinguishing graceful FIN versus RST from outside. Will merge soon

LekKit commented 9 months ago

Please test 89a01de, it should implement all the desired (so far) functionality (But no special arguments yet). You can ssh into the guest with ssh root@localhost -p 2022.

LekKit commented 8 months ago

Fixed a bunch of minor networking issues in staging branch.

LekKit commented 8 months ago

@X547 In upcoming patch it will be possible to have a separate IP address for the guest, so basically a non-NAT scenario without tap_linux being used.

It relies on

  1. Being able to create multiple IP addresses on one adapter
  2. Being able to bind tap_user sockets (both UDP/TCP) on that IP (See networking.h API for details)

The guest will be accessing the network from that address now on. It doesn't need to be a physical adapter either, so you can seat the guest onto a veth/tap/whatever interface you want. Firewall rules also can work specifically on this guest.

It is then possible to disable emulated NAT in tap_user and directly map it's IP to the assigned one, and map any ports directly without port translation (Which should be configured separately)

IPv6 is also worked on, the remaining bits are ICMPv6 and proper guest addr assignment. I am not sure yet if above technique will work for IPv6 ranges, but at least I'm sure /128 addresses will work.



3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e4:a7:a0:e7:25:2a brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.203/24 brd 192.168.0.255 scope global dynamic noprefixroute wlan0
       valid_lft 40563sec preferred_lft 40563sec
    inet 192.168.0.37/24 scope global secondary wlan0
       valid_lft forever preferred_lft forever

image

image

LekKit commented 8 months ago

When running in unprivileged environment without additional configuration, IPv6 will be still usable for internet access, but will be NATed - there is pretty much no other option in unprivileged case

fish4terrisa-MSDSM commented 8 months ago

A separate IP address would be great, but is it possible to choose the ip address to bind with? And, it may cause ip address conflict if the user is not sure about whether the ip address is available in the network. I think port forwarding is still useful, and options like -udpfwd and -tcpfwd would be great to see. Or, maybe a new option like -ip will help, and we may test whether the ip address is available before start the machine.

LekKit commented 8 months ago

A separate IP address would be great, but is it possible to choose the ip address to bind with? And, it may cause ip address conflict if the user is not sure about whether the ip address is available in the network. I think port forwarding is still useful, and options like -udpfwd and -tcpfwd would be great to see. Or, maybe a new option like -ip will help, and we may test whether the ip address is available before start the machine.

Basically there are going to be 2 options:

The latter option will be useful for more advanced usecases like firewalling the guest using host tools (iptables/netfilter/ipfw), accessing it as separate host without NAT, creating a virtual lan or putting it behind a VPN/proxy. It doesn't break simpler usecases (Which is to just get working internet everywhere)

fish4terrisa-MSDSM commented 8 months ago

Basically there are going to be 2 options:

  • Default: Guest simply has access to internet (can download stuff, etc) and is behind a virtual NAT (So it's not recognized as a separate network host from the outside point of view). You can use port forwarding to map some host ports to guest ports. Works on all OSes without special privileges.

Great. That will make my current use unaffected. Port forwarding will really help.

  • Interface passthrough: Guest sits on a separate address/interface so it is actually treated like a different machine in your LAN or even outside world in case of IPv6. Needs additional privileges, and unsure if it can be done on Windows. This is still WIP.

That seems attractive for me. Can you tell me which privileges (or, capabilities) in linux are needed?

The latter option will be useful for more advanced usecases like firewalling the guest using host tools (iptables/netfilter/ipfw), accessing it as separate host without NAT, creating a virtual lan or putting it behind a VPN/proxy. It doesn't break simpler usecases (Which is to just get working internet everywhere)

Hmm... It will also make it possible to use tcpdump to monitor the guest.

LekKit commented 8 months ago

That seems attractive for me. Can you tell me which privileges (or, capabilities) in linux are needed?

CAP_NET_ADMIN for adding an address to interface; CAP_NET_BIND_SERVICE if you want to make guest ports below 1024 accessible without mapping them to a different port externally.

Systems without epoll or kqueue support won't be able to bind port ranges efficiently.

LekKit commented 8 months ago

Implemented -portfwd CLI argument in c92727e.

Supported syntax: -portfwd tcp/2022=22 - Forward TCP 0.0.0.0:2022 to guest port 22 (DHCP guest address is assumed) -portfwd tcp/127.0.0.1:2022=22 - Forward TCP 127.0.0.1:2022 to guest port 22 (DHCP guest address is assumed) -portfwd 25565=25565 - Forward both UDP & TCP 0.0.0.0:25565 to the same guest port -portfwd udp/[::1]:6121=[fd9c:350d:af07::385]:6121 - Forward UDP [::1]:6121 to specific IPv6 address in guest network