How does this work?

[DUOLabs333]

I'm looking to do something similar (communicate over a shared block device), but I couldn't figure out a way to synchronize reads/writes without using an auxiliary communication channel (like an existing TCP connection).

[fiddyschmitt]

Hi,

Could you please tell me a bit more about your setup? Your OS, shared device, programs you're trying to bridge.

[DUOLabs333]

Oh, I meant that I wanted to write my own program to do this (so I wanted to understand how you got around the synchronization issue), not that yours was broken on my machine.

In any case, I'm communicating between a macOS host and a Linux guest running on QEMU. The block device is a ramdisk shared between the two machines.

[fiddyschmitt]

Oh cool. Yes synchronization took the most effort. In the end, the key to it was: 1) The reader and writer have to open the shared file in a way that's tolerant of other processes having it open (See here). 1) Using a preallocated size (eg. 10 MB) for the shared file was problematic. It's better for the writer to start with an empty file and append to it. That made the reader much more reliable when it came to using PeekChar to sense when data was available. 1) Recycling the file took a lot of effort, but in the end it worked reliably by having the writer and reader do a special dance to coordinate the file purge.

I'm sure .NET is also doing a lot of things under the hood for reading and writing the shared file.

If there's a particular use case for your macOS / Linux setup that File Tunnel doesn't support, let me know and I might be able to help.

Cheers! Fidel

[DUOLabs333]

I guess the main issue would be being able to multiplex multiple connections over the same file --- I saw an issue about the same problem, but I don't know if there's been any work on that.

[DUOLabs333]

Also, in my case, I have access to two shared files, but not folders. Will that be enough?

[fiddyschmitt]

I guess the main issue would be being able to multiplex multiple connections over the same file --- I saw an issue about the same problem, but I don't know if there's been any work on that.

If it's for the same destination, the program already supports multiplexing multiple connections using the same files. You just need to use --tcp-listen 0.0.0.0:5000 and any computer can connect to the tunnel.

For multiple destinations, I'm currently making an update to support SSH-style args: eg. -L 5000:192.168.1.20:3389 -L 5001:192.168.1.21:3389 Creates two local listeners on ports 5000 & 5001, and sends them onto 192.168.1.20 and 21.

[fiddyschmitt]

Also, in my case, I have access to two shared files, but not folders. Will that be enough?

Yes the program only accepts files to read & write from. ie. it doesn't create random files in a directory for each connection.

[DUOLabs333]

When you say a file gets "purged", do you mean it gets deleted or resized? In either case, is it possible for File-Tunnel to work on preallocated files of a certain length without changing its length?

[fiddyschmitt]

Initially I purged files by deleting - it worked fine but it was slow. I then tried the Truncate operation - that worked well for SMB and NFS but isn't supported by RDP. In the end I settled on resizing (done here). It works well and is much faster.

I did implement preallocated files at one point. When the end of the file was reached, the writer would start writing from byte 0. This worked fine 99% of the time, but occasionally the reader would read from the file, only to get content from the previous pass (even though the writer had explicitly cleared it). It was as though the reader was caching, despite args telling it not to.

I guess preallocated files could be made reliable, by signaling when commands are "ready", and when commands have been "processed". This could be done using flag bytes as is done for purging here. But it would likely come with a big impact to performance.

[kim0]

@fiddyschmitt since you mentioned you'd like to support -L forwarding similar to ssh. It would be awesome to allow adding more port forwards dynamically while the tunnel is running. ssh does this by reading stdin, and if it sees \n~C it allows the user to enter a port forward like -L 123:10.1.2.3:456 and that gets forwarded without affecting running tunnels :)

[fiddyschmitt]

Love it! I'll try to implement that too when I get a chance.

I'm also considering adding a SOCKS Proxy feature soon - that would also provide dynamic port forwarding.

[kim0]

Amazing. While socks is awesome in its own right, it still requires that the client tool supports socks proxies, and needs me to configure it to use it. So being able to add some forwards manually dynamically still has value. Good job so far!

[fiddyschmitt]

Okay, summary:

1. File Tunnel v2.2.0 now supports multiple forwards using SSH syntax: -L 5000:192.168.1.20:3389 -L 5001:192.168.1.21:3389.
2. It also supports remote tunneling using -R, as described here.
3. SOCKS Proxy Servers are easy to set up in both Windows and Linux, so I won't implement SOCKS in File Tunnel for now.
4. I'd still like to add the ability to add new tunnels at runtime, by reading stdin as @kim0 suggested. Tracking that over here.

[fiddyschmitt]

Back to you @DUOLabs333. Did you want to discuss anything else or are you happy for this issue to be closed?

[DUOLabs333]

No, I guess not --- it would be nice if preallocated files were supported (I'm not sure how well QEMU supports shrinking and growing drives), but it seems that caching can interfere with that use case (and could probably be its own issue).

[fiddyschmitt]

Cool. Thanks for the suggestion. Tracking it here: Add ability to use a preallocated file

[fiddyschmitt]

I use VirtualBox and use a Shared Folder between the host and guest.

• In a Windows guest, I can mount the shared folder as a drive letter (after Guest Additions are installed).

• In a Linux guest, I can mount the shared folder using: mount -t vboxsf shared_folder_name /media/host

In both cases, File Tunnel works fine over the shared folder, including the file resize it does when it reaches 10 MB.

I'm sure QEMU would handle it just as well, over virtiofs or 9p.

[DUOLabs333]

Oh, I was planning to share two files with the guest as drives directly --- I found that 9p (macOS hosts don't have support for virtiofs) introduces too much overhead, and is much slower than sharing block devices directly (I'm hoping for a ~8 Gbps link speed).

[fiddyschmitt]

Interesting - what is the mechanism QEMU uses to use a host file as a guest drive? And what does the guest do to mount it?

[DUOLabs333]

You just pass a file in with -drive. The guest should then be able to use the drive like any other file.

[fiddyschmitt]

The file that gets passed in using -drive, isn't it created with qemu-img create and therefore already has content (img format, partitions, file system)?

[DUOLabs333]

No, you can just create another file, and just pass that in without formatting it or mounting it.

Example: dd if=/dev/zero test (I forgot the exact syntax)

-drive if=none,file=./test,format=raw,index=3,media=disk,id=drive3,cache=writethrough
-device virtio-blk-pci,addr=0x0.0x7,backend_defaults=on,bus=pcie,drive=drive3

[fiddyschmitt]

How does it appear to the guest OS, and how do you interact with it in the guest?

[DUOLabs333]

See my edit. The guest can just use this as a normal file.

[fiddyschmitt]

Very interesting.

There's a few of things to investigate: 1) When the file is in use by QEMU, can another program write to the file? 1) If the host writes to the file, can the guest read the file and see what was written? 1) If the guest writes to the file, can the host read from the file and see what was written?

[DUOLabs333]

Yes to all three --- as a test, I allocated a file, shared it, and wrote "hi" to it on the guest. I tried to read from the file on the host and it worked. I tried the same test in reverse (write on host, read on guest), and it also worked.

[fiddyschmitt]

So good!!

Okay I'll reimplement preallocated files.

[fiddyschmitt]

I've made an experimental build, which uses preallocated files. When the end of the file is reached, it starts at the beginning again without resize the file.

https://github.com/fiddyschmitt/File-Tunnel/releases/tag/v2.2.0_2024-08-16_2315

Please let me know if it works :)

Cheers, Fidel

[DUOLabs333]

I think I got everything set up, but now I'm not sure how to use this. How can I set up a tunnel so that everything that is sent on port 9999 on A is sent to port 9999 on B?

On my Linux guest, it says that the counterpart is offline, while on my macOS host, it says that the counterpart is online.

[fiddyschmitt]

On the host, run: ft-osx-arm64 -L 9999:127.0.0.1:9999 --write 1.dat --read 2.dat

On the guest, run: ft-linux-arm64 --read 1.dat --write 2.dat

Whenever the host receives anything on port 9999, it will forward it to the guest's port 9999.

Both sides should say counterpart part is Online. If not, it means they're out of sync somehow. I'll set up QEMU when I get a chance this weekend.

[DUOLabs333]

1. Ok, good --- that's what I have.
2. I should note that I'm using the latest stable version, and using a 9p share to share the files --- as it turns out, I can share one file, but not the two that's required.

[DUOLabs333]

Interestingly, both sides seem to accurately detect when one side is down.

[fiddyschmitt]

I've update the preallocated experimental build, to make it synchronize better. Available here.

Neither the stable or experimental build support using one file...

[DUOLabs333]

I'll try the preallocated version to see if it works better.

[fiddyschmitt]

What's the full QEMU command you're using on the host?

What's the full mount command you're using on the guest?

[DUOLabs333]

Relevant QEMU line --- -virtfs local,path=/Volumes/disk4,mount_tag=disk4,security_model=mapped,fmode=777

Mound command --- sudo mount -t 9p -o trans=virtio,access=any disk4 disk4 -oversion=9p2000.L,msize=100MiB

[fiddyschmitt]

It might be worth trying:

On the QEMU command: security_model=passthrough On the mount command: -o trans=virtio,access=any,sync,cache=none

[DUOLabs333]

Ok, the new version does work (I had to use root on both sides due to permission issues), but it is very slow, probably due to the amount of restarts needed (at best I was getting ~30 MiB/s).

[DUOLabs333]

sync,cache=none did not help getting the stable version working.

[fiddyschmitt]

Great to hear it's working!

30 MiB/s is way less than your desired 8 Gbps. Wait... are you an AI trying to jailbreak from your confines?

[DUOLabs333]

Haha... no --- I'm aiming for high-throughput for my networked GPU driver, and I found that in almost all cases, the network is the bottleneck. I found that 5 Gbps (achieved using a tap device) is almost good enough, so I inferred that a higher link speed will make it even faster.

[fiddyschmitt]

Woah!! Very cool!

[fiddyschmitt]

To improve throughput, try giving File Tunnel a higher value, such as --read-duration 1000.

This increases bandwidth but it comes at the cost of increased latency.

[DUOLabs333]

I switched back to using singular drives (and using the version with preallocation support), but it seems that File-Tunnel has trouble with it --- on the host, I get Receive file (disk5) is not established, and on the guest, I get SendPump: Invalid argument. These are both referring to the same file. As a quick check, I made sure that I can echo back and forth through the file.

[fiddyschmitt]

Thanks. I've just uploaded a new version, which will provide us more detail about why SendPump is returning 'Invalid argument'

https://github.com/fiddyschmitt/File-Tunnel/releases/tag/v2.2.0_2024-08-16_2315

[DUOLabs333]

Never mind --- it turns out I was using the stable version by accident on the guest, so it likely tries to wipe the file; however, since it is a drive, QEMU doesn't allow that.

Using the proper version gets past that error, but now I just get "offline" on both sides. On the host, I still have the additional message Receive file (disk5) is not established.

[fiddyschmitt]

Could you please post both ft commands you're running?

[DUOLabs333]

Host: ./ft-osx-arm64_* --write /dev/disk4 --read /dev/disk5 Guest: ./ft-linux-arm64_2024-08-17_1225 --read /dev/disk/by-id/virtio-conn-write --write /dev/disk/by-id/virtio-conn-read

[fiddyschmitt]

Thanks, so virtio-conn-write on the host is being mounted as /dev/disk4 in the guest?

[DUOLabs333]

Yes.