Proposal: Direct TCP/IP Access

[oliverschmidt]

Hi,

https://www.irata.online/#connecting and the mentioning of tcpser indicate that IRATA.ONLINE is actually accessed via a TCP connection.

Instead of using a serial connection to some TCP proxy it is a viable alternative to have the 6502 machine do the TCP connection on its own.

There's a TCP/IP library available written in 6502 asm called IP65 (https://github.com/oliverschmidt/ip65) that is directly usable from cc65 C programs (https://github.com/oliverschmidt/ip65/blob/master/inc/ip65.h).

The IP65 library supports three Ethernet chips:

• Cirrus Logic CS8900A
• Microchip LAN91C96
• WIZnet W5100

From these three chips the CS8900A is the most popular. It's even so popular that it is emulated in all relevant emulators:

• C64: VICE
• Apple II: AppleWin & GSport
• ATARI 8bit: Altirra

If you are interested in Direct TCP/IP Access and have problems you can contact me...

Regards, Oliver

[tschak909]

Yup, it's most definitely on my list to support. I just need to make sure the basic terminal functionality is working correctly, and I chose the C64 to start, because it could directly leverage some of the Atari resources already extant (namely the default font).

If you'd like to see a demo of this service, contact me. It's very unique, and has the possibility of being the killer app for internet connected vintage computers. :)

[oliverschmidt]

It's very unique, and has the possibility of being the killer app for internet connected vintage computers. :)

Given that I'm both the maintainer of the IP65 library (with the recently added Telnet65 program) and Contiki-for-6502-machines (https://github.com/oliverschmidt/contiki) you don't need to make me interested in any 6502 online thingy ;-)

I've by the way noticed Pluto already before you started with cc65...

[tschak909]

Still..the offer still stands. Ping me if you want a demo. I have a Jitsi Meet server set up for this very purpose (all you need is chrome or firefox), to show the system, running on both modern and vintage machines. There are a lot of features that aren't immediately evident, such as the built in development environment, and built in collaboration features (even multi-user screen sharing).

[oliverschmidt]

Still..the offer still stands. [...]

Thanks :-) However, my time budget is very limited. So I think my time is best spend on assisting you on retro-front-end (aka client) topics rather then looking into the back-end (aka server).

[tschak909]

@oliverschmidt if I do an IP65 target, am I going to be mired in NMI hell?

I'm more than a bit frustrated, at the moment.

-Thom

[oliverschmidt]

I didn't want to actively propose that to you because I fear you may get stuck in different issues but...

The whole Ethernet stuff works without any interrupts. And on the Apple II IP65 doesn't use any interrupts at all. On the C64 and Atari IP65 uses the timer to get a notion of time passing by.

If I were you and wanted to move forward with the actual PLUTO protocol stuff then I personally would certainly start out with IP65 and do serial later. E.g. Contiki doesn't really work well over SLIP 'til today while it works like a charm over Ethernet for many years...

I'd say the biggest issue with Ethernet is supposed to be to get the emulation running - presuming you have no actual Ethernet device. Some background:

Vice, AppleWin and GSport all three use the same Ethernet device emulation core. It does Bridging (in contrast to NAT), so the virtual Ethernet device becomes visible on your current physical network with its own MAC address.

On Windows this doesn't work with a WiFi connection (in contrast to an Ethernet connection) as the usual Windows drivers don't allow the WiFi device to work in promiscuous mode. The two workarounds I know are to a) use some wireless bridge on the Ethernet port of your Windows machine or b) run the emulator inside a Windows virtual machine because programs like VMware offer a virtual Ethernet connection. A page on setting up the Vice Ethernet device: https://www.commodoreserver.com/BlogEntryView.asp?EID=CDB68A9028654CCEA758D37C6DB3E05B

Neither AppleWin nor GSport offer Ethernet emulation on Linux. I personally have no experience at all with Vice on Linux. It's likely not trivial to set up the Ethernet device there: http://www.lemon64.com/forum/viewtopic.php?t=53402&sid=8f01609bff9940bcd67014453eba6b73

Anyhow, you can rather easily check if things generally work for you: Set up Vice to support a virtual RR-Net. Get a recent cc65. Get a recent IP65. Use https://github.com/oliverschmidt/ip65/blob/master/apps/Makefile to build date65.prg and run it in Vice. If it successfully gets an IP address from your current DHCP server and then successfully gets the current date/time from the Internet then you can be sure that things work for you.

So you want to create a TCP client program. Check out https://github.com/oliverschmidt/ip65/blob/master/test/tcp.c

Many things are just the way you expect them:

• https://github.com/oliverschmidt/ip65/blob/master/test/tcp.c#L51
• https://github.com/oliverschmidt/ip65/blob/master/test/tcp.c#L57
• https://github.com/oliverschmidt/ip65/blob/master/test/tcp.c#L68
• https://github.com/oliverschmidt/ip65/blob/master/test/tcp.c#L92
• https://github.com/oliverschmidt/ip65/blob/master/test/tcp.c#L128

Surprising may be https://github.com/oliverschmidt/ip65/blob/master/test/tcp.c#L102 but IP65 doesn't do any "magic" background processing (-> no interrupt hell) so you need to make sure to call into IP65 regularily to keep things going (TCP handshake, ping answers, ...). Note that even if you aren't actually looking for new data this should be done as in https://github.com/oliverschmidt/ip65/blob/master/test/tcp.c#L115 which is inside a for() loop which may run for several seconds when it dumps a larger packet to the console.

But most interesting is how data is received. You provide a call back function on the tcp_connect(). If there's new data it will be called in the context of one of your ip65_process() calls. If you can process the data rather quickly then it's of course cool to do so right inside the call back function. E.g. the Telnet65 Telnet client does all it's protocol handling and screen output inside the call back function. However, if that approach isn't appropriate for some reason the call back function can as well just copy/append the data receive to some buffer (https://github.com/oliverschmidt/ip65/blob/master/test/tcp.c#L35) and the main() loop picks it up there (https://github.com/oliverschmidt/ip65/blob/master/test/tcp.c#L119).

[tschak909]

what's the difference between ip65.lib and ip65_tcp.lib ?

-Thom

[oliverschmidt]

ip65.lib supports only UDP. ip65_tcp.lib supports both UDP and TCP.

[tschak909]

Have made a new branch, ip65, and quickly grafted on support, using the RRNet driver for c64 for testing.

Connection happens, but I'm not seeing anything come across to be decoded...

-Thom

p.s. I find this in test/tcp.c

        if ((i % 11) == 0)
        {
          ip65_process();

          printf("\n$%04X:", i);
        }

Why the explicit process in this loop, or checking for this condition?

-Thom

[oliverschmidt]

Have made a new branch, ip65, and quickly grafted on support, using the RRNet driver for c64 for testing.

I see.

Connection happens,

Cool :-)

but I'm not seeing anything come across to be decoded...

I'd start out with two things as they are easy to do:

• Use Wireshark to make sure that there's actual incoming data on the line.

• Do the "usual" INC $D020 as first thing inside your TCP receive call back to understand if data makes it there.

Why the explicit process in this loop, or checking for this condition?

I explicitly explained that in detail before:

[...] you need to make sure to call into IP65 regularily to keep things going (TCP handshake, ping answers, ...). Note that even if you aren't actually looking for new data this should be done as in https://github.com/oliverschmidt/ip65/blob/master/test/tcp.c#L115 which is inside a for() loop which may run for several seconds when it dumps a larger packet to the console.

[oliverschmidt]

Looking quickly at your ip65 branch I see at least one bug. You don't set the len back to zero after processing the data. So you'll never process any data after the initial one.

Apart from that: You check in your tcp handler, if there's already data there (len != 0) and return right away. I presume that you are aware that this means that the new data will be lost. If you can't be sure that no additional data arrives while you're still processing the prior data then you need some extensible buffer management. Like using the heap to allocate buffers and link them together and have the main loop free the buffers after processing.

Or you need to know from the PLUTO protocol that never more than a certain amount of bytes will be sent for a single "request" from the terminal and append new data to the current data in the buffer. Then make sure that your terminal doesn't "accept" a new "request" before processing those bytes.

A usual TCP/IP stack modifies its TCP receive window to create back pressure on the sender - but IP65 (at least currrently) doesn't support that TCP feature.

[oliverschmidt]

Above I wrote:

If you can process the data rather quickly then it's of course cool to do so right inside the call back function. E.g. the Telnet65 Telnet client does all it's protocol handling and screen output inside the call back function. However, if that approach isn't appropriate for some reason [...]

Looking at your code I don't see a reason why the approach described isn't appropriate for you. Please don't just copy the code I pointed you to but read my text too ;-)

If you call decode() right from within you tcp handler then you don't have to fiddle with any buffering at all. If decode() takes "some" time this means that ip65_process() is called less often meaning the IP65 delays TCP ACKs for some time. This means that the peer waits some time for the ACKs and therefore doesn't send the next data right away. So as long as you don't delay things so much that the peer starts to re-transmit stuff you get a nice coupling of the peer sender and the IP65 receiver.

E.g. for the Telnet client doing a ls -lon a very long directory can easily take 1 minute to scroll by - without the Telnet client sending any data. But doing the screen output (incl. "slow" scrolling) inside the tcp handler makes the Telnet server send the data in just the right speed because it waits some time for the TCP ACKs.

[tschak909]

Have made a new version of the ip65 branch based on the new protocol decoder. It works, but it stops processing after roughly 10 seconds. Program is still active, as I can press F1 and quit the program.

This happens with buffer processing happening inside the receiving callback, or processing the resultant buffer in the main loop...what am I doing wrong?

-Thom

[oliverschmidt]

Have made a new version of the ip65 branch based on the new protocol decoder.

:-)

It works, but it stops processing after roughly 10 seconds. Program is still active, as I can press F1 and quit the program.

I see.

This happens with buffer processing happening inside the receiving callback, or processing the resultant buffer in the main loop...what am I doing wrong?

I don't see why it should help in your situation to buffer anything. So I changed your code to

void tcp_recv(const unsigned char* tcp_buf, int tcp_len)
{
  ShowPLATO((padByte*)tcp_buf, tcp_len);
}

and just ip65_process() plus kbhit()/cgetc()/tcp_send(..., 1) in the main loop.

Above I wrote:

Use Wireshark to make sure that there's actual incoming data on the line.

Have you actually done so? Likely not, because if you would have done you would have seen that the server closed the connection due to many TCP retransmissions.

It doesn't help if you try to do TCP programming without running Wireshark and at least roughly understanding its output !!!

Above I wrote:

So as long as you don't delay things so much that the peer starts to re-transmit stuff you get a nice coupling of the peer sender and the IP65 receiver.

You see, exactly that happens. So your code is just too slow. When I put the Vice speed to 500% then things seem to work pretty nicely. I get to the logon screen and can enter data. Use Wireshark to see and understand things !!!

I made a change to IP65 (https://github.com/oliverschmidt/ip65/commit/9148f9ef3aec0dd44eda3cab7110ecd63c0beb57) that seems to improve the situation. With that change in place I get to logon screen often with Vice speed set to 100%. So with optimizations in PLATOTerm necessary anyhow from the user experience POV it could work well.

Famous last words: Use Wireshark !!!

[tschak909]

Do I need to basically do a static build of PLATOTerm for each ethernet driver possibility?

-Thom

[oliverschmidt]

No, ip65_c64.lib contains all available Ethernet drivers (see https://github.com/oliverschmidt/ip65/blob/master/drivers/Makefile#L16).

In case you should have at one day several working cc65 SER drivers you could do the same there by doing a ser_install() with one after the other until one succeeds. This is what ip65_c64.lib does internally with the Ethernet drivers.

[tschak909]

@oliverschmidt at the end of my #@($%#@%(@ rope.

Spent two weeks trying to squeeze every spare cycle I could out of the drawing routines...ultimately had to back out unrolling the draw routines because it ballooned my code by 12K...

No matter what, I can't keep the connection stable.

This branch has the ip65 code: https://github.com/tschak909/platoterm64/tree/add_ip65

It automatically loads the c64combo lib, and sets the default preferences to dhcp...

I need help, or I'm going to start punching holes in walls...

-Thom

[tschak909]

The latest push seems to work best, but some stuff still gets lost. Adding additional ip65_process() calls throughout drawing code seems to worsen the output from wireshark. #@(%@#%(@#%@#( !!!!

-Thom

[oliverschmidt]

I wrote:

No, ip65_c64.lib contains all available Ethernet drivers (see https://github.com/oliverschmidt/ip65/blob/master/drivers/Makefile#L16).

Then you write:

It automatically loads the c64combo lib [...]

Any reason for using the wrong library?

No matter what, I can't keep the connection stable.

Are you testing with VICE? Can you reproduce my experience that increasing the VICE speed makes it work reliably? I went only to the logon screen. It would be interesting to know if it is really fully functional with increased VICE speed.

I need help, or I'm going to start punching holes in walls...

I wrote above:

I don't see why it should help in your situation to buffer anything. So I changed your code to

void tcp_recv(const unsigned char* tcp_buf, int tcp_len)
{
  ShowPLATO((padByte*)tcp_buf, tcp_len);
}

but looking at your current code I still see buffering. Any reason you don't follow my advice?

[tschak909]

If you look at the latest check out in the add_ip65 branch you will see the code that gets the farthest in the system. Turning on warp speed does not seem to help. You can't just call show Plato directly with the buffer, because double IAC escapes must be removed from the data before decoding. On Sat, Jun 30, 2018, 1:18 AM Oliver Schmidt notifications@github.com wrote:

I wrote:

No, ip65_c64.lib contains all available Ethernet drivers (see https://github.com/oliverschmidt/ip65/blob/master/drivers/Makefile#L16).

Then you write:

It automatically loads the c64combo lib [...]

Any reason for using the wrong library?

No matter what, I can't keep the connection stable.

Are you testing with VICE? Can you reproduce my experience that increasing the VICE speed makes it work reliably? I went only to the logon screen. It would be interesting to know if it is really fully functional with increased VICE speed.

I need help, or I'm going to start punching holes in walls...

I wrote above:

I don't see why it should help in your situation to buffer anything. So I changed your code to

void tcp_recv(const unsigned char tcp_buf, int tcp_len) { ShowPLATO((padByte)tcp_buf, tcp_len); }

but looking at your current code I still see buffering. Any reason you don't follow my advice?

— You are receiving this because you were assigned.

Reply to this email directly, view it on GitHub https://github.com/tschak909/platoterm64/issues/3#issuecomment-401520706, or mute the thread https://github.com/notifications/unsubscribe-auth/ACocZhz_1UHtfkJw60OehHQeMxmr_1Nxks5uBxgngaJpZM4UDLaw .

[tschak909]

I am respectfully asking for your help, and for you to bear with my temporary lapses of ignorance, because I am not familiar with the ip65 codebase. I can give you a username and password so you can see the effects logging into the system. Sometimes only data is lost, however sometimes the whole program will appear to hang waiting for input. Wireshark is showing more errors the more calls to IP 65 process that are attempted. This also coincides with the display appearing 2 Miss more information or outright hang.

[oliverschmidt]

Turning on warp speed does not seem to help.

That's maybe/likely because you still do that buffering stuff.

You can't just call show Plato directly with the buffer, because double IAC escapes must be removed from the data before decoding.

I saw your IAC comment. But as far as I understand you could still modify the buffer as you see fit and call showPlato from within the tcp receive function.

I can give you a username and password so you can see the effects logging into the system.

Please do so.

[..] the more calls to IP 65 process that are attempted [...]

I already tried to explain that this isn't the point! The server sends faster than the terminal can process the data. Calling IP 65 process more often rather makes the server believe that the terminal is idle waiting for data.

You've been trying to get flow control working for serial. With Ethernet you have sort of the same problem but on a different level. As I explained further above TCP has an explicit flow control mechanism but IP65 doesn't support it. So you have only very limited options to get the server and terminal in sync.

[tschak909]

The one flow control mechanism that PLATO supports, is XON/XOFF, but as of yet, haven't been able to get that to work properly. I've done this before, with other terminals (e.g. Android), where I'm just doing basic threshold checking against a ring buffer...but that doesn't seem to work here.

What if we added an API point in ip65 to set the receive window to 0?

Also, there's a test user you can use:

user: sam group: atari password: 12345

-Thom

[tschak909]

btw, make dist will make the appropriate d64 image.

[tschak909]

Okay, @oliverschmidt I did what you asked. I moved all the translation over to ShowPLATO itself, and there is nothing in the main loop, but ip65_process(), and the recv only consists of passing the buffer straight to ShowPLATO...

Adding the IAC escape check was enough to throw this thing into hang-ville...

and all I see in Wireshark is a sea of black and red (retries and RSTs). urgh.

-Thom

[oliverschmidt]

What if we added an API point in ip65 to set the receive window to 0?

That's exactly what would/could/should be done. However, it's not that easy, especially as IP65 is written in asm and not written by me.

Anway, if that's the only option I'll likely (try to) do it as I need it sooner or later myself. I want to implement a WGET and that almost certainly needs such an API to control the HTTP server speed during its own disk I/O.

I did what you asked.

I'll check it out when I have an option to do so. But this will not be in the next days :-(

[oliverschmidt]

It automatically loads the c64combo lib [...]

Any reason for using the wrong library?

As of today you're still using the wrong library.

Just in case the reason should be that replacing it with the correct ip65_c64.lib causes link failures: That's because your Makefile names the libraries on the cl65 command line in alphabetical order. And with "combo..." it happens to work while with "ip65..." it happens to fail.

Anyhow, you must make your Makefile to first name ip65_tcp.lib and then ip65_c64.lib.

[oliverschmidt]

The program fails on loading the serial driver even if in Ethernet mode. I suggest to change that behavior.

[oliverschmidt]

Hm, to me it seems to generally work. Looking at Wireshark the server seems to need many seconds "think time" now and then. But I have basically no clue what to expect...

[tschak909]

Folding in the ip65 support is enough to overflow every target except the c64 target by varying amounts. :( @#(%@#(% !!!!!

-Thom

[oliverschmidt]

Folding in the ip65 support is enough to overflow every target except the c64 target by varying amounts. :(

That's what I already anticipated. Therefore all the effort I put into my comments https://github.com/cc65/cc65/issues/698#issuecomment-403272492, https://github.com/cc65/cc65/issues/698#issuecomment-403272773, https://github.com/cc65/cc65/issues/698#issuecomment-403273144 and https://github.com/cc65/cc65/issues/698#issuecomment-403273540.

If you follow them you'll end up with roughly the same amout of memory you have on the C64.

The same goes in general for the Atari: Switch from the target atari to the target atarixl, read http://cc65.github.io/doc/atari.html#ss3.2 and http://cc65.github.io/doc/atari.html#ss10.2.

[tschak909]

If I switch to the atarixl target, that cuts off a sizable chunk of machines that can run this :(

-Thom

[oliverschmidt]

Yes, if the atarixl target is no option for you - and you already did the things you can do to reduce the code size like using only statically linked drivers (!) - then C is no option for you and you need to (at least partly) resort to asm to reduce the code size.

If it‘s only relatively few bytes then different compiler optimization options and reducing the stack size can help. And on the Atari you can go for a „minimized“ DOS / specific loader which allows for a lower start address. However, l‘m no specialist for the latter...

[oliverschmidt]

Or simply declare it as a non-problem: As far as I understand you memory problem is specific to the Ethernet transport. There have only been pretty few Ethernet carts been made for the Atari - and it doesn't look like more will follow - so Ethernet for the Atari might be seen as primarily a way to hook up an emulator more easily then the serial setup. And the only Atari emulator with Ethernet support of course emulates an Atari XL.

[oliverschmidt]

I wanted to let you know that IP65 isn“t located anymore at

http://oliverschmidt.github.io/ip65

but now at

http://cc65.github.io/ip65

The actual Git repo URL has changed accordingly.

Everything else is supposed to be just as before. In case you encounter URLs under my control still pointing the old location please let me know.

Sorry for any inconvenience - but the new GDPR made this necessary from my POV.

[tschak909]

hey @oliverschmidt on a lark, I did a quick re-port of the PLATOTerm code, now that the Atari has (MUCH FASTER) text output.

Due to the way Altirra exposes its ethernet device, I can't tap wireshark on it, but, I recorded a video showing the result.

https://www.youtube.com/watch?v=5Ewjh0OiCNM

-Thom

[oliverschmidt]

Yep, the speed looks pretty impressive.

[tschak909]

@oliverschmidt Silly question,. the W5100 support that you have for Apple II, does it work for the original Uthernet as well as the Uthernet II? (long winded way of asking will it work in AppleWin?)

If so, I can graft on ethernet support onto the apple2 version.

-Thom

[oliverschmidt]

The W5100 is the Ethernet chip on the Uthernet II. So my W5100 code neither runs on the Uthernet (I) nor the AppleWin emulator.

[tschak909]

damn. ok.

On Sun, Sep 16, 2018 at 2:37 PM Oliver Schmidt notifications@github.com wrote:

The W5100 is the Ethernet chip on the Uthernet II. So my W5100 code neither runs on the Uthernet (I) nor the AppleWin emulator.

— You are receiving this because you were assigned.

Reply to this email directly, view it on GitHub https://github.com/tschak909/platoterm64/issues/3#issuecomment-421819493, or mute the thread https://github.com/notifications/unsubscribe-auth/ACocZlwid3tn4Ge4jLd4ZMCTS2pkiRZ1ks5ubqhsgaJpZM4UDLaw .

[tschak909]

@oliverschmidt can I beg you to take a whack at trying to add IP65 to PLATOTERM? I have now tried five times to graft it on, and can't seem to get it to work fast enough in any case.

-Thom

[beretta42]

You know another way to flow control would be to not ack the tcp segments until plato is done with them. Changing the window size is a secondary method, more of a suggestion to the remote

[jonnosan]

Hi folks, I came across this while googling to see if people were doing anything interesting with ip65. It's been over a decade since I hacked up the ip65 tcp code, and I have done hardly any coding of any sort since, let alone 6502 assembler but I've been rereading the tcp.s code (and remembering what I was probably thinking at the time), and I think my intention was to advertise a recv buffer size that meant the other end of the connection only ever sent one packet at a time before waiting for an ACK - not at all efficient especially on high latency links but sort of good enough to work most of the time. Anyway I am now interested in why that nasty hack is failing here. Is there any chance one of you could send me a wireshark dump showing the problem (retransmits and RSTs) ? I will also start setting up a dev environment to see I can replicate myself .