search

Interlisp / medley

The main repo for the Medley Interlisp project. Wiki, Issues are here. Other repositories include maiko (the VM implementation) and Interlisp.github.io (web site sources)
https://Interlisp.org
MIT License
375 stars 19 forks source link

set up and test Interlisp networking tools #573

Closed masinter closed 1 year ago

masinter commented 2 years ago

Network: Medley has full protocol stacks, in Lisp, for three families of network protocols: PUP, XNS, and TCP/IP. Can we demonstrate?

nbriggs commented 2 years ago

At the moment, we have access to the (raw ethernet) network only on a Solaris system. In order to demonstrate network access on other systems we should look to the Berkeley Packet Filter interface, rather then the DLPI or NIT interfaces that are/were available on SunOS/Solaris.

tuexen commented 2 years ago

Where is the TCP/IP source code located?

nbriggs commented 2 years ago

@tuexen -- The Lisp implementation of TCP/IP was deleted from the repo -- better to use the underlying OS implementation for anything we can, I think. For PUP and XNS we have to use the Lisp implementation and the underlying OS raw ethernet interface (e.g., the /dev/bpf)

nbriggs commented 2 years ago

@tuexen -- In the medley repo: % git whatchanged | grep TCP

tuexen commented 2 years ago

@nbriggs Thanks for the information. I was looking for the TCP code and did not find it... I have not idea what PUP and XNS is, I was just wondering if there is interest in using the TCP stack from the Lisp code base. I'm somewhat familiar with TCP and have an interest in Lisp. So I was wondering if there is interest in getting the old codebase to work... But using the host stack gives you a much more modern TCP implementation...

Thanks for the quick response!

nbriggs commented 2 years ago

@tuexen -- PUP => PARC Universal Packet, the packets on the original 3 Mb ethernet (http://www.bitsavers.org/pdf/xerox/parc/techReports/CSL-79-10_Pup_An_Internetwork_Architecture_Jul79.pdf), and XNS => Xerox Network Services, the packets on the 10 Mb ethernet. Novell Netware copied the XNS Internet Datagram Protocol (IDP) for their IPX protocol. The old Lisp TCP/IP implementations were deprecated by the original maintainers after Interlisp started to be run on the emulator. On the original D-machines Interlisp was the OS, and had to implement everything including the raw ethernet packet handling, the page fault handlers, the disk controller operations, the file system etc., ... It became a lot easier to maintain when that could all be delegated to the OS.

Anzus commented 2 years ago

It followed pretty closely the design of the BSD stack from the early ‘80s, but implemented in Interlisp-D to work on 10-Mb Ethernet. With the transition to non-Xerox hardware the decision was made to use the stack provided by the host OS since the support and improvements would come from those OS maintainers. TCP/IP’s come a pretty long way since then and I wouldn’t be surprised if there aren’t some incompatibilities against current stacks.

On Dec 25, 2021, at 3:25 PM, Michael Tüxen @.***> wrote:

@nbriggs https://github.com/nbriggs Thanks for the information. I was looking for the TCP code and did not find it... I have not idea what PUP and XNS is, I was just wondering if there is interest in using the TCP stack from the Lisp code base. I'm somewhat familiar with TCP and have an interesting in Lisp. So I was wondering if there is interest in getting the old codebase to work... But using the host stack gives you a much more modern TCP implementation...

Thanks for the quick response!

— Reply to this email directly, view it on GitHub https://github.com/Interlisp/medley/issues/573#issuecomment-1001078856, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACBP2DFZQNP6X5KKJHZYZSTUSZAERANCNFSM5IITCKUQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you are subscribed to this thread.

tuexen commented 2 years ago

It followed pretty closely the design of the BSD stack from the early ‘80s, but implemented in Interlisp-D to work on 10-Mb Ethernet. With the transition to non-Xerox hardware the decision was made to use the stack provided by the host OS since the support and improvements would come from those OS maintainers. TCP/IP’s come a pretty long way since then and I wouldn’t be surprised if there aren’t some incompatibilities against current stacks.

That is what I'm interested in. I'm working on today's TCP stack of FreeBSD and would expect that it can interoperate with a stack just doing the initial specification (RFC 793)...

tuexen commented 2 years ago

@tuexen -- PUP => PARC Universal Packet, the packets on the original 3 Mb ethernet (http://www.bitsavers.org/pdf/xerox/parc/techReports/CSL-79-10_Pup_An_Internetwork_Architecture_Jul79.pdf), and XNS => Xerox Network Services, the packets on the 10 Mb ethernet. Novell Netware copied the XNS Internet Datagram Protocol (IDP) for their IPX protocol. The old Lisp TCP/IP implementations were deprecated by the original maintainers after Interlisp started to be run on the emulator. On the original D-machines Interlisp was the OS, and had to implement everything including the raw ethernet packet handling, the page fault handlers, the disk controller operations, the file system etc., ... It became a lot easier to maintain when that could all be delegated to the OS.

Thanks for the clarification. Unfortunately, I have no experience with PUP and XNS. My interest would have been the TCP stack in lisp...

nbriggs commented 2 years ago

You're more than welcome to check out the old code that you'll find from the git whatchanged command I mentioned above. I am looking to get the BPF interface for raw ethernet packets working, and that should allow the Lisp TCP/IP code to run again.

tuexen commented 2 years ago

You're more than welcome to check out the old code that you'll find from the git whatchanged command I mentioned above. I am looking to get the BPF interface for raw ethernet packets working, and that should allow the Lisp TCP/IP code to run again.

That should work... You have to make sure that the host stack does not interfere with the stack in the Lisp VM. Drop me a note once it is finished or if you have questions related to this...

masinter commented 2 years ago

you might want to check in the interlisp/history repository; the TCP files were moved to library/Obsolete/ by 2016. history-tcp-files.txt There might be some possibility of running Medley 2 on DOS, but the state of emulating old network cards in FreeDOS or DosBox is probably a problem.

What I remember was that trying to implement a full TCP protocol stack by reading RFCs was a learning experience. The XNS and PUP documents were a lot clearer (or maybe the implementors were down the hall).

masinter commented 2 years ago

The work on "dodo" (XNS in Java) noted above might provide an opportunity to revive Medley networking?

masinter commented 2 years ago

This issue has more useful information than #456 so I'll mark that one duplicate and close it.

devhawala commented 2 years ago

Hello,

the branch dodo-nethub-support in my clone of Maiko has a version of Maiko allowing to connect directly to a Dodo Nethub for using Dodo XNS services.
(for details about building and usage see Using Dodo-networking with Maiko)

The Dodo Nethub support was implemented and tested with x86_64-Linux, so the following information and tour through a demo apply to this platform.

The file dodo-and-medley.zip attached to this post contains the following items:

Unpacking the dodo-demo.tgz archive creates a directory dodo-demo with a prepared Dodo setup, based on the standard Dodo distribution with an additional file service medley having the file drawer internal holding some documentation files taken from the subdirectory docs/internal of the Medley github project. The print service poseidon is prepared to produce PDF files with support for standard Xerox fonts (Classic, Modern fonts).
Remark: producing PDF for printouts requires that the program ps2pdf is available through the shell PATH, so make sure that Ghostcript is installed on your system and the PATH environment variable is set up accordingly.

For running the Dodo demo system, open 2 shells, change to the dodo-demo directory in each and:

For shutting down the Dodo demo system, simply stop the programs with Ctrl-C in reverse order.

The Medley environment i used for my experiments was the release medley-211122 (loadups and runtime, downloaded 2021-11-24), using the full sysout. Newer releases also worked but were not my "daily" test environment. In any cases, XNS networking must be initialized by running the function \NSINIT when a fresh "loadup" sysout is started resp. XNS networking was not used so far (without \NSINIT Lisp will try forever to reach a Clearinghouse).

For a quick tour through the combined Medley/Dodo demo environment:

And then in the Lisp environment:

Experienced Interlisp-D users probably know more use-cases for using XNS services, the above were my standard tests for extending Dodo to support the little Medley-specialities at XNS filing and printing level.

Greetings
Hans

Attachment: dodo-and-medley.zip

masinter commented 2 years ago

Astounding!

devhawala commented 2 years ago

Hello,

looking again at the file i uploaded yesterday, i've seen that the filenames in the archive were messed up...

So please rename the files after unpacking as follows to match the names mentioned in my previous post:

from: to:
dodo-demo.TGZ dodo-demo.tgz
dodo-demo.sh run-medley
dodo-demo.pdf job_0183_7090_DD36.pdf

(Github does not accept all extensions for file uploads to a post, e.g. insisted on an uppercase .tgz extension and run-medley (or run-medley.sh) was simply rejected; so before putting them into a single ZIP-archive, i wanted to rename the .TGZ back to lower case: that's when scrambling the filenames must have happened, as it seems i had all 3 files selected when renaming with the Windows10-Explorer, which fired this super-Windows feature where all 3 got the same name and the .TGZ extension kept being upper case)

Sorry for the inconvenience Hans

masinter commented 2 years ago

Success!

I'm not sure what I did differently -- I was dying with a stack overflow trying to CH.FINDSERVER -- but today, after several similar failures, it all worked!

image debuglog.txt

demoing CoLab and things like it (didn't Notecards have some networking possibilities?) maybe TELERAID?

devhawala commented 2 years ago

Congratulations!

Yes, i also had stack overflows with some of the fresh loadup images, but others worked fine. Not sure if some of the failing ones have less things preloaded and one crucial thing is missing.

The important thing is to call \NSINIT first! Can this be included in the loadup generation so this can't be forgotten?

BTW: is there a mail client in Interlisp to check if exchanging mails with Star or Viewpoint works?

Hans

nbriggs commented 2 years ago

Larry can look back at how the network initialization used to work -- he removed the Lisp code that used to get the ethernet properly initialized in order to speed things up when the network wasn't present. It's NOT as simple as calling \NSINIT in the loadup.

Yes, Lafite is the mail client that understands XNS mail (and Grapevine on a PUP network).

nbriggs commented 2 years ago

Could you rebase onto the current maiko master branch -- all the Addr68k_from_LADDR etc. have been replaced by NativeAligned{2,4}FromLAddr to reduce the compiler warnings for incorrect alignment.

nbriggs commented 2 years ago

I'm looking at trying to reduce the entropy in the code, and this seems like a good time to look at splitting the ethernet handling code up in to separate files so that you don't have to guess which definition of something is active for which set of #ifdef options.

devhawala commented 2 years ago

just pushed a new Maiko version in my branch:

Compiles and works on Linux-x86_64 and Cygwin, the new command line option is untested however

ok, that's all for today...

nbriggs commented 2 years ago

OK, I'll look. It should probably reuse the "-t" flag that is currently (supposed to...) set the timer frequency.

The style for options should be '-x' (or '-xx') for short options or '--longer-option' (double dash). We're going to rationalize the option processing though, so you probably don't need to deal with that.

masinter commented 1 year ago

I'm going to this issue into a Discussion about possible networking goals (for TCP) since it's not clear what the goal is and open an issue to document the 'dodo' networking for PUP and XNS access via emulation