Khan/Conn: Working client for sending commands and returning output to an urbit from outside of it.

[zalberico]

Khan: The existing urbit vane part of the code Conn: The existing runtime part of the code Client: The external tool used to interact with conn in order to execute commands from outside of urbit.

Khan and Conn exist to allow a user to execute commands on an urbit from the outside and get the results. Currently though, there is no client for interacting with this service.

@ashelkovnykov is working on a client that will make it possible execute commands and get the relevant output from the already existing Khan/Conn service in urbit. The primary consumer of this client will be Tlon Hosting and its design is specifically targeted for our needs.

This issue will be the source of truth about the state of things and the place to discuss problems or questions. When a PR exists, this is the issue it will reference.

[ashelkovnykov]

Khan is the Arvo vane responsible for running threads (prior to Khan, running threads was a more cumbersome process involving directly calling the spider agent). conn.c is an I/O driver that handles all of the tasks in the 2x2 matrix formed by "read/write" and "Arvo/Vere". In particular, conn.c actually has two methods for writing to Arvo: launching threads through Khan, and directly injecting events into the event loop.

Work by @mopfel-winrux on #155 will shortly make using conn.c easier by allowing a very bash-scriptable way of generating jammed nouns. Using conn.c to launch threads via Khan also makes it possible to run arbitrary Hoon code from Earth. However, it's currently a more cumbersome process: the threads need to be written and |commit to a desk in Arvo, and then called by name via conn.c. @philipcmonk has proposed a new thread in %base that could parse and run arbitrary Hoon code, thus closing the loop: both methods in conn.c would allow running arbitrary Hoon code from Earth.

Tlon Hosting has three needs:

1. Easily send commands to one or more ships from Earth
2. Easily scry data from one or more ships from Earth
3. Passively consume observability data emitted by ships on Earth

Only needs 1 and 2 are relevant for this GitHub issue.

Before speaking with @istathar, the quickest way to meet these needs seemed to be a command-line thin client. Written in Python or Rust, this command line tool would contain light text formatting for the purposes of sending commands to conn.c as threads. It would help users create Hoon code as bash text which could be piped into the Vere eval command (after #155) to create jammed nouns, which could then be piped into the conn.c socket of a running ship. The pipeline would have looked something like these examples:

thin-cli 'hi' -a '~fed' | ./vere eval --jam | nc -U ~/urbit/zod/.urb/conn.sock | ./vere eval --cue
thin-cli --arbitrary '(add 2 2)' | ./vere eval --jam | nc -U ~/urbit/zod/.urb/conn.sock | ./vere eval --cue

After speaking with @istathar, this is not the direction in which we presently want to go. Hosting ultimately wants to move towards a thick client/hypervisor capable of more sophisticated automation. In addition, they already have access to a Haskell library for noun representation. This means that they already have the tools available to directly jam and cue nouns, handling the output from interactions with conn.c programmatically. Based on these circumstances, it seems that the better approach would be to immediately begin work on a thick client with the minimum viable number of features. The idea is that this thick client will start out rather skinny, but will rapidly bulk up as Hosting uncovers features they need and the Runtime team assists in providing them. It will also bulk up as the Hosting engineers add more features on their own (after all, they have both a library for representing nouns and a way to pass arbitrary nouns as events to Arvo).

One thing that's nice about this is that it works to keep the thin waist of Urbit as newt-encoded jammed nouns. newt-encoded jammed nouns are already the way in which King and Serf (soon to be Urth and Mars) communicate, and as Urth dissolves into multiple I/O drivers, the plan is for those to communicate using newt-encoded jammed nouns, too.

In summary: this has shifted to be a Tlon internal tooling discussion, and unless someone from UF or the community wants to jump in to take the first idea from above, I don't think there's more runway for this issue.

I'll note that @istathar intends for the Tlon Hosting team to open source as much of this tooling as possible.

[zalberico]

This is really great @ashelkovnykov thanks.

[istathar]

In passing, I'll note that I've discovered that one of our predecessors seems to have saved a copy of much of the original Haskell runtime work in an internal Tlon repo. It's rather annoying that he didn't credit the origin of this code at all in that repo (would have saved me a lot of digging and forensic comparisons) but as a result I've got access to what appears to be working code that knows how to jam and cue.

It'll need testing (yeah), but given that it came out of urbit and is MIT licensed we can re-release the necessary bits as appropriate. I'll start figuring out the logistics of that.

So we have both the pieces we need to support @ashelkovnykov's work and the ability to open source our helper when the time comes.

We'll continue this internally.

[zalberico]

We're reopening this and @ashelkovnykov is going to do the following (from conversation with @philipcmonk @lukebuehler and @ashelkovnykov)

spec:

• should be able to run a thread from a text file and produce the result as a jammed noun (written in rust)

goals:

• validate that conn works correctly
• document how to talk to conn
• communicate what you can do with conn
• have a little tool for doing this one-off outside of hosting

tasks:

• write a thread that runs a text string as a thread
• write a rust client to hook up to the socket, send nouns, receive nouns

[ashelkovnykov]

Expanding on what @zalberico wrote:

We've decided to reopen this issue for several reasons, but the ones which are publicly relevant are:

1. A public client would serve as documentation and an example for how to communicate with running ships via conn.c
2. Usage of such a client would verify that conn.c is working as intended under real use cases

Preliminary work involves producing a thread capable of running arbitrary Hoon code passed as text input (as proposed by @philipcmonk).

Afterwards, my plan is to produce two clients for interfacing with conn.c. The first will be a very simple thin client like the one I described in my above post. This first client likely doesn't even need to be in Rust, since it can lean heavily on the vere eval work by @mopfel-winrux (completing this client may involve lightly extending said work).

The second client will be written in Rust, taking advantage of the Rust noun representation library produced by @mcevoypeter. This client will form the basis for a thick client which has access to more sophisticated tooling (e.g. direct event injection, native noun representation, etc.).

[mopfel-winrux]

Do we need the --cue flag for the client?

[ashelkovnykov]

@mopfel-winrux I'd like to have it. Are you asking because you're planning to implement it?

[mopfel-winrux]

@ashelkovnykov Its on my todo list but I had it at a low priority because I didn't know of an immediate use for it. If it's gonna be used I can prioritize more.

[ashelkovnykov]

@mopfel-winrux and I synced offline:

Adding a --cue flag is a natural extension of the work he did for his UF grant (#155). However, the grant is complete and Native Planet does not currently have a need for --cue, so he was going to add it later just for fun/to keep contributing.

I might need it soon, so he shared his existing work/insight with me, and we agreed that I'll give him a heads up and take the ticket myself if need be.

[ashelkovnykov]

Feb 13 EOD update:

• Most basic version of a thread which can run arbitrary Hoon working
• Playing around with variations
  • Not sure which version is best, will get settled in code review
• Should be done tomorrow, will move on to most basic version of conn.c CLI

[ashelkovnykov]

Feb 14 EOD update:

• Additional variations for running arbitrary Hoon code working
• Still have 1 variation that needs to be completed
• Plan is to sync w/ @philipcmonk to get feedback regarding how to consolidate these variations into a PR

[ashelkovnykov]

Feb 16 EOD update:

• Synced with @philipcmonk regarding how to consolidate the variations of threads to run arbitrary Hoon into a few useful threads
• Applied feedback
• Submitted PR linked above

[ashelkovnykov]

Feb 17 EOD update:

• Merged arbitrary Hoon threads
  • Got feedback from @belisarius222 that I would like to apply during the second pass; could pass context as a single source file for processing by Ford
• Have a working first draft of a bash thin client which uses the aforementioned threads for sending arbitrary Hoon to Khan via conn.c
  • Has issues with certain data types/marks
  • Hard to debug, since output is given as jammed noun
  • Going to implement --cue flag (partner to --jam from #155)
    • Will also implement the feedback left by @joemfb on that PR

[ashelkovnykov]

Feb 21 EOD Update:

• Implemented --cue flag for Vere eval command
• Minor updates to eval threads in Urbit to work better with client
• 95% done with thin client
• Submitting PRs for all of the above tomorrow

[ashelkovnykov]

FEB 22 EOD Update:

• Ran into a bug after finishing thin client; took all day to track down and resolve
• Thin client done, submitting PRs tomorrow

[ashelkovnykov]

FEB 23 EOD Update:

• Added some more convenience features to thin client
• Tested some more scenarios
• PRs submitted for thin client dependencies; will submit PR for client itself after they're merged

[ashelkovnykov]

This isn't complete yet - after #255 and #6353 are both merged, I'll submit a PR with a thin client implementation. After that's merged, I think there's still a discussion to be had regarding whether that's resolved this issue: whether the UF or just Tlon want an additional first implementation of a thick client (i.e. something in not-bash that can natively read nouns).