Make exploring complex results convenient - inspiration: the CIDER Inspector.

[orestis]

As per https://github.com/BetterThanTomorrow/calva/issues/1327#issuecomment-1120498081, I am opening an issue to discuss the feature of a "data inspector", essentially mimicking the high-level goals of the CIDER inspector.

The value inspector allows you inspect and navigate the structure of data. While you can use it for pretty much anything (e.g. primitive data types, var, ref types) it’s most useful when you’re dealing with (deeply) nested collection-like data types (e.g. a vector of maps).

UPDATE by @PEZ: Moved the specific implementation suggestion to a comment. Keeping the problem description here.

[orestis]

@PEZ Apologies for making this 100% about Portal - I've renamed the issue to reflect it in the title. If you're reluctant to consider this for inclusion in Calva, I'm happy to keep my joyride script (a million thanks for having joyride, it makes cool things possible!).

Perhaps I'll need to open a few issues/PRs with Calva/Portal so that things can be a bit smoother, but even now I can use this in my daily workflow.

[PEZ]

Moving this to a comment on this issue, to keep it more general on the need to inspect results. From @orestis:

I never liked the actual UI of the CIDER inspector, but I think the restrictions of the Emacs UI are to blame. I instead would suggest building on top of Portal, which offers a turn-key solution with viewers, history of results and more.

The high level goal (for me) is to be able to use this everywhere that Calva can connect, that is, no runtime dependencies.

I have here a proof-of-concept joyride script that can send any EDN value over to portal, piggy-backing on the Calva command "calva.copyLastResults", since I couldn't find any API in Calva to do the same. It could be simplified and expanded quite a bit if Calva had an API, but even like this it works.

The script relies on having Portal installed via the Portal VSCode extension, and actually opened. I couldn't find a way to do this from the command palette, so I defined this keybinding:

  {"key": "Ctrl-Cmd-P P",
  "title": "Open Portal",
"command": "extension.portalOpen"
},

The script could try and detect if Portal is not open, and subsequently open it.

(ns portal-last-eval
  (:require ["vscode" :as vscode]
            ["http" :as http]
            ["fs" :as fs]
            [clojure.edn :as edn]
            [clojure.string :as string]
            ["path" :as path]))

(defonce DEBUG (atom false))
(reset! DEBUG true)

(defn info [& xs]
  (vscode/window.showInformationMessage (string/join " " xs)))

(defn dprintln [& args]
  (when
   @DEBUG
    (apply println args)))

(defn make-http-req [options data cb]
  (let [req (.request http options
                      (fn [res]
                        (cb (.-statusCode res))
                        (when @DEBUG
                          (println "gr.orestis.workflow res" res (.-statusCode res))
                          (.on res "data" (fn [data]
                                            (println ">>>" data))))))]
    (.on req "error" (fn [error]
                       (println "gr.orestis.workflow error" error)))
    (when data
      (.write req data))
    (.end req)))

(defn read-portal-host-port []
  (let [edn (str (fs/readFileSync (path/join vscode/workspace.rootPath
                                             ".portal" "vs-code.edn")))
        {:keys [host port]} (clojure.edn/read-string edn)]
    {:host host :port port}))

(defn send-portal-value [{:keys [host port]} edn]
  (let [http-options #js {:hostname host
                          :port port
                          :method "POST"
                          :path "/submit"
                          :headers #js {"Content-Type" "application/edn"}}]

    (make-http-req http-options edn (fn [status-code]
                                      (if (= 204 status-code)
                                        (info "Portal:" (subs edn 0 10))
                                        (info "Portal err:" status-code))))
    edn))

(defn print-clipboard []
  (.then (.readText (.. vscode -env -clipboard))
         (fn [text]
           (println "CLIP " text))))

(defn get-new-clipboard [prev-text-atom cb tries]
  (let [clipboard (.. vscode -env -clipboard)]
    (if (< tries 0)
      (cb (str ":" (name ::gave-up)))
      (->
       (.readText clipboard)
       (.then (fn [text]
                (dprintln "GET NEW CLIPBOARD" @prev-text-atom text)
                (if (and (not= ::pending @prev-text-atom)
                         (not= text @prev-text-atom))
                  (cb text)
                  (js/setTimeout
                   #(get-new-clipboard prev-text-atom cb (dec tries))
                   100))))))))

(defn get-last-eval-results []
  (let [clipboard (.. vscode -env -clipboard)
        prev-text-atom (atom ::pending)

        _ (-> (.readText clipboard)
              (.then (fn [text]
                       (dprintln "READ CLIPBOARD" text)
                       (dprintln "EXEC CIMMAND")
                       (reset! prev-text-atom text)
                       (vscode/commands.executeCommand "calva.copyLastResults")
                       (dprintln "DONE EXEc"))))]
    (js/Promise.
     (fn [resolve _reject]
       (dprintln "GET NEW CLIPBOARD")
       (get-new-clipboard prev-text-atom resolve 10)))))

(defn send-last-eval-to-portal []
  (.then (get-last-eval-results)
         (fn [results]
           (send-portal-value (read-portal-host-port) results))))

(send-last-eval-to-portal)

Please excuse the debugging prints :)

[PEZ]

Thanks for this awesome Joyride example, @orestis! Please consider contributing it to the Joyride repo. About opening portal. Maybe make the script do that as well? The script could check for if Portal is installed and suggest it be installed if not, and otherwise execute the command for opening it on the first script invokation. See the Joyride API example for some pointers on how to check for an extension's status.

[orestis]

Thanks for the kind words @PEZ. I will look into Portal integration of my joyride script and coordinate with @djblue - probably this joyride script could be embedded in the Portal VSCode extension.

The other main sticking point is that Calva doesn't provide a first-class API to evaluate forms, and getting the results as a Promise, resulting to that weird clipboard dance. Is that something that could be technically done for VSCode extensions?

[PEZ]

Yes, can and should be done!

Please file an issue about it. 😄

[orestis]

Yes, can and should be done!

Please file an issue about it. 😄

https://github.com/BetterThanTomorrow/calva/issues/1719 🎉

[seancorfield]

This seems like a lot of work to avoid just tap>ing a value which would go into directly into Portal -- and ctrl+shift+t t / ctrl+shift+t space are built into Calva for tap>ing. What am I missing here?

[orestis]

The main difference is that the editor becomes a conveyor of values. Meaning eg I can connect to a production instance over SSH but keep Portal running on my local machine.

[seancorfield]

OK. At the "cost" of having to serialize values to/from EDN -- which only works if you have representable values -- and I think you would also lose metadata? So no datafy/nav?

[orestis]

OK. At the "cost" of having to serialize values to/from EDN -- which only works if you have representable values -- and I think you would also lose metadata? So no datafy/nav?

Yes, precisely. Essentially an alternative to the way Calva handles pprint:

• result -> parse EDN -> pprint

Becomes

• result -> Portal (parses EDN)

I also have a proof of concept that sends data to Portal via HTTP directly through the JVM, via the built-in HTTP client, but given that you still have to go through EDN, I see little value in working on that more. If Calva gains an API that exposes evaluation results (removing the need for polling the System clipboard) then this script is complete in its goal.

[seancorfield]

OK, I just wanted to be clear on what you're losing -- I rely on datafy/nav far too much for this to work for me. The evaluation result (from nREPL) will still be a string that has to be "read" so you won't get actual data from that API. Right, @PEZ ?

The only way to get accurate data into Portal is via tap> inside the JVM with Portal as a listener. As soon as you go over the wire -- either nREPL or HTTP -- you're in string land.

[orestis]

Definitely. I see the string land as the lowest common denominator, guaranteed to work everywhere but not ideal. A safe fallback, if you will.

For me it gives me 80% of the value (because I don’t use datafy or metadata heavily) but perhaps the missing 20% is bigger than I know. So I’m eager to see a demo of more advanced usages of Portal.

[orestis]

Rather than opening a new issue, I want to raise that VSCode at some point introduced the ability to show web views on the panel and the sidebar https://github.com/microsoft/vscode-extension-samples/blob/main/webview-view-sample/src/extension.ts

• which opens up the ability to have a richer experience as a web view without the user having to manage another editor tab.

[PEZ]

Also the regular tree-views is already something to consider for navigating the structures. I have a branch somewhere where I experimented a bit with that...

[behrica]

For me the "cider-inspector" is as well the killer argument for Cider / Emacs. I do any code evaluation via the cider inspector.

I have tried Calva + Portal , but I am not sure, it is the same experience then with Cider inspector.

Specifically I would like to have the "remote development of vs code AND Calva with portal" working, on which I did not find a clear tutorial. The reason being "remote development" as killer feature of VsCode, no chance for Emacs.

So I would like to see: -> "VSCode + remote development + calva + clojure evaluation results on keypress into Portal"

I am not sure, if this can be done, and I did not find a setup / tutorial for this precise setup. Can somebody point me to any docu on this use case ?

[behrica]

Just to add : The Clerk tap inspector can be as well an alternative to Portal , having pros and cons.

[seancorfield]

@behrica Can you define what you mean by "remote development" here?

My daily dev setup is:

• VS Code on Windows
• Clojure files and REPL on Ubuntu (WSL2)
• Portal server runs in the JVM (on Ubuntu) and displays into VS Code (on Windows)

When I want to debug stuff in a remote process, I can:

• Start an ssh tunnel
• Start Simple Browser in VS Code connected to a Portal server running in the remote process
• Connect Calva to an nREPL server running in the remote process

In both cases, any evaluations I do from the editor show their results in Portal in VS Code, regardless of whether that's "locally" (WSL2-Remote) or "remote" (QA or production server).

I documented how I was approaching this back in December https://corfield.org/blog/2022/12/18/calva-joyride-portal/ and I just posted an update to that, with an updated "local" workflow, and noting that the "remote" stuff, via Joyride, is published to my https://github.com/seancorfield/vscode-calva-setup repo.

Does any of that help?

[behrica]

@seancorfield "remote development" using this extension: https://code.visualstudio.com/docs/remote/remote-overview

So a "local vscode" connected to a remote server (or a Docker container, the same thing) via ssh. In this "everything happens" on the remote machine (code, execution) but is shown locally.

This makes "portal" or similar more complex , I even think it does not fully work: (at least the vscode portal extension): https://github.com/djblue/portal/issues/157

Maybe some "port forwarding" magic can even show a "remote portal" locally.

[behrica]

@behrica Can you define what you mean by "remote development" here?

My daily dev setup is:

• VS Code on Windows
• Clojure files and REPL on Ubuntu (WSL2)
• Portal server runs in the JVM (on Ubuntu) and displays into VS Code (on Windows)

The last point eventually does not work in my "remote development setup". Due to https://github.com/djblue/portal/issues/157 (vscode portal extension). Without the extension I could use the "normal portal viewer" and get it via X11 forwarding from the server to my local machine (which is as well cumbersome and slow)

[seancorfield]

For the "completely remote" setup I described (for working with QA/production), our Portal server starts "headless" on a known port, and then as long as you have HTTP traffic accessible on that port, you can use a regular browser to connect to it.

I use Simple Browser in VS Code, just so that the browser UI stays in the editor, but it has enough limitations that I wouldn't want to work that way all the time -- I'd switch to a full-featured browser and just tile that next to VS Code. You don't need to use an X11 browser with forwarding (I agree that can be slow/cumbersome).

But, yeah, if you use the Portal extension in VS Code, it needs to know the host/port of the remote Portal server. You'd need to read .portal/vs-code.edn from the remote side and lay it down locally, but you'd also need the host entry set to a name or IP address. For a while, I was doing that when starting Portal by passing :host (-> (sh/sh "hostname" "-I") :out (str/trim)) to portal/api.open -- to workaround not having localhost forwarding enabled on WSL2.

I suspect you could use Joyride to ask the remote JVM to give you that file's contents via the REPL and then write it out locally, and then fire up the Portal extension UI... but you'd need to work with @djblue on the details of that.

[djblue]

@behrica I think https://github.com/djblue/portal/commit/c2ab152df5260ae9cd9faf4201020e85794ff084 should be enough to use the portal vscode extension with https://code.visualstudio.com/docs/remote/remote-overview via the ssh protocol. I tested it locally by ssh-ing into my macbook from a windows machine and the UI seems to work just fine.

[djblue]

@seancorfield, I might also be able to expose the functionality via a command to allow scripting via joyride 🤔 Although, https://github.com/seancorfield/vscode-calva-setup/pull/3 might be enough.

[behrica]

@behrica I think djblue/portal@c2ab152 should be enough to use the portal vscode extension with https://code.visualstudio.com/docs/remote/remote-overview via the ssh protocol. I tested it locally by ssh-ing into my macbook from a windows machine and the UI seems to work just fine.

yes, it works now in 0.4.0 of portal extension. So it is now full functional in vscode remote development extension

[seancorfield]

Yup, I now have the Portal VS Code extension working with remote servers too, thanks to Slack help from @djblue ! Very happy!

[seancorfield]

It looks like the Portal folks are happy with the state of the world at this point -- is there still active interest in the OP topic?

[PEZ]

It looks like the Portal folks are happy with the state of the world at this point -- is there still active interest in the OP topic?

Active, idk, but I am certainly still interested in having something that is better than what we have today, that works right out of the box, without any further installation, setup and configuration.

[behrica]

I think that an interactive "cider-inspector" in Calva shoul dbe doable. I looked a bit into the "Orchard API" for "data inspection", and it seems to me that it has done te most part of the work to "dig" into data. So that "only" the visualisation part needed to be re-done in Calva

[PEZ]

We're about to release a version of Calva with an inspector view. It's nothing like the CIDER inspector, and it lives entirely in string land. But it covers a lot of use cases when you just need convenience for exploring data. You also can just paste some data in there.

• Preview docs: https://deploy-preview-2513--calva-docs.netlify.app/inspector/

[orestis]

On first glance this looks very useful and well-designed! Living in the background, not incurring a performance penalty unless you really want to inspect something, uses native UI... Looking forward to taking it for a spin.