RobotLab 2021: JupyterLab 3, Robot Framework 4

[bollwyvl]

Let's get the band back together!

There have been so many nice things published recently that it's worth warming this back up.

Some ideas (please feel free to add more):

Packages

new?	package	version	conda-forge?	rf4?	notes
0	Python	3.8	1	1
0	JupyterLab 3	3.0.x	1	-
0	Robot Framwork	4.x	1	1
0	Firefox	78 LTS	1	-
?	xeus-robot	0.3.3	1	1	discussion below
0	robotkernel	1.5.0	1	1
0	robotframework-jupyterlibrary	0.3.1	1	1
1	jupyterlab_robotmode	0.3.1	1	1
0	jupyterlab-starters	1.0.2	1	-
1	jupyterlab-tour	3.0.1	1	-
1	jupyterlab-lsp	3.6.0	1	-
1	robotframework-lsp	0.15.0	1	1
1	robotframework-robocop	1.7.1	1	1
0	restinstance	1.0.2	1	?

Infra

• adapt build from gt-coarl-lab
  • use doit
  • use GitHub Actions (it actually generates most of the workflow scripts)

for posterity:

Hooray JupyterLab 2.0!

https://pypi.org/project/jupyterlab/2.0.0/

Looks like RF 3.2 is also dangerously close:

https://github.com/robotframework/robotframework/milestone/48

It may also be worth considering moving to python 3.8, as the whole Jupyter stack (at least) is good to go on windows (that was a mess :stuck_out_tongue_closed_eyes:)

I can't commit to a bunch of time to move the ball forward right now, but am going to start tinkering on various things (like jupyterlab-starters) so that we're in a better spot when it's go-time.

[bollwyvl]

started on starters: https://github.com/deathbeds/jupyterlab-starters/pull/39

[datakurre]

I’ll probably try to fix RF 3.2b2 compat on Tuesday. It allows robotkernel to work fully on public API.

Update: Fell into rabbit holes. Continuing later.

[bollwyvl]

:tada: RF 3.2! Guess it's time to take a proper look at actually doing a new release.

I see robotkernel 1.4.0 is already out (congrats!), but some things we may want to address first:

• [ ] #65
• [ ] #68
• [ ] https://github.com/robots-from-jupyter/robotkernel/issues/47
• [ ] https://github.com/robots-from-jupyter/robotframework-jupyterlibrary/issues/20
• [x] https://github.com/conda-forge/staged-recipes/pull/11394

[datakurre]

65 could be skipped for this.

[datakurre]

Btw, what is the state of JupyterLab LSP?

[bollwyvl]

jlsp is working pretty well, and supports lab 2.x: there's some clutch work going on to support dynamic configuration by the client, which certain language servers pretty much require for some of their cooler features. i haven't checked in on rflsp in a while...

[bollwyvl]

ha, and 3.2 broke our pipeline on jlsp. funny.

[bollwyvl]

Here's a little thing that puts together both jupyterlab-lsp and jupyterlab-debugger:

https://github.com/blink1073/jupyter-ide-demo

It would be super rad, of course, to be able to add breakpoint debugging to robotkernel, but the plumbing under the current debugger is based on https://github.com/jupyter-xeus/xeus-python, not ipykernel :crying_cat_face: so would be a non-trivial amount of work.

[datakurre]

Cool!

I have assumed that break point debugging would require turning kernel async and implementing custom robot runner 🤔 On 30. Apr 2020, 1.43 +0300, Nicholas Bollweg notifications@github.com, wrote:

Here's a little thing that puts together both jupyterlab-lsp and jupyterlab-debugger: https://github.com/blink1073/jupyter-ide-demo It would be super rad, of course, to be able to add breakpoint debugging to robotkernel, but the plumbing under the current debugger is based on https://github.com/jupyter-xeus/xeus-python, not ipykernel 😿 so would be a non-trivial amount of work. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[bollwyvl]

Hey hey! Long time no talk!

Some updates: xeus-robot has been released, and supports the debug adapter protocol, and works with the jupyterlab debugger! I :heart: irobotframework and robotkernel, but seems we should probably consider a build that includes this new kernel (and robotkernel).

JupyterLab 3 (in RC) has been (partially) internationalized, and most importantly to me, doesn't need end-user nodejs to add extensions. I'm currently trying to smooth things out with the weirdness that is codemirror's extend-by-import approach with Lab 3 by just shipping the simple mode addon, and hopefully it makes the cut. Once that's through, i think maybe we make a new jupyterlab_robotmode repo and start tinkering there (#47).

These sound like some very interesting things coming together!

[datakurre]

So great to hear from you!

All these changes have proven that RobotLab as a stablish and out of the box working release was a great thing.

Sounds like a lot of work ahead. I hope to get more close to robot stuff again later this year. The next RoboCon will be online event in January, but probably still too early for me to fix everything. Another big change is that RobotFramework community has now a playwright based alternative (robotframework-browser) for Selenium for browser testing and I have not been able to look into that yet. Also Whit me has some replacement for Windows testing / automation so all my “magic” needs rewelriting in the future 🙈

I’ve also been following RoboCorp development. As I expected, they had to heavily fork the kernel to match their goals. But it also looks like their product is so tied to their platform that there remains room for RobotLab.

[bollwyvl]

Sounds like a lot of work ahead.

Yeah, it never ends! Nearest term, are you good with me starting a third repo for the syntax stuff for lab 3, targeting a pypi release? This would also let me warm robotframework-jupyterlibrary back up before trying to do the whole enchilada on this repo.

Over here, we've already done pretty well to start reducing the complexity of doing these releases by upstreaming things, and Lab 3 will bring that down even further, as we won't have to rebuild lab with the custom extensions. I've been experimenting with some different approaches (doit, conda-lock) to make some of these steps more reproducible... it still ain't nix, but what is? :rofl:

playwright based alternative

ha, and I keep trying to get nodejs out of my projects!

so all my “magic” needs rewelriting in the future

these are pretty much the coolest things ever, though. good luck trying to make it work with everything!

there remains room for RobotLab.

absolutely! even if we aren't putting out releases all the time, this work has been instrumental in demonstrating some key overall system concepts. It remains one of my favorite testing examples.

[datakurre]

@bollwyvl Yes, it is completely ok to create a new repo for the syntax stuff. The current package inside robotkernel is already almost 1:1 copy of ipythonrobotframework's version.

[datakurre]

I am curious, how did JupyterLab solve the re-compilation after every extension issue? A link to somewhere is enough :-)

[bollwyvl]

It took a change to webpack core. Here's the magic sauce:

https://webpack.js.org/concepts/module-federation/

And where the bulk of the work happened:

https://github.com/jupyterlab/jupyterlab/pull/8385

Basically, an extension needs a devDependency on @jupyterlab/builder, the dev runs "labextension build" to generate an opaque remoteEntry.js file pointing at all of the bundles it might need, and then makes sure those all those files end up in data_files for a pip package.

At runtime, it grabs all of the remoteEntry files, served from a composite static_path like how nbextensions work, and as each dependency is loaded, it does a real semver solve... So it's still possible to get duplicates or unpredictable behavior... as we've seen, codemirror can still create some issues, as it relies on import side-effects. but for the most part will Just Work.

[datakurre]

:scream: I already forgot that we have this pull open.

Maybe "one more" release with legacy robotkernel. Until we are confident with xeus-robot.

RobotKernel binder is still working with JupyterLab 2 and RF 4: https://github.com/robots-from-jupyter/robotkernel/blob/master/binder/environment.yml Maybe upgrade that to JupyterLab 3 at first?

I see no reason, why we couldn't use the latest versions of the previously bundled packages, unless their latest versions have conflicting dependencies.

With one exception: We could use robotmode from the marketplace. It saw no problems with it https://gitlab.com/atsoukka/robot-rpa-playground/-/blob/master/robotlab/Dockerfile#L13

I see that jupyterlab-starters is now JupyterLab 3.x compatible. Are there any backwards incompatibilities or should we expect everything to just work?

(And no QWeb. RESTinstance was too much hurdle with its version pinnings, but those are probably no gone and it is more downstream friendly.)

[bollwyvl]

Maybe "one more" release with legacy robotkernel. Until we are confident with xeus-robot.

I'd say we try shipping it... having the debugger is almost worth the price of admission, as we have discussed elsewhere.

use the latest versions of the previously bundled packages

yes. that's pretty much the idea, but at least we have some forensic info... and the old installers! Updated the matrix for key ones... like, I don't think we go to py 3.9 yet... unless we want to do an (untestable) MacOS M1 build (grumble grumble proprietary grumble grumble).

jupyterlab-starters is now JupyterLab 3.x compatible

Yep, i've even tested it against the one you built, and have pointed it out as a superb open source example of using the state-machine-y side of things in a notebook. Heck, I should add it to the docs! And indeed, we can even fire off jupyterlab-tours from starters, as it exposes a Lumino command.

RESTinstance

Yeah, restinstance is on conda-forge, too!

[datakurre]

@bollwyvl Would you think that both xeus-robot and robotkernel could be available or do they conflict with both reserving robot-mimetype?

Or a release with robotkernel and THEN a "beta" release with xeus :)

[bollwyvl]

They both work fine, together, you just have to choose when you pick a kernel. I believe robotkernel ends up winning in an undefined situation.

[bollwyvl]

Would be blocked by robotframework-lsp, but might start this up this weekend...

[datakurre]

I have still mixed feelings about having confusing two different robot kernels, but maybe that encourages me to contribute xeus-robot or migrate robotkernel on top of xeus-robot so that we can have all the nice things.

Are jupyterlab-starters and -tour overlapping? Should I plan to migrate from starters to tour?

Robocop is definitely something that should be better integrated at some point. I have no idea about all the feature in current JupyterLab, but I guess that it would be possible to have robocop warnings pinned on related rows.

[bollwyvl]

Well, we're all set up to do multiple variants... On COAR, the windows build has A+B+C while osx-CPU has A+B+D+E while Linux-gpu has A+B+E+F+G. So we can ship "all you need to do the starter," with as few buttons on the screen as possible, as well as "all the things". Heck, maybe JupyterLab Classic is the default UI.

Starters is still the way, in my experience, to package and ship versioned, tested, templated content. But since one can start a tour with a Lab command, we might be able to pull back some of the starter notebook/schema complexity, and use tour to replace some narrative instructions/screenshots with THIS BUTTON IS GLOWING, YOU KNOW YOU WANT TO CLICK IT. luckily we know some things about the Lab DOM 😂.

[bollwyvl]

Sorry, no progress this weekend, but I did ship my newest GTCOARLab (al 8gb of it across 4 distributions) late on friday, so feeling pretty good about the prospects... but now maybe next weekend. :blush:

[bollwyvl]

robotframework-lsp 0.15 just shipped, so we're ready to go... when there's time!