search

robots-from-jupyter / robotlab

Experiments in building installers for JupyterLab, Robot Framework and Friends
BSD 3-Clause "New" or "Revised" License
24 stars 3 forks source link

Consider multiple size-based releases 🔬🔍🔭📡 #42

Open bollwyvl opened 4 years ago

bollwyvl commented 4 years ago

RobotLab is big and likely to get bigger as more cool features need more (exotic) dependencies.

We could just as easily spin a couple different intstallers a la Miniconda/Anaconda in CI.

MicrobotLab 🔬

RobotLabLite 🔍

RobotLab 🔭

RobotHub 📡

datakurre commented 4 years ago

My workshop for the next robot conference was accepted, which confirms that I will be attending the sprints there. Apparently there is still no installer available for Robot Framework, so I've been thinking, if I should try to fork this project and try this "multiple size-based releases" for that. For plain robot it would make sense to have minimal version with just robot framework, but also a version for getting started with selenium testing (the real reason for the installer).

bollwyvl commented 4 years ago

workshop for the next robot conference was accepted

Hooray!

fork this project

You know I'm glad to help, and a fork is just more repos to watch: I don't see any reason why we wouldn't start it here, and why we wouldn't want synchronized, automated releases of the whole shooting match. If somebody wants to then fork that, at some point, that's great.

We can even stop doing the kinda gross templating, and just manage the yaml generation more simply in python directly, if we start feeling confident. The CI build would take longer, but i think there's value in managing a single pipeline for all the stuff together.

minimal version with just robot framework

yup, as a CI thing, that would be great. MiniRobotConda. Even if it doesn't ship conda (which it probably should, if only for activation and administration chores) it can't get much smaller than about 70mb, but that's not so bad!

version for getting started with selenium testing

so even there i would imagine two flavors:

i'm really liking having firefox in our installer, and if we can do our little part to help more people test on firefox, i'm all for it. The only gotcha is on Linux, you do still have to have gtk3 installed, even when running --headless, but i'll still take it.

Unless we ship all the chromedrivers, we can't really ship any chromedrivers, but we could ship webdrivermanager, and indeed ship all of them pre-cached.

datakurre commented 4 years ago

Thanks. I can agree with your reasoning. I also prefer Firefox being bundled because now things just work. It also makes sense that whatever version or browser you use for surfing in the web should not interfere with learning Selenium testing. Finally, promoting Firefox would not hurt the diversity in browser ecosystem...

That said I cannot really focus on this before November, but will then.

bollwyvl commented 4 years ago

That said I cannot really focus on this before November, but will then.

I hear you, my friend. Who knows, maybe i'll be able to throw some stuff at it. What is your priority for getting something out? Maybe hack on the summary of this issue to make it clear. I guess I would lean towards the smallest functioning robot environment, just so we have a size/complexity baseline, but if the web one is more useful we could start there.

datakurre commented 4 years ago

I’d vote for the baseline. We already know that we can make a GB sized installer with everything :)

After the baseline, another baseline with SL, gecko and Firefox, and then documented approach to fork and extend those baselines.

Finally, at the conf it makes sense to discuss what libraries should be bundled with the default starter (for learning and getting started purposes):

bollwyvl commented 4 years ago

i did a little playing with this last night:

609M Sep 30 23:58 RobotLab-2019.9.1-Linux-x86_64.sh
 84M Oct  1 00:05 MiniRobot-2019.9.1-Linux-x86_64.sh
161M Oct  1 00:35 ExoRobot-2019.9.1-Linux-x86_64.sh

Mini is rf + conda. not shipping conda only saves about 2mb. Exo is Mini, plus every optional dependency i could find in the rf codebase. The heavy hitter is definitely wx, which screenshots uses. Might as well throw in RIDE at that point...

datakurre commented 4 years ago

Very cool! Which screenshots feature relies on wx? Does dialogs library work? The point for a installer is to give environment that just works, so from that point of view 200M is not much nowadays.

And now that you mentioned RIDE... does it already support Python 3? Sure many would like it to be included and it’s been hard to install on many systems because of wx. (Although that probably was because for long it required very old version of that.)

bollwyvl commented 4 years ago
here's the full package list of `Exo` (on linux) 

    asn1crypto-0.24.0-py36_1003.tar.bz2
    atk-2.32.0-haf93ef1_0.tar.bz2
    ca-certificates-2019.9.11-hecc5488_0.tar.bz2
    cairo-1.16.0-hfb77d84_1002.tar.bz2
    certifi-2019.9.11-py36_0.tar.bz2
    cffi-1.12.3-py36h8022711_0.tar.bz2
    chardet-3.0.4-py36_1003.tar.bz2
    conda-4.7.12-py36_0.tar.bz2
    conda-package-handling-1.6.0-py36h516909a_0.tar.bz2
    cryptography-2.7-py36h72c5cf5_0.tar.bz2
    docutils-0.15.2-py36_0.tar.bz2
    expat-2.2.5-he1b5a44_1003.tar.bz2
    fontconfig-2.13.1-h86ecdb6_1001.tar.bz2
    freetype-2.10.0-he983fc9_1.tar.bz2
    fribidi-1.0.5-h516909a_1002.tar.bz2
    gdk-pixbuf-2.36.12-h7a26e22_1003.tar.bz2
    gettext-0.19.8.1-hc5be6a0_1002.tar.bz2
    glib-2.58.3-h6f030ca_1002.tar.bz2
    gobject-introspection-1.58.2-py36h5503ade_1002.tar.bz2
    graphite2-1.3.13-hf484d3e_1000.tar.bz2
    gst-plugins-base-1.14.5-h0935bb2_0.tar.bz2
    gstreamer-1.14.5-h36ae1b5_0.tar.bz2
    gtk2-2.24.32-h586f36d_1.tar.bz2
    harfbuzz-2.4.0-h9f30f68_3.tar.bz2
    icu-64.2-he1b5a44_1.tar.bz2
    idna-2.8-py36_1000.tar.bz2
    jpeg-9c-h14c3975_1001.tar.bz2
    libffi-3.2.1-he1b5a44_1006.tar.bz2
    libgcc-ng-9.1.0-hdf63c60_0.tar.bz2
    libglu-9.0.0-hf484d3e_1000.tar.bz2
    libiconv-1.15-h516909a_1005.tar.bz2
    libpng-1.6.37-hed695b0_0.tar.bz2
    libstdcxx-ng-9.1.0-hdf63c60_0.tar.bz2
    libtiff-4.0.10-h57b8799_1003.tar.bz2
    libuuid-2.32.1-h14c3975_1000.tar.bz2
    libxcb-1.13-h14c3975_1002.tar.bz2
    libxml2-2.9.9-hee79883_5.tar.bz2
    libxslt-1.1.33-h31b3aaa_0.tar.bz2
    lxml-4.4.1-py36h7ec2d77_0.tar.bz2
    lz4-c-1.8.3-he1b5a44_1001.tar.bz2
    ncurses-6.1-hf484d3e_1002.tar.bz2
    openssl-1.1.1c-h516909a_0.tar.bz2
    pango-1.42.4-ha030887_1.tar.bz2
    pathlib2-2.3.5-py36_0.tar.bz2
    pcre-8.41-hf484d3e_1003.tar.bz2
    pip-19.2.3-py36_0.tar.bz2
    pixman-0.38.0-h516909a_1003.tar.bz2
    pthread-stubs-0.4-h14c3975_1001.tar.bz2
    pycosat-0.6.3-py36h14c3975_1001.tar.bz2
    pycparser-2.19-py36_1.tar.bz2
    pyopenssl-19.0.0-py36_0.tar.bz2
    pypubsub-4.0.3-py_0.tar.bz2
    pysocks-1.7.1-py36_0.tar.bz2
    pyte-0.8.0-py_0.tar.bz2
    python-3.6.7-h357f687_1005.tar.bz2
    pyyaml-5.1.2-py36h516909a_0.tar.bz2
    readline-8.0-hf8c457e_0.tar.bz2
    requests-2.22.0-py36_1.tar.bz2
    robotframework-3.1.2-py_0.tar.bz2
    ruamel_yaml-0.15.71-py36h14c3975_1000.tar.bz2
    setuptools-41.2.0-py36_0.tar.bz2
    six-1.12.0-py36_1000.tar.bz2
    sqlite-3.29.0-hcee41ef_1.tar.bz2
    tk-8.6.9-hed695b0_1003.tar.bz2
    tqdm-4.36.1-py_0.tar.bz2
    urllib3-1.25.6-py36_0.tar.bz2
    wcwidth-0.1.7-py_1.tar.bz2
    wheel-0.33.6-py36_0.tar.bz2
    wxpython-4.0.6-py36h8f66242_1.tar.bz2
    xorg-kbproto-1.0.7-h14c3975_1002.tar.bz2
    xorg-libice-1.0.10-h516909a_0.tar.bz2
    xorg-libsm-1.2.3-h84519dc_1000.tar.bz2
    xorg-libx11-1.6.8-h516909a_0.tar.bz2
    xorg-libxau-1.0.9-h14c3975_0.tar.bz2
    xorg-libxdmcp-1.1.3-h516909a_0.tar.bz2
    xorg-libxext-1.3.4-h516909a_0.tar.bz2
    xorg-libxrender-0.9.10-h516909a_1002.tar.bz2
    xorg-renderproto-0.11.1-h14c3975_1002.tar.bz2
    xorg-xextproto-7.3.0-h14c3975_1002.tar.bz2
    xorg-xproto-7.0.31-h14c3975_1007.tar.bz2
    xz-5.2.4-h14c3975_1001.tar.bz2
    yaml-0.1.7-h14c3975_1001.tar.bz2
    zlib-1.2.11-h516909a_1006.tar.bz2
    zstd-1.4.0-h3b9ef0a_0.tar.bz2

Which screenshots feature relies on wx?

Looks like wx, gtk or pillow would work. I don't think gtk is supported on windows, but pillow is handy, anyhow...

Does dialogs library work?

if all it takes is tk! haven't tested.

RIDE... does it already support Python 3?

looks like they test with 3.6. and don't pin wx. I haven't tried packaging it in a while... couldn't get hidpi to work correctly..

bollwyvl commented 4 years ago

No I didn't put selenium in one of the new ones, but I do think shipping Firefox/geckodriver with it is probably the right thing to do, if it should Just Work after downloading (e.g in CI). Do you think wx/ride are worth the size hit for the web excursion? Or just also add it to Exo? You wouldn't actually use wx in CI.

Also what about SSHLibrary? It's first party, solid as hell, relatively light, super useful, and makes it easy to test iot stuff, as well as vms d cloud stuff... But you kinda have to know what you're doing because passwords. Put in Exo?

WhiteRobot? I've still not made it work on win10, and don't like how it's packages. But sounds so good!

SikuliRobot: I always consider it the Last Argument of Kings of test execution. Hard to maintain, but can probably do the thing. Packaging Java stuff is no problem, though jython probably is or will be soon, if you expect to be able to run anything else after 89 days!

The whole *Robot branding thing is a bit silly, of course, and wouldn't help people. If we were doing this for real, we'd probably need a choose-your-own-adventure page "what are you testing?" "What operating system are you testing on?" That kind of thing. Have a "just show me the robots" escape hatch, and let people pick the one they want from a table of key packages. Do the classic up-sell and point them towards RobotLab because it has more check marks :)

I definitely want to belay the Hub one, for now. I've made a totally self-contained jupyterhub work with ansible/systems spawner (haven't tried a k8s version yet). in certain kinds of training (schools), the participants may only have an older Chromebook, and wouldn't be able to install random software. Also building for ARM is still a touch outside of our capabilities. Anyhow: set up a NUC, run the installer, throw in your course content, turn on WiFi, and you're good to go in seconds per participant rather than tens-of-minutes.

On Thu, Oct 3, 2019, 00:22 Asko Soukka notifications@github.com wrote:

Am I correct that ExoRobot does not yet include Selenium testing dependencies? I recall robot users have a long hoped for an installer with Selenium dependencies. But adding Firefox with geckodriver into mix would probably make ExoRobot fat? So, should there be "SeleniumRobot"?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/robots-from-jupyter/robotlab/issues/42?email_source=notifications&email_token=AAALCRCIQ3CBAGLUSQZUKOTQMVXQZA5CNFSM4IYDQJ6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAG5ZWI#issuecomment-537779417, or mute the thread https://github.com/notifications/unsubscribe-auth/AAALCRC53EF77KHDIGYLF4TQMVXQZANCNFSM4IYDQJ6A .

datakurre commented 4 years ago

Ok Let's not guess too far ahead :) I could see two separate installer tracks: one for RobotLab and the other one for more conservative Robot Framework users.

Quoting Nicholas Bollweg (2019-09-18 23:42:00)

MicrobotLab ð¬

  • for CI environments

But "how micro" this could be, because "Lab"-branding would probably make this need nbconvert and nbrobot (the latter from robotkernel) to be able to execute notebooks. (I do actually use nbrobot in production RPA already.) No wx required here.

RobotLabLite ð

I'm not quite convinced that this is really needed. This could be used as a base to build custom environment, but why not just install miniconda then?

RobotLab ð

  • as it is now!
    • even shipping Firefox is great!

I agree everything on this. Big is good here. It's so good that the download is worth waiting for.

RobotHub ð¡

  • ship a whole [3]littlest jupyter hub

This is interesting. I have never set up JupyterHub myself yet, but this would have very interesting educational and enterprise use cases :)

Then the more conservative Robot Framework track. Robot Framework is completely missing standalone installers and your work seems to be the best there is so far. It feels a bit weird to design and build it here, but I understood that it would be better than forking this into another repository to follow.

MiniRobot as it is. It's good to have something minimal with at least pip to install some libraries when you know what to do.

ExoRobot was interesting. Like "MiniRobot" was just a proof-of-concept and "ExoRobot" was the actually practical base to build an environment. I feel this makes more sense thatn "RobotLabLite", but I'm not sure if ride really belongs to here (even when it comes with wx).

WebRobot (terrible name). Comparable to RobotLab. Should come at least with Ride, SeleniumLibrary, geckodriver and Firefox. But maybe also with SSHLibrary, RESTinstance and SeleniumTestability. Really hard to draw line where to end here...

Should I ask the other workshops at the next Robocon if they would like to have your installer experience?

bollwyvl commented 4 years ago

A lot to chew on there!

I can rework some of the existing ones with your guidance above.... Adding the web one is a no-brainer, we already have everything.

Maybe we need a further piece of the name on the non-lab ones, e.g WebRobotBox or something, as "*robot" is super overloaded, and we don't need to get into any trouble.

The only bounding condition on us shipping more installers is supporting them, to my mind. What we would not want is upstreams getting complaints from users when things don't work, if the installers are not a suggested install mechanism.

The workshop case is ideal, though, as the instructor is on the hook to provide support, and can't disappear. You should certainly offer it.

On Thu, Oct 3, 2019, 13:36 Asko Soukka notifications@github.com wrote:

Ok Let's not guess too far ahead :) I could see two separate installer tracks: one for RobotLab and the other one for more conservative Robot Framework users.

Quoting Nicholas Bollweg (2019-09-18 23:42:00)

MicrobotLab ð¬

  • for CI environments

But "how micro" this could be, because "Lab"-branding would probably make this need nbconvert and nbrobot (the latter from robotkernel) to be able to execute notebooks. (I do actually use nbrobot in production RPA already.) No wx required here.

RobotLabLite ð

I'm not quite convinced that this is really needed. This could be used as a base to build custom environment, but why not just install miniconda then?

RobotLab ð

  • as it is now!
  • even shipping Firefox is great!

I agree everything on this. Big is good here. It's so good that the download is worth waiting for.

RobotHub ð¡

  • ship a whole [3]littlest jupyter hub

This is interesting. I have never set up JupyterHub myself yet, but this would have very interesting educational and enterprise use cases :)

Then the more conservative Robot Framework track. Robot Framework is completely missing standalone installers and your work seems to be the best there is so far. It feels a bit weird to design and build it here, but I understood that it would be better than forking this into another repository to follow.

MiniRobot as it is. It's good to have something minimal with at least pip to install some libraries when you know what to do.

ExoRobot was interesting. Like "MiniRobot" was just a proof-of-concept and "ExoRobot" was the actually practical base to build an environment. I feel this makes more sense thatn "RobotLabLite", but I'm not sure if ride really belongs to here (even when it comes with wx).

WebRobot (terrible name). Comparable to RobotLab. Should come at least with Ride, SeleniumLibrary, geckodriver and Firefox. But maybe also with SSHLibrary, RESTinstance and SeleniumTestability. Really hard to draw line where to end here...

Should I ask the other workshops at the next Robocon if they would like to have your installer experience?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/robots-from-jupyter/robotlab/issues/42?email_source=notifications&email_token=AAALCRDICGDT7ZE7NF53HK3QMYUSRA5CNFSM4IYDQJ6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAI7MKI#issuecomment-538048041, or mute the thread https://github.com/notifications/unsubscribe-auth/AAALCRDJHTFILBLS42GYN33QMYUSRANCNFSM4IYDQJ6A .

datakurre commented 4 years ago

@bollwyvl To avoid misunderstandings, don’t waste your time to rework anything yet. As you said, all the pieces are already there and probably even I could put them together if I really need to. Let’s continue to focus on what this should mean for the “Lab” and only return to the generic installers when there is real need (and a possibly a maintainer).

Disclaimer about the CI-version: I don’t have personal use for it, because we have Nix on our CI machines :/

bollwyvl commented 4 years ago

Great, focus on lab good. I like the nbrobot-forward idea. Re: nix: indeed, once nix supports one click installers on windows, I'll happily use it to train people on things other than nix. :P

Back on topic: I guess to the marketing-as-design point, if we sketch up what we'd like a blog post/landing page to contain, that would help resolve some of the ambiguity... Will take a crack at it. Documentation need still remains the same, but maybe there's a story in there... The (somewhat neglected) irobotframework docs built with nbsphinx might be a good prototype.

bollwyvl commented 4 years ago

Sry, have been deep in the language server stuff. I'll try to get back to this after a bit of a holiday this weekend: basically, I intend to keep everything in #52, but not actually build any of the general purpose stuff, land that, and then have a look at some of the other concepts we're developing here. Maybe do some design sketches on the road :)

Back on the LSP stuff: it's taken some really nasty stuff down in python/r/node land, and am only now getting to where I can help out with the browser testing. Been stealing a lot from this repo :P The end game there is to get all of that into JupyterLab itself, but there are a lot of nuts and bolts to figure out before then.

When that's good to go (on pip and npm) some time next month, I think we'll want the UI things it already offers (jump to definition, highlight references, etc) for .robot, .robot.ipynb, output.xml and more possible stuff down the pipe (outline view, tidy). Shipping the existing robot language server implementation in node is no problem for robotlab, but there are things it can't do that kernels can, e.g. you have to libdoc your 3rd party stuff ahead of time. We can at least check it out! It should just be:

I can also imagine a future where kernels become implicit language servers and clients: with what we ended up with, it's a many-to-many relationship between documents and language servers, everything goes through the notebook server, what's a few more? Heck, there are even nix language servers (though can't tell which of the rust, go, or haskell implementations will win). Additionally, to that end, here's a cool thread about a guix kernel: https://groups.google.com/forum/#!topic/jupyter/jUfLBPSlbXY

bollwyvl commented 4 years ago

And 'cuz I know you love looking at robot reports...

bollwyvl commented 4 years ago

Whoops, meant to post this here, not there:

I did do some drawing: nothing super solid to show for it. There's not much you can do with a feature table, and there's certainly bounded complexity to how much you can stuff in there. There are a few hierarchies that came to mind, but i think one of the more logical ones was...

A funny thing that came to mind, sort of related to the Hub concept, is a "Pro" spin (think a second, full day workshop), which includes the things you would need to extract, build, test, and package new robot libraries (or extensions to kernel) as part/in support of a team. On the backend, you'd have your tooling like git, while on the front-end, I'd stuff in jupyterlab-git. Ship pytest, robotframework-statuschecker, robotframework-lint, black, etc. Ship conda-build, and maybe even constructor (since we can't ship nix :stuck_out_tongue_winking_eye: ). Anyhow :pizza: for thought.