search

galaxyproject / galaxy

Data intensive science for everyone.
https://galaxyproject.org
Other
1.39k stars 1k forks source link

GIE Hardening #4651

Open natefoo opened 7 years ago

natefoo commented 7 years ago

After a few attempts to teach workshops (mainly overseas) we've discovered some robustness problems with GIEs on usegalaxy.org. Here's a list of identified issues and potential resolutions, as well as planning for enhancements, in order of priority:

natefoo commented 7 years ago

ping @nekrut

nekrut commented 7 years ago

@erasche @bgruening - we need to solve the mystery of the green button. @davebx volunteered to help so can you guide him?

hexylena commented 7 years ago

We're happy to help, but this isn't a good week for us. Next week will be better for me at least, I think @bgruening will still be at conferences/meetings then though.

blankenberg commented 7 years ago

Would be neat to enable checkpointing and restoring, including filesystem changes.

martenson commented 7 years ago

This should be the green button fix: https://github.com/bgruening/docker-jupyter-notebook/pull/23

natefoo commented 7 years ago

Client/timeout fixes are now on Test/Main.

natefoo commented 7 years ago

The container timeout is now 5 minutes on Test/Main, and both now use the 17.09 image, which includes the fixed magic button.

martenson commented 7 years ago

mockup of adjusted UI, this would allow non-uniform approach based on what environment you select (some allow selecting datasets, some whole histories, some nothing etc.). You could also initialize the dialogue based on what is passed as params and validate it against the already available IE configs

ie_launcher

martenson commented 7 years ago

vector: ie_launcher.pdf

natefoo commented 7 years ago

@martenson I like this. Any thoughts on how the UI might display any running containers you have and potentially any stopped/relaunchable containers (assuming we decide to offer that capability)?

martenson commented 7 years ago

@natefoo ? screenshot 2017-10-06 14 31 17

hexylena commented 7 years ago
martenson commented 7 years ago

@erasche

hexylena commented 7 years ago

@martenson

yes, you can get(), but I really believe that knowing which datasets it was initially launched on will be a good reminder. More than "It was running for 8 hours on tuesday", I don't remember what I did tuesday, but I might remember that I started analysing my (e.g.) htseq data. I know it is extra data to store, but if we're doing this as UX improvements to GIEs, I think it'd be important.

jxtx commented 7 years ago

Tags? On Fri, Oct 6, 2017 at 3:09 PM Eric Rasche notifications@github.com wrote:

@martenson https://github.com/martenson

yes, you can get(), but I really believe that knowing which datasets it was initially launched on will be a good reminder. More than "It was running for 8 hours on tuesday", I don't remember what I did tuesday, but I might remember that I started analysing my (e.g.) htseq data

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/galaxyproject/galaxy/issues/4651#issuecomment-334844225, or mute the thread https://github.com/notifications/unsubscribe-auth/AAE4ZUP-iPVvAkKrNqvAyq3m71mJQG02ks5spnrJgaJpZM4Pc7Ot .

jmchilton commented 7 years ago

I'd make the Environment + version one thing

I don't know - I'd keep the version separate if there are more than one. You can imagine different flavors of any of these. I just wouldn't call it version. I'd imagine different flavors of Jupyter for instance with different libraries installed. If there is just one version - then definitely collapse - that is totally cool by me.

hexylena commented 7 years ago

You can imagine different flavours (indeed we did during implementation), but I am yet to see anyone deploy multiple flavours. Even we only have one jupyter flavour.

martenson commented 7 years ago

@jxtx something like this? screenshot 2017-10-06 15 18 21

natefoo commented 7 years ago

I'd consider having more if they weren't so big. They are very big.

On Fri, Oct 6, 2017 at 15:16 Eric Rasche notifications@github.com wrote:

You can imagine different flavours (indeed we did during implementation), but I am yet to see anyone deploy multiple flavours. Even we only have one jupyter flavour.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/galaxyproject/galaxy/issues/4651#issuecomment-334845810, or mute the thread https://github.com/notifications/unsubscribe-auth/AARg-Y7y5xzxfkxxwKYs7L6brZHNB1UXks5spnxrgaJpZM4Pc7Ot .

jxtx commented 7 years ago

I was thinking explicit tags for GIE instances. Seems like a better reminder than inferring from associated datasets. Could initialize automatically that way though.

On Fri, Oct 6, 2017 at 3:18 PM Martin Cech notifications@github.com wrote:

@jxtx https://github.com/jxtx something like this? [image: screenshot 2017-10-06 15 18 21] https://user-images.githubusercontent.com/1814954/31294623-9d5d2c32-aaa9-11e7-972d-7103ba12755c.png

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/galaxyproject/galaxy/issues/4651#issuecomment-334846364, or mute the thread https://github.com/notifications/unsubscribe-auth/AAE4Zf0QjgQbpCWuVCb5vfewwoIXXvjQks5spn0KgaJpZM4Pc7Ot .

martenson commented 7 years ago

screenshot 2017-10-06 15 48 26

yvanlebras commented 7 years ago

Hi @natefoo , everyone,

Maybe here is the place to post a comment related to the fact that we can't save into a Galaxy history a Jupyter notebook .ipynb if the kernel is R... It seems that the "save the current notebook in Galaxy" (the so-called magix button ;) ) function works only if we reconnect the Python 2 kernel.... If you want I create a dedicated issue, don't hesitate.

In the meantime, as the user can (switching to the Python 2 kernel) save the ipynb, this is not a priority I think to work on it..

jmchilton commented 6 years ago

When testing loading of Jupyter notebooks on main today from Selenium I encountered a bunch of different transient failures getting the notebook to load:

natefoo commented 6 years ago

I added a new to-do item, there are some ways we can improve the launch experience that all generally pertain to the "readiness check" (i.e. spinning on /interactive_environments/ready):

  1. There's a hack in place (#5677) that prevents the GIE template from returning until the container/service's ports are returned by Docker, since this may not actually succeed immediately after container/service creation. We should instead return immediately and allow /interactive_environments/ready to update the proxy map with the ports once they are available.
  2. Right now we perform a readiness check to see if the container is running. Once that's done, we stop checking the container state and immediately attempt to connect to the notebook. However, this may never succeed due to proxy/network/whatever problems, and the container may shut down. The user has no way of knowing that it's shut down and that it will never succeed and instead just stares at the spinner until they get fed up and quit. So instead of having the readiness check stop once it's ready, it would be better to continue checking state concurrently with the connection attempts (and keepalive requests?).
  3. Instead of the toast notifications, it'd be nice to have something like a persistent box centered in the middle pane that has a "row" for each state change and our familiar colors/icons to indicate what's happening.

For example (mocked as a table using some github emoji, where each table shows the progression, but hopefully you get the idea):

Launching Jupyter
:curly_loop: Starting a container in which to run Jupyter
:clock5: Connecting to container
Launching Jupyter
:curly_loop: Starting a container in which to run Jupyter
:information_source: Container state is PENDING
:information_source: Container state is ASSIGNED
:clock5: Connecting to container

Once running, the state progression can collapse:

Launching Jupyter
:heavy_check_mark: Starting a container in which to run Jupyter: RUNNING v
:curly_loop: Connecting to container
Launching Jupyter
:heavy_check_mark: Starting a container in which to run Jupyter: RUNNING v
:x: Connecting to container
:exclamation: The container is no longer running, click here to attempt to launch again, or report this error