Link to open a session not fully working when notebook to open is specified

[burnout87]

When composing a link to open a session on renkulab when not logged-in, we get the following behavior:

The link is so composed:

https://renkulab.io/projects/astronomy/mmoda/mmoda-request/sessions/new?autostart=1&branch=mmoda_request_c570580ab66720b3_76adbfd3aecd840a&commit=8507b7803d4b67d7d77c305e5568c00cc11307ec&notebook=api_code_c570580ab66720b3.ipynb&env%5BODA_TOKEN%5D=secret

We noticed that by removing the notebook we want to have opened, the link then works as expected, e.g.:

https://renkulab.io/projects/astronomy/mmoda/mmoda-request/sessions/new?autostart=1&branch=mmoda_request_c570580ab66720b3_76adbfd3aecd840a&commit=8507b7803d4b67d7d77c305e5568c00cc11307ec&env%5BODA_TOKEN%5D=secret

and interestingly, the notebook I was previously attempting to open, is now opened

Please ask any detail I might have missed

Thanks

[lorenzo-cavazzi]

Thank you for reporting this! I tried to dig deeper because I could easily start a session using the link you provided and everything seemed to work fine.

Apparently, the link breaks for non-authenticated users. May I ask you to do a test just to confirm this? Could you try to log in to https://renkulab.io/login and do the following:

• Open a new tab and copy/paste the session link with the embedded data
• Open an incognito window and copy/paste the session link with the embedded data

I expect the first to succeed (also opening the correct notebook) and the second to fail with the "token authentication" error that you reported

[olevski]

I can confirm that the notebook parameter is not used at all. So whether you include it or not should make no difference. This is a leftover from older versions and we kept it so that we do not break the API.

[burnout87]

Apparently, the link breaks for non-authenticated users. May I ask you to do a test just to confirm this? Could you try to log in to https://renkulab.io/login and do the following:

* Open a new tab and copy/paste the session link with the embedded data

* Open an incognito window and copy/paste the session link with the embedded data

I expect the first to succeed (also opening the correct notebook) and the second to fail with the "token authentication" error that you reported

Thanks Lorenzo for your reply.

I can confirm the behavior you predicted.

[lorenzo-cavazzi]

Ok, I fear we hit a very special case here. Theoretically, the correct case is the anonymous user case, and this should always fail. As @olevski mentioned, it's working just because this takes advantage of a leftover in our APIs.

The actual plan on the UI is to "sanitize" the URL by removing all the non-allowed parameters (that would be the notebook in this case). In general, the notebook parameter works just for JupyterLab and breaks in other cases. We are trying to detach from JupyterLab in order to extend the support to support other development environments.

An alternative solution would be passing the notebook name through an env variable and using a startup script on the image to load the corresponding notebook. Or (much easier if you always target the same notebook) modify the default URL so that it already includes the path, like in the following screenshot -- to do that, you can use the command in the first example here https://renku.readthedocs.io/en/latest/renku-python/docs/reference/commands/config.html#examples

Would any of these 2 alternatives be reasonable for you?

[burnout87]

Ok, I fear we hit a very special case here. Theoretically, the correct case is the anonymous user case, and this should always fail. As @olevski mentioned, it's working just because this takes advantage of a leftover in our APIs.

So, if I correctly understood, accessing anonymously should never be permitted, correct?

Would any of these 2 alternatives be reasonable for you?

Of the solutions you suggested, the second seems to be the more reasonable one, and easier to implement.

[olevski]

So, if I correctly understood, accessing anonymously should never be permitted, correct?

You are supposed to be able to start a session with a link even if you are anonymous. And you are supposed to be able to access the anonymous session afterwards as well (as long as you doing so as the same anonymous user).

The problem is only in the notebook parameter which is not handled properly in the anonymous case. The intention so far is to fully deprecate this and potentially add another method that would indicate that a specific file or path should be open in that session.

I will open another issue specifically for this - i.e. to remove or strengthen the treatment of the notebook parameter.

And I agree that the 2nd alternative that Lorenzo mentioned is the most robust one we can currently offer.

[volodymyrss]

Thanks for explaining! Since we are creating the repository just before generating the link, there is no problem to write the config. If this works for us (we with @burnout87 should experiment) and it is clear that this API is not likely to be deprecated soon, we can use this approach.

[burnout87]

Or (much easier if you always target the same notebook) modify the default URL so that it already includes the path, like in the following screenshot -- to do that, you can use the command in the first example here https://renku.readthedocs.io/en/latest/renku-python/docs/reference/commands/config.html#examples

Ciao @lorenzo-cavazzi,

I have implemented the approach you recommended, and it seems to be working, but I noticed the following behavior:

After pushed the modified version of the renku.ini file, having generated the session-starting link, that no longer includes the starting notebook, I experience the following:

The session does look as it is not started, but, if I try again the same link, then the labstudio is opened and pointing to the notebook I set within the renku.ini file. Is tihs behavior expected? What could be the reason for it?

[lorenzo-cavazzi]

We had an offline conversation about this.

What happens here?

As a recap, the problem is very specific to the flow. @burnout87 uses a platform that creates new branches/commits and an autostart link. When clicking the link immediately, it occasionally happens that the Docker image built for the new commit by RenkuLab (and required to start the new session) is not ready yet.

Accidentally, this process usually needs just a few seconds more, so the autostart fails, and UI shows for a moment the "Docker image building" warning

That message goes away as soon as the image is ready. From that point on, the quickstart links work fine again.

The user's feeling here is that the behavior is random since the failure with the autostart doesn't always happen and, when it happens, the warning accidentally goes away very quickly most of the time, leaving a "Docker image available" behind.

This actually is the intended behavior, and it works well in most cases.

How to solve this?

There are a couple of possible solutions:

1. The platform used to launch the sessions could check the status of the image for the newly created commits. This requires invoking the "project pipelines" API from GitLab. For public projects, this is trivial.
   • There is a caveat in using the pipelines API, but it doesn't seem relevant for the user. I provided more detailed info to @burnout87 on another channel.
2. When using the quickstart link, RenkuLab could keep the user waiting when the image is building.
   • After looking into this possible solution, it doesn't seem easy because only authenticated users can access job details. This means anonymous users would necessarily have a different experience and we can't differentiate between building and failed images.
   • There are a few other complications with this solution, and it might not even be the best solution for other scenarios -- we would need to discuss it further

If the first solution is acceptable to you, I suggest checking the pipeline's status before opening the link and repeating the action every few seconds (5 seconds to avoid frame limiting?) until the status turns from "running" to "success".

[volodymyrss]

Thanks for the explanations! By the way, I have an impression I am seeing this "image not ready" message more often than before. For example, when just creating a repository. But I might be wrong. Anyway, it also means that autostart URL does not autostart in cases like this. Just wondering, could it be made so that renku UI waits until the image is ready and actually concludes an autostart?

[lorenzo-cavazzi]

Thanks for the suggestion @volodymyrss . The 2nd solution I listed suggests the same approach. The main issue is with non-logged users. Due to a GitLab limitation (for security reasons?), they can't access the API returning the building status for the Docker image. So we can't always be sure whether the image is currently building or not.

[volodymyrss]

Right, indeed. I still do not quite understand: if the users have enough rights to end up seeing the message that the image is building, is it not possible for the UI to act on this information to retry end eventually execute autostart?

[lorenzo-cavazzi]

Oh, you won't see that message when you are not logged in. Just a generic "Docker image missing".

Our idea is to keep the flow for starting sessions coherent for all users. When something fails, we fall back to the "session start" page where different users have different experiences. Ofc implementing different starting session flows is technically feasible, but it complicates the logic quite a bit.

[volodymyrss]

Ok, I see, thanks.

In principle, image build may take some time (our images in particular). So we need to wait for some time, informing user about updates we get from renkulab. It's probably best if it looks a bit like the renkulab session start sequence (stages being checked).

From my own user perspective, it seems a bit unfortunate that I am ending up fairly often in the page with "image building" message, only to come back there later to click run it. I had this also with URLs starting from renku. Natively I'd imagine this being one of the checkboxes in session start up. But I guess I am not seeing all the possibilities clearly. But it's a small detail and user experience may differ.

But, I am noticing something else: were we not fixing the image? Why do we have to build it often now?

[lorenzo-cavazzi]

From my own user perspective, it seems a bit unfortunate that I am ending up fairly often on the page with an "image building" message, only to come back there later to click run it.

That's a fair point. Probably no other users have an automated system that creates a new commit and tries to start a session right away. It's just a matter of a few seconds, so any further delay is enough to make the UX smooth. Still, showing a warning about the building image and resuming the flow after that, would be more natural. We don't have a quick solution, but we will keep this in mind and discuss it internally to assess other alternatives. P.S. We very much appreciate feedback like this, I hope my previous comments don't sound too dismissive.

Natively I'd imagine this being one of the checkboxes in session start-up.

One of the points checks the image availability, but it's limited to "available"/"not available" due to the limitations of non-logged users.

But, I am noticing something else: were we not fixing the image? Why do we have to build it often now?

Ha! With a fixed image, building the image shouldn't be an issue. Actually, the project I saw yesterday doesn't fix the image in the renku-ini file. This is all I see there

@burnout87 could you give it a try with a fixed image? You should not experience this problem anymore.

[burnout87]

@burnout87 could you give it a try with a fixed image? You should not experience this problem anymore.

Currently I only modify the default_url entry. Would you give me an example on how to specify the image in the config file? I n particular how to find its name... I will try right away...

[volodymyrss]

From my own user perspective, it seems a bit unfortunate that I am ending up fairly often on the page with an "image building" message, only to come back there later to click run it.

That's a fair point. Probably no other users have an automated system that creates a new commit and tries to start a session right away.

I also experienced it when starting new session from a freshly created repository.

It's just a matter of a few seconds, so any further delay is enough to make the UX smooth.

Unfortunately, our software is large, and it's typical for a build to take 5min. We were also thinking about crafting more elaborate repositories, per-request added-value elaborations (it's part of the goal of our common project with SDSC) - this will make the build longer.

Still, showing a warning about the building image and resuming the flow after that, would be more natural. We don't have a quick solution, but we will keep this in mind and discuss it internally to assess other alternatives.

Indeed, and thank you for your effort to potentially accommodate our special case. I think this seamless, frictionless session start can increase user engagement with renkulab. As we are directing the users to renkulab, we want them to have smooth and natural experience, when possible. For many people coming from MMODA to renkulab, this is going to be the first experience of renkulab, and we want this to be a very good experience. Else they are likely to just close the tab.

P.S. We very much appreciate feedback like this, I hope my previous comments don't sound too dismissive.

We appreciate your help, and also this comment! It's hard to be clear on the tone sometimes, especially in this online text-only medium, with people having differ habits in communication. To me, there was nothing unnecessarily dismissive in our communication here. That said, I am totally prepared for any of my suggestions being potentially dismissed, as long as we can keep discussing it (within reason, of course).

Natively I'd imagine this being one of the checkboxes in session start-up.

One of the points checks the image availability, but it's limited to "available"/"not available" due to the limitations of non-logged users.

So I guess it would be natural if this box would have an option "preparing" or something like that, and this option would occur when it is possible to derive this state (i.e. if the repository is public or logged in used has the access). Maybe it's too unnatural from the implementation side though.

But, I am noticing something else: were we not fixing the image? Why do we have to build it often now?

Ha! With a fixed image, building the image shouldn't be an issue. Actually, the project I saw yesterday doesn't fix the image in the renku-ini file. This is all I see there

@burnout87 could you give it a try with a fixed image? You should not experience this problem anymore.

I suppose it was effectively a regression on our side.

However it does highlight that if we do want to change something in the image (e.g. requirements) for the auto-generated code, we will have this small UX issue.

[volodymyrss]

@burnout87 could you give it a try with a fixed image? You should not experience this problem anymore.

Currently I only modify the default_url entry. Would you give me an example on how to specify the image in the config file? I n particular how to find its name... I will try right away...

This is what I thought and that is why it is strange it changed.

You can see an example in what is created in the current production version, there is no delay there.

[burnout87]

You can see an example in what is created in the current production version, there is no delay there.

Thanks, but I am trying to understand where I can see the name of the image within my project, so that I can specify within the renku ini file

[lorenzo-cavazzi]

This is a (verbose) documentation page on how to do that https://renku.readthedocs.io/en/stable/reference/templates.html#pin-docker-image

In short, you need to add an image field after the default_url. Something similar to this screenshot I took from another project

If you are re-using an image from a previous commit, the easiest way to find the registry URL would be to start a session from there, click on the "running" icon, and copy the "image source" with the copy button. The same information is available from the "Registry" section in GitLab

In your specific case, the URL should be something like the following, just swap the 7-chars commit with the one you need

registry.renkulab.io/gabriele.barni/test-dispatcher-endpoint:7fd550b

[volodymyrss]

You can see an example in what is created in the current production version, there is no delay there.

Thanks, but I am trying to understand where I can see the name of the image within my project, so that I can specify within the renku ini file

You do not need to put the image corresponding to the current project version. This image might not be built yet. Instead, you should put there another, pre-defined, fixed ("pinned") image. The image which is certain to exist. Since you are reusing the repository master branch, together with existing renku.ini, you can use the value set there.

In here: https://renkulab.io/gitlab/astronomy/mmoda/mmoda-request/-/blob/master/.renku/renku.ini

(but you should not set it, just re-use)

[volodymyrss]

In your specific case, the URL should be something like the following, just swap the 7-chars commit with the one you need

registry.renkulab.io/gabriele.barni/test-dispatcher-endpoint:7fd550b

I used these sort of images before, but it turns out that eventually they disappear, breaking the flow. So I switched to images in dockerhub. It rather makes sense, since you can not keep all images forever. How are they cleaned?

But, maybe it is possible to keep persistent image on renkulab? It might even lead to a performance boost (the image would always be nearby, though it depends).

[burnout87]

(but you should not set it, just re-use)

Following your suggestion, I now have implemented the approach we discussed.

I am running some tests, but so far, it seems to be working

[lorenzo-cavazzi]

I used these sort of images before, but it turns out that eventually they disappear, breaking the flow. So I switched to images in dockerhub. It rather makes sense, since you can not keep all images forever. How are they cleaned?

I've just double-checked this and there is no retention time. The Docker images should never be purged automatically from https://renkulab.io/gitlab (unless that was manually set in the project options in a page like the following)

[volodymyrss]

Thanks @lorenzo-cavazzi, good to know. It did disappear for me, maybe we added cleaning in that case (for the big images as they are).