Server stops quickly if not actively connected to Hub

[zstatmanweil]

When I was initially started using Planetary Computer Hub, I was able to leave a script running on my server but close my connection to the Hub on my browser. In the last couple weeks I have noticed when I do this, my server stops soon after. I have tested closing it and returning to it within a couple minutes and it is fine, but when I leave it for a couple hours, it always seems to stop. Is this intended? Should I not expect to be able to leave a long-running script on the server? Thanks!

[TomAugspurger]

Hmm I don't exactly know what would have changed recently. The Hub does include an "idle-timeout" to close notebook servers that are idle (https://github.com/microsoft/planetary-computer-hub/blob/7d08e5f985a49c70d9736bfa4666b54b820720bc/helm/values.yaml#L279, uses https://github.com/jupyterhub/jupyterhub-idle-culler). I haven't changed that value recently.

It's not documented yet, but we're working on https://kbatch.readthedocs.io/en/latest/, which is designed for these non-interactive workflows. You could try that if you want, but you'll likely hit some rough edges. If you go this route, I'm happy to provide support.

Alternatively, you could insert a bit of code into your script, to tell the Hub that you're actually still active (untested):

headers = {"Authorization": "token " + os.environ["JUPYTERHUB_API_TOKEN"]}

body = {
    "last_activity": datetime.datetime.utcnow().isoformat() + "Z"
}
r = requests.post(
    os.environ["JUPYTERHUB_API_URL"] + "/users/" + os.environ["JUPYTERHUB_USER"] + "/activity", headers=headers,
    json=body
)

That would need to be run in some kind of loop that runs at least every <20 minutes.

[zstatmanweil]

Thanks for the suggestions! I think I will try your last recommendation as it seems pretty straight-forward. I will let you know how it goes

[TomAugspurger]

OK, let me know if that doesn't work and we can try other options.

[zstatmanweil]

Hi Tom,

I wanted to follow up on this and let you know that adding that bit of code does keep my Hub server active when I leave the tab open and don't shut down my laptop. I believe previously in this circumstance the server was still shutting down so this is a good thing.

When I close the tab though, the server does shut down. It is not a huge deal on my end, just wanted you to know for future reference! Unless you have another relatively simple idea, I think I will wait for you all to implement/document the kbatch solution. Thanks :)

[TomAugspurger]

Hi Zoe,

We've been working a bit on kbatch as a way to submit asynchronous jobs to run on the Hub's compute (see https://planetarycomputer.microsoft.com/docs/overview/batch/ and https://kbatch.readthedocs.io/en/latest/). It's very alpha, but might be worth looking into if you need something without a timeout.