EOX: Headless execution for UC3

[BachirNILU]

Hi,

In UC3, we want to test the headless execution. @Schpidi provided a comprehensive step-by-step guide on running a notebook headlessly in the following issue: How to?: Headless execution. I understand there are two options for UC3:

1. Use UC2 headless server link, in that case, we need the credentials to access it.
2. Set up a new headless server for UC3.

Can you help us with this?

Thanks in advance.

Best regards,

-Bachir.

[eox-cs1]

A headless access for UC3 has now been set up ans is ready for use:

curl -X POST -v https://headless-fairicubeuc3.hub.eox.at/processes/execute-notebook/jobs \
    -u USERNAME:PASSWORD \
     --header 'Content-Type: application/json' \
     --data-raw '{"inputs": {
  "notebook": "s3/NOTEBOOK_IN_BUCKET.ipynb",
  "cpu_requests": "1",
  "cpu_limit": "1",
  "mem_requests": "4G",
  "mem_limit": "4G",
  "node_purpose": "user",
  "kernel": "conda-env-eurodatacube8-torch-py",
  "parameters_json": {"a": "b", "c": 123}
}}'

Instructions the same as for UC2, see: https://github.com/FAIRiCUBE/flux-config/issues/1#issuecomment-1689488002 Username and PW are provided via personal communication

[BachirNILU]

Thanks @eox-cs1! I have tested it and it works, thanks! I only have an issue regarding the kernal. I understand "conda-env-eurodatacube8-torch-py" is the only available kernal as for now, right? A library that we use is missing, how can I install the library into this kernal?

Thanks in advance.

[eox-cs1]

Yes, "conda-env-eurodatacube8-torch-py" is the only kernel for your headless usage what libraries are you missing?

[BachirNILU]

@eox-cs1 it is "codecarbon". I was able to run our code, but I need to install the same libraries for each run (by dedicating the first notebook cell to install missing libraries).

[BachirNILU]

Before closing this issue, I came across a minor problem. In our code, we use the S3 common bucket environment variables to read/write. Headless running of the code did not provide us with access to these variables. I had to copy/past the name, key and secret codes to the script (which is obviously not safe). Is there a way to fix this? (Maybe passing the environment variables through parameters directly in the curl command?)

[eox-cs1]

Hey @BachirNILU we have now upgraded the kernels for headless (conda-env-eurodatacube8-torch-py) and jupyter-lab (fairicubeuc3-torch)

• adding codecarbon to both

• also the availability of the environmental variables in the headless execution should now be fixed

[BachirNILU]

I have tested it and it works. I close the issue here. Thanks for your help.