[BUG] running on EGI Binder failing

[rsignell]

When I try to run this notebook using the EGI binder link, the notebook fails to launch with

Installing pip dependencies: ...working... Pip subprocess error:
  Running command git clone --filter=blob:none --quiet https://github.com/icenet-ai/icenet /tmp/pip-install-czcwp4v6/icenet_6b44d5c07b4f4ac1baf288917f8cd475
  Running command git rev-parse -q --verify 'sha^0bac48fbc6447d937498b2c7240a2bd946430700'
  Running command git fetch -q https://github.com/icenet-ai/icenet 0bac48fbc6447d937498b2c7240a2bd946430700
  Running command git checkout -q 0bac48fbc6447d937498b2c7240a2bd946430700
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.

[ricardobarroslourenco]

Hi @rsignell !

FYI, I was able to launch and begin running on the regular Binder instance (link) up to cell 21 (where the kernel breaks, likely due to OOM error, given the tiny size of the container).

I believe the error may be on the routine that initializes the container on EGI. I do not have access to EGI, so I cannot debug it further. @acocac can you take a look on this?

[acocac]

@sebastian-luna-valero can you assist here? It seems EGI binder fails to launch the virtual image of one of the EDS book notebooks. It was working in the regular Binder.

[sebastian-luna-valero]

I have been having a look. In the logs, pip seems to be updated to the latest version:

  - pip                                 22.2.2  pyhd8ed1ab_0             conda-forge                   
  + pip                                   24.2  pyhd8ed1ab_0             conda-forge/noarch         1MB

And a local install of the environment with conda works, so I am not sure what's the issue.

@enolfc could you please have a look as well?

[acocac]

Hi @rsignell !

FYI, I was able to launch and begin running on the regular Binder instance (link) up to cell 21 (where the kernel breaks, likely due to OOM error, given the tiny size of the container).

I believe the error may be on the routine that initializes the container on EGI. I do not have access to EGI, so I cannot debug it further. @acocac can you take a look on this?

@ricardobarroslourenco thanks for joining the conversation and support the EDS book community (:

I double checked the notebook in the standard Binder and it works. It initially crashed at the cell running train_model, but it worked after I restarted the kernel and run all cells. Can you validate this?

[ricardobarroslourenco]

Hi @rsignell ! FYI, I was able to launch and begin running on the regular Binder instance (link) up to cell 21 (where the kernel breaks, likely due to OOM error, given the tiny size of the container). I believe the error may be on the routine that initializes the container on EGI. I do not have access to EGI, so I cannot debug it further. @acocac can you take a look on this?

@ricardobarroslourenco thanks for joining the conversation and support the EDS book community (:

I double checked the notebook in the standard Binder and it works. It initially crashed at the cell running train_model, but it worked after I restarted the kernel and run all cells. Can you validate this?

Happy to help @acocac !

I have re-run on the regular Binder, and it actually completed running (rendered Jupyter notebook) :

Looking at the host CPU specs, I believe I got lucky, and the platform gave me a larger instance (in which kernel hasn't been suspended). I believe that last time I have got the error when running:

!icenet_output -m -o ./results/predict custom_run_forecast api_dataset predict_dates.csv

At cell 21.

[acocac]

When I try to run this notebook using the EGI binder link, the notebook fails to launch with

Installing pip dependencies: ...working... Pip subprocess error:
  Running command git clone --filter=blob:none --quiet https://github.com/icenet-ai/icenet /tmp/pip-install-czcwp4v6/icenet_6b44d5c07b4f4ac1baf288917f8cd475
  Running command git rev-parse -q --verify 'sha^0bac48fbc6447d937498b2c7240a2bd946430700'
  Running command git fetch -q https://github.com/icenet-ai/icenet 0bac48fbc6447d937498b2c7240a2bd946430700
  Running command git checkout -q 0bac48fbc6447d937498b2c7240a2bd946430700
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.

@rsignell thanks for reporting the issue. I think the notebook computational environment seems to be affected by upgrades of pip in the repo2docker image of the EGI binder. In the meantime, I suggest trying the regular Binder which runs the minimal training example of the notebook.

@bnubald can you confirm how vulnerable is the icenet-ai version 0.2.9_dev to upgrades in pip? I recently saw the latest version of pip prefers pyproject.toml, but you might have a better understanding of this.

@sebastian-luna-valero I've pinned the pip version to 22.2.2 in a forked repository. Despite the change, I still have the same issue. @enolfc your insights would be welcome!

[sebastian-luna-valero]

Discussing with @enolfc this morning it looks like the issue in EGI is caused by the old version of repo2docker installed. We are currently discussing its upgrade to a recent version.

[acocac]

Discussing with @enolfc this morning it looks like the issue in EGI is caused by the old version of repo2docker installed. We are currently discussing its upgrade to a recent version.

Great - please feel free to share updates here. The EGI binder is a fundamental resource to make EDS book notebooks more accessible. Thank you!

[bnubald]

When I try to run this notebook using the EGI binder link, the notebook fails to launch with

Installing pip dependencies: ...working... Pip subprocess error:
  Running command git clone --filter=blob:none --quiet https://github.com/icenet-ai/icenet /tmp/pip-install-czcwp4v6/icenet_6b44d5c07b4f4ac1baf288917f8cd475
  Running command git rev-parse -q --verify 'sha^0bac48fbc6447d937498b2c7240a2bd946430700'
  Running command git fetch -q https://github.com/icenet-ai/icenet 0bac48fbc6447d937498b2c7240a2bd946430700
  Running command git checkout -q 0bac48fbc6447d937498b2c7240a2bd946430700
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.

@rsignell thanks for reporting the issue. I think the notebook computational environment seems to be affected by upgrades of pip in the repo2docker image of the EGI binder. In the meantime, I suggest trying the regular Binder which runs the minimal training example of the notebook.

@bnubald can you confirm how vulnerable is the icenet-ai version 0.2.9_dev to upgrades in pip? I recently saw the latest version of pip prefers pyproject.toml, but you might have a better understanding of this.

@sebastian-luna-valero I've pinned the pip version to 22.2.2 in a forked repository. Despite the change, I still have the same issue. @enolfc your insights would be welcome!

@acocac, thanks for linking me into this, it shouldn't be sensitive to upgrades in pip and we do have update to pyproject.toml in another dev branch for a release later on. Once stable, I'll update the notebook submission to point to the stable version of icenet.

[rsignell]

Thanks everyone for working on this! Can't wait to try it out on EGI Binder!

[acocac]

@sebastian-luna-valero do you have any updates about the EGI binder. I just double checked if it launches the notebook, but it returns the following lines:

7c47-2d4ea9-2d8df8-2de868326bc90b-5c0580:821721d3724f7b24e0789d9c6c568083f51d047a" already present on machine
2024-09-03T07:50:06Z [Normal] Created container notebook
2024-09-03T07:50:06Z [Normal] Started container notebook
2024-09-03T07:50:11Z [Warning] Back-off restarting failed container
2024-09-03T07:50:15Z [Normal] Stopping container oneclient

[sebastian-luna-valero]

Indeed @acocac the issue is still there, sorry.

The team behind the EGI Binder is currently busy with other high priority tasks.

We will report back when it's solved.

Apologies for the inconvenience caused.

[acocac]

Apologies for the inconvenience caused.

No worries - Thanks for the update!

[sebastian-luna-valero]

Hi,

Sorry for the delay; could you double check whether this works?

https://github.com/eds-book-gallery/67a1e320-7c47-4ea9-8df8-e868326bc90b/pull/11

[sebastian-luna-valero]

Thanks for testing and merging @acocac

I believe this can now be closed.

[acocac]

Thanks for testing and merging @acocac

I believe this can now be closed.

You're right! Thanks for the support on this. It's interesting that EGI binder work for the rest EDS book notebooks without modifying the environment.yml file.

[acocac]

@rsignell the notebook works using EGI binder. Can you confirm?