search

ersilia-os / ersilia

The Ersilia Model Hub, a repository of AI/ML models for infectious and neglected disease research.
https://ersilia.io
GNU General Public License v3.0
225 stars 147 forks source link

🐛 Bug: Error when Ersilia tries to automatically re-fetch a model #1374

Open GemmaTuron opened 2 weeks ago

GemmaTuron commented 2 weeks ago

Describe the bug.

Hi @Abellegese this is related to #1373 Once Ersilia detects the model is there, by default it should say Yes to re-fetching it in any case, but this error appears. I am guessing this might have to do with the latest re-factoring of the Ersilia CLI and maybe only happens in those models for which Ersilia catalog does not detect them...

8:05:10 | DEBUG    | Trying to get metadata from: /home/gturon/eos/dest/eos7kpb
⬇️  Fetching model eos7kpb: h3d-virtual-screening-cascade-light
18:05:11 | DEBUG    | Initialized with URL: None
18:05:11 | DEBUG    | Getting model source
18:05:11 | DEBUG    | Model getting fetched from DockerHub
18:05:11 | DEBUG    | Starting fetching procedure
18:05:11 | DEBUG    | Decided to fetch from DockerHub
18:05:11 | DEBUG    | Fetching from DockerHub
18:05:11 | DEBUG    | Pulling model image from DockerHub
18:05:11 | DEBUG    | Image ersiliaos/eos7kpb:latest is available locally
Requested image eos7kpb is available locally. Do you still want to fetch it?[Y/n]Traceback (most recent call last):
  File "/home/gturon/miniconda3/envs/ersilia/bin/ersilia", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/gturon/miniconda3/envs/ersilia/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gturon/miniconda3/envs/ersilia/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/gturon/miniconda3/envs/ersilia/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gturon/miniconda3/envs/ersilia/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gturon/miniconda3/envs/ersilia/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gturon/github/ersilia-os/ersilia/ersilia/cli/commands/__init__.py", line 22, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/gturon/github/ersilia-os/ersilia/ersilia/cli/commands/fetch.py", line 121, in fetch
    _fetch(mf, model_id)
  File "/home/gturon/github/ersilia-os/ersilia/ersilia/cli/commands/fetch.py", line 15, in _fetch
    asyncio.run(mf.fetch(model_id))
  File "/home/gturon/miniconda3/envs/ersilia/lib/python3.11/site-packages/nest_asyncio.py", line 30, in run
    return loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gturon/miniconda3/envs/ersilia/lib/python3.11/site-packages/nest_asyncio.py", line 98, in run_until_complete
    return f.result()
           ^^^^^^^^^^
  File "/home/gturon/miniconda3/envs/ersilia/lib/python3.11/asyncio/futures.py", line 203, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/home/gturon/miniconda3/envs/ersilia/lib/python3.11/asyncio/tasks.py", line 277, in __step
    result = coro.send(None)
             ^^^^^^^^^^^^^^^
  File "/home/gturon/github/ersilia-os/ersilia/ersilia/hub/fetch/fetch.py", line 240, in fetch
    await self._fetch(model_id)  
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gturon/github/ersilia-os/ersilia/ersilia/hub/fetch/fetch.py", line 226, in _fetch
    await self._fetch_from_dockerhub(model_id=model_id)
  File "/home/gturon/github/ersilia-os/ersilia/ersilia/hub/fetch/fetch.py", line 164, in _fetch_from_dockerhub
    await self.model_dockerhub_fetcher.fetch(model_id=model_id)
  File "/home/gturon/github/ersilia-os/ersilia/ersilia/hub/fetch/lazy_fetchers/dockerhub.py", line 142, in fetch
    await mp.async_pull()
  File "/home/gturon/github/ersilia-os/ersilia/ersilia/hub/pull/pull.py", line 92, in async_pull
    do_pull = yes_no_input(
              ^^^^^^^^^^^^^
  File "/home/gturon/github/ersilia-os/ersilia/ersilia/utils/terminal.py", line 82, in yes_no_input
    ans = raw_input_with_timeout(
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gturon/github/ersilia-os/ersilia/ersilia/utils/terminal.py", line 75, in raw_input_with_timeout
    answer = inputimeout(prompt=prompt, timeout=timeout)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gturon/miniconda3/envs/ersilia/lib/python3.11/site-packages/inputimeout/inputimeout.py", line 24, in posix_inputimeout
    sel.register(sys.stdin, selectors.EVENT_READ)
  File "/home/gturon/miniconda3/envs/ersilia/lib/python3.11/selectors.py", line 359, in register
    self._selector.register(key.fd, poller_events)
PermissionError: [Errno 1] Operation not permitted

Describe the steps to reproduce the behavior

No response

Operating environment

Ubuntu 22.04 LTS

Abellegese commented 2 weeks ago

Hi @GemmaTuron I will take a look and fix the issue.

GemmaTuron commented 2 weeks ago

Hi @Abellegese

Thanks for the fast response. More into the error. I have double checked and indeed the catalog command does not show for example model eos4e40. But then when I try to fetch it I run into the error mentioned above. However:

The model IS NOT there indeed:

(ersilia) gturon@raluy:~/github/ersilia-os$ ersilia delete eos4e40
💁 Model eos4e40 is not available locally. No delete is necessary

If I try to fetch it and I can input manually the Yes when it asks if I want to re-fetch the model, the procedure continues normally

(ersilia) gturon@raluy:~/github/ersilia-os$ ersilia -v fetch eos4e40 --from_dockerhub
10:57:43 | DEBUG    | Trying to get metadata from: /home/gturon/eos/dest/eos4e40
⬇️  Fetching model eos4e40: chemprop-antibiotic
10:57:44 | DEBUG    | Initialized with URL: None
10:57:44 | DEBUG    | Getting model source
10:57:44 | DEBUG    | Model getting fetched from DockerHub
10:57:44 | DEBUG    | Starting fetching procedure
10:57:44 | DEBUG    | Decided to fetch from DockerHub
10:57:44 | DEBUG    | Fetching from DockerHub
10:57:44 | DEBUG    | Pulling model image from DockerHub
10:57:44 | DEBUG    | Image ersiliaos/eos4e40:latest is available locally
Requested image eos4e40 is available locally. Do you still want to fetch it? [Y/n]y
10:57:47 | DEBUG    | Deleting locally available image ersiliaos/eos4e40:latest
Untagged: ersiliaos/eos4e40:latest
Untagged: ersiliaos/eos4e40@sha256:3771206150556c381dfc8855120cedbfe89812789b8cdfe9510134162a18d8de
Deleted: sha256:5b2f5884fa50c8ab5eaa04988edbccff2445fd7cc751e2f165cc6562e09c84fe
Deleted: sha256:04bf490d7c8ab9db90fa69abe50aa7cd46f28a596af097b6ffd255190fa7a637
10:57:49 | DEBUG    | The docker image ersiliaos/eos4e40:latest exists in DockerHub
10:57:49 | DEBUG    | Pulling image ersiliaos/eos4e40:latest from DockerHub...
10:57:49 | DEBUG    | Trying to pull image ersiliaos/eos4e40

It can be there is a problem with models that were fetched from older version of Ersilia that are now not recognised? @DhanshreeA tagging you here as well as this might have to do with backward comp. of the Ersilia Pack

GemmaTuron commented 2 weeks ago

Indeed I can confirm that the model images are in my DockerHub (which I see using docker image ls) but they are not recognised by Ersilia therefore triggering this error. Then this seems more to do with ersilia pack @DhanshreeA than the changes in the cli by @Abellegese but I let you have a look at it.

An option if you think this will be impossible to reproduce is for me to simply delete all images and start anew. But I think we need to make sure there is not an error in the code that could easily fix this first.

DhanshreeA commented 2 weeks ago

@GemmaTuron can you please give me the following details?

  1. Version for the library 'inputimeout' in your ersilia environment
  2. OS and version this was run on, and how this was run?

I ask these because this really seems more like an OS level permission issue, from these last few logs:

  File "/home/gturon/miniconda3/envs/ersilia/lib/python3.11/site-packages/inputimeout/inputimeout.py", line 24, in posix_inputimeout
    sel.register(sys.stdin, selectors.EVENT_READ)
  File "/home/gturon/miniconda3/envs/ersilia/lib/python3.11/selectors.py", line 359, in register
    self._selector.register(key.fd, poller_events)
PermissionError: [Errno 1] Operation not permitted

It seems like the shell that this was running on does not have permission to wait for user input, or is non-interactive.

GemmaTuron commented 2 weeks ago

mm that would be surprising as is the workstation (Ubuntu 22) I always use and it had never given problems before. I have not changed any permissions on my user. I run it through the CLI command ersilia run. I reinstalled ersilia, inputtimeout is 1.0.4