problems with compose.yaml and the nolabs backend

[vilmosbozsik]

First of all, compose.yaml has an invalid (currently unreleased) version of nolabs and reinvent which of course docker-compose can't find. After editing compose.yaml, I can install the containers, but I can't access the webui, since it jsut gets stuck in an infinite loop over loading the nonexistent experiments. (nolabs: set version to 2.0.4, reinvent: set version to 1.0.0) After analyzing the logs, turns out the backend crashes out and the webui simply can't reach the backend at localhost:8000. I've already tried all released versions of your docker containers, nothing works... It seems to me like there's a problem with the python libraries inside the container. This project has enormous potential and I really hope these issues can get resolved. Here's the nolabs log from the container: nolabs-1 | INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) nolabs-1 | INFO: Started parent process [8] nolabs-1 | nolabs-1 | » Reported at............ 8/12/2024 2:30:33 PM nolabs-1 | » App dir................ /app/frontend nolabs-1 | » App URL................ http://localhost:9000/ nolabs-1 | http://192.168.0.109:9000/ nolabs-1 | http://192.168.67.244:9000/ nolabs-1 | » Dev mode............... spa nolabs-1 | » Pkg quasar............. v2.15.2 nolabs-1 | » Pkg @quasar/app-vite... v1.8.0 nolabs-1 | » Browser target......... es2019|edge88|firefox78|chrome87|safari13.1 nolabs-1 | nolabs-1 | Process SpawnProcess-1: nolabs-1 | Process SpawnProcess-2: nolabs-1 | Process SpawnProcess-3: nolabs-1 | Process SpawnProcess-4: nolabs-1 | Traceback (most recent call last): nolabs-1 | Traceback (most recent call last): nolabs-1 | Traceback (most recent call last): nolabs-1 | Traceback (most recent call last): nolabs-1 | File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap nolabs-1 | self.run() nolabs-1 | File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap nolabs-1 | self.run() nolabs-1 | File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run nolabs-1 | self._target(*self._args, self._kwargs) nolabs-1 | File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run nolabs-1 | self._target(*self._args, *self._kwargs) nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/_subprocess.py", line 76, in subprocess_started nolabs-1 | target(sockets=sockets) nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/_subprocess.py", line 76, in subprocess_started nolabs-1 | target(sockets=sockets) nolabs-1 | File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap nolabs-1 | self.run() nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/server.py", line 61, in run nolabs-1 | return asyncio.run(self.serve(sockets=sockets)) nolabs-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/server.py", line 61, in run nolabs-1 | return asyncio.run(self.serve(sockets=sockets)) nolabs-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nolabs-1 | File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run nolabs-1 | return runner.run(main) nolabs-1 | ^^^^^^^^^^^^^^^^ nolabs-1 | File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run nolabs-1 | self._target(self._args, self._kwargs) nolabs-1 | File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run nolabs-1 | return runner.run(main) nolabs-1 | ^^^^^^^^^^^^^^^^ nolabs-1 | File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap nolabs-1 | self.run() nolabs-1 | File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run nolabs-1 | return self._loop.run_until_complete(task) nolabs-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/_subprocess.py", line 76, in subprocess_started nolabs-1 | target(sockets=sockets) nolabs-1 | File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run nolabs-1 | return self._loop.run_until_complete(task) nolabs-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nolabs-1 | File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run nolabs-1 | self._target(*self._args, **self._kwargs) nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/server.py", line 61, in run nolabs-1 | return asyncio.run(self.serve(sockets=sockets)) nolabs-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nolabs-1 | File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete nolabs-1 | return future.result() nolabs-1 | ^^^^^^^^^^^^^^^ nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/_subprocess.py", line 76, in subprocess_started nolabs-1 | target(sockets=sockets) nolabs-1 | File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run nolabs-1 | return runner.run(main) nolabs-1 | ^^^^^^^^^^^^^^^^ nolabs-1 | File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete nolabs-1 | return future.result() nolabs-1 | ^^^^^^^^^^^^^^^ nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/server.py", line 68, in serve nolabs-1 | config.load() nolabs-1 | File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run nolabs-1 | return self._loop.run_until_complete(task) nolabs-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/config.py", line 467, in load nolabs-1 | self.loaded_app = import_from_string(self.app) nolabs-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/server.py", line 61, in run nolabs-1 | return asyncio.run(self.serve(sockets=sockets)) nolabs-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/server.py", line 68, in serve nolabs-1 | config.load() nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/importer.py", line 24, in import_from_string nolabs-1 | raise exc from None nolabs-1 | File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run nolabs-1 | return runner.run(main) nolabs-1 | ^^^^^^^^^^^^^^^^ nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/importer.py", line 21, in import_from_string nolabs-1 | module = importlib.import_module(module_str) nolabs-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nolabs-1 | File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete nolabs-1 | return future.result() nolabs-1 | ^^^^^^^^^^^^^^^ nolabs-1 | File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run nolabs-1 | return self._loop.run_until_complete(task) nolabs-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/config.py", line 467, in load nolabs-1 | self.loaded_app = import_from_string(self.app) nolabs-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nolabs-1 | File "/usr/local/lib/python3.11/importlib/init.py", line 126, in import_module nolabs-1 | return _bootstrap._gcd_import(name[level:], package, level) nolabs-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/server.py", line 68, in serve nolabs-1 | config.load() nolabs-1 | File "", line 1204, in _gcd_import nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/importer.py", line 24, in import_from_string nolabs-1 | raise exc from None nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/config.py", line 467, in load nolabs-1 | self.loaded_app = import_from_string(self.app) nolabs-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nolabs-1 | File "", line 1176, in _find_and_load nolabs-1 | File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete nolabs-1 | return future.result() nolabs-1 | ^^^^^^^^^^^^^^^ nolabs-1 | File "", line 1147, in _find_and_load_unlocked nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/importer.py", line 24, in import_from_string nolabs-1 | raise exc from None nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/importer.py", line 21, in import_from_string nolabs-1 | module = importlib.import_module(module_str) nolabs-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nolabs-1 | File "", line 690, in _load_unlocked nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/server.py", line 68, in serve nolabs-1 | config.load() nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/importer.py", line 21, in import_from_string nolabs-1 | module = importlib.import_module(module_str) nolabs-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nolabs-1 | File "", line 940, in exec_module nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/config.py", line 467, in load nolabs-1 | self.loaded_app = import_from_string(self.app) nolabs-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nolabs-1 | File "/usr/local/lib/python3.11/importlib/init.py", line 126, in import_module nolabs-1 | return _bootstrap._gcd_import(name[level:], package, level) nolabs-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nolabs-1 | File "/usr/local/lib/python3.11/importlib/init.py", line 126, in import_module nolabs-1 | return _bootstrap._gcd_import(name[level:], package, level) nolabs-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nolabs-1 | File "", line 241, in _call_with_frames_removed nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/importer.py", line 24, in import_from_string nolabs-1 | raise exc from None nolabs-1 | File "", line 1204, in _gcd_import nolabs-1 | File "", line 1204, in _gcd_import nolabs-1 | File "/app/nolabs/api.py", line 4, in nolabs-1 | from nolabs.controllers.conformations.conformations import router as conformations_router nolabs-1 | File "/root/.cache/pypoetry/virtualenvs/nolabsserver-9TtSrW0h-py3.11/lib/python3.11/site-packages/uvicorn/importer.py", line 21, in import_from_string nolabs-1 | module = importlib.import_module(module_str) nolabs-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nolabs-1 | File "", line 1176, in _find_and_load nolabs-1 | File "", line 1176, in _find_and_load nolabs-1 | File "", line 1147, in _find_and_load_unlocked nolabs-1 | File "/usr/local/lib/python3.11/importlib/init.py", line 126, in import_module nolabs-1 | return _bootstrap._gcd_import(name[level:], package, level) nolabs-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ nolabs-1 | File "", line 1147, in _find_and_load_unlocked nolabs-1 | File "", line 690, in _load_unlocked nolabs-1 | File "", line 1204, in _gcd_import nolabs-1 | File "", line 940, in exec_module nolabs-1 | File "", line 1176, in _find_and_load nolabs-1 | ModuleNotFoundError: No module named 'nolabs.controllers' nolabs-1 | File "", line 690, in _load_unlocked nolabs-1 | File "", line 241, in _call_with_frames_removed nolabs-1 | File "", line 1147, in _find_and_load_unlocked nolabs-1 | File "", line 690, in _load_unlocked nolabs-1 | File "/app/nolabs/api.py", line 4, in nolabs-1 | from nolabs.controllers.conformations.conformations import router as conformations_router nolabs-1 | File "", line 940, in exec_module nolabs-1 | File "", line 940, in exec_module nolabs-1 | File "", line 241, in _call_with_frames_removed nolabs-1 | File "", line 241, in _call_with_frames_removed nolabs-1 | File "/app/nolabs/api.py", line 4, in nolabs-1 | from nolabs.controllers.conformations.conformations import router as conformations_router nolabs-1 | File "/app/nolabs/api.py", line 4, in nolabs-1 | from nolabs.controllers.conformations.conformations import router as conformations_router nolabs-1 | ModuleNotFoundError: No module named 'nolabs.controllers' nolabs-1 | ModuleNotFoundError: No module named 'nolabs.controllers' nolabs-1 | ModuleNotFoundError: No module named 'nolabs.controllers'

[timurishmuratov7]

Is this the latest version of master (we fixed this issue a couple of weeks ago)? I ran it in a clean directory and it worked for me. What's your platform you are running it from? And what's the docker command (does it include mongo and redis from the README)?

[vilmosbozsik]

I've tried docker compose up and docker compose up nolabs redis mongo. None of them worked. I just git cloned the repo, so I assume it was the latest version

[timurishmuratov7]

What's your OS?

[vilmosbozsik]

Ubuntu 24.04, in a clean LXD container

[timurishmuratov7]

Looking into it now.

[JaktensTid]

Hi @vilmosbozsik are you trying to use nolabs with reinvent?

[vilmosbozsik]

I just tried to install everything at once because why not. I'm pretty new to bioinformatics, an I'm just trying to explore everything.

[JaktensTid]

Thank you for your patience. Give me some time, I will fix the issue with main docker container version and reinvent.

What about installing everything at a time - images with AI models that we have a pretty big, so it is better to use docker compose up <dependencies that you need>. For example if you want to test reinvent only it is better to use docker compose up nolabs mongo redis reinvent

[vilmosbozsik]

Yeah I know, I've also tried the "barebones" version, but it didn't work. I was actually just going to try every tool out and delete the unnecessary ones afterwards. I just want a tool for docking simulation in the end.

[timurishmuratov7]

Ok I think I figured it out.

It will get a bit awkward in terms of workaround since you are running in a container, but it's fine, I just did that on a VM in a similar setting for Ubuntu, so here are the steps to fix:

Clear cache 1) Delete the containers for nolabs, redis and mongo that have been created with docker compose sudo docker ps -a <List of containers, you need to look for IDs of the ones which have "nolabs-" prefix> sudo docker rm <container_id1> <container_id2> <container_id3> 2) Delete the images which have been created for nolabs with previous docker compose runs sudo docker images <List of images, you need to look for IDs of the ones with names "ghcr.io/basedlabs/nolabs", "mongo", "redis"> sudo docker image rm <id1> <id2> <id3>

Adjusting for remote connection Then, since you are running in the container, you need to re-build nolabs with a correct version (we just published 2.1.5 which has blast in it and should not contain the error) and a correct IP address of your container.

If you open UI in the same container environment, then everything is fine, no need to adjust anything (skip to instructions of running NoLabs).

If you are connecting to this Ubuntu machine via SSH, from some other machine, you need to adjust the file https://github.com/BasedLabs/NoLabs/blob/master/frontend/src/refinedApi/constants.ts Instead of 'http://127.0.0.1:8000' you would need 'http://192.168.0.109:8000' (I assume this is the public IP of your machine)

Running NoLabs sudo docker compose build --no-cache nolabs sudo docker compose up nolabs mongo redis

Spinning up microservices As far as I undestood, you want to run docking. Therefore, you don't need to spin up other containers than 'esmfold_light' and 'diffdock', since they would occupy GPU (we are working on infra right now so that it is done way smarter, but for now it is the best way to manage your computational resources)

In a separate bash window, navigate to nolabs directory and run sudo docker compose up esmfold_light diffdock

This would start the containers to which the backend will talk to

Let me know if that solves the issue. And also feel free to reach out if you have other bugs/questions

[vilmosbozsik]

Thanks, that seemed to have done the trick! I've also had a bit of trouble getting gpu passthrough going, so I though I'd share my fix for that. I just modified the compose.yaml slightly: added nvidia_driver: image: nvidia/cuda:11.0-base runtime: nvidia volumes:

• /dev/nvidia0:/dev/nvidia0 changed all services needing gpu like conformations: image: 'ghcr.io/basedlabs/conformations:1.1.0' network_mode: host runtime: nvidia restart: unless-stopped build: context: microservices/conformations dockerfile: build/Dockerfile command: --host=0.0.0.0 --port=5785 volumes:
• /dev/nvidia0:/dev/nvidia0 I'll now try to run a docking simulation and we'll see if it works. I think this issue can be closed.

[timurishmuratov7]

Nice! Also the product is still in it's early stages so feel free to reach out directly if you need to see any tools/functionality we could add for your research. There's a tutorial available on YouTube: https://www.youtube.com/watch?v=DF1_q8qBdmM

And beware that when you first start diffdock it will warm up the first job for ~5 mins (it will cache the weights and stuff), so don't worry if the first docking job is running a bit long, the following docking jobs are running for 1 min approximately.

Based on how diffdock works, you'd need to upload your fasta files first and run them through esmfold/esmfold_light (esmfold_light supports sequences less than 400 amino acids in length, but esmfold does not have this limitation).

[JaktensTid]

Thanks, that seemed to have done the trick! I've also had a bit of trouble getting gpu passthrough going, so I though I'd share my fix for that. I just modified the compose.yaml slightly: added nvidia_driver: image: nvidia/cuda:11.0-base runtime: nvidia volumes: - /dev/nvidia0:/dev/nvidia0 changed all services needing gpu like conformations: image: 'ghcr.io/basedlabs/conformations:1.1.0' network_mode: host runtime: nvidia restart: unless-stopped build: context: microservices/conformations dockerfile: build/Dockerfile command: --host=0.0.0.0 --port=5785 volumes: - /dev/nvidia0:/dev/nvidia0 I'll now try to run a docking simulation and we'll see if it works. I think this issue can be closed.

Great, that it worked! May I know what issue with GPU have you experienced? You can open another issue