search

SpikeInterface / spikeinterface

A Python-based module for creating flexible and robust spike sorting pipelines.
https://spikeinterface.readthedocs.io
MIT License
527 stars 187 forks source link

Using spikeinterface with a spike sorter installed in a different environment #1890

Open grg2rsr opened 1 year ago

grg2rsr commented 1 year ago

Hi, I have a simple question regarding the installation of spike sorters that I can't seem to find an answer for in th docs. I have pykilosort previously installed in a different environment, and now looking into spikeinterface. Managing the CUDA mess was nothing that I necessarily want to go through again. Can I somehow tell spikeinterface where to find the sorter, as in tell the wrapper the environment of installation or something similar? If I understood the docs correctly, the sorters are started as a different process in any case. I was hoping that something like this exists:

sorting_pyks = ss.run_sorter("pykilosort", recording, output_folder="spikeinterface_pyks", env_name= "pyks2")

but doesn't seem like. Any chance I missed something? Thanks

alejoe91 commented 1 year ago

Hi @grg2rsr

Unfortunately that is not an option. The sorters are started in the same process when run locally, so selecting an a different environment would require some changes and some additional flexibility (why not supporting venv too?)

Have you looked into using docker/singularity? That should solve any installation issue, since the sorter doesn't need to be installed.

Cheers Alessio

grg2rsr commented 1 year ago

after some lost and won battles with docker, I am getting the following error from spikeinterface

---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
File [~/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/api/client.py:268](https://file+.vscode-resource.vscode-cdn.net/home/georg/Projects/FreelyMoving/~/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/api/client.py:268), in APIClient._raise_for_status(self, response)
    [267](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/api/client.py?line=266) try:
--> [268](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/api/client.py?line=267)     response.raise_for_status()
    [269](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/api/client.py?line=268) except requests.exceptions.HTTPError as e:

File [~/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/requests/models.py:1021](https://file+.vscode-resource.vscode-cdn.net/home/georg/Projects/FreelyMoving/~/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/requests/models.py:1021), in Response.raise_for_status(self)
   [1020](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/requests/models.py?line=1019) if http_error_msg:
-> [1021](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/requests/models.py?line=1020)     raise HTTPError(http_error_msg, response=self)

HTTPError: 400 Client Error: Bad Request for url: http+docker://localhost/v1.41/containers/17839a28557d56af412be33ba18377bf1e9e711fd6beefe4b751d770438d9906/start

The above exception was the direct cause of the following exception:

APIError                                  Traceback (most recent call last)
[/home/georg/Projects/FreelyMoving/spikeinterface_testing.py](https://file+.vscode-resource.vscode-cdn.net/home/georg/Projects/FreelyMoving/spikeinterface_testing.py) in line 2
      [36](file:///home/georg/Projects/FreelyMoving/spikeinterface_testing.py?line=35) # %%
----> [37](file:///home/georg/Projects/FreelyMoving/spikeinterface_testing.py?line=36) sorting_KS2_5 = ss.run_sorter("pykilosort", recording, output_folder="[/home/georg/data_local/D1Chr2_6351_240723/spikeinterface_ks25/](https://file+.vscode-resource.vscode-cdn.net/home/georg/data_local/D1Chr2_6351_240723/spikeinterface_ks25/)", docker_image=True)

File [~/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py:142](https://file+.vscode-resource.vscode-cdn.net/home/georg/Projects/FreelyMoving/~/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py:142), in run_sorter(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, delete_container_files, with_output, **sorter_params)
    [140](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py?line=139)         else:
    [141](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py?line=140)             container_image = singularity_image
--> [142](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py?line=141)     return run_sorter_container(
    [143](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py?line=142)         container_image=container_image,
    [144](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py?line=143)         mode=mode,
    [145](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py?line=144)         **common_kwargs,
    [146](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py?line=145)     )
    [148](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py?line=147) return run_sorter_local(**common_kwargs)

File [~/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py:489](https://file+.vscode-resource.vscode-cdn.net/home/georg/Projects/FreelyMoving/~/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py:489), in run_sorter_container(sorter_name, recording, mode, container_image, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, delete_container_files, extra_requirements, **sorter_params)
    [487](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py?line=486) if verbose:
    [488](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py?line=487)     print("Starting container")
--> [489](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py?line=488) container_client.start()
    [491](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py?line=490) if verbose and install_si_from_source:
    [492](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py?line=491)     print("******")

File [~/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py:310](https://file+.vscode-resource.vscode-cdn.net/home/georg/Projects/FreelyMoving/~/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py:310), in ContainerClient.start(self)
    [308](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py?line=307) def start(self):
    [309](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py?line=308)     if self.mode == "docker":
--> [310](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py?line=309)         self.docker_container.start()
    [311](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py?line=310)     elif self.mode == "singularity":
    [312](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py?line=311)         self.client_instance.start()

File [~/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/models/containers.py:406](https://file+.vscode-resource.vscode-cdn.net/home/georg/Projects/FreelyMoving/~/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/models/containers.py:406), in Container.start(self, **kwargs)
    [397](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/models/containers.py?line=396) def start(self, **kwargs):
    [398](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/models/containers.py?line=397)     """
    [399](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/models/containers.py?line=398)     Start this container. Similar to the ``docker start`` command, but
    [400](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/models/containers.py?line=399)     doesn't support attach options.
   (...)
    [404](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/models/containers.py?line=403)             If the server returns an error.
    [405](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/models/containers.py?line=404)     """
--> [406](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/models/containers.py?line=405)     return self.client.api.start(self.id, **kwargs)

File [~/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/utils/decorators.py:19](https://file+.vscode-resource.vscode-cdn.net/home/georg/Projects/FreelyMoving/~/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/utils/decorators.py:19), in check_resource..decorator..wrapped(self, resource_id, *args, **kwargs)
     [15](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/utils/decorators.py?line=14) if not resource_id:
     [16](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/utils/decorators.py?line=15)     raise errors.NullResource(
     [17](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/utils/decorators.py?line=16)         'Resource ID was not provided'
     [18](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/utils/decorators.py?line=17)     )
---> [19](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/utils/decorators.py?line=18) return f(self, resource_id, *args, **kwargs)

File [~/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/api/container.py:1127](https://file+.vscode-resource.vscode-cdn.net/home/georg/Projects/FreelyMoving/~/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/api/container.py:1127), in ContainerApiMixin.start(self, container, *args, **kwargs)
   [1125](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/api/container.py?line=1124) url = self._url("[/containers/](https://file+.vscode-resource.vscode-cdn.net/containers/){0}[/start](https://file+.vscode-resource.vscode-cdn.net/start)", container)
   [1126](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/api/container.py?line=1125) res = self._post(url)
-> [1127](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/api/container.py?line=1126) self._raise_for_status(res)

File [~/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/api/client.py:270](https://file+.vscode-resource.vscode-cdn.net/home/georg/Projects/FreelyMoving/~/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/api/client.py:270), in APIClient._raise_for_status(self, response)
    [268](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/api/client.py?line=267)     response.raise_for_status()
    [269](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/api/client.py?line=268) except requests.exceptions.HTTPError as e:
--> [270](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/api/client.py?line=269)     raise create_api_error_from_http_exception(e) from e

File [~/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/errors.py:39](https://file+.vscode-resource.vscode-cdn.net/home/georg/Projects/FreelyMoving/~/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/errors.py:39), in create_api_error_from_http_exception(e)
     [37](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/errors.py?line=36)     else:
     [38](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/errors.py?line=37)         cls = NotFound
---> [39](file:///home/georg/anaconda3/envs/spikeinterface/lib/python3.9/site-packages/docker/errors.py?line=38) raise cls(e, response=response, explanation=explanation) from e

APIError: 400 Client Error for http+docker://localhost/v1.41/containers/17839a28557d56af412be33ba18377bf1e9e711fd6beefe4b751d770438d9906/start: Bad Request ("failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown")

I have all nvidia related things installed, including the nvidia-container-toolkit

alejoe91 commented 1 year ago

What OS are you using? Are you able to run other sorters via Docker? Does this command work?

docker run -it --gpus all spikeinterface/pykilosort-base:latest

And inside the container run: nvidia-smi?

Have you followed all steps here (https://spikeinterface.readthedocs.io/en/latest/modules/sorters.html#containerizedsorters) to set up docker with the GPU?

grg2rsr commented 1 year ago

thanks for your response!

What OS are you using?

Ubuntu 22.04.2 LTS

Are you able to run other sorters via Docker?

at least what I tried now is klusta (which crashes as it needs pandas, very unrelated issue but maybe interesting for you that it misses it), now trying mountainsort4, which ends up stuck in pulling forever.

Does this command work?

no:

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown.
ERRO[0002] error waiting for container: context canceled

currently looking into https://github.com/NVIDIA/nvidia-docker/issues/1648 and https://github.com/NVIDIA/nvidia-container-toolkit/issues/154 ... I think probably related.

samuelgarcia commented 1 year ago

Hi @grg2rsr. Thanks you for the suggestion. running a sorter in a differents env should be not so a big patch in our API for python based sorters.

alejoe91 commented 1 year ago

@samuelgarcia we would still need a run_sorter_in_subprocess() function and make a script similar to the container one. No?

samuelgarcia commented 1 year ago

@alejoe91 : I would have in mind more something than we do for matlab prepare a script and run it in a diffrent env like we do for matlab not somethign complicated like docker where SI must be installed in the other envs.