search

TuragaLab / DECODE

This is the official implementation of our publication "Deep learning enables fast and dense single-molecule localization with high accuracy" (Nature Methods)
GNU General Public License v3.0
89 stars 26 forks source link

Install numba in DECODE environment? #217

Closed cleterrier closed 1 year ago

cleterrier commented 1 year ago

Hi, I am trying to locally install the COMET drift-correction notebook (https://github.com/gpufit/Comet/blob/master/Colab_notebooks/COMET.ipynb) to chain it with DECODE output and correct drift on localization data.

The "from numba import cuda" fails because numba is not intalled in the DECODE environment. I tried a simple "conda install numba" but it does not seem to resolve properly. Any idea of what I should do?

Here is everything that is installed in my current DECODE environment: absl-py 0.12.0 pyhd8ed1ab_0 conda-forge aiohttp 3.7.4 py38h294d835_0 conda-forge anyio 2.2.0 py38haa244fe_0 conda-forge appdirs 1.4.4 pyh9f0ad1d_0 conda-forge argon2-cffi 20.1.0 py38h294d835_2 conda-forge async-timeout 3.0.1 py_1000 conda-forge async_generator 1.10 py_0 conda-forge atomicwrites 1.4.0 pyh9f0ad1d_0 conda-forge attrs 20.3.0 pyhd3deb0d_0 conda-forge babel 2.9.0 pyhd3deb0d_0 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 py_2 conda-forge backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge blas 2.106 mkl conda-forge bleach 3.3.0 pyh44b312d_0 conda-forge blinker 1.4 py_1 conda-forge blosc 1.21.0 h0e60522_0 conda-forge brotli 1.0.9 h0e60522_4 conda-forge brotlipy 0.7.0 py38h294d835_1001 conda-forge bzip2 1.0.8 h8ffe710_4 conda-forge ca-certificates 2020.12.5 h5b45459_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cachetools 4.2.1 pyhd8ed1ab_0 conda-forge certifi 2020.12.5 py38haa244fe_1 conda-forge cffi 1.14.5 py38hd8c33c5_0 conda-forge chardet 4.0.0 py38haa244fe_1 conda-forge charls 2.2.0 h39d44d4_0 conda-forge click 7.1.2 pyh9f0ad1d_0 conda-forge cloudpickle 1.6.0 py_0 conda-forge colorama 0.4.4 pyh9f0ad1d_0 conda-forge cryptography 3.4.7 py38hd7da0ea_0 conda-forge cudatoolkit 11.0.3 h3f58a73_8 conda-forge cycler 0.10.0 py_2 conda-forge cytoolz 0.11.0 py38h294d835_3 conda-forge dask-core 2021.4.1 pyhd8ed1ab_0 conda-forge decode 0.10.0 np119py_0 turagalab decorator 4.4.2 py_0 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge deprecated 1.2.12 pyh44b312d_0 conda-forge deprecation 2.1.0 pyh9f0ad1d_0 conda-forge entrypoints 0.3 pyhd8ed1ab_1003 conda-forge freetype 2.10.4 h546665d_1 conda-forge fsspec 2021.4.0 pyhd8ed1ab_0 conda-forge giflib 5.2.1 h8d14728_2 conda-forge gitdb 4.0.7 pyhd8ed1ab_0 conda-forge gitpython 3.1.15 pyhd8ed1ab_0 conda-forge google-auth 1.28.0 pyh44b312d_0 conda-forge google-auth-oauthlib 0.4.1 py_2 conda-forge grpcio 1.37.0 py38he5377a8_0 conda-forge h5py 3.2.1 nompi_py38he6c2248_100 conda-forge hdf5 1.10.6 nompi_h5268f04_1114 conda-forge icu 68.1 h0e60522_0 conda-forge idna 2.10 pyh9f0ad1d_0 conda-forge imagecodecs 2021.3.31 py38hccd6b70_0 conda-forge imageio 2.9.0 py_0 conda-forge importlib-metadata 4.0.1 py38haa244fe_0 conda-forge importlib_resources 5.1.2 py38haa244fe_0 conda-forge iniconfig 1.1.1 pyh9f0ad1d_0 conda-forge intel-openmp 2021.2.0 h57928b3_616 conda-forge ipykernel 5.5.3 py38h43734a8_0 conda-forge ipython 7.22.0 py38h43734a8_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge ipywidgets 7.6.3 pypi_0 pypi jedi 0.18.0 py38haa244fe_2 conda-forge jinja2 2.11.3 pyh44b312d_0 conda-forge joblib 1.0.1 pyhd8ed1ab_0 conda-forge jpeg 9d h8ffe710_0 conda-forge json5 0.9.5 pyh9f0ad1d_0 conda-forge jsonschema 3.2.0 pyhd8ed1ab_3 conda-forge jupyter-packaging 0.9.2 pyhd8ed1ab_0 conda-forge jupyter_client 6.1.12 pyhd8ed1ab_0 conda-forge jupyter_core 4.7.1 py38haa244fe_0 conda-forge jupyter_server 1.6.4 py38haa244fe_0 conda-forge jupyterlab 3.0.14 pyhd8ed1ab_0 conda-forge jupyterlab-widgets 1.0.0 pypi_0 pypi jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge jupyterlab_server 2.4.0 pyhd8ed1ab_0 conda-forge jxrlib 1.1 h8ffe710_2 conda-forge kiwisolver 1.3.1 py38hbd9d945_1 conda-forge krb5 1.17.2 hbae68bd_0 conda-forge lcms2 2.12 h2a16943_0 conda-forge lerc 2.2.1 h0e60522_0 conda-forge libaec 1.0.4 h39d44d4_1 conda-forge libblas 3.9.0 6_mkl conda-forge libcblas 3.9.0 6_mkl conda-forge libclang 11.1.0 default_h5c34c98_0 conda-forge libcurl 7.76.1 hf1763fc_1 conda-forge libdeflate 1.7 h8ffe710_5 conda-forge liblapack 3.9.0 6_mkl conda-forge liblapacke 3.9.0 6_mkl conda-forge libpng 1.6.37 h1d00b33_2 conda-forge libprotobuf 3.15.8 h7755175_0 conda-forge libsodium 1.0.18 h8d14728_1 conda-forge libssh2 1.9.0 h680486a_6 conda-forge libtiff 4.2.0 hc10be44_1 conda-forge libuv 1.41.0 h8ffe710_0 conda-forge libwebp-base 1.2.0 h8ffe710_2 conda-forge libzopfli 1.0.3 h0e60522_0 conda-forge locket 0.2.0 py_2 conda-forge lz4-c 1.9.3 h8ffe710_0 conda-forge m2w64-gcc-libgfortran 5.3.0 6 conda-forge m2w64-gcc-libs 5.3.0 7 conda-forge m2w64-gcc-libs-core 5.3.0 7 conda-forge m2w64-gmp 6.1.0 2 conda-forge m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge markdown 3.3.4 pyhd8ed1ab_0 conda-forge markupsafe 1.1.1 py38h294d835_3 conda-forge matplotlib 3.4.1 py38haa244fe_0 conda-forge matplotlib-base 3.4.1 py38heae8d8c_0 conda-forge mistune 0.8.4 py38h294d835_1003 conda-forge mkl 2020.4 hb70f87d_311 conda-forge more-itertools 8.7.0 pyhd8ed1ab_1 conda-forge msys2-conda-epoch 20160418 1 conda-forge multidict 5.1.0 py38h294d835_1 conda-forge nbclassic 0.2.7 pyhd8ed1ab_0 conda-forge nbclient 0.5.3 pyhd8ed1ab_0 conda-forge nbconvert 6.0.7 py38haa244fe_3 conda-forge nbformat 5.1.3 pyhd8ed1ab_0 conda-forge nd2reader 3.3.0 pypi_0 pypi nest-asyncio 1.5.1 pyhd8ed1ab_0 conda-forge networkx 2.5.1 pyhd8ed1ab_0 conda-forge ninja 1.10.2 h5362a0b_0 conda-forge notebook 6.3.0 pyha770c72_1 conda-forge numpy 1.19.5 py38h0cc643e_1 conda-forge oauthlib 3.0.1 py_0 conda-forge olefile 0.46 pyh9f0ad1d_1 conda-forge openjpeg 2.4.0 h48faf41_0 conda-forge openssl 1.1.1k h8ffe710_0 conda-forge packaging 20.9 pyh44b312d_0 conda-forge pandas 1.2.4 py38h60cbd38_0 conda-forge pandoc 2.13 h8ffe710_0 conda-forge pandocfilters 1.4.2 py_1 conda-forge parso 0.8.2 pyhd8ed1ab_0 conda-forge partd 1.2.0 pyhd8ed1ab_0 conda-forge patsy 0.5.1 py_0 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 8.1.2 py38h9273828_1 conda-forge pims 0.5 pypi_0 pypi pip 21.1 pyhd8ed1ab_0 conda-forge pluggy 0.13.1 py38haa244fe_4 conda-forge pooch 1.3.0 pyhd8ed1ab_0 conda-forge prometheus_client 0.10.1 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.18 pyha770c72_0 conda-forge protobuf 3.15.8 py38h885f38d_0 conda-forge py 1.10.0 pyhd3deb0d_0 conda-forge pyasn1 0.4.8 py_0 conda-forge pyasn1-modules 0.2.7 py_0 conda-forge pycparser 2.20 pyh9f0ad1d_2 conda-forge pygments 2.8.1 pyhd8ed1ab_0 conda-forge pyjwt 2.0.1 pyhd8ed1ab_1 conda-forge pyopenssl 20.0.1 pyhd8ed1ab_0 conda-forge pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge pyqt 5.12.3 py38haa244fe_7 conda-forge pyqt-impl 5.12.3 py38h885f38d_7 conda-forge pyqt5-sip 4.19.18 py38h885f38d_7 conda-forge pyqtchart 5.12 py38h885f38d_7 conda-forge pyqtwebengine 5.12.1 py38h885f38d_7 conda-forge pyreadline 2.1 py38haa244fe_1003 conda-forge pyrsistent 0.17.3 py38h294d835_2 conda-forge pysocks 1.7.1 py38haa244fe_3 conda-forge pytest 6.2.3 py38haa244fe_0 conda-forge python 3.8.8 h7840368_0_cpython conda-forge python-dateutil 2.8.1 py_0 conda-forge python_abi 3.8 1_cp38 conda-forge pytorch 1.7.1 py3.8_cuda110_cudnn8_0 pytorch pytz 2021.1 pyhd8ed1ab_0 conda-forge pywavelets 1.1.1 py38h347fdf6_3 conda-forge pywin32 300 py38h294d835_0 conda-forge pywinpty 0.5.7 py38haa244fe_1 conda-forge pyyaml 5.4.1 py38h294d835_0 conda-forge pyzmq 22.0.3 py38h09162b1_1 conda-forge qt 5.12.9 h5909a2a_4 conda-forge requests 2.25.1 pyhd3deb0d_0 conda-forge requests-oauthlib 1.3.0 pyh9f0ad1d_0 conda-forge rsa 4.7.2 pyh44b312d_0 conda-forge scikit-image 0.18.1 py38h4c96930_0 conda-forge scikit-learn 0.24.1 py38ha09990b_0 conda-forge scipy 1.6.2 py38he847743_0 conda-forge seaborn 0.10.1 py_0 conda-forge send2trash 1.5.0 py_0 conda-forge setuptools 49.6.0 py38haa244fe_3 conda-forge six 1.15.0 pyh9f0ad1d_0 conda-forge slicerator 1.0.0 pypi_0 pypi smmap 3.0.5 pyh44b312d_0 conda-forge snappy 1.1.8 ha925a31_3 conda-forge sniffio 1.2.0 py38haa244fe_1 conda-forge spline 0.10.0 np119py38ha925a31_1 turagalab sqlite 3.35.5 h8ffe710_0 conda-forge statsmodels 0.12.2 py38h347fdf6_0 conda-forge tensorboard 2.4.1 pyhd8ed1ab_0 conda-forge tensorboard-plugin-wit 1.8.0 pyh44b312d_0 conda-forge terminado 0.9.4 py38haa244fe_0 conda-forge testpath 0.4.4 py_0 conda-forge threadpoolctl 2.1.0 pyh5ca1d4c_0 conda-forge tifffile 2021.4.8 pyhd8ed1ab_0 conda-forge tk 8.6.10 h8ffe710_1 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge tomlkit 0.7.0 py38haa244fe_3 conda-forge toolz 0.11.1 py_0 conda-forge tornado 6.1 py38h294d835_1 conda-forge tqdm 4.60.0 pyhd8ed1ab_0 conda-forge traitlets 5.0.5 py_0 conda-forge typing-extensions 3.7.4.3 0 conda-forge typing_extensions 3.7.4.3 py_0 conda-forge urllib3 1.26.4 pyhd8ed1ab_0 conda-forge vc 14.2 hb210afc_4 conda-forge vs2015_runtime 14.28.29325 h5e1d092_4 conda-forge wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge webencodings 0.5.1 py_1 conda-forge werkzeug 1.0.1 pyh9f0ad1d_0 conda-forge wheel 0.36.2 pyhd3deb0d_0 conda-forge widgetsnbextension 3.5.1 pypi_0 pypi win_inet_pton 1.1.0 py38haa244fe_2 conda-forge wincertstore 0.2 py38haa244fe_1006 conda-forge winpty 0.4.3 4 conda-forge wrapt 1.12.1 py38h294d835_3 conda-forge xmltodict 0.12.0 pypi_0 pypi xz 5.2.5 h62dcd97_1 conda-forge yaml 0.2.5 he774522_0 conda-forge yarl 1.6.3 py38h294d835_1 conda-forge zeromq 4.3.4 h0e60522_0 conda-forge zfp 0.5.5 h0e60522_5 conda-forge zipp 3.4.1 pyhd8ed1ab_0 conda-forge zlib 1.2.11 h62dcd97_1010 conda-forge zstd 1.4.9 h6255e5f_0 conda-forge

cleterrier commented 1 year ago

Here's the end of the output of "conda install numba":

The following specifications were found to be incompatible with your system:

Your installed version is: 0

Note that strict channel priority may have removed packages required for satisfiability.

Haydnspass commented 1 year ago

@cleterrier hmm, that's a good point. I don't think we can do anything about it in the released DECODE version, as conda states that the packages are, in fact, incompatible; probably because of some side requirements. As a workaround you could save your localizations and then load them in a new environment with for the drift correction. The .h5 files are fairly easy to read, you can just take a look at the DECODE .h5 reader.

In a future release I might be able to change the requirements and/or relax them, such that combining both DECODE and the drift correction is possible.

cleterrier commented 1 year ago

@Haydnspass thanks for the tip, I'm trying to set up an independent environment for this. I'm wondering how I can get the relevant variables from an emitter set (x,y,z coordinates) into arrays that are the entry format of COMET? Relevant import code is currently this:

import numpy as np 
import pandas as pd 
import io

filepath = 'C:/Users/chris/Desktop/test.csv'

data = pd.read_csv(filepath)

localizations = np.zeros((len(data['frame']), 4))
localizations[:, 0] = np.asarray(data['x [nm]'])
localizations[:, 1] = np.asarray(data['y [nm]'])
localizations[:, 2] = np.asarray(data['z [nm]'])
localizations[:, 3] = np.asarray(data['frame'])
frames = np.unique(localizations[:, -1])
n_frames = len(frames)
print(f"{filepath} import successful, {len(localizations[:, 0])} localizations, {n_frames} frames")

So basically I'd need a way to feed x,y,z coordinates as arrays after opening the h5 file. Is there a command for that?

Haydnspass commented 1 year ago

I will have to look into that

nolan1999 commented 1 year ago

@cleterrier This should put x, y, z, and frame index from a .h5 file into a Nx4 numpy array:

import h5py
filename = <path_to_emitters.h5>

with h5py.File(filename, "r") as f:
    x, y, z = np.array(f["data"]["xyz"]).T
    f = np.array(f["data"]["frame_ix"])
    localizations = np.stack([x, y, z, f]).T

Does this help you?