search

HySonLab / ML4Insects

Machine Learning for signal processing on insects' behaviors
https://www.biorxiv.org/content/10.1101/2024.06.10.598170v1.full.pdf
MIT License
7 stars 1 forks source link

can't install #1

Closed bjarthur closed 1 month ago

bjarthur commented 1 month ago

trying to install on a Mac and am having problems because i'm not terribly familiar with python (usually use Julia). i start out by installing miniforge, python, and then cloning your repo, all of which works:

brew install miniforge

conda create --name epg
conda activate epg
conda install python

git clone https://github.com/HySonLab/ML4Insects.git
cd ML4Insects

but then how do i install your python package? python setup.py install gives me a warning about this not being the right way to do it. so i google and it says to do pip install . instead. so i do that, but then your code can't find "easydict":

>>> from PyEPG.utils import process_config
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/arthurb/projects/stern/ML4Insects/PyEPG/utils/__init__.py", line 1, in <module>
    from .configs_utils import *
  File "/Users/arthurb/projects/stern/ML4Insects/PyEPG/utils/configs_utils.py", line 4, in <module>
    from easydict import EasyDict
ModuleNotFoundError: No module named 'easydict'

so i try:

pip install -r PyEPG/requirements.txt

now it complains it can't find "cv2":

>>> from PyEPG.utils import process_config   # not ML4Insects
Matplotlib is building the font cache; this may take a moment.
>>> config = process_config('cnn1d.soybeanAphid.raw.256.json')
>>> config['dataset_name']='dataset'
>>> config['data_path'] = '/Users/arthurb/projects/stern/data'
>>> from PyEPG.models import EPGSegment 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/arthurb/projects/stern/ML4Insects/PyEPG/models/__init__.py", line 1, in <module>
    from .Segmentation import EPGSegment
  File "/Users/arthurb/projects/stern/ML4Insects/PyEPG/models/Segmentation.py", line 14, in <module>
    from .Dataset import EPGDataset
  File "/Users/arthurb/projects/stern/ML4Insects/PyEPG/models/Dataset.py", line 13, in <module>
    from ..dataset_utils.datagenerator import generate_sliding_windows_single
  File "/Users/arthurb/projects/stern/ML4Insects/PyEPG/dataset_utils/datagenerator.py", line 4, in <module>
    import cv2 
    ^^^^^^^^^^
ModuleNotFoundError: No module named 'cv2'

so i pip install opencv-python and now get "ImportError: attempted relative import beyond top-level package":

>>> from PyEPG.utils import process_config   # not ML4Insects
>>> config = process_config('cnn1d.soybeanAphid.raw.256.json')
>>> config['dataset_name']='dataset'
>>> config['data_path'] = '/Users/arthurb/projects/stern/data'
>>> 
>>> from PyEPG.models import EPGSegment 
>>> epgs = EPGSegment(config, inference = False)
Training mode.
Loading data ...
100%|█████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 16.59it/s]
Done! Elapsed: 0.19384466693736613 s
>>> epgs.load_checkpoint('cnn1d.soybeanAphid.json')
/Users/arthurb/projects/stern/ML4Insects/PyEPG/models/Segmentation.py:356: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  self.model = torch.load(f'./checkpoints/{path}')
/Users/arthurb/projects/stern/ML4Insects/PyEPG/models/Segmentation.py:358: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  self.model = torch.load(f'./checkpoints/{self.config.arch}/{path}')
Traceback (most recent call last):
  File "/Users/arthurb/projects/stern/ML4Insects/PyEPG/models/Segmentation.py", line 356, in load_checkpoint
    self.model = torch.load(f'./checkpoints/{path}')
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/epg/lib/python3.12/site-packages/torch/serialization.py", line 1065, in load
    with _open_file_like(f, 'rb') as opened_file:
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/epg/lib/python3.12/site-packages/torch/serialization.py", line 468, in _open_file_like
    return _open_file(name_or_buffer, mode)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/epg/lib/python3.12/site-packages/torch/serialization.py", line 449, in __init__
    super().__init__(open(name, mode))
                     ^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: './checkpoints/cnn1d.soybeanAphid.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/arthurb/projects/stern/ML4Insects/PyEPG/models/Segmentation.py", line 358, in load_checkpoint
    self.model = torch.load(f'./checkpoints/{self.config.arch}/{path}')
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/epg/lib/python3.12/site-packages/torch/serialization.py", line 1097, in load
    return _load(
           ^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/epg/lib/python3.12/site-packages/torch/serialization.py", line 1525, in _load
    result = unpickler.load()
             ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/epg/lib/python3.12/site-packages/torch/serialization.py", line 1515, in find_class
    return super().find_class(mod_name, name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/epg/lib/python3.12/site-packages/models/__init__.py", line 1, in <module>
    from .Segmentation import EPGSegment
  File "/opt/homebrew/Caskroom/miniforge/base/envs/epg/lib/python3.12/site-packages/models/Segmentation.py", line 14, in <module>
    from .Dataset import EPGDataset
  File "/opt/homebrew/Caskroom/miniforge/base/envs/epg/lib/python3.12/site-packages/models/Dataset.py", line 12, in <module>
    from ..dataset_utils.datahelper import format_data, read_signal, get_index
ImportError: attempted relative import beyond top-level package

any help would be appreciated. thanks!

DINHQuangDung1999 commented 1 month ago

Hi Ben, Thank you for your interest. I am sorry that you encounter these error as the repo was not updated on time. I have just published a new version of our package, for which you can install easily without conda as 

pip install DiscoEPG

The only requirements is that you need to have Python installed on your computer (preferably independently from conda)

bjarthur commented 1 month ago

can't find torch after installing with pip install DiscoEPG:

% pip install DiscoEPG
Collecting DiscoEPG
  Downloading DiscoEPG-0.0.6-py3-none-any.whl.metadata (8.7 kB)
Downloading DiscoEPG-0.0.6-py3-none-any.whl (61 kB)
Installing collected packages: DiscoEPG
Successfully installed DiscoEPG-0.0.6

% python
Python 3.12.6 | packaged by conda-forge | (main, Sep 11 2024, 04:55:15) [Clang 17.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from DiscoEPG import EPGSegment # Importing trainer objects
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/homebrew/Caskroom/miniforge/base/envs/epg/lib/python3.12/site-packages/DiscoEPG/__init__.py", line 1, in <module>
    from .models import *
  File "/opt/homebrew/Caskroom/miniforge/base/envs/epg/lib/python3.12/site-packages/DiscoEPG/models/__init__.py", line 1, in <module>
    from .Segmentation import EPGSegment
  File "/opt/homebrew/Caskroom/miniforge/base/envs/epg/lib/python3.12/site-packages/DiscoEPG/models/Segmentation.py", line 1, in <module>
    import torch 
    ^^^^^^^^^^^^
ModuleNotFoundError: No module named 'torch'
DINHQuangDung1999 commented 1 month ago

it seems that you have not installed PyTorch you can do that with this command 

pip3 install torch torchvision torchaudio

A small reminder. Since you are working on Mac, it will be difficult to run the Deep Learning algorithms as a key component for PyTorch called CUDA is not supported on Mac.

bjarthur commented 1 month ago

i get the same error on ubuntu 22.04:

% pip install DiscoEPG

Collecting DiscoEPG
  Downloading DiscoEPG-0.0.6-py3-none-any.whl.metadata (8.7 kB)
Downloading DiscoEPG-0.0.6-py3-none-any.whl (61 kB)
Installing collected packages: DiscoEPG
Successfully installed DiscoEPG-0.0.6
(epg) 
arthurb-ws3 stern % python               
Python 3.12.5 | packaged by Anaconda, Inc. | (main, Sep 12 2024, 18:27:27) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from DiscoEPG import EPGSegment
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/arthurb@hhmi.org/miniconda3/envs/epg/lib/python3.12/site-packages/DiscoEPG/__init__.py", line 1, in <module>
    from .models import *
  File "/home/arthurb@hhmi.org/miniconda3/envs/epg/lib/python3.12/site-packages/DiscoEPG/models/__init__.py", line 1, in <module>
    from .Segmentation import EPGSegment
  File "/home/arthurb@hhmi.org/miniconda3/envs/epg/lib/python3.12/site-packages/DiscoEPG/models/Segmentation.py", line 1, in <module>
    import torch 
    ^^^^^^^^^^^^
ModuleNotFoundError: No module named 'torch'

moreover, if i manually install torch as you suggested, it now complains it can't find pandas. if i then manually install pandas, it complains about matplotlib. this on both mac and linux.

bjarthur commented 1 month ago

have you tried installing in a fresh environment (pyenv or conda) on your machine?

DINHQuangDung1999 commented 1 month ago

Perhaps you can locate into your environment, put this requirements.txt file inside, and run

pip install -r requirements.txt

requirements.txt

This will install all the necessary dependencies for DiscoEPG.

bjarthur commented 1 month ago

i'm not sure how pyenv works, but in a fresh conda env i see none of DiscoEPG's requirements installed in sys.path:

% pip install DiscoEPG

Collecting DiscoEPG
  Using cached DiscoEPG-0.0.6-py3-none-any.whl.metadata (8.7 kB)
Using cached DiscoEPG-0.0.6-py3-none-any.whl (61 kB)
Installing collected packages: DiscoEPG
Successfully installed DiscoEPG-0.0.6

% python
Python 3.12.6 | packaged by conda-forge | (main, Sep 11 2024, 04:55:15) [Clang 17.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/opt/homebrew/Caskroom/miniforge/base/envs/epg/lib/python312.zip', '/opt/homebrew/Caskroom/miniforge/base/envs/epg/lib/python3.12', '/opt/homebrew/Caskroom/miniforge/base/envs/epg/lib/python3.12/lib-dynload', '/opt/homebrew/Caskroom/miniforge/base/envs/epg/lib/python3.12/site-packages']
>>> 

% ls /opt/homebrew/Caskroom/miniforge/base/envs/epg/lib/python3.12/site-packages
DiscoEPG                pip-24.2.dist-info
DiscoEPG-0.0.6.dist-info        pkg_resources
README.txt              setuptools
_distutils_hack             setuptools-73.0.1-py3.12.egg-info
distutils-precedence.pth        wheel
pip                 wheel-0.44.0.dist-info
DINHQuangDung1999 commented 1 month ago

Can you try this solution?

Perhaps you can locate into your environment, put this requirements.txt file inside, and run

pip install -r requirements.txt

requirements.txt

This will install all the necessary dependencies for DiscoEPG.

bjarthur commented 1 month ago

that almost works. lots of pkgs in my sys.path now. but pywt is missing, and there doesn't seem to be one available for Mac.

i think your fresh pyenv environment must somehow still be accessing your development environment.

% ls /opt/homebrew/Caskroom/miniforge/base/envs/epg/lib/python3.12/site-packages
DiscoEPG                    nbformat
DiscoEPG-0.0.6.dist-info            nbformat-5.10.4.dist-info
IPython                     nest_asyncio-1.6.0.dist-info
MarkupSafe-2.1.5.dist-info          nest_asyncio.py
PIL                     networkx
README.txt                  networkx-3.3.dist-info
__pycache__                 numba
_cffi_backend.cpython-312-darwin.so     numba-0.60.0.dist-info
_distutils_hack                 numpy
_plotly_future_                 numpy-1.26.4.dist-info
_plotly_utils                   opencv_python-4.10.0.84.dist-info
_soundfile.py                   orjson
_soundfile_data                 orjson-3.10.7.dist-info
asttokens                   packaging
asttokens-2.4.1.dist-info           packaging-24.1.dist-info
attr                        pandas
attrs                       pandas-2.2.2.dist-info
attrs-24.2.0.dist-info              parso
audioread                   parso-0.8.4.dist-info
audioread-3.0.1.dist-info           pexpect
blinker                     pexpect-4.9.0.dist-info
blinker-1.8.2.dist-info             pillow-10.4.0.dist-info
certifi                     pip
certifi-2024.8.30.dist-info         pip-24.2.dist-info
cffi                        pkg_resources
cffi-1.17.1.dist-info               platformdirs
charset_normalizer              platformdirs-4.3.2.dist-info
charset_normalizer-3.3.2.dist-info      plotly
click                       plotly-5.22.0.dist-info
click-8.1.7.dist-info               plotly_resampler
comm                        plotly_resampler-0.10.0.dist-info
comm-0.2.2.dist-info                pooch
contourpy                   pooch-1.8.2.dist-info
contourpy-1.3.0.dist-info           prompt_toolkit
cv2                     prompt_toolkit-3.0.47.dist-info
cycler                      ptyprocess
cycler-0.12.1.dist-info             ptyprocess-0.7.0.dist-info
dash                        pure_eval
dash-2.18.1.dist-info               pure_eval-0.2.3.dist-info
dash_core_components                pycparser
dash_core_components-2.0.0.dist-info        pycparser-2.22.dist-info
dash_html_components                pygments
dash_html_components-2.0.0.dist-info        pygments-2.18.0.dist-info
dash_table                  pylab.py
dash_table-5.0.0.dist-info          pyparsing
dateutil                    pyparsing-3.1.4.dist-info
decorator-5.1.1.dist-info           python_dateutil-2.9.0.post0.dist-info
decorator.py                    pyts
distutils-precedence.pth            pyts-0.13.0.dist-info
easydict                    pytz
easydict-1.13.dist-info             pytz-2024.2.dist-info
executing                   referencing
executing-2.1.0.dist-info           referencing-0.35.1.dist-info
fastjsonschema                  requests
fastjsonschema-2.20.0.dist-info         requests-2.32.3.dist-info
filelock                    retrying-1.3.4.dist-info
filelock-3.16.0.dist-info           retrying.py
flask                       rpds
flask-3.0.3.dist-info               rpds_py-0.20.0.dist-info
fontTools                   scikit_learn-1.5.1.dist-info
fonttools-4.53.1.dist-info          scipy
fsspec                      scipy-1.14.0.dist-info
fsspec-2024.9.0.dist-info           seaborn
functorch                   seaborn-0.13.2.dist-info
idna                        setuptools
idna-3.8.dist-info              setuptools-73.0.1-py3.12.egg-info
importlib_metadata              six-1.16.0.dist-info
importlib_metadata-8.5.0.dist-info      six.py
ipython-8.27.0.dist-info            sklearn
ipywidgets                  soundfile-0.12.1.dist-info
ipywidgets-8.1.3.dist-info          soundfile.py
isympy.py                   soxr
itsdangerous                    soxr-0.5.0.post1.dist-info
itsdangerous-2.2.0.dist-info            stack_data
jedi                        stack_data-0.6.3.dist-info
jedi-0.19.1.dist-info               sympy
jinja2                      sympy-1.13.2.dist-info
jinja2-3.1.4.dist-info              tenacity
joblib                      tenacity-9.0.0.dist-info
joblib-1.4.2.dist-info              threadpoolctl-3.5.0.dist-info
jsonschema                  threadpoolctl.py
jsonschema-4.23.0.dist-info         torch
jsonschema_specifications           torch-2.3.1.dist-info
jsonschema_specifications-2023.12.1.dist-info   torchgen
jupyter.py                  tqdm
jupyter_core                    tqdm-4.66.5.dist-info
jupyter_core-5.7.2.dist-info            traitlets
jupyterlab_plotly               traitlets-5.14.3.dist-info
jupyterlab_widgets              tsdownsample
jupyterlab_widgets-3.0.13.dist-info     tsdownsample-0.1.3.dist-info
kiwisolver                  typing_extensions-4.12.2.dist-info
kiwisolver-1.4.7.dist-info          typing_extensions.py
lazy_loader                 tzdata
lazy_loader-0.4.dist-info           tzdata-2024.1.dist-info
librosa                     urllib3
librosa-0.10.2.post1.dist-info          urllib3-2.2.3.dist-info
llvmlite                    wcwidth
llvmlite-0.43.0.dist-info           wcwidth-0.2.13.dist-info
markupsafe                  werkzeug
matplotlib                  werkzeug-3.0.4.dist-info
matplotlib-3.9.1.dist-info          wheel
matplotlib_inline               wheel-0.44.0.dist-info
matplotlib_inline-0.1.7.dist-info       widgetsnbextension
mpl_toolkits                    widgetsnbextension-4.0.13.dist-info
mpmath                      xgboost
mpmath-1.3.0.dist-info              xgboost-2.1.0.dist-info
msgpack                     zipp
msgpack-1.1.0.dist-info             zipp-3.20.1.dist-info

% python
Python 3.12.6 | packaged by conda-forge | (main, Sep 11 2024, 04:55:15) [Clang 17.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from DiscoEPG import EPGDataset
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/homebrew/Caskroom/miniforge/base/envs/epg/lib/python3.12/site-packages/DiscoEPG/__init__.py", line 1, in <module>
    from .models import *
  File "/opt/homebrew/Caskroom/miniforge/base/envs/epg/lib/python3.12/site-packages/DiscoEPG/models/__init__.py", line 1, in <module>
    from .Segmentation import EPGSegment
  File "/opt/homebrew/Caskroom/miniforge/base/envs/epg/lib/python3.12/site-packages/DiscoEPG/models/Segmentation.py", line 14, in <module>
    from .Dataset import EPGDataset
  File "/opt/homebrew/Caskroom/miniforge/base/envs/epg/lib/python3.12/site-packages/DiscoEPG/models/Dataset.py", line 13, in <module>
    from ..dataset_utils.datagenerator import generate_sliding_windows_single
  File "/opt/homebrew/Caskroom/miniforge/base/envs/epg/lib/python3.12/site-packages/DiscoEPG/dataset_utils/datagenerator.py", line 5, in <module>
    import pywt 
    ^^^^^^^^^^^
ModuleNotFoundError: No module named 'pywt'

% pip install pywt
ERROR: Could not find a version that satisfies the requirement pywt (from versions: none)
ERROR: No matching distribution found for pywt
DINHQuangDung1999 commented 1 month ago

Sorry, it miswrote that package. pip install PyWavelets should work!

bjarthur commented 1 month ago

that works! thanks so much. you should probably either update the docs with the requirement.txt step, or somehow get those deps into the wheel.

bjarthur commented 1 month ago

i'm not going to re-open this issue b/c the docs on installation to use the requirements.txt file work, but i will say that having to do so is an unusual way to install a python package. matplotlib's docs, for example, simply say to pip install. no mention of a requirements.txt file at all. i know nothing about creating packages in python (it's so much easier in Julia), but somehow there must be a way to get your dependencies into the wheel, and doing so would make it a lot easier for users of this package.

also, your docs for a conda install need an extra step to install python itself. conda is a package manager meant to support all programming languages, and so it doesn't install any by default.

DINHQuangDung1999 commented 1 month ago

Hi Ben,

Thank you very much for your suggestion. I think your opinions are reasonable. I will make sure the issues are taken care of.

Best,

On Mon, Sep 16, 2024 at 3:18 PM Ben Arthur @.***> wrote:

i'm not going to re-open this issue b/c the docs on installation to use the requirements.txt file work, but i will say that having to do so is an unusual way to install a python package. matplotlib's docs https://matplotlib.org/stable/install/index.html, for example, simply say to pip install. i know nothing about creating packages in python (it's so much easier in Julia), but somehow there must be a way to get your dependencies into the wheel, and doing so would make it a lot easier for users of this package.

also, your docs for a conda install need an extra step to install python itself. conda is a package manager meant to support all programming languages, and so it doesn't install any by default.

— Reply to this email directly, view it on GitHub https://github.com/HySonLab/ML4Insects/issues/1#issuecomment-2352897147, or unsubscribe https://github.com/notifications/unsubscribe-auth/BDAVC4JKX4RYW3PIKDTRU2TZW3LCBAVCNFSM6AAAAABOEAOXGGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJSHA4TOMJUG4 . You are receiving this because you commented.Message ID: @.***>

-- DINH Quang-Dung M.Sc. student Institut Galilée Université Sorbonne Paris Nord *99 Av. Jean Baptiste Clément, 93430 Villetaneuse, *Paris, France