Luckytek commented 2 months ago

When running the Train_clf.py script I am getting constantly the following attribute error: File "C:\Users\Pavel\anaconda3\Lib\site-packages\h5torch\dataset.py", line 132, in getitem if self.f["central"].attrs["mode"] == "N-D": ^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'AttrArray' object has no attribute 'attrs'. RKIpeaks.h5 file has been generated by the process_RKI.py script successfully (see HDFView printscreen) Please recommend what could be the problem. Thanks. Pavel

gdewael commented 2 months ago

Hi Pavel,

Thanks for testing out our code. It's curious, because I can't reproduce the error upon re-running using a fresh conda environment. What version of python and h5torch are you using? Additionally, how exactly have you used the train_clf script?

Kind regards, Gaetan

Luckytek commented 2 months ago

Hi Gaetan,

Thank you for your prompt response and for looking into this.

I’m using the following versions:

Python: version 3.11.5
h5torch: version 0.2.14

Regarding the usage of the train_clf.py script, here is the approach I followed:

Preprocessing: I first ran the process_RKI.py script with the command „python process_RKI.py F:\RKI_ROOT\ F:\DRIAMS_ROOT\RKIraw.h5 F:\DRIAMS_ROOT\RKIbin.h5 F:\DRIAMS_ROOT\RKIpeaks.h5“ to generate the HDF5 files (RKI_raw.h5, RKI_bin.h5, RKI_peaks.h5). The files appeared to be correctly structured, as verified using HDFView.
Training: After generating these files, I executed the train_clf.py script with the command „python train_clf.py F:/DRIAMS_ROOT/RKIpeaks.h5 logs/ trf M --devices "cpu" --lr 0.0005

This is when the AttributeError related to AttrArray lacking the attrs attribute occurred.

Kind regards,

Pavel

From: Gaetan De Waele @.*** Sent: Monday, August 19, 2024 10:03 AM To: gdewael/maldi-nn Cc: Luckytek; Author Subject: Re: [gdewael/maldi-nn] Train_clf.py issue (Issue #2)

Hi Pavel,

Thanks for testing out our code. It's curious, because I can't reproduce the error upon re-running using a fresh conda environment. What version of python and h5torch are you using? Additionally, how exactly have you used the train_clf script?

Kind regards, Gaetan

— Reply to this email directly, view it on GitHub https://github.com/gdewael/maldi-nn/issues/2#issuecomment-2295918703 , or unsubscribe https://github.com/notifications/unsubscribe-auth/A7HTH2RILDKISWD6AN5ZG5LZSGRCLAVCNFSM6AAAAABMWV6NKSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJVHEYTQNZQGM . You are receiving this because you authored the thread. https://github.com/notifications/beacon/A7HTH2QRLMZOWWCZIVPABXTZSGRCLA5CNFSM6AAAAABMWV6NKSWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTUI3DYG6.gif Message ID: @.***>

gdewael commented 2 months ago

You are using our code correctly. Could you provide me with your complete environment setup (e.g. the output of pip list), so I can reproduce?

gdewael commented 2 months ago

In addition, can you run this piece of code without error?

import h5torch

f = h5torch.File("F:/DRIAMS_ROOT/RKIpeaks.h5").to_dict()
f["central"].attrs

Luckytek commented 2 months ago

Hi Gaetan,

I've attached the complete environment setup as requested, including the output of pip list.

Please let me know if there's any further information you need or if there are additional steps I should take.

Thanks again for your support.

Kind regards,

Pavel

From: Gaetan De Waele @.*** Sent: Monday, August 19, 2024 10:43 PM To: gdewael/maldi-nn Cc: Luckytek; Author Subject: Re: [gdewael/maldi-nn] Train_clf.py issue (Issue #2)

You are using our code correctly. Could you provide me with your complete environment setup (e.g. the output of pip list), so I can reproduce?

— Reply to this email directly, view it on GitHub https://github.com/gdewael/maldi-nn/issues/2#issuecomment-2297417466 , or unsubscribe https://github.com/notifications/unsubscribe-auth/A7HTH2QYEOWLC4H4YIHVIJDZSJKEPAVCNFSM6AAAAABMWV6NKSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJXGQYTONBWGY . You are receiving this because you authored the thread. https://github.com/notifications/beacon/A7HTH2Q6W6ZOGMUASLN6CDTZSJKEPA5CNFSM6AAAAABMWV6NKSWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTUI57HPU.gif Message ID: @.***>

Package Version Editable project location

absl-py 2.1.0 aext-assistant 4.0.15 aext-assistant-server 4.0.15 aext-core 4.0.15 aext-core-server 4.0.15 aext-panels 4.0.15 aext-panels-server 4.0.15 aext-share-notebook 4.0.15 aext-share-notebook-server 4.0.15 aext-shared 4.0.15 aiobotocore 2.5.0 aiofiles 22.1.0 aiohttp 3.8.5 aioitertools 0.7.1 aiosignal 1.2.0 aiosqlite 0.18.0 alabaster 0.7.12 anaconda-anon-usage 0.4.2 anaconda-catalogs 0.2.0 anaconda-client 1.12.1 anaconda-cloud-auth 0.5.0 anaconda-navigator 2.6.1 anaconda-project 0.11.1 anyio 3.5.0 appdirs 1.4.4 argon2-cffi 21.3.0 argon2-cffi-bindings 21.2.0 arrow 1.2.3 asciitree 0.3.3 astroid 2.14.2 astropy 5.1 asttokens 2.0.5 async-lru 2.0.4 async-timeout 4.0.2 atomicwrites 1.4.0 attrs 23.1.0 Automat 20.2.0 autopep8 1.6.0 Babel 2.11.0 backcall 0.2.0 backports.functools-lru-cache 1.6.4 backports.tempfile 1.0 backports.weakref 1.0.post1 bcrypt 3.2.0 beautifulsoup4 4.12.2 binaryornot 0.4.4 bio-attention 0.1.6 black 0.0 bleach 4.1.0 bokeh 3.2.1 boltons 23.0.0 botocore 1.29.76 Bottleneck 1.3.5 brotlipy 0.7.0 certifi 2024.7.4 cffi 1.15.1 chardet 4.0.0 charset-normalizer 2.0.4 click 8.0.4 cloudpickle 2.2.1 clyent 1.2.2 colorama 0.4.6 colorcet 3.0.1 comm 0.1.2 conda 23.7.4 conda-build 3.26.1 conda-content-trust 0.2.0 conda_index 0.3.0 conda-libmamba-solver 23.7.0 conda-pack 0.6.0 conda-package-handling 2.2.0 conda_package_streaming 0.9.0 conda-repo-cli 1.0.75 conda-token 0.4.0 conda-verify 3.4.2 configmypy 0.1.0 constantly 15.1.0 contourpy 1.0.5 cookiecutter 1.7.3 cryptography 41.0.3 cssselect 1.1.0 cycler 0.11.0 cytoolz 0.12.0 daal4py 2023.1.1 dask 2023.6.0 datasets 2.12.0 datashader 0.15.2 datashape 0.5.4 debugpy 1.6.7 decorator 5.1.1 deepsmiles 1.0.1 defusedxml 0.7.1 diff-match-patch 20200713 dill 0.3.6 distributed 2023.6.0 distro 1.8.0 docker-pycreds 0.4.0 docstring-to-markdown 0.11 docutils 0.18.1 einops 0.8.0 entrypoints 0.4 et-xmlfile 1.1.0 executing 0.8.3 fasteners 0.19 fastjsonschema 2.16.2 filelock 3.9.0 flake8 6.0.0 Flask 2.2.2 fonttools 4.25.0 frozendict 2.4.0 frozenlist 1.3.3 fsspec 2023.4.0 future 0.18.3 gensim 4.3.0 gitdb 4.0.10 GitPython 3.1.37 glob2 0.7 graphviz 0.20.3 greenlet 2.0.1 grpcio 1.65.1 h11 0.14.0 h5py 3.9.0 h5torch 0.2.14 HeapDict 1.0.1 holoviews 1.17.1 html5lib 1.1 httpcore 1.0.2 httpx 0.25.2 huggingface-hub 0.15.1 hvplot 0.8.4 hyperlink 21.0.0 idna 3.4 imagecodecs 2023.1.23 imageio 2.26.0 imagesize 1.4.1 imbalanced-learn 0.10.1 importlib-metadata 6.0.0 incremental 21.3.0 inflection 0.5.1 iniconfig 1.1.1 intake 0.6.8 intervaltree 3.1.0 ipykernel 6.25.0 ipython 8.15.0 ipython-genutils 0.2.0 ipywidgets 8.0.4 isort 5.9.3 itemadapter 0.3.0 itemloaders 1.0.4 itsdangerous 2.0.1 jaraco.classes 3.2.1 jedi 0.18.1 jellyfish 1.0.1 Jinja2 3.1.2 jinja2-time 0.2.0 jmespath 0.10.0 joblib 1.2.0 json5 0.9.6 jsonpatch 1.32 jsonpointer 2.1 jsonschema 4.19.2 jsonschema-specifications 2023.7.1 jupyter 1.0.0 jupyter-cache 0.6.1 jupyter_client 7.4.9 jupyter-console 6.6.3 jupyter_core 5.3.0 jupyter-events 0.10.0 jupyter-lsp 2.2.0 jupyter_server 2.14.1 jupyter_server_fileid 0.9.0 jupyter_server_terminals 0.4.4 jupyter_server_ydoc 0.8.0 jupyter-ydoc 0.2.4 jupyterlab 4.0.11 jupyterlab-pygments 0.1.2 jupyterlab_server 2.25.1 jupyterlab-widgets 3.0.10 kaleido 0.2.1 keyring 23.13.1 kiwisolver 1.4.4 lazy_loader 0.2 lazy-object-proxy 1.6.0 libarchive-c 2.9 libmambapy 1.5.1 lightgbm 4.3.0 lightning 2.3.3 lightning-utilities 0.11.6 linkify-it-py 2.0.0 llvmlite 0.40.0 lmdb 1.4.1 locket 1.0.0 lxml 4.9.3 lz4 4.3.2 maldi-nn 0.2.4 Markdown 3.4.1 markdown-it-py 2.2.0 MarkupSafe 2.1.1 matplotlib 3.7.2 matplotlib-inline 0.1.6 mccabe 0.7.0 mdit-py-plugins 0.3.0 mdurl 0.1.0 menuinst 1.4.19 mistune 0.8.4 mkl-fft 1.3.8 mkl-random 1.2.4 mkl-service 2.4.0 more-itertools 8.12.0 mpmath 1.3.0 msgpack 1.0.3 multidict 6.0.2 multipledispatch 0.6.0 multiprocess 0.70.14 multitasking 0.0.11 munkres 1.1.4 mypy-extensions 1.0.0 myst-nb 0.17.2 myst-parser 0.18.1 navigator-updater 0.4.0 nbclassic 0.5.5 nbclient 0.5.13 nbconvert 6.5.4 nbformat 5.9.2 nest-asyncio 1.5.6 networkx 3.1 neuraloperator 0.3.0 C:\Users\Pavel\neuraloperator nltk 3.8.1 nose 1.3.7 notebook 7.0.8 notebook_shim 0.2.2 numba 0.57.1 numcodecs 0.12.0 numexpr 2.8.4 numpy 1.24.3 numpydoc 1.5.0 openai 1.3.7 opencv-python 4.8.1.78 openpyxl 3.0.10 opt-einsum 3.3.0 overrides 7.4.0 packaging 23.1 pandas 2.0.3 pandocfilters 1.5.0 panel 1.2.3 param 1.13.0 paramiko 2.8.1 parsel 1.6.0 parso 0.8.3 partd 1.4.0 pathlib 1.0.1 pathspec 0.10.3 pathtools 0.1.2 patsy 0.5.3 peewee 3.17.1 pep8 1.7.1 pexpect 4.8.0 pickleshare 0.7.5 pillow 10.3.0 pip 23.2.1 pkce 1.0.3 pkginfo 1.9.6 platformdirs 3.10.0 playsound 1.3.0 plotly 5.9.0 pluggy 1.0.0 ply 3.11 poyo 0.5.0 prometheus-client 0.14.1 prompt-toolkit 3.0.36 Protego 0.1.16 protobuf 4.24.4 psutil 5.9.0 ptyprocess 0.7.0 pure-eval 0.2.2 py-cpuinfo 8.0.0 pyarrow 11.0.0 pyasn1 0.4.8 pyasn1-modules 0.2.8 pycodestyle 2.10.0 pycosat 0.6.4 pycparser 2.21 pyct 0.5.0 pycurl 7.45.2 pydantic 1.10.8 PyDispatcher 2.0.5 pydocstyle 6.3.0 pyerfa 2.0.0 pyflakes 3.0.1 pygame 2.5.2 Pygments 2.15.1 PyJWT 2.4.0 pylint 2.16.2 pylint-venv 2.3.0 pyls-spyder 0.4.0 PyNaCl 1.5.0 pyodbc 4.0.34 pyOpenSSL 23.2.0 pyparsing 3.0.9 PyQt5 5.15.10 PyQt5-sip 12.13.0 PyQtWebEngine 5.15.6 pyrsistent 0.18.0 PySocks 1.7.1 pytest 7.4.0 pytest-mock 3.11.1 python-dateutil 2.8.2 python-dotenv 0.21.0 python-json-logger 2.0.7 python-lsp-black 1.2.1 python-lsp-jsonrpc 1.0.0 python-lsp-server 1.7.2 python-slugify 5.0.2 python-snappy 0.6.1 pytoolconfig 1.2.5 pytorch-lightning 2.3.3 pytz 2023.3.post1 pyviz-comms 2.3.0 PyWavelets 1.4.1 pywin32 305.1 pywin32-ctypes 0.2.0 pywinpty 2.0.10 PyYAML 6.0 pyzmq 25.1.2 QDarkStyle 3.0.2 qstylizer 0.2.2 QtAwesome 1.2.2 qtconsole 5.4.2 QtPy 2.2.0 queuelib 1.5.0 rdkit 2024.3.3 referencing 0.30.2 regex 2022.7.9 requests 2.31.0 requests-file 1.5.1 requests-toolbelt 1.0.0 responses 0.13.3 rfc3339-validator 0.1.4 rfc3986-validator 0.1.1 rope 1.7.0 rpds-py 0.10.6 Rtree 1.0.1 ruamel.yaml 0.17.21 ruamel-yaml-conda 0.17.21 s3fs 2023.4.0 safetensors 0.3.2 scikit-image 0.20.0 scikit-learn 1.3.0 scikit-learn-intelex 20230426.121932 scipy 1.11.1 Scrapy 2.8.0 seaborn 0.12.2 selfies 2.1.2 semver 3.0.2 Send2Trash 1.8.2 sentry-sdk 1.31.0 service-identity 18.1.0 setproctitle 1.3.3 setuptools 68.0.0 sip 6.7.12 six 1.16.0 smart-open 5.2.1 smmap 5.0.1 sniffio 1.2.0 snowballstemmer 2.2.0 sortedcontainers 2.4.0 soupsieve 2.4 Sphinx 5.0.2 sphinx-gallery 0.14.0 sphinxcontrib-applehelp 1.0.2 sphinxcontrib-devhelp 1.0.2 sphinxcontrib-htmlhelp 2.0.0 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-qthelp 1.0.3 sphinxcontrib-serializinghtml 1.1.5 spyder 5.4.3 spyder-kernels 2.4.4 SQLAlchemy 1.4.39 stack-data 0.2.0 statsmodels 0.14.0 sympy 1.11.1 tables 3.8.0 tabulate 0.8.10 TBB 0.2 tblib 1.7.0 tenacity 8.2.2 tensorboard 2.17.0 tensorboard-data-server 0.7.2 tensorly 0.8.1 tensorly-sphinx-theme 0.2.3 tensorly-torch 0.4.0 terminado 0.17.1 text-unidecode 1.3 textdistance 4.2.1 threadpoolctl 2.2.0 three-merge 0.1.1 tifffile 2023.4.12 tinycss2 1.2.1 tldextract 3.2.0 tokenizers 0.13.2 toml 0.10.2 tomlkit 0.11.1 toolz 0.12.0 torch 2.1.0 torch-harmonics 0.6.2 torchmetrics 1.4.0.post0 tornado 6.3.2 tqdm 4.65.0 traitlets 5.7.1 transformers 4.32.1 Twisted 22.10.0 twisted-iocpsupport 1.0.2 typing_extensions 4.7.1 tzdata 2023.3 uc-micro-py 1.0.1 ujson 5.4.0 Unidecode 1.2.0 urllib3 1.26.16 w3lib 1.21.0 wandb 0.15.12 watchdog 2.1.6 wcwidth 0.2.5 webencodings 0.5.1 websocket-client 1.8.0 Werkzeug 2.2.3 whatthepatch 1.0.2 wheel 0.38.4 widgetsnbextension 4.0.5 win-inet-pton 1.1.0 wrapt 1.14.1 xarray 2023.6.0 xgboost 2.1.0 xlwings 0.29.1 xxhash 2.0.2 xyzservices 2022.9.0 y-py 0.5.9 yapf 0.31.0 yarl 1.8.1 yfinance 0.2.36 ypy-websocket 0.8.2 zarr 2.16.1 zict 2.2.0 zipp 3.11.0 zope.interface 5.4.0 zstandard 0.19.0

Luckytek commented 2 months ago

It seems the code ran without immediate errors (see bellow).

When I add the print(f["central"].attrs) to the code the output or attributes is visible.

PS C:\Users\Pavel\maldi-nn\maldi_nn> c:; cd 'c:\Users\Pavel\maldi-nn\maldi_nn';

& 'c:\Users\Pavel\Anaconda3\python.exe' 'c:\Users\Pavel.vscode\extensions\ms-python.debugpy-2024.10.0-win32-x64\bundled\libs\debugpy\adapter/../..\debugpy\launcher' '59313' '--' 'C:\Users\Pavel\gaetan_code.py'

{'dtypes': array(['int32', 'int64'], dtype=object), 'filled_to': 11055, 'mode': 'N-D', 'shape': array([11055])}

From: Gaetan De Waele @.*** Sent: Monday, August 19, 2024 10:55 PM To: gdewael/maldi-nn Cc: Luckytek; Author Subject: Re: [gdewael/maldi-nn] Train_clf.py issue (Issue #2)

In addition, can you run this piece of code without error?

import h5torch

f = h5torch.File("F:/DRIAMS_ROOT/RKIpeaks.h5").to_dict() f["central"].attrs

— Reply to this email directly, view it on GitHub https://github.com/gdewael/maldi-nn/issues/2#issuecomment-2297439171 , or unsubscribe https://github.com/notifications/unsubscribe-auth/A7HTH2QWME7PLTM7IWAEJ73ZSJLRZAVCNFSM6AAAAABMWV6NKSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJXGQZTSMJXGE . You are receiving this because you authored the thread. https://github.com/notifications/beacon/A7HTH2SPFAAU7DOIHFR43OTZSJLRZA5CNFSM6AAAAABMWV6NKSWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTUI6AR4G.gif Message ID: @.***>

gdewael commented 2 months ago

Then I must admit I'm quite stumped, as it is clear that -- unlike the error claims -- f["central"] as AttrArray does have an attrs attribute.

Do you encounter the same issue running the classifier on binned data + MLP? I.e.: python train_clf.py F:/DRIAMS_ROOT/RKIbin.h5 logs/ mlp M --devices "cpu" --lr 0.0005

Luckytek commented 2 months ago

When I run python train_clf.py F:/DRIAMS_ROOT/RKIbin.h5 logs/ mlp M --devices "cpu" --lr 0.0005

I am getting KeyError: "Unable to open object (object 'mz' doesn't exist)" as follows:

PS C:\Users\Pavel\maldi-nn\maldi_nn\scripts> python train_clf.py F:/DRIAMS_ROOT/RKIbin.h5 logs/ mlp M --devices "cpu" --lr 0.0005 Group 0 mode: vlen Central dataset mode: N-D Unstructured group mode: N-D Intensity dataset mode: N-D Traceback (most recent call last): File "C:\Users\Pavel\maldi-nn\maldi_nn\scripts\train_clf.py", line 292, in main() File "C:\Users\Pavel\maldi-nn\maldi_nn\scripts\train_clf.py", line 147, in main check_and_log_attributes(f) File "C:\Users\Pavel\maldi-nn\maldi_nn\scripts\train_clf.py", line 56, in check_and_log_attributes mz_mode = safe_get_attr(f["0/mz"], "mode", "vlen") ~^^^^^^^^ File "h5py_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "C:\Users\Pavel\anaconda3\Lib\site-packages\h5py_hl\group.py", line 357, in getitem oid = h5o.open(self.id, self._e(name), lapl=self._lapl) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "h5py_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py\h5o.pyx", line 190, in h5py.h5o.open KeyError: "Unable to open object (object 'mz' doesn't exist)"

When I modify the process_RKI.py script so that the mz dataset is placed under the 0 group in the RKIbin.h5 file as follows:

def RKI_raw_to_binned(rawfile, processed_file):

binner = SequentialPreprocessor(

    VarStabilizer(method="sqrt"),

    Smoother(halfwindow=10),

    BaselineCorrecter(method="SNIP", snip_n_iter=20),

    Trimmer(),

    Binner(),

    Normalizer(sum=1),

)

shutil.copy(rawfile, processed_file)

f = h5torch.File(processed_file, "a")

if '0/mz' not in f or '0/intensity' not in f:

    print("Datasets '0/mz' or '0/intensity' not found in the file. Aborting.")

    print(f"Available datasets in the file: {list(f.keys())}")

    f.close()

    return

len_ = f["0/mz"].shape[0]

ints = []

mzs = None  # Initialize mzs variable

for i in range(len_):

    mz = f["0/mz"][i]

    intensity = f["0/intensity"][i]

    s = SpectrumObject(mz=mz, intensity=intensity)

    processed_s = binner(s)

    ints.append(processed_s.intensity)

    if mzs is None:

        mzs = processed_s.mz  # Save the mz values once

    if (i + 1) % 1000 == 0:

        print(i, end=" ", flush=True)

# Delete the old datasets

del f["0/mz"]

del f["0/intensity"]

# Register the new intensity dataset under group "0"

f.register(np.stack(ints), 0, name="intensity", dtype_save="float32")

f["0/intensity"].attrs["mode"] = "N-D"

# Register the mz dataset under group "0"

f.register(mzs, 0, name="mz", dtype_save="float32")

f["0/mz"].attrs["mode"] = "N-D"

f.close()

return None """

I am getting the ValueError: the number of rows in the given data does not equal the number of elements in the central data object along its alignment axis like this J

PS C:\Users\Pavel\maldi-nn\maldi_nn\scripts> python process_RKI.py F:\RKI_ROOT\ F:\DRIAMS_ROOT\RKIraw.h5 F:\DRIAMS_ROOT\RKIbin.h5 F:\DRIAMS_ROOT\RKIpeaks.h5

Species: 11055, Locations: 11055, Splits: 11055

999 1999 2999 3999 4999 5999 6999 7999 8999 9999 10999 Traceback (most recent call last):

File "C:\Users\Pavel\maldi-nn\maldi_nn\scripts\process_RKI.py", line 237, in

main()

File "C:\Users\Pavel\maldi-nn\maldi_nn\scripts\process_RKI.py", line 233, in main

RKI_raw_to_binned(spectraraw, spectrabin)

File "C:\Users\Pavel\maldi-nn\maldi_nn\scripts\process_RKI.py", line 171, in RKI_raw_to_binned f.register(mzs, 0, name="mz", dtype_save="float32")

raise ValueError(

ValueError: the number of rows in the given data does not equal the number of elements in the

central data object along its alignment axis.

From: Gaetan De Waele @.*** Sent: Tuesday, August 20, 2024 9:42 AM To: gdewael/maldi-nn Cc: Luckytek; Author Subject: Re: [gdewael/maldi-nn] Train_clf.py issue (Issue #2)

Then I must admit I'm quite stumped, as it is clear that -- unlike the error claims -- f["central"] as AttrArray does have an attrs attribute.

Do you encounter the same issue running the classifier on binned data + MLP? I.e.: python train_clf.py F:/DRIAMS_ROOT/RKIbin.h5 logs/ mlp M --devices "cpu" --lr 0.0005

— Reply to this email directly, view it on GitHub https://github.com/gdewael/maldi-nn/issues/2#issuecomment-2298182766 , or unsubscribe https://github.com/notifications/unsubscribe-auth/A7HTH2VNIGFHH3L2FHP7UTDZSLXKZAVCNFSM6AAAAABMWV6NKSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJYGE4DENZWGY . You are receiving this because you authored the thread. https://github.com/notifications/beacon/A7HTH2QOYYB4RPITVL6P2LTZSLXKZA5CNFSM6AAAAABMWV6NKSWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTUI7N6G4.gif Message ID: @.***>

gdewael commented 2 months ago

I can not reproduce any of your errors. Normally, process_RKI should not be changed in order to make the scripts work, as I've tested out everything in a fresh conda environment.

Could you be so kind to try and install maldi-nn in a fresh new environment and try again?

Also, normally, you should be able to run process_RKI and train_clf as is (i.e. without preceding it by python and without including the .py. The pip install includes these scripts as executables in your PATH. This will make sure you're executing the right code.

Thanks.

Luckytek commented 2 months ago

Thank you for your guidance. I will follow your suggestion and set up a fresh conda environment to reinstall maldi-nn and run the scripts again.

I’ll let you know how it goes.

From: Gaetan De Waele @.*** Sent: Tuesday, August 20, 2024 1:28 PM To: gdewael/maldi-nn Cc: Luckytek; Author Subject: Re: [gdewael/maldi-nn] Train_clf.py issue (Issue #2)

I can not reproduce any of your errors. Normally, process_RKI should not be changed in order to make the scripts work, as I've tested out everything in a fresh conda environment.

Could you be so kind to try and install maldi-nn in a fresh new environment and try again?

Also, normally, you should be able to run process_RKI and train_clf as is (i.e. without preceding it by python and without including the .py. The pip install includes these scripts as executables in your PATH. This will make sure you're executing the right code.

Thanks.

— Reply to this email directly, view it on GitHub https://github.com/gdewael/maldi-nn/issues/2#issuecomment-2298626366 , or unsubscribe https://github.com/notifications/unsubscribe-auth/A7HTH2URXLJVMMC7FCPXCXTZSMR4JAVCNFSM6AAAAABMWV6NKSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJYGYZDMMZWGY . You are receiving this because you authored the thread. https://github.com/notifications/beacon/A7HTH2US2CVVP5DJ7ZQHENTZSMR4JA5CNFSM6AAAAABMWV6NKSWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTUJAJAT4.gif Message ID: @.***>

Luckytek commented 2 months ago

Hello Gaetan,

I wanted to update you on the progress I’ve made with the train_clf.

Fresh Installations

I have performed a fresh installation of Anaconda and set up the maldi-nn environment from scratch as you suggested (see the attached base_environment_packages list and maldi_nn_environment_packages list).

Current Issue

Despite these fresh installations, I am still encountering the same issue when trying to access attributes in the central dataset within an HDF5 file. Specifically:

When running the following script:

import h5torch

f = h5torch.File("F:/DRIAMS_ROOT/RKIpeaks.h5").to_dict()

print(f["central"].attrs)

I get the expected output with the attributes listed correctly:

{'dtypes': array(['int32', 'int32'], dtype=object), 'filled_to': np.int32(11055), 'mode': 'N-D', 'shape': array([11055], dtype=int32)}

However, when I run train_clf I get AttributeError: 'AttrArray' object has no attribute 'attrs' as follows:

Exception has occurred: AttributeError

Caught AttributeError in DataLoader worker process 0. Original Traceback (most recent call last): File "c:\Users\Pavel\anaconda3\envs\maldi-nn\Lib\site-packages\torch\utils\data_utils\worker.py", line 309, in _worker_loop data = fetcher.fetch(index) # type: ignore[possibly-undefined] ^^^^^^^^^^^^^^^^^^^^ File "c:\Users\Pavel\anaconda3\envs\maldi-nn\Lib\site-packages\torch\utils\data_utils\fetch.py", line 52, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] ~~~~^^^^^ File "c:\Users\Pavel\anaconda3\envs\maldi-nn\Lib\site-packages\h5torch\dataset.py", line 133, in getitem if self.f["central"].attrs["mode"] == "N-D": ^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'AttrArray' object has no attribute 'attrs'

File "C:\Users\Pavel\train_clf.py", line 267, in main trainer.fit(model, dm.train_dataloader(), dm.val_dataloader()) File "C:\Users\Pavel\train_clf.py", line 292, in main() AttributeError: Caught AttributeError in DataLoader worker process 0. Original Traceback (most recent call last): File "c:\Users\Pavel\anaconda3\envs\maldi-nn\Lib\site-packages\torch\utils\data_utils\worker.py", line 309, in _worker_loop data = fetcher.fetch(index) # type: ignore[possibly-undefined] ^^^^^^^^^^^^^^^^^^^^ File "c:\Users\Pavel\anaconda3\envs\maldi-nn\Lib\site-packages\torch\utils\data_utils\fetch.py", line 52, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] ~~~~^^^^^ File "c:\Users\Pavel\anaconda3\envs\maldi-nn\Lib\site-packages\h5torch\dataset.py", line 133, in getitem if self.f["central"].attrs["mode"] == "N-D": ^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'AttrArray' object has no attribute 'attrs'

When I integrate similar code to to access attributes into the dataset.py file within h5torch, the script fails to print these attributes, and instead, I receive an output indicating that the attributes do not exist or are inaccessible:

print(f"self.f['central']: {self.f['central']}")

print(f"type(self.f['central']): {type(self.f['central'])}")

print(self.f["central"].attrs)

The output only shows:

self.f['central']: [ 0 0 0 ... 269 269 269]

type(self.f['central']): <class 'h5torch.file.AttrArray'>

Attributes do not exist or are not accessible.

Given that the attributes are correctly accessible when using a simple script but not when accessed through the dataset.py module, I’m unsure where the disconnect might be happening.

Could you please provide further guidance on how to resolve this issue? Given the fresh installation, I’m hoping there might be a specific aspect of the h5torch package or the dataset interaction that I’m missing.

Thank you in advance for your help.

Pavel

From: @. @. Sent: Tuesday, August 20, 2024 1:47 PM To: 'gdewael/maldi-nn'; 'gdewael/maldi-nn' Cc: 'Author' Subject: RE: [gdewael/maldi-nn] Train_clf.py issue (Issue #2)

Thank you for your guidance. I will follow your suggestion and set up a fresh conda environment to reinstall maldi-nn and run the scripts again.

I’ll let you know how it goes.

From: Gaetan De Waele @.*** Sent: Tuesday, August 20, 2024 1:28 PM To: gdewael/maldi-nn Cc: Luckytek; Author Subject: Re: [gdewael/maldi-nn] Train_clf.py issue (Issue #2)

I can not reproduce any of your errors. Normally, process_RKI should not be changed in order to make the scripts work, as I've tested out everything in a fresh conda environment.

Could you be so kind to try and install maldi-nn in a fresh new environment and try again?

Also, normally, you should be able to run process_RKI and train_clf as is (i.e. without preceding it by python and without including the .py. The pip install includes these scripts as executables in your PATH. This will make sure you're executing the right code.

Thanks.

— Reply to this email directly, view it on GitHub https://github.com/gdewael/maldi-nn/issues/2#issuecomment-2298626366 , or unsubscribe https://github.com/notifications/unsubscribe-auth/A7HTH2URXLJVMMC7FCPXCXTZSMR4JAVCNFSM6AAAAABMWV6NKSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJYGYZDMMZWGY . You are receiving this because you authored the thread. https://github.com/notifications/beacon/A7HTH2US2CVVP5DJ7ZQHENTZSMR4JA5CNFSM6AAAAABMWV6NKSWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTUJAJAT4.gif Message ID: @.***>

# packages in environment at C:\Users\Pavel\anaconda3: #

# packages in environment at C:\Users\Pavel\anaconda3\envs\maldi-nn: #

Name Version Build Channel

absl-py 2.1.0 pypi_0 pypi aiohappyeyeballs 2.4.0 pypi_0 pypi aiohttp 3.10.5 pypi_0 pypi aiosignal 1.3.1 pypi_0 pypi attrs 24.2.0 pypi_0 pypi bio-attention 0.1.7 pypi_0 pypi blas 1.0 mkl
brotli-python 1.0.9 py312hd77b12b_8
bzip2 1.0.8 h2bbff1b_6
ca-certificates 2024.7.2 haa95532_0
certifi 2024.7.4 py312haa95532_0
charset-normalizer 3.3.2 pyhd3eb1b0_0
colorama 0.4.6 pypi_0 pypi contourpy 1.2.1 pypi_0 pypi cuda-cccl 12.4.127 0 nvidia cuda-cudart 11.8.89 0 nvidia cuda-cudart-dev 11.8.89 0 nvidia cuda-cupti 11.8.87 0 nvidia cuda-libraries 11.8.0 0 nvidia cuda-libraries-dev 11.8.0 0 nvidia cuda-nvrtc 11.8.89 0 nvidia cuda-nvrtc-dev 11.8.89 0 nvidia cuda-nvtx 11.8.86 0 nvidia cuda-profiler-api 12.4.127 0 nvidia cuda-runtime 11.8.0 0 nvidia cycler 0.12.1 pypi_0 pypi deepsmiles 1.0.1 pypi_0 pypi einops 0.8.0 pypi_0 pypi expat 2.6.2 hd77b12b_0
filelock 3.15.4 pypi_0 pypi fonttools 4.53.1 pypi_0 pypi freetype 2.12.1 ha860e81_0
frozenlist 1.4.1 pypi_0 pypi fsspec 2024.6.1 pypi_0 pypi grpcio 1.66.0 pypi_0 pypi h5py 3.11.0 pypi_0 pypi h5torch 0.2.14 pypi_0 pypi idna 3.8 pypi_0 pypi intel-openmp 2023.1.0 h59b6b97_46320
jinja2 3.1.4 py312haa95532_0
joblib 1.4.2 pypi_0 pypi jpeg 9e h827c3e9_3
kiwisolver 1.4.5 pypi_0 pypi lcms2 2.12 h83e58a3_0
lerc 3.0 hd77b12b_0
libcublas 11.11.3.6 0 nvidia libcublas-dev 11.11.3.6 0 nvidia libcufft 10.9.0.58 0 nvidia libcufft-dev 10.9.0.58 0 nvidia libcurand 10.3.5.147 0 nvidia libcurand-dev 10.3.5.147 0 nvidia libcusolver 11.4.1.48 0 nvidia libcusolver-dev 11.4.1.48 0 nvidia libcusparse 11.7.5.86 0 nvidia libcusparse-dev 11.7.5.86 0 nvidia libdeflate 1.17 h2bbff1b_1
libffi 3.4.4 hd77b12b_1
libjpeg-turbo 2.0.0 h196d8e1_0
libnpp 11.8.0.86 0 nvidia libnpp-dev 11.8.0.86 0 nvidia libnvjpeg 11.9.0.86 0 nvidia libnvjpeg-dev 11.9.0.86 0 nvidia libpng 1.6.39 h8cc25b3_0
libtiff 4.5.1 hd77b12b_0
libuv 1.48.0 h827c3e9_0
libwebp-base 1.3.2 h2bbff1b_0
lightning 2.4.0 pypi_0 pypi lightning-utilities 0.11.6 pypi_0 pypi lz4-c 1.9.4 h2bbff1b_1
maldi-nn 0.2.4 pypi_0 pypi markdown 3.7 pypi_0 pypi markupsafe 2.1.5 pypi_0 pypi matplotlib 3.9.2 pypi_0 pypi mkl 2023.1.0 h6b88ed4_46358
mkl-service 2.4.0 py312h2bbff1b_1
mkl_fft 1.3.8 py312h2bbff1b_0
mkl_random 1.2.4 py312h59b6b97_0
mpmath 1.3.0 py312haa95532_0
multidict 6.0.5 pypi_0 pypi networkx 3.3 py312haa95532_0
numpy 2.1.0 pypi_0 pypi openjpeg 2.5.2 hae555c5_0
openssl 3.0.14 h827c3e9_0
packaging 24.1 pypi_0 pypi pandas 2.2.2 pypi_0 pypi pillow 10.4.0 py312h827c3e9_0
pip 24.2 py312haa95532_0
protobuf 5.27.3 pypi_0 pypi pyparsing 3.1.4 pypi_0 pypi pysocks 1.7.1 py312haa95532_0
python 3.12.4 h14ffc60_1
python-dateutil 2.9.0.post0 pypi_0 pypi pytorch 2.4.0 py3.12_cuda11.8_cudnn9_0 pytorch pytorch-cuda 11.8 h24eeafa_5 pytorch pytorch-lightning 2.4.0 pypi_0 pypi pytorch-mutex 1.0 cuda pytorch pytz 2024.1 pypi_0 pypi pyyaml 6.0.2 pypi_0 pypi rdkit 2024.3.5 pypi_0 pypi requests 2.32.3 py312haa95532_0
scikit-learn 1.5.1 pypi_0 pypi scipy 1.14.1 pypi_0 pypi selfies 2.1.2 pypi_0 pypi setuptools 72.1.0 py312haa95532_0
six 1.16.0 pypi_0 pypi sqlite 3.45.3 h2bbff1b_0
sympy 1.13.2 pypi_0 pypi tbb 2021.8.0 h59b6b97_0
tensorboard 2.17.1 pypi_0 pypi tensorboard-data-server 0.7.2 pypi_0 pypi threadpoolctl 3.5.0 pypi_0 pypi tk 8.6.14 h0416ee5_0
torch 2.4.0 pypi_0 pypi torchaudio 2.4.0 pypi_0 pypi torchmetrics 1.4.1 pypi_0 pypi torchvision 0.19.0 pypi_0 pypi tqdm 4.66.5 pypi_0 pypi typing-extensions 4.12.2 pypi_0 pypi typing_extensions 4.11.0 py312haa95532_0
tzdata 2024.1 pypi_0 pypi urllib3 2.2.2 py312haa95532_0
vc 14.40 h2eaa2aa_0
vs2015_runtime 14.40.33807 h98bb1dd_0
werkzeug 3.0.4 pypi_0 pypi wheel 0.43.0 py312haa95532_0
win_inet_pton 1.1.0 py312haa95532_0
xgboost 2.1.1 pypi_0 pypi xz 5.4.6 h8cc25b3_1
yaml 0.2.5 he774522_0
yarl 1.9.4 pypi_0 pypi zlib 1.2.13 h8cc25b3_1
zstd 1.5.5 hd43e919_2

gdewael commented 2 months ago

Hi Pavel,

You set up the tests correctly: one script says that "in-memory loaded h5torch objects" (represented by a nested dict of AttrArrays) have the .attrs attribute, while the other does not. I have no clue what's going wrong as I also cannot reproduce your issues.

As a possible work-around, I have pushed a new release of maldi-nn (0.2.5) which allows you to set --data_in_memory False for the train_clf script. Can you check if updating pip install -U maldi-nn and running train_clf with this flag as False resolves your issue?

Luckytek commented 2 months ago

Hi Gaetan,Thank you very much, I will try your solution and I am back to you.Kind Regards,Pavel Sent from Android deviceDne 29. 8. 2024 15:35 napsal uživatel Gaetan De Waele @.***>: Hi Pavel, You set up the tests correctly: one script says that "in-memory loaded h5torch objects" (represented by a nested dict of AttrArrays) have the .attrs attribute, while the other does not. I have no clue what's going wrong as I also cannot reproduce your issues. As a possible work-around, I have pushed a new release of maldi-nn (0.2.5) which allows you to set --data_in_memory False for the train_clf script. Can you check if updating pip install -U maldi-nn and running train_clf with this flag as False resolves your issue?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

gdewael / maldi-nn

Train_clf.py issue #2

Name Version Build Channel

Name Version Build Channel