search

Quasars / orange-spectroscopy

Other
51 stars 58 forks source link

HyperSpectra: handle degenerate (all nan) coordinates and data #687

Closed markotoplak closed 11 months ago

markotoplak commented 11 months ago

Fixes #686

Handle (and also test) degenerate inputs.

686 described errors that happened for the SNR widget because the widget is not meant to be used on normal data (without data point repetitions). HyperSpectra now does not crash but in the (before crashing) cases does not do anything useful either.

codecov[bot] commented 11 months ago

Codecov Report

Merging #687 (c602791) into master (5cdfb1a) will increase coverage by 0.01%. Report is 4 commits behind head on master. The diff coverage is 100.00%.

:exclamation: Current head c602791 differs from pull request most recent head dc9d6b7. Consider uploading reports for the commit dc9d6b7 to get more accurate results

@@            Coverage Diff             @@
##           master     #687      +/-   ##
==========================================
+ Coverage   89.35%   89.37%   +0.01%     
==========================================
  Files          72       72              
  Lines       12019    12032      +13     
==========================================
+ Hits        10740    10754      +14     
+ Misses       1279     1278       -1
borondics commented 11 months ago

I still get an error with the current master.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN" "http://www.w3.org/TR/REC-html40/strict.dtd">

Exception: | TypeError: 'NoneType' object is not subscriptable -- | -- Module: | orangecontrib.spectroscopy.widgets.owhyper:925 Version: | 3.37.0.dev Environment: | Python 3.11.0 on Darwin 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul 5 22:21:56 PDT 2023; root:xnu-8796.141.3~6/RELEASE_X86_64 x86_64 Installed Packages: | AnyQt==0.2.0, Babel==2.12.1, Bottleneck==1.3.7, CacheControl==0.13.0, Deprecated==1.2.14, Flask-Cors==3.0.10, Flask==2.3.2, Genesis-PyAPI==1.2.1, Jinja2==3.1.2, Markdown==3.4.3, MarkupSafe==2.1.2, Orange-Spectroscopy==0.6.10, Orange3-Bioinformatics==4.8.0, Orange3==3.37.0.dev0+536040c, Pebble==5.0.3, Pillow==9.5.0, PyQt5-sip==12.11.0, PyQt5==5.15.7, PySocks==1.7.1, PyWavelets==1.4.1, PyYAML==6.0, Pygments==2.15.1, QtPy==2.3.1, Send2Trash==1.8.0, Werkzeug==2.3.6, XlsxWriter==3.1.2, absl-py==1.4.0, aiodns==3.0.0, aiohttp-retry==2.8.3, aiohttp==3.8.5, aiosignal==1.3.1, alabaster==0.7.13, aleph-alpha-client==2.17.0, anthropic==0.2.10, anyio==3.6.2, appnope==0.1.3, argon2-cffi-bindings==21.2.0, argon2-cffi==21.3.0, arrow==1.2.3, asteval==0.9.29, asttokens==2.2.1, astunparse==1.6.3, async-timeout==4.0.2, attrs==22.2.0, awscrt==0.16.26, backcall==0.2.0, backports.functools-lru-cache==1.6.4, baycomp==1.0.2, beautifulsoup4==4.12.2, bibtexparser==1.4.0, bleach==6.0.0, blinker==1.6.2, bokeh==3.1.1, boto3-stubs==1.28.47, boto3==1.28.47, botocore-stubs==1.31.47, botocore==1.31.47, brotlipy==0.7.0, cachetools==5.3.1, catboost==1.2.1.1, cattrs==23.1.2, certifi==2022.12.7, cffi==1.15.1, chardet==5.1.0, charset-normalizer==3.1.0, click==8.1.3, cloudpickle==2.2.1, colorama==0.4.6, colorcet==3.0.1, comm==0.1.3, commonmark==0.9.1, contourpy==1.0.7, cryptography==40.0.2, cycler==0.11.0, cytoolz==0.12.0, dask==2023.4.0, debugpy==1.6.7, decorator==5.1.1, defusedxml==0.7.1, dictdiffer==0.9.0, docutils==0.18.1, entrypoints==0.4, et-xmlfile==1.1.0, exceptiongroup==1.1.2, executing==1.2.0, extranormal3==0.0.3, fabio==2023.6.0, fake-useragent==1.1.3, fastjsonschema==2.16.3, filelock==3.12.0, flatbuffers==23.5.26, flit-core==3.8.0, fonttools==4.39.3, free-proxy==1.1.1, frozenlist==1.4.0, fsspec==2023.4.0, future==0.18.3, gast==0.4.0, google-auth-oauthlib==1.0.0, google-auth==2.20.0, google-pasta==0.2.0, graphviz==0.20.1, grpcio==1.54.2, gwyfile==0.2.0, h11==0.14.0, h5py==3.8.0, httpcore==0.17.2, httpx==0.24.1, huggingface-hub==0.13.4, idna==3.4, imagecodecs==2023.1.23, imageio==2.27.0, imagesize==1.4.1, importlib-metadata==6.5.0, importlib-resources==5.12.0, ipykernel==6.22.0, ipython-genutils==0.2.0, ipython==8.12.0, ipywidgets==8.0.6, itsdangerous==2.1.2, jaraco.classes==3.2.3, jax==0.4.12, jedi==0.18.2, jmespath==1.0.1, joblib==1.2.0, jsonschema==4.17.3, jupyter-client==8.2.0, jupyter-console==6.6.3, jupyter-core==5.3.0, jupyter-events==0.6.3, jupyter-server-terminals==0.4.4, jupyter-server==2.5.0, jupyter==1.0.0, jupyterlab-pygments==0.2.2, jupyterlab-widgets==3.0.7, keras==2.12.0, keyring==23.13.1, keyrings.alt==4.2.0, kiwisolver==1.4.4, lazy-loader==0.2, libclang==16.0.0, llvmlite==0.40.1rc1, lmfit==1.2.1, locket==1.0.0, lxml==4.9.3, matplotlib-inline==0.1.6, matplotlib==3.7.1, mecom==0.1, mistune==2.0.5, ml-dtypes==0.2.0, more-itertools==9.1.0, mpmath==1.3.0, msgpack==1.0.5, multidict==6.0.4, munkres==1.1.4, mypy-boto3-s3==1.28.36, nbclassic==0.5.5, nbclient==0.7.3, nbconvert==7.3.1, nbformat==5.8.0, nest-asyncio==1.5.6, networkx==3.1, notebook-shim==0.2.2, notebook==6.5.4, numba==0.57.0, numpy==1.23.5, oauthlib==3.2.2, openTSNE==1.0.0, openai==0.27.8, opencv-python==4.8.0.74, openplayground==0.1.5, openpyxl==3.1.2, opt-einsum==3.3.0, orange-canvas-core==0.1.33.dev0, orange-widget-base==4.23.0.dev0+fd1fb70, outcome==1.2.0, packaging==23.1, pandas==2.0.2, pandocfilters==1.5.0, param==1.13.0, parso==0.8.3, partd==1.4.0, pexpect==4.8.0, pickleshare==0.7.5, pip==23.1, pkgutil-resolve-name==1.3.10, platformdirs==3.2.0, plotly==5.14.1, ply==3.11, point-annotator==2.0.0, pooch==1.7.0, prettytable==3.8.0, prometheus-client==0.16.0, prompt-toolkit==3.0.38, protobuf==4.23.3, psutil==5.9.5, ptyprocess==0.7.0, pure-eval==0.2.2, pyOpenSSL==23.1.1, pyasn1-modules==0.3.0, pyasn1==0.5.0, pycares==4.3.0, pyclipper==1.3.0.post5, pycparser==2.21, pyct==0.5.0, pydub==0.25.1, pyparsing==3.0.9, pyqtgraph==0.13.3, pyrsistent==0.19.3, python-dateutil==2.8.2, python-dotenv==1.0.0, python-json-logger==2.0.7, python-louvain==0.16, pythoncrc==1.21, pytz==2023.3, pyzmq==25.0.2, qasync==0.24.0, qtconsole==5.4.2, regex==2023.6.3, renishawWiRE==0.1.16, requests-cache==1.1.0, requests-oauthlib==1.3.1, requests==2.28.2, resdk==19.0.1, rfc3339-validator==0.1.4, rfc3986-validator==0.1.1, rsa==4.9, s3transfer==0.6.2, scapy==2.5.0, scholarly==1.7.11, scikit-image==0.20.0, scikit-learn==1.1.3, scipy==1.10.1, selenium==4.10.0, serverfiles==0.3.1, setuptools==67.6.1, silx==1.1.2, single-sample-gsea==0.2.0, sip==6.7.9, six==1.16.0, slumber==0.7.1, sniffio==1.3.0, snowballstemmer==2.2.0, sortedcontainers==2.4.0, soupsieve==2.3.2.post1, spectral==0.23.1, sphinx-rtd-theme==1.2.2, sphinx==6.2.1, sphinxcontrib-applehelp==1.0.4, sphinxcontrib-devhelp==1.0.2, sphinxcontrib-htmlhelp==2.0.1, sphinxcontrib-jquery==4.1, sphinxcontrib-jsmath==1.0.1, sphinxcontrib-qthelp==1.0.3, sphinxcontrib-serializinghtml==1.1.5, sseclient==0.0.27, stack-data==0.6.2, sympy==1.12, tenacity==8.2.2, tensorboard-data-server==0.7.1, tensorboard==2.12.3, tensorflow-estimator==2.12.0, tensorflow-io-gcs-filesystem==0.32.0, tensorflow==2.12.0, termcolor==2.3.0, terminado==0.17.1, threadpoolctl==3.1.0, tifffile==2023.4.12, tinycss2==1.2.1, tmm==0.1.8, tokenizers==0.13.3, toml==0.10.2, tomli==2.0.1, toolz==0.12.0, torch==2.0.1, tornado==6.3, tqdm==4.65.0, traitlets==5.9.0, transformers==4.29.0, trio-websocket==0.10.3, trio==0.22.2, types-awscrt==0.19.1, types-s3transfer==0.6.2, typing-extensions==4.5.0, tzdata==2023.3, tzlocal==5.0.1, uncertainties==3.1.7, url-normalize==1.4.3, urllib3==1.26.15, wcwidth==0.2.6, webencodings==0.5.1, websocket-client==1.5.1, wheel==0.40.0, widgetsnbextension==4.0.7, wrapt==1.14.1, wsproto==1.2.0, xgboost==2.0.0, xlrd==2.0.1, xyzservices==2023.5.0, yarl==1.9.2, zipp==3.15.0 Machine ID: | 73436674-0c73-4b39-b24f-5aec33812e78 Stack Trace: | Traceback (most recent call last):  File "/Users/borondics/Documents/code/python/orange3/Orange/widgets/utils/concurrent.py", line 547, in _on_task_done    self.on_exception(ex)  File "/Users/borondics/Documents/code/python/orange-spectroscopy/orangecontrib/spectroscopy/widgets/owhyper.py", line 994, in on_exception    raise ex  File "/Users/borondics/miniconda3/envs/master/lib/python3.11/concurrent/futures/thread.py", line 58, in run    result = self.fn(*self.args, **self.kwargs)             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/Users/borondics/Documents/code/python/orange-spectroscopy/orangecontrib/spectroscopy/widgets/owhyper.py", line 925, in compute_image    if lsx[-1] * lsy[-1] > IMAGE_TOO_BIG:       ~~~^^^^TypeError: 'NoneType' object is not subscriptable Local Variables: | OrderedDict([('Result',              .Result'>),             ('attr_x',              DiscreteVariable(name='type', values=('DNA', 'collagen', 'glycogen', 'lipids'))),             ('attr_y',              DiscreteVariable(name='type', values=('DNA', 'collagen', 'glycogen', 'lipids'))),             ('data',              [[5.12271, 5.11292, 5.10893, 5.04445, 5.00195, ... \| ?]]),             ('extract_col',              .extract_col at 0x1785d1d00>),             ('image_values',              Integrate(methods=, limits=[[902.5606, 1801.264]])),             ('image_values_fixed_levels', None),             ('lsx', None),             ('lsy', None),             ('progress_interrupt',              .progress_interrupt at 0x1785ba660>),             ('res',              .Result object at 0x15934a0d0>),             ('state',              ),             ('xat',              DiscreteVariable(name='type', values=('DNA', 'collagen', 'glycogen', 'lipids'))),             ('yat',              DiscreteVariable(name='type', values=('DNA', 'collagen', 'glycogen', 'lipids')))])
borondics commented 11 months ago

The workflow was image

using the "liver spectroscopy dataset"

markotoplak commented 11 months ago

@borondics, this was not merged yet. Could you check if it works with this branch?

markotoplak commented 11 months ago

Ok, I am just merging this into master, and test there. :)