3dem / relion

Image-processing software for cryo-electron microscopy
https://relion.readthedocs.io/en/latest/
GNU General Public License v2.0
456 stars 203 forks source link

relion 5 pick tomograms: RuntimeError: no validator found for <class 'numpy.ndarray'>, see `arbitrary_types_allowed` in Config #1194

Open RichStone99 opened 1 month ago

RichStone99 commented 1 month ago

Installed current relion 5 on centOS 9. Am trying the subtomogram averaging tutorial with the HIV VLP data set. Worked well up to "Reconstruct tomograms". Tomograms have been generated and look fine. But when trying to launch "Pick tomograms", napari didn't open and got the following error message. How can we fix it? Seems to relate to pydantic and/or tomography_python_programs. Btw, napari ran well during, for example, the "Exclude tilt-images" step and inspection of generated tomo in the "Reconstruct tomograms" step. IMOD was used in "Reconstruct tomograms".

Traceback (most recent call last): File "", line 1, in File "/home/tangl/sw/miniforge3/envs/relion-5.0/lib/python3.10/site-packages/tomography_python_programs/pick/init.py", line 1, in from .particles import pick_particles File "/home/tangl/sw/miniforge3/envs/relion-5.0/lib/python3.10/site-packages/tomography_python_programs/pick/particles.py", line 7, in from napari_threedee.annotators import PointAnnotator File "/home/tangl/sw/miniforge3/envs/relion-5.0/lib/python3.10/site-packages/napari_threedee/init.py", line 1, in from . import data_models File "/home/tangl/sw/miniforge3/envs/relion-5.0/lib/python3.10/site-packages/napari_threedee/data_models/init.py", line 1, in from .points import N3dPoints File "/home/tangl/sw/miniforge3/envs/relion-5.0/lib/python3.10/site-packages/napari_threedee/data_models/points.py", line 8, in from napari_threedee.annotators.base import N3dDataModel File "/home/tangl/sw/miniforge3/envs/relion-5.0/lib/python3.10/site-packages/napari_threedee/annotators/init.py", line 5, in from .surfaces import SurfaceAnnotator File "/home/tangl/sw/miniforge3/envs/relion-5.0/lib/python3.10/site-packages/napari_threedee/annotators/surfaces/init.py", line 1, in from .annotator import SurfaceAnnotator File "/home/tangl/sw/miniforge3/envs/relion-5.0/lib/python3.10/site-packages/napari_threedee/annotators/surfaces/annotator.py", line 12, in from morphosamplers.surface_spline import GriddedSplineSurface File "/home/tangl/sw/miniforge3/envs/relion-5.0/lib/python3.10/site-packages/morphosamplers/init.py", line 12, in from morphosamplers.models import Path, Sphere, Surface, Dipole File "/home/tangl/sw/miniforge3/envs/relion-5.0/lib/python3.10/site-packages/morphosamplers/models/init.py", line 1, in from .path import Path File "/home/tangl/sw/miniforge3/envs/relion-5.0/lib/python3.10/site-packages/morphosamplers/models/path.py", line 7, in class Path(MorphoModel): File "pydantic/main.py", line 197, in pydantic.main.ModelMetaclass.new File "pydantic/fields.py", line 504, in pydantic.fields.ModelField.infer File "pydantic/fields.py", line 434, in pydantic.fields.ModelField.init File "pydantic/fields.py", line 555, in pydantic.fields.ModelField.prepare File "pydantic/fields.py", line 829, in pydantic.fields.ModelField.populate_validators File "pydantic/validators.py", line 765, in find_validators RuntimeError: no validator found for <class 'numpy.ndarray'>, see arbitrary_types_allowed in Config

labzw commented 4 weeks ago

Having exactly the same error. I feel this might be a bug in morphosamplers. It seems natively Pydantic does not support ndarray.

Solved or bypassed this issue by adding

class Config:
        arbitrary_types_allowed = True

in dipole.py path.py sphere.py surface.py files under /YOUR/CONDA/PATH/envs/relion-5.0/lib/python3.10/site-packages/morphosamplers/models

alisterburt commented 4 weeks ago

@RichStone99 @labzw apologies for this - could you please run pip show morphosamplers and pip show pydantic in your RELION 5 conda environment and tell me the output? We recently moved to only supporting pydantic>=2 which may be the root cause of this issue for you

CACNYC commented 4 weeks ago

@alisterburt I have an identical error to @RichStone99 @labzw : Traceback (most recent call last): File "", , line 1, in < module» File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/tomography_python_programs/pick/_init_py", line 1, in from particles import pick_particles File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/tomography_python_programs/pick/particles.py", line 7, in from napari_threedee.annotators import PointAnnotator File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari_threedee/init.py", line 1, in from. import data models File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari_threedee/data_models/_init_py", line 1, in from points import N3dPoints File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari_threedee/data_models/points.py", line 8, in from napari threedee.annotators.base import N3dDataModel File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari_threedee/annotators/_init.py", line 5, in from surfaces import SurfaceAnnotator File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari_threedee/annotators/surfaces/_init_py", line 1, in < module> from annotator import SurfaceAnnotator File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari_threedee/annotators/surfaces/annotator.py", line 12, in from morphosamplers.surface_spline import GriddedSplineSurface File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/morphosamplers/_init_py", line 12, in from morphosamplers.models import Path, Sphere, Surface, Dipole File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/morphosamplers/models/_init_py", line 1, in from path import Path File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/morphosamplers/models/path.py", line 7, in class Path(MorphoModel): File "pydantic/main.py", line 197, in pydantic.main.ModelMetaclass.new File "pydantic/fields.py", line 504, in pydantic.fields.ModelField.infer File "pydantic/fields.py", line 434, in pydantic.fields.ModelField.__init File "pydantic/fields.py", line 555, in pydantic.fields.ModelField prepare File "pydantic/fields.py", line 829, in pydantic.fields.ModelField.populate_validators File "pydantic/validators.py", line 765, in find_validators RuntimeError: no validator found for <class 'numpy.ndarray >, see 'arbitrary_types_allowed* in Config

Output of pip show morphosamplers: (relion-5.0) [spuser@spgpu TS01 projecti$ pip show morphosamplers Name: morphosamplers Version: 0.0.15 Summary: A library for sampling image data along morphological objects such as splines and surfaces. Home-page: https: //github.com/kevinyamauchi/morphosamplers Author: Kevin Yamauchi Author-email: kevin.yamauchi@gmail.com License: BSD 3-Clause License Location: /opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages Requires: einops, numpy, psygnal, pydantic, pydantic-compat, scipy, typing-extensions •Required-by: napari-threedee, relion

Output of pip show pydantic: (relion-5.0) [spuser@spgpu TS_01 projecti$ pip show pydantic -Name: pydantic Version: 1.10.18 Summary: Data validation and settings management using python type hints Home-page: https: //github.com/pydantic/pydantic rAuthor: Samuel Colvin l Author-email: s@muelcolvin.com License: MIT Location: /opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages Requires: typing-extensions Required-by: app-model, mdocfile, morphosamplers, napari, napari-threedee, npez, pydantic-compat

Ah, then pydantic 1.x is the culprit?

Thanks for your help Alister!

CACNYC commented 4 weeks ago

Hmm, it looks like I have a napari version constraint with napari 0.4.18

[spuser@spgpu TS_01_project]$ pip install --upgrade pydantic Defaulting to user installation because normal site-packages is not writeable Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com Requirement already satisfied: pydantic in /opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages (1.10.18) Collecting pydantic Downloading pydantic-2.9.2-py3-none-any.whl.metadata (149 kB) Collecting annotated-types>=0.6.0 (from pydantic) Downloading annotated_types-0.7.0-py-none-any. whl.metadata (15 kB) Collecting pydantic-core==2.23.4 (from pydantic) Downloading pydantic_core-2.23.4-cp310-c310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB) Requirement already satisfied: typing-extensions>=4.6.1 in /opt/miniconda3/envs/relion-5.0/Lib/python3.10/site-packages (from pydantic) (4.12.2) Downloading pydantic-2.9.2-py3-none-any.whl (434 kB) DownLoading pydantic_core-2.23,4-cp310-cp310-manyLinux_2_17_x86_64.manylinux2014_x86_64.wh1 (2.1 MB) Downloading annotated_types-0.7.0-py3-none-any.whl (13 kB) Installing collected packages: pydantic-core, annotated-types, pydantic ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. napari 0.4.18 requires pydantic<2,>=1.9.0, but you have pydantic 2.9.2 which is incompatible. Successfully installed annotated-types-0.7.0 pydantic-2.9.2 pydantic-core-2.23.4

In relion5: Traceback (most recent call last): File "", line 1, in ‹module> File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/tomography_python_programs/pick/_init_py". line 1, in from particles import pick_particles File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/tomography_python_programs/pick/particles.py", line 7, in from napari threedee.annotators import PointAnnotator File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari_threedee/_init_py", line 1, in from. import data models File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari_threedee/data_models/_init_py", line 1, in < module> from points import N3dPoints File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari_threedee/data_models/points.py", line 3, in import napari.layers File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari/layers/init.py", line 9, in < module > from napari.layers.base import Layer File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari/layers/base/_init_py", line 1, in from napari.layers.base._base_constants import ActionType File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari/layers/base/_base_constants.py", line 4, in < module> from napari.utils.misc import StringEnum File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari/utils/_init_py", line 2, in from napari.utils.colormaps import Colormap File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari/utils/colormaps/_init-py", line 2, in from napari.utils.co/ormaps.colormap import Colormap File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari/utils/colormaps/colormap.py", line 9, in from napari.utils.events import EventedModel File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari/utils/events/_init_py", line 17, in from napari.utils.events.evented_model import EventedModel File "/opt/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari/utils/events/evented_model.py", line 66, in class EventedMetaclass(main.ModelMetaclass): File "/home/spuser/.local/lib/python3.10/site-packages/pydantic/_migration.py", line 306, in wrapper raise AttributeError(f'module {module!r} has no attribute {name!r}") AttributeError: module 'pydantic.main' has no attribute "ModelMetaclass"

bogdantoader commented 3 weeks ago

Hello, downgrading to the following versions in the conda environment should fix the issue:

morphosamplers.version '0.0.13' pydantic.version '1.10.18' napari.version '0.4.18'

CACNYC commented 3 weeks ago

Hello, downgrading to the following versions in the conda environment should fix the issue:

morphosamplers.version '0.0.13' pydantic.version '1.10.18' napari.version '0.4.18'

Thanks Bogdan, I can confirm that rolling back morphosamplers to version 0.0.13, combined with pydantic 1.10.18 and napari 0.4.18, allowed both the Exclude tilt and Pick tomograms job types to run and complete successfully.

labzw commented 1 week ago

Update:

Remove and recreate conda environment with latest environment.yml works!

So after I rebuild the latest version and roll back morphosamplers to 0.0.13, I am getting this error now.......

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/bin/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/tomography_python_programs/exclude_tilt_images/__init__.py", line 1, in <module>
    from ._cli import cli
  File "/usr/bin/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/tomography_python_programs/exclude_tilt_images/_cli.py", line 8, in <module>
    from .relion_tilt_image_excluder import RelionTiltImageExcluderWidget
  File "/usr/bin/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/tomography_python_programs/exclude_tilt_images/relion_tilt_image_excluder.py", line 5, in <module>
    from .._qt.tilt_image_excluder import TiltImageExcluderWidget
  File "/usr/bin/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/tomography_python_programs/_qt/tilt_image_excluder.py", line 8, in <module>
    from .components.tilt_image_selection_widget import TiltImageSelectionWidget
  File "/usr/bin/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/tomography_python_programs/_qt/components/tilt_image_selection_widget.py", line 12, in <module>
    class TiltImageSelectionWidget(QWidget):
  File "/usr/bin/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/tomography_python_programs/_qt/components/tilt_image_selection_widget.py", line 17, in TiltImageSelectionWidget
    viewer: napari.Viewer,
  File "/usr/bin/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari/_lazy.py", line 45, in __getattr__
    submod = import_module(
  File "/usr/bin/miniconda3/envs/relion-5.0/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/bin/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari/viewer.py", line 8, in <module>
    from napari.components.viewer_model import ViewerModel
  File "/usr/bin/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari/components/__init__.py", line 19, in <module>
    from napari.components.layerlist import LayerList
  File "/usr/bin/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari/components/layerlist.py", line 9, in <module>
    from napari.layers import Layer
  File "/usr/bin/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari/layers/__init__.py", line 12, in <module>
    from napari.layers.points import Points
  File "/usr/bin/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari/layers/points/__init__.py", line 1, in <module>
    from napari.layers.points import _points_key_bindings
  File "/usr/bin/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari/layers/points/_points_key_bindings.py", line 4, in <module>
    from napari.layers.points.points import Points
  File "/usr/bin/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari/layers/points/points.py", line 38, in <module>
    from napari.layers.utils.text_manager import TextManager
  File "/usr/bin/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari/layers/utils/text_manager.py", line 29, in <module>
    class TextManager(EventedModel):
  File "/usr/bin/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari/layers/utils/text_manager.py", line 83, in TextManager
    string: StringEncoding = ConstantStringEncoding(constant='')
  File "/usr/bin/miniconda3/envs/relion-5.0/lib/python3.10/site-packages/napari/utils/events/evented_model.py", line 242, in __init__
    super().__init__(**kwargs)
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for ConstantStringEncoding
constant
  Unable to avoid copy while creating an array as requested.
If using `np.array(obj, copy=False)` replace it with `np.asarray(obj)` to allow a copy when needed (no behavior change in NumPy 1.x).
For more details, see https://numpy.org/devdocs/numpy_2_0_migration_guide.html#adapting-to-changes-in-the-copy-keyword. (type=value_error)