search

IsoNet-cryoET / IsoNet

Self-supervised learning for isotropic cryoET reconstruction
https://www.nature.com/articles/s41467-022-33957-8
MIT License
67 stars 12 forks source link

Issue with Keras / function mae #59

Open jmcdonal opened 1 month ago

jmcdonal commented 1 month ago

Hi, I'm trying to help some users use the isonet.py program. We are running on a Rocky Linux 9 platform (RHEL linux 9 clone). The user is trying to run a refine job and we're seeing errors like the ones below. In a naive google search looking at the issue it seems that some of the loss functions need to be serialized unless I should be using a different package version (below is the full list of packages in the conda environment). Please advise, Jeff

$ isonet.py refine ./subtomo.star --gpuID 0 --preprocessing_ncpus 16 --iterations 4 --noise_start_iter 10,15,20,25 --noise_level 0.05,0.1,0.15,0.2 07-10 15:44:31, INFO

Isonet starts refining

07-10 15:44:34, INFO Start Iteration1! 07-10 15:44:34, WARNING The results folder already exists
The old results folder will be renamed (to results~) /panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/layers/activations/leaky_relu.py:41: UserWarning: Argument alpha is deprecated. Use negative_slope instead. warnings.warn( 07-10 15:44:36, WARNING You are saving your model as an HDF5 file via model.save() or keras.saving.save_model(model). This file format is considered legacy. We recommend using instead the native Keras format, e.g. model.save('my_model.keras') or keras.saving.save_model(model, 'my_model.keras'). 07-10 15:44:39, INFO Noise Level:0.0 07-10 15:45:22, INFO Done preparing subtomograms! 07-10 15:45:22, INFO Start training! 07-10 15:45:23, ERROR Traceback (most recent call last): File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/IsoNet/bin/refine.py", line 128, in run history = train_data(args) #train based on init model and save new one as model_iter{num_iter}.h5 ^^^^^^^^^^^^^^^^ File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/IsoNet/models/unet/train.py", line 93, in train_data history = train3D_continue('{}/model_iter{:0>2d}.h5'.format(settings.result_dir,settings.iter_count), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/IsoNet/models/unet/train.py", line 38, in train3D_continue model = load_model( model_file) ^^^^^^^^^^^^^^^^^^^^^^^ File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/saving/saving_api.py", line 189, in load_model return legacy_h5_format.load_model_from_hdf5( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/legacy/saving/legacy_h5_format.py", line 155, in load_model_from_hdf5 **saving_utils.compile_args_from_training_config( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/legacy/saving/saving_utils.py", line 143, in compile_args_from_training_config loss = _deserialize_nested_config(losses.deserialize, loss_config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/legacy/saving/saving_utils.py", line 202, in _deserialize_nested_config return deserialize_fn(config) ^^^^^^^^^^^^^^^^^^^^^^ File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/losses/init.py", line 149, in deserialize return serialization_lib.deserialize_keras_object( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/saving/serialization_lib.py", line 575, in deserialize_keras_object return deserialize_keras_object( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/saving/serialization_lib.py", line 678, in deserialize_keras_object return _retrieve_class_or_fn( ^^^^^^^^^^^^^^^^^^^^^^ File "/panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo/lib/python3.12/site-packages/keras/src/saving/serialization_lib.py", line 812, in _retrieve_class_or_fn raise TypeError( TypeError: Could not locate function 'mae'. Make sure custom classes are decorated with @keras.saving.register_keras_serializable(). Full object config: {'module': 'keras.metrics', 'class_name': 'function', 'config': 'mae', 'registered_name': 'mae'}

$ conda list

packages in environment at /panfs/hisoftware/rocky9/miniconda3/py39_23.1.0/envs/tomo:

#

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge absl-py 2.1.0 pypi_0 pypi astunparse 1.6.3 pypi_0 pypi bzip2 1.0.8 hd590300_5 conda-forge ca-certificates 2024.6.2 hbcca054_0 conda-forge certifi 2024.6.2 pypi_0 pypi charset-normalizer 3.3.2 pypi_0 pypi fire 0.6.0 pypi_0 pypi flatbuffers 24.3.25 pypi_0 pypi gast 0.6.0 pypi_0 pypi google-pasta 0.2.0 pypi_0 pypi grpcio 1.64.1 pypi_0 pypi h5py 3.11.0 pypi_0 pypi idna 3.7 pypi_0 pypi imageio 2.34.2 pypi_0 pypi keras 3.4.1 pypi_0 pypi lazy-loader 0.4 pypi_0 pypi ld_impl_linux-64 2.40 hf3520f5_7 conda-forge libclang 18.1.1 pypi_0 pypi libexpat 2.6.2 h59595ed_0 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libgcc-ng 14.1.0 h77fa898_0 conda-forge libgomp 14.1.0 h77fa898_0 conda-forge libnsl 2.0.1 hd590300_0 conda-forge libsqlite 3.46.0 hde9e2c9_0 conda-forge libuuid 2.38.1 h0b41bf4_0 conda-forge libxcrypt 4.4.36 hd590300_1 conda-forge libzlib 1.3.1 h4ab18f5_1 conda-forge markdown 3.6 pypi_0 pypi markdown-it-py 3.0.0 pypi_0 pypi markupsafe 2.1.5 pypi_0 pypi mdurl 0.1.2 pypi_0 pypi ml-dtypes 0.3.2 pypi_0 pypi mrcfile 1.5.0 pypi_0 pypi namex 0.0.8 pypi_0 pypi ncurses 6.5 h59595ed_0 conda-forge networkx 3.3 pypi_0 pypi numpy 1.26.4 pypi_0 pypi openssl 3.3.1 h4ab18f5_1 conda-forge opt-einsum 3.3.0 pypi_0 pypi optree 0.11.0 pypi_0 pypi packaging 24.1 pypi_0 pypi pillow 10.4.0 pypi_0 pypi pip 24.0 pyhd8ed1ab_0 conda-forge protobuf 4.25.3 pypi_0 pypi pygments 2.18.0 pypi_0 pypi pyqt5 5.15.10 pypi_0 pypi pyqt5-qt5 5.15.14 pypi_0 pypi pyqt5-sip 12.13.0 pypi_0 pypi python 3.12.4 h194c7f8_0_cpython conda-forge readline 8.2 h8228510_1 conda-forge requests 2.32.3 pypi_0 pypi rich 13.7.1 pypi_0 pypi scikit-image 0.24.0 pypi_0 pypi scipy 1.14.0 pypi_0 pypi setuptools 70.1.1 pyhd8ed1ab_0 conda-forge six 1.16.0 pypi_0 pypi tensorboard 2.16.2 pypi_0 pypi tensorboard-data-server 0.7.2 pypi_0 pypi tensorflow 2.16.1 pypi_0 pypi termcolor 2.4.0 pypi_0 pypi tifffile 2024.6.18 pypi_0 pypi tk 8.6.13 noxft_h4845f30_101 conda-forge tqdm 4.66.4 pypi_0 pypi typing-extensions 4.12.2 pypi_0 pypi tzdata 2024a h0c530f3_0 conda-forge urllib3 2.2.2 pypi_0 pypi werkzeug 3.0.3 pypi_0 pypi wheel 0.43.0 pyhd8ed1ab_1 conda-forge wrapt 1.16.0 pypi_0 pypi xz 5.2.6 h166bdaf_0 conda-forge

procyontao commented 1 month ago

Hi,

Thanks for reporting. I have been using tensorflow2.5 to test everything. If you can roll back to a older version, that would be great.

We are working on reproducing your issue and update IsoNet so that it can matches latest versions of dependencies.