search

YunyunGao / AUSPEX

Newly forged AUSPEX by THORNLAB
Other
1 stars 1 forks source link

Test file failing #1

Open erikssod opened 3 weeks ago

erikssod commented 3 weeks ago

auspex fails when executed with the suggested test file:

COMMAND LINE: auspex test/4puc.mtz
mtz
Traceback (most recent call last):
  File "/home/de/miniconda3/envs/auspex/bin/auspex", line 8, in <module>
    sys.exit(run())
             ^^^^^
  File "/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/auspex/__init__.py", line 12, in run
    import Parser
  File "/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/auspex/Parser.py", line 217, in <module>
    ice_info.run_helcaraxe()
  File "/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/auspex/Auspex.py", line 111, in run_helcaraxe
    = cnn_predict(ires, iobs,
      ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/auspex/Helcaraxe.py", line 46, in cnn_predict
    model_iobs = keras.models.load_model(os.path.join(current_path, "Helcaraxe_models/final_models/Helcaraxe_Iobs_model"))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/keras/src/saving/saving_api.py", line 206, in load_model
    raise ValueError(
ValueError: File format not supported: filepath=/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/auspex/Helcaraxe_models/final_models/Helcaraxe_Iobs_model. Keras 3 only supports V3 `.keras` files and legacy H5 format files (`.h5` extension). Note that the legacy SavedModel format is not supported by `load_model()` in Keras 3. In order to reload a TensorFlow SavedModel as an inference-only layer in Keras 3, use `keras.layers.TFSMLayer(/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/auspex/Helcaraxe_models/final_models/Helcaraxe_Iobs_model, call_endpoint='serving_default')` (note that your `call_endpoint` might have a different name).

My environment is:

>uname -a
Linux MX3-DEV-DE2 6.5.0-41-generic #41~22.04.2-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun  3 11:32:55 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

>conda info
     active environment : auspex
    active env location : /home/de/miniconda3/envs/auspex
            shell level : 2
       user config file : /home/de/.condarc
 populated config files : 
          conda version : 24.5.0
    conda-build version : not installed
         python version : 3.12.4.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=zen2
                          __conda=24.5.0=0
                          __glibc=2.35=0
                          __linux=6.5.0=0
                          __unix=0=0
       base environment : /home/de/miniconda3  (writable)
      conda av data dir : /home/de/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/de/miniconda3/pkgs
                          /home/de/.conda/pkgs
       envs directories : /home/de/miniconda3/envs
                          /home/de/.conda/envs
               platform : linux-64
             user-agent : conda/24.5.0 requests/2.32.2 CPython/3.12.4 Linux/6.5.0-41-generic ubuntu/22.04.4 glibc/2.35 solver/libmamba conda-libmamba-solver/24.1.0 libmambapy/1.5.8 aau/0.4.4 c/. s/. e/.
                UID:GID : 1002:1002
             netrc file : None
           offline mode : False
YunyunGao commented 3 weeks ago

Hi Daniel, thank you for reporting the bug. It seems in your case keras 3+ was automatically installed when you built the conda environment. Instead of asking you to downgrade keras, I have updated the code and models so it should work for both keras 2 and 3. The test has past on my building env. Could you pull the new commit and install the package then test again?

erikssod commented 3 weeks ago

Thanks, that worked!

Might be unrelated, but I also sent a job to the Hamburg server and it seems to be stuck in the queue.

erikssod commented 3 weeks ago

Found a few more issues using the same test data:

COMMAND LINE: auspex ../test/4puc.mtz --score-figure
mtz
/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/keras/src/optimizers/base_optimizer.py:86: UserWarning: Argument `decay` is no longer supported and will be ignored.
  warnings.warn(
WARNING:absl:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.
WARNING:absl:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.
_______________________________________________________________________________

                          QUANTITATIVE ICE RING SCORE                          

The severity of ice ring contamination at the corresponding resolution ranges.
from 0.0 (no ice ring) to 1.0 (significant ice ring or truncation)

| reosolution (Ang)   | score   |
|---------------------|---------|
| 3.95-3.81           | 1.0     |
| 3.75-3.58           | 1.0     |
| 3.48-3.37           | 1.0     |
| 2.68-2.64           | 1.0     |
| 2.29-2.21           | 0.0     |
_______________________________________________________________________________

                                GENERATING PLOTS:                              
Set of plots is generated with 103607 intensities.
834.8807
23.141397
8.429095
Traceback (most recent call last):
  File "/home/de/miniconda3/envs/auspex/bin/auspex", line 8, in <module>
    sys.exit(run())
             ^^^^^
  File "/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/auspex/__init__.py", line 12, in run
    import Parser
  File "/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/auspex/Parser.py", line 297, in <module>
    plot.generate(ice_info, nemo_info_F, nemo_info_I)
  File "/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/auspex/Plotter.py", line 666, in generate
    self.generate_standardised_mean_plot(iobs, reso_data, ax=ax1)
  File "/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/auspex/Plotter.py", line 438, in generate_standardised_mean_plot
    for idx in range(self.icefinder_handle.Size()):

and 

COMMAND LINE: auspex ../test/4puc.mtz --disable_helcaraxe
mtz
Traceback (most recent call last):
  File "/home/de/miniconda3/envs/auspex/bin/auspex", line 8, in <module>
    sys.exit(run())
             ^^^^^
  File "/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/auspex/__init__.py", line 12, in run
    import Parser
  File "/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/auspex/Parser.py", line 280, in <module>
    report_ice_ring(ice_info.quantitative_score(), ice_info.max_ires())
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/auspex/Auspex.py", line 189, in quantitative_score
    icefinder_scores_in_ice = self.icefinder_scores()[self._args_ice_by_icefinderscore]
                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'IceFinder' object has no attribute '_args_ice_by_icefinderscore'. Did you mean: 'ice_range_by_icefinderscore'?
YunyunGao commented 3 weeks ago

Thank you for the testing.

I conclude the errors were mainly due to the migration ramification from 3.7 to 3.11 and partly because disabling helcaraxe is deprecated (hence a bit ignored) as the CNN model has better performance.
After some hotfixes, these options should work as intended now.

In regards to the Hamburg server, I am currently responsible to the development as well. Our redis server dead without any stack trace. It is a know issue which happened a few times before. Now the queuing system should work again. Please try to re-submit your task.

erikssod commented 3 weeks ago

Still seeing errors after pulling latest version (2.3.1)

COMMAND LINE: auspex ../test/4puc.HKL --score-figure
xds_hkl
#####################################################################################################
#####################################   Intensity Statistics   ######################################
#####################################################################################################
  Resolution    #Data   %Complete    Redundancy      <I>    <I/s>    cc1/2    Rmerge     Rpim    Rmeas
------------  -------  -----------  ------------  ------  -------  -------  --------  -------  -------
 1.80 - 1.74     9264     42.4          1.14        5.57     0.24   0.2070    1.2727   1.2040   1.7554
 1.84 - 1.80     9304     59.6          1.19        7.44     0.33   0.2726    1.0383   0.9603   1.4179
 1.89 - 1.84     9520     70.3          1.24       10.84     0.47   0.4167    0.8280   0.7547   1.1235
 1.94 - 1.89     9464     73.1          1.28       -4.66    -0.17   0.9252   -1.1735  -1.0927  -1.6073
 1.98 - 1.94     9339     88.4          1.48       18.95     0.79   0.6051    0.6849   0.6180   0.9253
 2.03 - 1.98     9311     95.7          1.74       20.94     0.88   0.6297    0.5769   0.5327   0.7873
 2.08 - 2.03     9431     95.9          1.80       29.25     1.21   0.6925    0.4974   0.4603   0.6795
 2.13 - 2.08     9246     97.2          1.80       29.26     1.23   0.7865    0.4358   0.4049   0.5964
 2.19 - 2.13     9464     96.8          1.79       39.87     1.68   0.8228    0.3296   0.3069   0.4514
 2.28 - 2.19     9585     81.0          1.67       83.73     2.68   0.9767    0.5016   0.4743   0.6917
 2.36 - 2.28     9274     94.2          1.72       51.83     2.19   0.8869    0.2532   0.2325   0.3447
 2.45 - 2.36     9588     97.7          1.83       60.91     2.75   0.9297    0.1983   0.1837   0.2710
 2.56 - 2.45     9460     97.3          1.82       74.06     3.40   0.9507    0.1616   0.1498   0.2209
 2.69 - 2.56     9205     96.0          1.79       91.94     4.22   0.9659    0.1353   0.1259   0.1853
 2.86 - 2.69     9359     94.2          1.70      115.80     5.37   0.9779    0.1046   0.0967   0.1428
 3.07 - 2.86     9560     97.2          1.84      167.44     7.43   0.9880    0.0774   0.0721   0.1060
 3.37 - 3.07     9475     96.1          1.81      280.73    11.01   0.9939    0.0526   0.0495   0.0724
 3.87 - 3.37     9308     88.3          1.71      525.00    14.80   0.9957    0.0422   0.0400   0.0583
 4.92 - 3.87     9490     92.0          1.82      693.70    17.82   0.9971    0.0306   0.0289   0.0422
 7.34 - 4.92     6494     93.5          1.79      475.22    17.89   0.9972    0.0298   0.0283   0.0412
49.78 - 7.34     2782     93.6          1.77      937.28    20.23   0.9987    0.0176   0.0171   0.0245
------------  -------  -----------  ------------  ------  -------  -------  --------  -------  -------
49.78 - 1.74   187886     83.7          1.65      145.80     5.53   0.9957    0.0830   0.0775   0.1138
/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/keras/src/optimizers/base_optimizer.py:86: UserWarning: Argument `decay` is no longer supported and will be ignored.
  warnings.warn(
WARNING:absl:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.
WARNING:absl:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.
/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/scipy/integrate/_quadpack_py.py:1233: IntegrationWarning: Extremely bad integrand behavior occurs at some points of the
  integration interval.
  quad_r = quad(f, low, high, args=args, full_output=self.full_output,
_______________________________________________________________________________

                          QUANTITATIVE ICE RING SCORE                          

The severity of ice ring contamination at the corresponding resolution ranges.
from 0.0 (no ice ring) to 1.0 (significant ice ring or truncation)

| reosolution (Ang)   | score   |
|---------------------|---------|
| 3.95-3.81           | 1.0     |
| 3.75-3.58           | 1.0     |
| 3.48-3.37           | 1.0     |
| 2.68-2.64           | 1.0     |
| 2.29-2.21           | 0.07    |
| 2.09-2.04           | 1.0     |
| 1.954-1.939         | 1.0     |
| 1.935-1.897         | 0.53    |
| 1.889-1.863         | 0.89    |
_______________________________________________________________________________

                                GENERATING PLOTS:                              
Set of plots is generated with 309546 intensities.
374.87106576686114
10.050754143284198
7.684254588834294
Traceback (most recent call last):
  File "/home/de/miniconda3/envs/auspex/bin/auspex", line 8, in <module>
    sys.exit(run())
             ^^^^^
  File "/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/auspex/__init__.py", line 12, in run
    import Parser
  File "/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/auspex/Parser.py", line 297, in <module>
    plot.generate(ice_info, nemo_info_F, nemo_info_I)
  File "/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/auspex/Plotter.py", line 666, in generate
    self.generate_standardised_mean_plot(iobs, reso_data, ax=ax1)
  File "/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/auspex/Plotter.py", line 438, in generate_standardised_mean_plot
    for idx in range(self.icefinder_handle.Size()):
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'IceFinder' object has no attribute 'Size'

and 

COMMAND LINE: auspex ../test/4puc.mtz --disable_helcaraxe
mtz
Traceback (most recent call last):
  File "/home/de/miniconda3/envs/auspex/bin/auspex", line 8, in <module>
    sys.exit(run())
             ^^^^^
  File "/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/auspex/__init__.py", line 12, in run
    import Parser
  File "/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/auspex/Parser.py", line 280, in <module>
    report_ice_ring(ice_info.quantitative_score(), ice_info.max_ires())
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/de/miniconda3/envs/auspex/lib/python3.11/site-packages/auspex/Auspex.py", line 189, in quantitative_score
    icefinder_scores_in_ice = self.icefinder_scores()[self._args_ice_by_icefinderscore]
                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'IceFinder' object has no attribute '_args_ice_by_icefinderscore'. Did you mean: 'ice_range_by_icefinderscore'?
YunyunGao commented 3 weeks ago

Hi, perhaps you just forgot reinstalling the package? After pulling python install . and try again.

However, I do realized that by enabling score-figure helcaraxe should be automatically disabled because the score figure is basically plotting icefinder scores. Wondering why helcaraxe is superior to icefinder score? Try comparing the results from these three lines: auspex test/4puc.cif auspex test/4puc.cif --score-figure auspex test/4puc.cif --score-figure --cutoff 2.5