Error in BGMN refinement on a fresh install.

[JoonatanL]

Hi Nathan, Was trying out the package, and after installing as per instructions, and trying to run the Example, I get the following lengthy error when running run_CNN.py (relevant part bold). Any thoughts? I've tried training a model, and get the same problem (but training itself is fine). I'm wondering if it's some kind of setup/hardware difference error that I'm missing, (in e.g. multiprocessing). Have also checked with elevated privileges.

Joonatan

python run_CNN.py _2024-04-08 23:17:09.515084: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0. 2024-04-08 23:17:10.519070: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0. Running XRD analysis 0%| | 0/8 [00:00<?, ?it/s]WARNING:tensorflow:From C:\Users\joona\anaconda3\envs\xrd\lib\site-packages\keras\src\backend\tensorflow\core.py:187: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. [x8]

WARNING:absl:No training configuration found in the save file, so the model was not compiled. Compile it manually. [x8] 0%| | 0/8 [00:56<?, ?it/s] multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "C:\Users\joona\anaconda3\envs\xrd\lib\site-packages\autoXRD\dara__init.py", line 72, in do_refinement return get_result(control_file_path) File "C:\Users\joona\anaconda3\envs\xrd\lib\site-packages\autoXRD\dara__init__.py", line 492, in get_result "lst_data": parse_lst(lst_path, phase_names), **File "C:\Users\joona\anaconda3\envs\xrd\lib\site-packages\autoXRD\dara\init__.py", line 535, in parse_lst result[var] = float(re.search(rf"{var}=(\d+(.\d+)?)%", texts).group(1)) AttributeError: 'NoneType' object has no attribute 'group**'

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "C:\Users\joona\anaconda3\envs\xrd\lib\multiprocessing\pool.py", line 125, in worker result = (True, func(*args, kwds)) File "C:\Users\joona\anaconda3\envs\xrd\lib\site-packages\autoXRD\spectrum_analysis__init.py", line 736, in classify_mixture mixtures, confidences, backup_mixtures, scalings, spectra = spec_analysis.suspected_mixtures File "C:\Users\joona\anaconda3\envs\xrd\lib\site-packages\autoXRD\spectrum_analysis__init.py", line 99, in suspected_mixtures prediction_list, confidence_list, backup_list, scale_list, spec_list = self.enumerate_routes(spectrum) File "C:\Users\joona\anaconda3\envs\xrd\lib\site-packages\autoXRD\spectrum_analysis\init.py", line 324, in enumerate_routes reduced_spectrum, norm, scaling_constant, is_done = self.get_reduced_pattern(predicted_cmpd, xrd_spectrum, last_normalization=normalization) File "C:\Users\joona\anaconda3\envs\xrd\lib\site-packages\autoXRD\spectrum_analysis\init__.py", line 406, in get_reduced_pattern result = do_refinement_no_saving( File "C:\Users\joona\anaconda3\envs\xrd\lib\site-packages\autoXRD\dara\init.py", line 87, in do_refinement_no_saving return do_refinement( File "C:\Users\joona\anaconda3\envs\xrd\lib\site-packages\autoXRD\dara\init__.py", line 74, in do_refinement raise RuntimeError(f"Error in BGMN refinement for {control_file_path}") from e RuntimeError: Error in BGMN refinement for C:\Users\joona\AppData\Local\Temp\tmp7p1spume\Li2MnO3+MnO+TiO2.sav** """

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "C:\Users\joona\OneDrive\Computer\altrove\code\xrd_analyzer\XRD-AutoAnalyzer-main\Example\run_CNN.py", line 71, in results['XRD']['scale_factors'], results['XRD']['reduced_spectra'] = spectrum_analysis.main('Spectra', 'References', File "C:\Users\joona\anaconda3\envs\xrd\lib\site-packages\autoXRD\spectrum_analysis__init.py", line 881, in main spectrum_names, predicted_phases, confidences, backup_phases, scale_factors, reduced_spectra = phase_id.all_predictions File "C:\Users\joona\anaconda3\envs\xrd\lib\site-packages\autoXRD\spectrum_analysis\init.py", line 702, in all_predictions all_info = list(tqdm(pool.imap(self.classify_mixture, spectrum_filenames), File "C:\Users\joona\anaconda3\envs\xrd\lib\site-packages\tqdm\std.py", line 1181, in iter__ for obj in iterable: File "C:\Users\joona\anaconda3\envs\xrd\lib\multiprocessing\pool.py", line 870, in next raise value RuntimeError: Error in BGMN refinement for C:\Users\joona\AppData\Local\Temp\tmp7p1spume\Li2MnO3+MnO+TiO2.sav

[njszym]

Thanks for bringing this issue up. Can you try upgrading to the latest version of the package? We just pushed some changes to the repo a few days ago (on April 6th) that should resolve this error!

Please let me know if the issue persists after doing that.

[Qais-Ali-5]

Hi Nathan, Thank you for your really nice work. Sorry for a rather lengthy post, but I am also facing similar issues. When I run the trained model in Example folder it works fine. After that I tried to train my own model. 1- Firstly, I tried to train the model using reference phases directly python construct_model.py --inc_pdf --skip_filter I first see following

2024-06-09 13:15:01.572975: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-06-09 13:15:01.575971: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-06-09 13:15:01.617877: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-06-09 13:15:02.377848: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/home/qali/.local/lib/python3.10/site-packages/pymatgen/io/cif.py:1168: UserWarning: Issues encountered while parsing CIF: Some fractional coordinates rounded to ideal values to avoid issues with finite precision.
  warnings.warn("Issues encountered while parsing CIF: " + "\n".join(self.warnings))
/home/qali/.local/lib/python3.10/site-packages/pymatgen/io/cif.py:716: UserWarning: No _symmetry_equiv_pos_as_xyz type key found. Spacegroup from _symmetry_space_group_name_H-M used.
  warnings.warn(msg)
/home/qali/.local/lib/python3.10/site-packages/pymatgen/io/cif.py:1168: UserWarning: Issues encountered while parsing CIF: No _symmetry_equiv_pos_as_xyz type key found. Spacegroup from _symmetry_space_group_name_H-M used.

Here I would like to ask should I install TensorRT libraries to make use of GPU as I can also try to run it on cluster where I do have GPU access and which CUDA drivers are nescessary? 2- Secondly, what about those warnings regarding CIF files, as I took them from COD database will it still be an issue for final predictions? 3- Thirdly, at the end of 50 epochs I see following WARNING:absl:You are saving your model as an HDF5 file viamodel.save()orkeras.saving.save_model(model). This file format is considered legacy. We recommend using instead the native Keras format, e.g.model.save('my_model.keras')orkeras.saving.save_model(model, 'my_model.keras'). `

And it never finishes after this step. I cancelled from keyboard. There exists only Model.h5 file. I was also expected PDF model. 4- After this when tried to run the model for predictions I see following error.

Traceback (most recent call last):
  File "/home/xrd/AutoAnalyzer/analyzer_2/autoXRD/dara/__init__.py", line 72, in do_refinement
    return get_result(control_file_path)
  File "/home/xrd/AutoAnalyzer/analyzer_2/autoXRD/dara/__init__.py", line 492, in get_result
    "lst_data": parse_lst(lst_path, phase_names),
  File "/home/xrd/AutoAnalyzer/analyzer_2/autoXRD/dara/__init__.py", line 535, in parse_lst
    result[var] = float(re.search(rf"{var}=(\d+(\.\d+)?)%", texts).group(1))
AttributeError: 'NoneType' object has no attribute 'group'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/xrd/AutoAnalyzer/analyzer_2/run_CNN.py", line 141, in <module>
    visualizer.main('Spectra', spectrum_fname, phasenames, heights, final_spectrum,
  File "/home/xrd/AutoAnalyzer/analyzer_2/autoXRD/visualizer/__init__.py", line 504, in main
    result = do_refinement_no_saving(
  File "/home/xrd/AutoAnalyzer/analyzer_2/autoXRD/dara/__init__.py", line 87, in do_refinement_no_saving
    return do_refinement(
  File "/home/xrd/AutoAnalyzer/analyzer_2/autoXRD/dara/__init__.py", line 74, in do_refinement
    raise RuntimeError(f"Error in BGMN refinement for {control_file_path}") from e
RuntimeError: Error in BGMN refinement for /tmp/tmpalqtm28t/xrd_data_file.sav

Thank you for your feedback in advnace!

[Qais-Ali-5]

And sometimes for some cif files I do see following warning

home/xrd/lib/python3.11/site-packages/pymatgen/io/cif.py:1222: UserWarning: Incorrect stoichiometry:
  CIF={'Fe': 1.0, 'O': 3.0}
  PMG={'Fe': 2.0, 'O': 3.0}
  ratios={'Fe': 2.0, 'O': 1.0}

Would it be better to change your code such that it prints out the name of exact cif file, which is problematic such that user can make changes exactly where its required? Thanks!

[Qais-Ali-5]

Just little more update. If I do not use the Rietveld refinement option, i.e. to skip --weights and also the --plot flag then the generated Model.h5 is able to make predictions for phases. However, out of 15 samples, for almost half of them it identifies no phase at all and for the ones it predicts phases the confidence is ranging between 40% to 70% with WARNING: some peaks (I ~ [ranging between 30-40]%) were not identified.

[njszym]

Thanks for raising this issue. Would you mind uploading your All_CIFs folder (or your References folder if --skip_filter is being used) and your XRD patterns (the Spectra folder) so that I may help debug?

A couple of notes off the bat: 1) The warnings you see usually are not critical, they occur when site occupancies in a CIF do not sum to 1, but pymatgen can round these values to avoid errors. 2) If your Rietveld refinement installation is not working properly, it will also affect phase identification (even if you turn off --plot and --weights) because refinement is still used to subtract the peaks of known phases in mixtures.