Closed sgbaird closed 2 years ago
shyuep
commented
2 years ago The tensorflow warnings can be captured with a catch warnings. But the FIRE: is output from ASE. I am trying to figure out how to control the verbosity, but I can't see an obvious way to do it (beyond supplying a log file, which I don't want to do).
sgbaird
commented
2 years ago For the FIRE:, if nothing on the ASE side, maybe filter out lines that begin with FIRE: and then reset the stdout after the function is called? Not the most elegant, but I verified it and it seems to works as expected:
import re
class Filter(object):
"""https://stackoverflow.com/a/63662744/13697228
For reassigning default, https://stackoverflow.com/a/51340381/13697228"""
def __init__(self, stream, re_pattern):
self.stream = stream
self.pattern = (
re.compile(re_pattern) if isinstance(re_pattern, str) else re_pattern
)
self.triggered = False
def __getattr__(self, attr_name):
return getattr(self.stream, attr_name)
def write(self, data):
if data == "\n" and self.triggered:
self.triggered = False
else:
if self.pattern.search(data) is None:
self.stream.write(data)
self.stream.flush()
else:
# caught bad pattern
self.triggered = True
def flush(self):
self.stream.flush() if self.relax_on_decode:
sys.stdout = Filter(sys.stdout, r"^FIRE:| Step Time Energy fmax|\*Force-consistent energies used in optimization.") # type: ignore # noqa: E501
relaxer = Relaxer() # This loads the default pre-trained model
# relaxation
if self.relax_on_decode:
# restore default https://stackoverflow.com/a/51340381/13697228
sys.stdout = sys.__stdout__2022-06-16 20:35:23.393379: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found 2022-06-16 20:35:23.397911: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform. Skipping registering GPU devices... 2022-06-16 20:35:23.405877: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("PartitionedCall:4", shape=(None,), dtype=int32), values=Tensor("PartitionedCall:3", shape=(None, 3, 3), dtype=float32), dense_shape=Tensor("PartitionedCall:5", shape=(3,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory. C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("PartitionedCall:1", shape=(3024,), dtype=int32), values=Tensor("Neg:0", shape=(3024, 3), dtype=float32), dense_shape=Tensor("PartitionedCall:2", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory. C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/three_d_interaction_2/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/three_d_interaction_2/GatherV2_1_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/three_d_interaction_2/GatherV2_1_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory. C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory. C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_1_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_1_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory. C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_2/concat_atoms_2/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_2/concat_atoms_2/GatherV2_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_2/concat_atoms_2/GatherV2_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory. C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/three_d_interaction_1/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/three_d_interaction_1/GatherV2_1_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/three_d_interaction_1/GatherV2_1_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory. C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory. C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_1_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_1_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory. C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_1/concat_atoms_1/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_1/concat_atoms_1/GatherV2_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_1/concat_atoms_1/GatherV2_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory. C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/concat_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/concat:0", shape=(None,), dtype=float32), dense_shape=Tensor("gradients/m3g_net/three_d_interaction_2/GatherV2_2_grad/Cast:0", shape=(1,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory. C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/GatherV2_5_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/GatherV2_5_grad/Reshape:0", shape=(None,), dtype=float32), dense_shape=Tensor("gradients/m3g_net/GatherV2_5_grad/Cast:0", shape=(1,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory. C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/GatherV2_6_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/GatherV2_6_grad/Reshape:0", shape=(None,), dtype=float32), dense_shape=Tensor("gradients/m3g_net/GatherV2_6_grad/Cast:0", shape=(1,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory. C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/GatherV2_3_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/GatherV2_3_grad/Reshape:0", shape=(None, 3), dtype=float32), dense_shape=Tensor("gradients/m3g_net/GatherV2_3_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory. C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/GatherV2_4_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/GatherV2_4_grad/Reshape:0", shape=(None, 3), dtype=float32), dense_shape=Tensor("gradients/m3g_net/GatherV2_4_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory. C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("PartitionedCall:1", shape=(None,), dtype=int32), values=Tensor("Neg:0", shape=(None, 3), dtype=float32), dense_shape=Tensor("PartitionedCall:2", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory. C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/three_d_interaction/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/three_d_interaction/GatherV2_1_grad/Reshape:0", shape=(None, 9), dtype=float32), dense_shape=Tensor("gradients/m3g_net/three_d_interaction/GatherV2_1_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory. C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_grad/Reshape:0", shape=(None, 64), dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory. C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_1_grad/Reshape:0", shape=(None, 64), dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_1_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory. C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer/concat_atoms/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer/concat_atoms/GatherV2_grad/Reshape:0", shape=(None, 64), dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer/concat_atoms/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
sgbaird
commented
2 years ago For the tensorflow warnings, I went with:
import tensorflow as tf
...
if not self.verbose:
tf.get_logger().setLevel(logging.ERROR)which just leaves:
2022-06-16 20:48:47.442172: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found 2022-06-16 20:48:47.446997: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform. Skipping registering GPU devices... 2022-06-16 20:48:47.460638: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
I'm pretty sure this is just during the one call to Relax() (not the repeated calls to relaxer.relax()), and I'm fine with that.
shyuep
commented
2 years ago I actually found a more elegant solution. Using contextlib.redirect_stdout....
shyuep
commented
2 years ago Yeah for the TF, I will leave them for now since it is relatively easy to silence by the end user if you wish. Tensorflow has a lot of deprecations and warnings and sometimes those are useful, especially for the developer end....
sgbaird
commented
2 years ago @shyuep great! That works for me. The contextlib solution is definitely a lot cleaner for the FIRE lines.
shyuep
commented
2 years ago Ok, I just released v0.06 that should have the verbosity control.
This isn't time-sensitive (I'm grateful it's running!), but I'm planning to loop through many structures, and the default output I get for a single relaxation is: