kmdalton
commented
1 year ago Hi @gyuhyeokcho,
This error happens when careless tries to standardize the metadata columns in the input. It will attempt to convert each of the columns into z-scores by subtracting the mean and dividing by the std deviation. If the column has standard deviation of zero, this leads to a divide by zero and the column will be set to NaN. If you remove the column with zero standard deviation from the metadata keys, it should run without issue. You can figure out which column is problematic by looking at rs.mtzdump. Let me know if that doesn't solve your problem.
I agree this is not a very informative error message. I'll look into changing the default behavior to prevent this error for future users.
Thanks, Kevin
gyuhyeokcho
When attempting to run the Careless in Google Colab with data obtained from XDS INTEGRATE (converted to an mtz file using Pointless), I encountered an issue. The command !rs.mtzdump INTEGRATE.mtz seems to be fine, but when I ran Careless using the following command:
!careless mono \ --studentt-likelihood-dof=16 \ --disable-image-scales \ --merge-half-datasets \ --iterations=15_000 \ --wilson-prior-b 65.522 \ "BATCH,dHKL,Hobs,Kobs,Lobs,XDET,YDET,ROT,LP,FLAG,FRACTIONCALC" \ INTEGRATE.mtz \ test/INTEGRATE
I received the following error message:
Careless version 0.2.6 /usr/local/lib/python3.8/dist-packages/careless/io/formatter.py:312: RuntimeWarning: invalid value encountered in true_divide metadata = (metadata - metadata.mean(0)) / metadata.std(0) Training: 0% 1/15000 [00:04<18:08:09, 4.35s/it, loss=nan, F KLDiv=7.82e+02, NLL=nan] Traceback (most recent call last): File "/usr/local/bin/careless", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/careless/careless.py", line 9, in main
run_careless(parser)
File "/usr/local/lib/python3.8/dist-packages/careless/careless.py", line 52, in run_careless
history = model.train_model(
File "/usr/local/lib/python3.8/dist-packages/careless/models/merging/variational.py", line 172, in train_model
_history = train_step((self, data))
File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 54, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: Graph execution error:
Detected at node 'variational_merging_model/TruncatedNormal_CONSTRUCTED_AT_top_level/sample/stateless_parameterized_truncated_normal/StatelessParameterizedTruncatedNormal' defined at (most recent call last): File "/usr/local/bin/careless", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/careless/careless.py", line 9, in main
run_careless(parser)
File "/usr/local/lib/python3.8/dist-packages/careless/careless.py", line 52, in run_careless
history = model.train_model(
File "/usr/local/lib/python3.8/dist-packages/careless/models/merging/variational.py", line 172, in train_model
_history = train_step((self, data))
File "/usr/local/lib/python3.8/dist-packages/careless/models/merging/variational.py", line 159, in train_step
history = model.train_step((data,))
File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 889, in train_step
y_pred = self(x, training=True)
File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler
return fn(*args, kwargs)
File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 490, in call
return super().call(*args, *kwargs)
File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler
return fn(args, kwargs)
File "/usr/local/lib/python3.8/dist-packages/keras/engine/base_layer.py", line 1014, in call
outputs = call_fn(inputs, *args, kwargs)
File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 92, in error_handler
return fn(*args, *kwargs)
File "/usr/local/lib/python3.8/dist-packages/careless/models/merging/variational.py", line 121, in call
z_f = self.surrogate_posterior.sample(self.mc_sample_size)
File "/usr/local/lib/python3.8/dist-packages/careless/models/merging/surrogate_posteriors.py", line 50, in sample
s = self.distribution.sample(args, kwargs)
File "/usr/local/lib/python3.8/dist-packages/tensorflow_probability/python/distributions/distribution.py", line 1234, in sample
return self._call_sample_n(sample_shape, seed, kwargs)
File "/usr/local/lib/python3.8/dist-packages/tensorflow_probability/python/distributions/distribution.py", line 1211, in _call_sample_n
samples = self._sample_n(
File "/usr/local/lib/python3.8/dist-packages/tensorflow_probability/python/distributions/truncated_normal.py", line 251, in _sample_n
return tf.random.stateless_parameterized_truncated_normal(
Node: 'variational_merging_model/TruncatedNormal_CONSTRUCTED_AT_top_level/sample/stateless_parameterized_truncated_normal/StatelessParameterizedTruncatedNormal'
Detected at node 'variational_merging_model/TruncatedNormal_CONSTRUCTED_AT_top_level/sample/stateless_parameterized_truncated_normal/StatelessParameterizedTruncatedNormal' defined at (most recent call last):
File "/usr/local/bin/careless", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/careless/careless.py", line 9, in main
run_careless(parser)
File "/usr/local/lib/python3.8/dist-packages/careless/careless.py", line 52, in run_careless
history = model.train_model(
File "/usr/local/lib/python3.8/dist-packages/careless/models/merging/variational.py", line 172, in train_model
_history = train_step((self, data))
File "/usr/local/lib/python3.8/dist-packages/careless/models/merging/variational.py", line 159, in train_step
history = model.train_step((data,))
File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 889, in train_step
y_pred = self(x, training=True)
File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler
return fn(*args, *kwargs)
File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 490, in call
return super().call(args, kwargs)
File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler
return fn(*args, kwargs)
File "/usr/local/lib/python3.8/dist-packages/keras/engine/base_layer.py", line 1014, in call
outputs = call_fn(inputs, *args, *kwargs)
File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 92, in error_handler
return fn(args, kwargs)
File "/usr/local/lib/python3.8/dist-packages/careless/models/merging/variational.py", line 121, in call
z_f = self.surrogate_posterior.sample(self.mc_sample_size)
File "/usr/local/lib/python3.8/dist-packages/careless/models/merging/surrogate_posteriors.py", line 50, in sample
s = self.distribution.sample(*args, kwargs)
File "/usr/local/lib/python3.8/dist-packages/tensorflow_probability/python/distributions/distribution.py", line 1234, in sample
return self._call_sample_n(sample_shape, seed, kwargs)
File "/usr/local/lib/python3.8/dist-packages/tensorflow_probability/python/distributions/distribution.py", line 1211, in _call_sample_n
samples = self._sample_n(
File "/usr/local/lib/python3.8/dist-packages/tensorflow_probability/python/distributions/truncated_normal.py", line 251, in _sample_n
return tf.random.stateless_parameterized_truncated_normal(
Node: 'variational_merging_model/TruncatedNormal_CONSTRUCTED_AT_top_level/sample/stateless_parameterized_truncated_normal/StatelessParameterizedTruncatedNormal'
2 root error(s) found.
(0) INVALID_ARGUMENT: Invalid parameters
[[{{node variational_merging_model/TruncatedNormal_CONSTRUCTED_AT_top_level/sample/stateless_parameterized_truncated_normal/StatelessParameterizedTruncatedNormal}}]]
[[variational_merging_model/TruncatedNormal_CONSTRUCTED_AT_top_level/sample/stateless_parameterized_truncated_normal/StatelessParameterizedTruncatedNormal/_14]]
(1) INVALID_ARGUMENT: Invalid parameters
[[{{node variational_merging_model/TruncatedNormal_CONSTRUCTED_AT_top_level/sample/stateless_parameterized_truncated_normal/StatelessParameterizedTruncatedNormal}}]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_step_5385]