applying a scale function to new input

[DHekstra]

I am seeking to jointly process "light" and "dark" integrated intensities previously processed using CrystFEL.

1. Careless appears to run fine on a single "dark" input MTZ generated from the stream files (e.g. job 24317437_9082). An existing structure can be refined against these dark structure factor amplitudes.

2. I am able to run a next Careless run on the dark data where I initialize the scales using the scales from the preceding run (--scale-file="./merge_crystfel_24317437_9082_mono_grid_3/dhfr_grid_scale")--see job id 24362624_115. The resulting amplitudes again permit for refinement in Phenix with similar statistics (a little better, FWIW).

3. When I run Careless on light and dark data (without DW), I get a numerical issues warning and early termination:

   Encountered numerical issues, terminating optimization early!

   and progress bar:

   Training:   0%|          | 0/30000 [00:03<?, ?it/s, Grad Norm=nan, loss=nan, F KLDiv=6.07e+02, NLL=nan, NLL_val=nan]

4. I then tried to prevent this by providing careless with an approximate initial scale function based on just the dark data (using the same metadata keys). In this case, I get the following error (job ID 24368387_30733)

   Traceback (most recent call last):
   File "/n/hekstra_lab/people/dhekstra/conda_envs/careless-12/bin/careless", line 8, in <module>
   sys.exit(main())
   File "/n/hekstra_lab/people/dhekstra/conda_envs/careless-12/lib/python3.9/site-packages/careless/careless.py", line 9, in main
   run_careless(parser)
   File "/n/hekstra_lab/people/dhekstra/conda_envs/careless-12/lib/python3.9/site-packages/careless/careless.py", line 41, in run_careless
   model.scaling_model.load_weights(parser.scale_file)
   File "/n/hekstra_lab/people/dhekstra/conda_envs/careless-12/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler
   raise e.with_traceback(filtered_tb) from None
   File "/n/hekstra_lab/people/dhekstra/conda_envs/careless-12/lib/python3.9/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 720, in _restore_from_tensors
   raise ValueError(
   ValueError: Received incompatible tensor with shape (2884,) when attempting to restore variable with shape (6292,) and name Variable:0.
   WARNING:tensorflow:Detecting that an object or model or tf.train.Checkpoint is being deleted with unrestored values. See the following logs for the specific values in question. To sil
   ence these warnings, use `status.expect_partial()`. See https://www.tensorflow.org/api_docs/python/tf/train/Checkpoint#restorefor details about the status object returned by the resto
   re function.
   WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-0.kernel
   WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-0.bias
   WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-1.kernel
   WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-1.bias
   WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-2.kernel
   WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-2.bias
   WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-3.kernel
   WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-3.bias
   WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-4.kernel
   WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-4.bias
   WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-5.kernel
   WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-5.bias
   WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-6.kernel
   WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-6.bias
   WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-7.kernel
   WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-7.bias
   WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-8.kernel
   WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-8.bias
   WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-9.kernel
   WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.network.layer_with_weights-9.bias
   WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.distribution.layer_with_weights-0.kernel
   WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).mlp_scaler.distribution.layer_with_weights-0.bias

[DHekstra]

Careless v 0.4.1 with

Args: mono 
--separate-files 
--mc-samples=10 
--mlp-layers=10 
--image-layers=0 
--dmin=2.5 
--iterations=30_000 
--test-fraction=0.1 
--seed=5165 
--mlp-width=20

[kmdalton]

Sorry if I missed it, but can you process the light data alone?

[DHekstra]

let me try

[DHekstra]

oddly, the error on step 3 above appears to have been caused by some of the input MTZs missing metadata columns.

[kmdalton]

your last message seems to indicate this is an issue with the input data formatting and not with the careless CLI. i will close this issue for now.