extra_parameters_file

[canismarko]

I'm looking for a way to specify per-file recon parameters, and I have a solution to propose. It's not technically breaking backwards compatibility, but we should deprecate the rotation_axis_file to avoid confusion.

Use Case: I have a large number (>100) tomograms to reconstruct from an in-situ experiment. I have a single recon.conf file that is close to the right configuration for all the tomograms, but each tomogram has a few parameters that are different (e.g. rotation_axis, nsino, stripe_removal).

My solution using YAML file: My first thought was to extend the rotation_axis.json approach already present. The problem is that the current schema of {"0": {"my_tomogram001.h5": 1024}} only allows for specifying a rotation axis. I would have to break the schema in order to include other parameters (and then need to include some version info to detect which schema is being used). Since I'm rewriting the schema anyway, why not switch to a more human-readable markup language, like YAML. The same rotation axis info would then be::

my_tomogram001.h5:
  rotation-axis: 1024

Extra parameters can be added as well:

my_tomogram001.h5:
  rotation-axis: 1024
my_tomogram002.h5:
  rotation-axis: 1015
  remove-stripe-method: none

I create new branch yaml-parameter-file that implements this approach. At the start of each reconstruction, a new argparse.Namespace object is created that is a deep copy of the main args. Then if args.extra_parameters_file is given, the YAML file is loaded for the current value of args.file_name and the extra parameters (if any) are swapped out in the local Namespace used for the reconstruction. After each reconstruction, the local Namespace is discarded.

Any argument given directly via the command-line takes precedence over those found in the YAML file, and the YAML file takes precedence over the .conf file.

The same YAML file can also be provided as the argument to args.file_name. In this case, the list of HDF5 files to reconstruct will be taken from the YAML file, but the parameters will be ignored (unless the same file is given for args.extra_parameters_file). This matches the current behavior for if a JSON file is given as args.file_name.

Open Questions:

1. Is YAML the best approach here? Are there situations where YAML is too constrained? It would be nice not to change the schema again if this is accepted.
2. After a full reconstruction, a new .conf file is saved alongside the TIFFs for reproducibility. Should this reflect the main args Namespace, or the temporary local Namespace? I think it should be the local one but could see an argument for either one.
3. How best to deprecate args.rotation_axis_file? One option is to put a check and logging in config.update_config so the parameter is not saved. That way $tomopy init doesn't save that parameter. It's a bit fragmented, though.
4. What to do with args.rotation_axis_auto? Currently, if args.rotation_axis_auto == "json", then the rotation axis is taken from the JSON file. I've done the same for the YAML file if args.rotation_axis_auto == "yaml", but this seems needlessly complicated since this only applies to rotation_axis and requires someone to be aware of this behavior. If we removed this, though, it would be difficult to specify a manual rotation center in the .conf file and still use args.extra_parameters_file.
5. Is args.extra_parameters_file the right name for the parameter or is there something more clear?
6. Does order matter? The current rotation_axis.json file includes an index for each tomogram, but I don't see a reason that this would matter. Even if order was important, it seems that the order in which the files are listed could be respected.

Backwards Compatibility:

I don't think there's any breaking changes here. The $ tomopy find_center command will save rotation centers into the YAML file specified by args.extra_parameters_file. The args.rotation_axis_file parameter is effectively deprecated, though will still be used for rotation centers if given, with the YAML file taking precedence.

Other Strategies: Here are some other things I considered but ultimately didn't like for one reason or another.

1. I could create a .conf file for each tomogram. The problem here is that if I need to change some common parameter (e.g. number of iterations) I need to update every .conf file.
2. Shell script or makefile that contains a series of lines to call tomopy recon mytomogramNNN.h5 and then add each overidden parameter as an argument to the relevant script line. This would probably work, but rotation centers would still be stored in the rotation_axis.json file created by tomopy find_center. Would be nice to have everything in one file and all handled by tomopy-cli.

[decarlof]

1. YAML is much easier to read/edit by humans and with dump() and load() as easy to handle in python as json.
2. Not sure, I see +/- in both …
3. I came up with the json file as a quick solution to run a batch of reconstructions after running the find_rotation_axis on a folder of raw data. I use it regularly at 2-BM to do user reconstruction, but I am not sure how much is used by users themselves => if there is not a strong opposition in this thread I would not worry much about deprecation and backwards compatibility and move forward to a better solution that includes other per-dataset parameters.
4. See above
5. How about just args.parameter_file
6. Order is not important. What I wanted to make sure is that when I run a batch reconstruction the raw data are processed in alphabetic order. In this case if the batch stops for some reason is easy to find which data sets still need processing. To accomplish this I sorted the file name list at folder reading time here. There is no need to add the index in the file as the execution order is still respected.

Another advantage of handling multiple parameters besides the center is that we can create a default extracting values from the hdf file itself; let's say someone wants to run paganin => we can populate the yaml with data like energy, sample detector distance, pixel size from the file, (now to find these values I have to open the hdf and read them manually) run the reconstruction with those values and if there is a need to re-run with a different parameter (let's say the energy was not calibrated right) I can just modify that single parameter and re-run the all batch.

[canismarko]

Thanks for the feedback. I'll clean up some of the code and merge it into the main branch soon.

I don't think your answer to (3) quite solves (4), but in thinking it through I can't come up with a use case for having rotation_axis_auto=manual in the config file but not reading it from the parameters file if it's present, so I will remove the "yaml" and "json" options. I'll also get rid of the json version entirely.

How about just args.parameter_file

Sure, sounds good.

Order is not important. What I wanted to make sure is that when I run a batch reconstruction the raw data are processed in alphabetic order. In this case if the batch stops for some reason is easy to find which data sets still need processing. To accomplish this I sorted the file name list at folder reading time here. There is no need to add the index in the file as the execution order is still respected.

I've also added a feature where if there is an exception during reconstruction, tomopy-cli will log the exception and keep going, then at the end it logs a list of all the tomograms that failed. This avoids the infuriating situation where I start a long batch of reconstructions and come back 2 days later only to learn that the script ran into a problem 20 minutes after I left and didn't finish any tomograms.

[canismarko]

Fix: 1d314c96787dff4f5041a04fbe013f274df45154