Closed yousefmoazzam closed 2 months ago
For an example of how the logging looks so far (I've only added the basics), with the following pipeline (sweeps over 10 center
values in the recon):
- method: standard_tomo
module_path: httomo.data.hdf.loaders
parameters:
name: tomo
data_path: entry1/tomo_entry/data/data
image_key_path: entry1/tomo_entry/instrument/detector/image_key
rotation_angles:
data_path: /entry1/tomo_entry/data/rotation_angle
dimension: 1
pad: 0
preview:
detector_y:
start: 50
stop: 57
- method: normalize
module_path: httomolibgpu.prep.normalize
parameters:
cutoff: 10.0
minus_log: true
nonnegativity: false
remove_nans: false
- method: paganin_filter_tomopy
module_path: httomolibgpu.prep.phase
parameters:
pixel_size: 0.0001
dist: 50.0
energy: 53.0
alpha: 0.001
- method: remove_all_stripe
module_path: httomolibgpu.prep.stripe
parameters:
snr: 3.0
la_size: 61
sm_size: 21
dim: 1
- method: FBP
module_path: httomolibgpu.recon.algorithm
save_result: False
parameters:
center: !SweepRange
start: 10
stop: 20
step: 1
filter_freq_cutoff: 0.6
recon_size: null
recon_mask_radius: null
- method: save_to_images
module_path: httomolib.misc.images
parameters:
subfolder_name: images
axis: 1
file_format: tif
bits: 8
perc_range_min: 0.0
perc_range_max: 100.0
jpeg_quality: 95
offset: 0
asynchronous: true
the following terminal output is produced:
See the full log file at: /dls/tmp/twi18192/sweep-tests/08-07-2024_16_35_08_output/user.log
Loading data with shape (1801, 7, 2560)
Running data_reducer (httomolib)
Running normalize (httomolibgpu)
Running paganin_filter_tomopy (httomolibgpu)
Running remove_all_stripe (httomolibgpu)
Running FBP (httomolibgpu)
Parameter sweep over 10 values of parameter: center
0%| | 0/10 [00:00<?, ?value/s, center=10]
10%|# | 1/10 [00:02<00:21, 2.36s/value, center=11]
20%|## | 2/10 [00:03<00:12, 1.51s/value, center=12]
30%|### | 3/10 [00:04<00:08, 1.25s/value, center=13]
40%|#### | 4/10 [00:05<00:06, 1.12s/value, center=14]
50%|##### | 5/10 [00:06<00:05, 1.04s/value, center=15]
60%|###### | 6/10 [00:06<00:03, 1.00value/s, center=16]
70%|####### | 7/10 [00:07<00:02, 1.03value/s, center=17]
80%|######## | 8/10 [00:08<00:01, 1.05value/s, center=18]
90%|######### | 9/10 [00:09<00:00, 1.06value/s, center=19]
Finished parameter sweep
Running save_to_images (httomolib)
Pipeline finished. Took 32.329s
Feel free to make any suggestions on tweaks/additions for improvement :slightly_smiling_face:
Attempt to fix #362
Main changes:
ParamSweepWriter
+ParamSweepReader
for writing + reading parameter sweep results using blocksStages
to group methods that are before the sweep, in the sweep, and after the sweepParamSweepRunner
to orchestrate the parameter sweep runSideOutputManager
to separate side output logic from runnerDataSetBlock
and introduce three interfaces that describe the distinct sets of behaviour that a block needs to be processable by an implementor ofMethodWrapper
(see commit message of 4123373 for more info)ParamSweepBlock
to hold data during a parameter sweep runrun
command to choose between executing high-throughput or param sweep runThings left to do:
!Sweep
and!SweepRange
YAML "tags"!Sweep
and!SweepRange
tagsAcceptance criteria checklist