landmanbester
commented
2 years ago @o-smirnov not sure how I broke the stimelafication but I am now ending up with a dict of dicts when calls to workers are made through stimela (even when I check out my commit where I has these things running through). For example, if I put a break point here
the kw dict contains
# (Pdb)
kw
# {'nworkers': {'nworkers': 1, 'nthreads': 8, 'scheduler': 'sync', 'ms': 'MS/point_gauss_nb.MS_p0', 'output_filename': 'out/stimflow', 'nband': 4, 'data_column': 'DATA', 'weight_column': 'WEIGHT_SPECTRUM', 'flag_column': 'FLAG', 'product': 'I', 'utimes_per_chunk': -1, 'row_out_chunk': 10000, 'group_by_field': True, 'group_by_ddid': True, 'group_by_scan': True, 'precision': 'double', 'bda_decorr': 1.0, 'max_field_of_view': 1.5}, 'nthreads': {'nworkers': 1, 'nthreads': 8, 'scheduler': 'sync', 'ms': 'MS/point_gauss_nb.MS_p0', 'output_filename': 'out/stimflow', 'nband': 4, 'data_column': 'DATA', 'weight_column': 'WEIGHT_SPECTRUM', 'flag_column': 'FLAG', 'product': 'I', 'utimes_per_chunk': -1, 'row_out_chunk': 10000, 'group_by_field': True, 'group_by_ddid': True, 'group_by_scan': True, 'precision': 'double', 'bda_decorr': 1.0, 'max_field_of_view': 1.5}, 'scheduler': {'nworkers': 1, 'nthreads': 8, 'scheduler': 'sync', 'ms': 'MS/point_gauss_nb.MS_p0', 'output_filename': 'out/stimflow', 'nband': 4, 'data_column': 'DATA', 'weight_column': 'WEIGHT_SPECTRUM', 'flag_column': 'FLAG', 'product': 'I', 'utimes_per_chunk': -1, 'row_out_chunk': 10000, 'group_by_field': True, 'group_by_ddid': True, 'group_by_scan': True, 'precision': 'double', 'bda_decorr': 1.0, 'max_field_of_view': 1.5}, 'ms': {'nworkers': 1, 'nthreads': 8, 'scheduler': 'sync', 'ms': 'MS/point_gauss_nb.MS_p0', 'output_filename': 'out/stimflow', 'nband': 4, 'data_column': 'DATA', 'weight_column': 'WEIGHT_SPECTRUM', 'flag_column': 'FLAG', 'product': 'I', 'utimes_per_chunk': -1, 'row_out_chunk': 10000, 'group_by_field': True, 'group_by_ddid': True, 'group_by_scan': True, 'precision': 'double', 'bda_decorr': 1.0, 'max_field_of_view': 1.5}, 'output_filename': {'nworkers': 1, 'nthreads': 8, 'scheduler': 'sync', 'ms': 'MS/point_gauss_nb.MS_p0', 'output_filename': 'out/stimflow', 'nband': 4, 'data_column': 'DATA', 'weight_column': 'WEIGHT_SPECTRUM', 'flag_column': 'FLAG', 'product': 'I', 'utimes_per_chunk': -1, 'row_out_chunk': 10000, 'group_by_field': True, 'group_by_ddid': True, 'group_by_scan': True, 'precision': 'double', 'bda_decorr': 1.0, 'max_field_of_view': 1.5}, 'nband': {'nworkers': 1, 'nthreads': 8, 'scheduler': 'sync', 'ms': 'MS/point_gauss_nb.MS_p0', 'output_filename': 'out/stimflow', 'nband': 4, 'data_column': 'DATA', 'weight_column': 'WEIGHT_SPECTRUM', 'flag_column': 'FLAG', 'product': 'I', 'utimes_per_chunk': -1, 'row_out_chunk': 10000, 'group_by_field': True, 'group_by_ddid': True, 'group_by_scan': True, 'precision': 'double', 'bda_decorr': 1.0, 'max_field_of_view': 1.5}, 'data_column': {'nworkers': 1, 'nthreads': 8, 'scheduler': 'sync', 'ms': 'MS/point_gauss_nb.MS_p0', 'output_filename': 'out/stimflow', 'nband': 4, 'data_column': 'DATA', 'weight_column': 'WEIGHT_SPECTRUM', 'flag_column': 'FLAG', 'product': 'I', 'utimes_per_chunk': -1, 'row_out_chunk': 10000, 'group_by_field': True, 'group_by_ddid': True, 'group_by_scan': True, 'precision': 'double', 'bda_decorr': 1.0, 'max_field_of_view': 1.5}, 'weight_column': {'nworkers': 1, 'nthreads': 8, 'scheduler': 'sync', 'ms': 'MS/point_gauss_nb.MS_p0', 'output_filename': 'out/stimflow', 'nband': 4, 'data_column': 'DATA', 'weight_column': 'WEIGHT_SPECTRUM', 'flag_column': 'FLAG', 'product': 'I', 'utimes_per_chunk': -1, 'row_out_chunk': 10000, 'group_by_field': True, 'group_by_ddid': True, 'group_by_scan': True, 'precision': 'double', 'bda_decorr': 1.0, 'max_field_of_view': 1.5}, 'flag_column': {'nworkers': 1, 'nthreads': 8, 'scheduler': 'sync', 'ms': 'MS/point_gauss_nb.MS_p0', 'output_filename': 'out/stimflow', 'nband': 4, 'data_column': 'DATA', 'weight_column': 'WEIGHT_SPECTRUM', 'flag_column': 'FLAG', 'product': 'I', 'utimes_per_chunk': -1, 'row_out_chunk': 10000, 'group_by_field': True, 'group_by_ddid': True, 'group_by_scan': True, 'precision': 'double', 'bda_decorr': 1.0, 'max_field_of_view': 1.5}, 'product': {'nworkers': 1, 'nthreads': 8, 'scheduler': 'sync', 'ms': 'MS/point_gauss_nb.MS_p0', 'output_filename': 'out/stimflow', 'nband': 4, 'data_column': 'DATA', 'weight_column': 'WEIGHT_SPECTRUM', 'flag_column': 'FLAG', 'product': 'I', 'utimes_per_chunk': -1, 'row_out_chunk': 10000, 'group_by_field': True, 'group_by_ddid': True, 'group_by_scan': True, 'precision': 'double', 'bda_decorr': 1.0, 'max_field_of_view': 1.5}, 'utimes_per_chunk': {'nworkers': 1, 'nthreads': 8, 'scheduler': 'sync', 'ms': 'MS/point_gauss_nb.MS_p0', 'output_filename': 'out/stimflow', 'nband': 4, 'data_column': 'DATA', 'weight_column': 'WEIGHT_SPECTRUM', 'flag_column': 'FLAG', 'product': 'I', 'utimes_per_chunk': -1, 'row_out_chunk': 10000, 'group_by_field': True, 'group_by_ddid': True, 'group_by_scan': True, 'precision': 'double', 'bda_decorr': 1.0, 'max_field_of_view': 1.5}, 'row_out_chunk': {'nworkers': 1, 'nthreads': 8, 'scheduler': 'sync', 'ms': 'MS/point_gauss_nb.MS_p0', 'output_filename': 'out/stimflow', 'nband': 4, 'data_column': 'DATA', 'weight_column': 'WEIGHT_SPECTRUM', 'flag_column': 'FLAG', 'product': 'I', 'utimes_per_chunk': -1, 'row_out_chunk': 10000, 'group_by_field': True, 'group_by_ddid': True, 'group_by_scan': True, 'precision': 'double', 'bda_decorr': 1.0, 'max_field_of_view': 1.5}, 'group_by_field': {'nworkers': 1, 'nthreads': 8, 'scheduler': 'sync', 'ms': 'MS/point_gauss_nb.MS_p0', 'output_filename': 'out/stimflow', 'nband': 4, 'data_column': 'DATA', 'weight_column': 'WEIGHT_SPECTRUM', 'flag_column': 'FLAG', 'product': 'I', 'utimes_per_chunk': -1, 'row_out_chunk': 10000, 'group_by_field': True, 'group_by_ddid': True, 'group_by_scan': True, 'precision': 'double', 'bda_decorr': 1.0, 'max_field_of_view': 1.5}, 'group_by_ddid': {'nworkers': 1, 'nthreads': 8, 'scheduler': 'sync', 'ms': 'MS/point_gauss_nb.MS_p0', 'output_filename': 'out/stimflow', 'nband': 4, 'data_column': 'DATA', 'weight_column': 'WEIGHT_SPECTRUM', 'flag_column': 'FLAG', 'product': 'I', 'utimes_per_chunk': -1, 'row_out_chunk': 10000, 'group_by_field': True, 'group_by_ddid': True, 'group_by_scan': True, 'precision': 'double', 'bda_decorr': 1.0, 'max_field_of_view': 1.5}, 'group_by_scan': {'nworkers': 1, 'nthreads': 8, 'scheduler': 'sync', 'ms': 'MS/point_gauss_nb.MS_p0', 'output_filename': 'out/stimflow', 'nband': 4, 'data_column': 'DATA', 'weight_column': 'WEIGHT_SPECTRUM', 'flag_column': 'FLAG', 'product': 'I', 'utimes_per_chunk': -1, 'row_out_chunk': 10000, 'group_by_field': True, 'group_by_ddid': True, 'group_by_scan': True, 'precision': 'double', 'bda_decorr': 1.0, 'max_field_of_view': 1.5}, 'precision': {'nworkers': 1, 'nthreads': 8, 'scheduler': 'sync', 'ms': 'MS/point_gauss_nb.MS_p0', 'output_filename': 'out/stimflow', 'nband': 4, 'data_column': 'DATA', 'weight_column': 'WEIGHT_SPECTRUM', 'flag_column': 'FLAG', 'product': 'I', 'utimes_per_chunk': -1, 'row_out_chunk': 10000, 'group_by_field': True, 'group_by_ddid': True, 'group_by_scan': True, 'precision': 'double', 'bda_decorr': 1.0, 'max_field_of_view': 1.5}, 'bda_decorr': {'nworkers': 1, 'nthreads': 8, 'scheduler': 'sync', 'ms': 'MS/point_gauss_nb.MS_p0', 'output_filename': 'out/stimflow', 'nband': 4, 'data_column': 'DATA', 'weight_column': 'WEIGHT_SPECTRUM', 'flag_column': 'FLAG', 'product': 'I', 'utimes_per_chunk': -1, 'row_out_chunk': 10000, 'group_by_field': True, 'group_by_ddid': True, 'group_by_scan': True, 'precision': 'double', 'bda_decorr': 1.0, 'max_field_of_view': 1.5}, 'max_field_of_view': {'nworkers': 1, 'nthreads': 8, 'scheduler': 'sync', 'ms': 'MS/point_gauss_nb.MS_p0', 'output_filename': 'out/stimflow', 'nband': 4, 'data_column': 'DATA', 'weight_column': 'WEIGHT_SPECTRUM', 'flag_column': 'FLAG', 'product': 'I', 'utimes_per_chunk': -1, 'row_out_chunk': 10000, 'group_by_field': True, 'group_by_ddid': True, 'group_by_scan': True, 'precision': 'double', 'bda_decorr': 1.0, 'max_field_of_view': 1.5}}So every key contains a dict with all other arguments. This does not happen if I call the workers through the cmd line interface. Any ideas on what could be going wrong? Also pinging @SpheMakh for ideas. I have up to date master branches of both stimela2 and scabha2 installed
Adds functionality to execute workers through stimela recipes (thanks to @o-smirnov's clickification magic in scabha).