casangi / xradio

Xarray Radio Astronomy Data IO
https://xradio.readthedocs.io/en/latest/
Other
9 stars 5 forks source link

Allow partitioning by ANTENNA1 #220

Open tnakazato opened 1 month ago

tnakazato commented 1 month ago

Antenna partition only processes autocorrelation, i.e., all cross-correlation data will be discarded. So it is useful only for single-dish data processing.

This is related to Issue #9. Please note that this is a draft PR just to show you an idea on what is actually requested in #9 (please see below for missing development items). Point is to allow separating data per antenna both on memory and disk.

This PR allows to set "ANTENNA1" to partition_scheme. If "ANTENNA1" is specified, processing set is separated by antenna and each processing set includes only auto-correlation data.

convert_msv2_to_processing_set(
    in_file=in_file,
    out_file=out_file,
    partition_scheme=['FIELD_ID', 'ANTENNA1'],
    parallel=False,
    overwrite=True,
)

The above command will produce more granular processing sets separated by antenna and ANTENNA_ID will be shown in the log message like below.

[2024-08-09 15:09:41,986]     INFO      client:  Partition scheme that will be used: ['DATA_DESC_ID', 'OBS_MODE', 'OBSERVATION_ID', 'FIELD_ID', 'ANTENNA1'] 
[2024-08-09 15:09:43,453]     INFO      client:  Number of partitions: 156 
[2024-08-09 15:09:43,453]     INFO      client:  OBSERVATION_ID [0], DDI [0], STATE [10], FIELD [1], SCAN [ 2  4  6  8 10 12], ANTENNA [0] 
[2024-08-09 15:09:43,895]     INFO      client:  OBSERVATION_ID [0], DDI [0], STATE [10], FIELD [1], SCAN [ 2  4  6  8 10 12], ANTENNA [1] 
[2024-08-09 15:09:44,318]     INFO      client:  OBSERVATION_ID [0], DDI [0], STATE [10], FIELD [1], SCAN [ 2  4  6  8 10 12], ANTENNA [2] 
[2024-08-09 15:09:44,744]     INFO      client:  OBSERVATION_ID [0], DDI [0], STATE [11], FIELD [1], SCAN [ 2  4  6  8 10 12], ANTENNA [0] 
[2024-08-09 15:09:45,188]     INFO      client:  OBSERVATION_ID [0], DDI [0], STATE [11], FIELD [1], SCAN [ 2  4  6  8 10 12], ANTENNA [1] 
[2024-08-09 15:09:45,611]     INFO      client:  OBSERVATION_ID [0], DDI [0], STATE [11], FIELD [1], SCAN [ 2  4  6  8 10 12], ANTENNA [2]

Development is incomplete. At least, convert_msv2_to_processing_set should be updated so that,

CLAassistant commented 1 month ago

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.