Internal flags in QuartiCal are now represented by a np.int8 array rather than a np.bool array. This has zero impact on memory footprint (numpy atomicity is already at the byte level). I have also removed the correlation axis from the internal flag representation as we take the union of the flags over the correlation axis anyway. This results in the flags being a factor of 32 smaller than the data (in the four correlation case). In addition, internal flags now have three valid states:
-1 a temporarily flagged value. This will principally be used for reversible selection operations such as uv cuts.
0 an unflagged value.
1 a genuinely flagged value. These will usually be the input flags but QuartiCal may augment these with e.g. flags based on robust reweighting.
This change is necessary to cope with several use cases which place strain on using the weights to implement flagging. These use cases are:
Flagging based on the robust weights.
Scaling the input weights by the gain amplitudes.
Prior to this PR, operations like uv selection would simply set the relevant weights to zero but this is not reversible. Consequently, uv selection would interact weirdly with the cases above. This should no longer be a problem as uv selection is now implemented as a temporary flagging operation which doesn't preclude subsequent operations on the weights.
For the canny amongst you, this is a limited form of bitflagging. This will NEVER be exposed to users and my intention is to keep it to strictly three states, which I believe will be sufficient.
This change also makes the solvers faster on heavily flagged data.
Internal flags in QuartiCal are now represented by a
np.int8array rather than anp.boolarray. This has zero impact on memory footprint (numpyatomicity is already at the byte level). I have also removed the correlation axis from the internal flag representation as we take the union of the flags over the correlation axis anyway. This results in the flags being a factor of 32 smaller than the data (in the four correlation case). In addition, internal flags now have three valid states:-1a temporarily flagged value. This will principally be used for reversible selection operations such as uv cuts.0an unflagged value.1a genuinely flagged value. These will usually be the input flags but QuartiCal may augment these with e.g. flags based on robust reweighting.This change is necessary to cope with several use cases which place strain on using the weights to implement flagging. These use cases are:
Prior to this PR, operations like uv selection would simply set the relevant weights to zero but this is not reversible. Consequently, uv selection would interact weirdly with the cases above. This should no longer be a problem as uv selection is now implemented as a temporary flagging operation which doesn't preclude subsequent operations on the weights.
For the canny amongst you, this is a limited form of bitflagging. This will NEVER be exposed to users and my intention is to keep it to strictly three states, which I believe will be sufficient.
This change also makes the solvers faster on heavily flagged data.