Detects the wrong number of solution intervals

[Joshuaalbert]

Describe the bug In some cases the number of solution intervals is incorrectly detected.

Data description (if applicable) It's an MS with a single scan of 10 time integrations and 1 channel:

2023-10-23 12:43:46 | INFO | summary:field_info | 
Field summary:
    ID   NAME             PHASE_DIR        REF_DIR          DELAY_DIR       
    0    source           0.0000 0.6480    0.0000 0.6480    0.0000 0.6480   

2023-10-23 12:43:46 | WARNING | reads:_single_dataset | 
Ignoring 'LOG': Unable to infer shape of column '{column}'
2023-10-23 12:43:46 | WARNING | reads:_single_dataset | 
Ignoring 'SCHEDULE': Unable to infer shape of column '{column}'
2023-10-23 12:43:46 | INFO | summary:polarization_info | 
Polarization summary:
    INDEX    CORR_TYPE                     
    0        [ 9 10 11 12] -> ['XX', 'XY', 'YX', 'YY']

2023-10-23 12:43:46 | INFO | summary:spw_info | 
Spectral window summary:
    INDEX    CHANNELS   BANDWIDTH        REF_FREQ        
    0        1          8576000.0        704221000.0     

2023-10-23 12:43:46 | WARNING | reads:_dataset_variable_factory | 
The shape of column 'TARGET' is unconstrained (ndim == -1). Assuming shape is (1, 2) from exemplar
2023-10-23 12:43:51 | INFO | summary:dimension_summary | 
Dimension summary:
    DATA_DESC_ID SCAN_NUMBER  FIELD_ID     ROW      TIME   CHAN   CORR
    0            1            0            20961280 10     1      4   

2023-10-23 12:43:52 | INFO | summary:flagging_summary | 
Flagging summary:
    DATA_DESC_ID SCAN_NUMBER  FIELD_ID     PERC_FLAGGED
    0            1            0            0.00        
                              TOTAL        0.00

To Reproduce

Case 1 - Wrong: Finds only T0F0, but expected 10 chunks.

input_ms.time_chunk=0
input_ms.freq_chunk=0

solver.terms=[G]

G.type=phase
G.time_interval=1
G.freq_interval=1

Case 2 - Wrong: Finds only T0F0, but expected 5 chunks.

input_ms.time_chunk=0
input_ms.freq_chunk=0

solver.terms=[G]

G.type=phase
G.time_interval=2
G.freq_interval=1

Case 3 - Correct: Finds the expected 10 chunks.

input_ms.time_chunk=1 # upper limit for time interval, so we do expect 10 chunks
input_ms.freq_chunk=1

solver.terms=[G]

G.type=phase
G.time_interval=2
G.freq_interval=1

Case 4 - Correct: Finds the expected 5 chunks.

input_ms.time_chunk=2
input_ms.freq_chunk=1

solver.terms=[G]

G.type=phase
G.time_interval=2
G.freq_interval=1

Case 5 - Correct: Finds the expected 10 chunks.

input_ms.time_chunk=1 # upper limit for time interval, so we do expect 10 chunks
input_ms.freq_chunk=1

solver.terms=[G]

G.type=phase
G.time_interval=1
G.freq_interval=1

Version

aiobotocore==2.7.0
aiohttp==3.8.6
aioitertools==0.11.0
aiosignal==1.3.1
antlr4-python3-runtime==4.9.3
anyio==4.0.0
appdirs==1.4.4
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asciitree==0.3.3
astLib==0.11.10
astro-kittens==1.4.6
astro-tigger-lsm==1.7.2
astropy==5.3.4
asttokens==2.4.0
async-lru==2.0.4
async-timeout==4.0.3
attrs==23.1.0
Babel==2.13.0
backcall==0.2.0
beautifulsoup4==4.12.2
bleach==6.1.0
bokeh==3.3.0
botocore==1.31.64
certifi==2023.7.22
cffi==1.16.0
charset-normalizer==3.3.1
cli-ui==0.17.2
click==8.1.7
cloudpickle==3.0.0
codex-africanus==0.3.4
colorama==0.4.6
Columnar==1.4.1
comm==0.1.4
commonmark==0.9.1
configparser==6.0.0
contextlib2==21.6.0
contourpy==1.1.1
cycler==0.12.1
dask==2023.10.0
dask-ms==0.2.18
debugpy==1.8.0
decorator==5.1.1
defusedxml==0.7.1
dill==0.3.7
distributed==2023.10.0
docopt==0.6.2
donfig==0.7.0
exceptiongroup==1.1.3
executing==2.0.0
fasteners==0.19
fastjsonschema==2.18.1
fonttools==4.43.1
fqdn==1.5.1
frozenlist==1.4.0
fsspec==2023.10.0
future==0.18.3
idna==3.4
importlib-metadata==6.8.0
iniconfig==2.0.0
ipykernel==6.25.2
ipython==8.16.1
ipython-genutils==0.2.0
ipywidgets==8.1.1
isoduration==20.11.0
jedi==0.19.1
Jinja2==3.1.2
jmespath==1.0.1
json5==0.9.14
jsonpointer==2.4
jsonschema==4.19.1
jsonschema-specifications==2023.7.1
jupyter==1.0.0
jupyter-console==6.6.3
jupyter-events==0.8.0
jupyter-lsp==2.2.0
jupyter_client==8.4.0
jupyter_core==5.4.0
jupyter_server==2.8.0
jupyter_server_terminals==0.4.4
jupyterlab==4.0.7
jupyterlab-pygments==0.2.2
jupyterlab-widgets==3.0.9
jupyterlab_server==2.25.0
kiwisolver==1.4.5
llvmlite==0.41.1
locket==1.0.0
loguru==0.7.2
MarkupSafe==2.1.3
matplotlib==3.8.0
matplotlib-inline==0.1.6
mistune==3.0.2
msgpack==1.0.7
multidict==6.0.4
munch==2.5.0
nbclient==0.8.0
nbconvert==7.9.2
nbformat==5.9.2
nest-asyncio==1.5.8
notebook==7.0.6
notebook_shim==0.2.3
numba==0.58.1
numcodecs==0.12.1
numpy==1.23.5
omegaconf==2.3.0
overrides==7.4.0
packaging==23.2
pandas==2.1.1
pandocfilters==1.5.0
parso==0.8.3
partd==1.4.1
pexpect==4.8.0
pickleshare==0.7.5
Pillow==10.1.0
platformdirs==3.11.0
pluggy==1.3.0
prometheus-client==0.17.1
prompt-toolkit==3.0.39
psutil==5.9.6
ptyprocess==0.7.0
pure-eval==0.2.2
pycparser==2.21
pydantic==1.10.13
pyerfa==2.0.1.1
Pygments==2.16.1
pyparsing==3.1.1
pytest==7.4.2
python-casacore==3.5.2
python-dateutil==2.8.2
python-json-logger==2.0.7
pytz==2023.3.post1
PyYAML==6.0.1
pyzmq==25.1.1
qtconsole==5.4.4
QtPy==2.4.0
quartical==0.2.0
referencing==0.30.2
requests==2.31.0
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rich==12.6.0
rpds-py==0.10.6
ruamel.yaml==0.17.40
ruamel.yaml.clib==0.2.8
s3fs==2023.10.0
schema==0.7.5
scipy==1.11.3
Send2Trash==1.8.2
six==1.16.0
sniffio==1.3.0
sortedcontainers==2.4.0
soupsieve==2.5
stack-data==0.6.3
stimela==2.0rc4
tabulate==0.8.10
tblib==3.0.0
tbump==6.11.0
terminado==0.17.1
tinycss2==1.2.1
tomli==2.0.1
tomlkit==0.11.8
toolz==0.12.0
tornado==6.3.3
traitlets==5.11.2
types-python-dateutil==2.8.19.14
typing_extensions==4.8.0
tzdata==2023.3
Unidecode==1.3.7
uri-template==1.3.0
urllib3==2.0.7
wcwidth==0.2.8
webcolors==1.13
webencodings==0.5.1
websocket-client==1.6.4
widgetsnbextension==4.0.9
wrapt==1.15.0
xarray==2023.10.1
xyzservices==2023.10.0
yarl==1.9.2
zarr==2.16.1
zict==3.0.0
zipp==3.17.0

[JSKenyon]

I think that this may stem from a misconception. The solution intervals do not control the chunking and the run-end statistics are per chunk. I have taken the liberty of adding some case labels to your comment to simplify the reply.

• Case 1 - Data is not chunked, (1,1) solution intervals, only T0F0 is found. This is the expected behaviour. There is only a single chunk. That chunk contains 10 solution intervals and the printed statistics are per-chunk, hence only seeing T0F0.
• Case 2 - Data is not chunked, (2,1) solution intervals, only T0F0 is found. This is the expected behaviour. There is only a single chunk. That chunk contains 5 solution intervals and the printed statistics are per-chunk, hence only seeing T0F0.
• Case 3 - (1,1) chunks, (2,1) solution intervals, T0F0 - T9F0 found. This is the expected behaviour. There are 10 chunks. Each chunk contains (and limits) a single solution interval. The printed statistics are per-chunk, but in this case that corresponds to per-interval.
• Case 4 - (2,1) chunks, (2,1) solution intervals, T0F0 - T4F0 found. This is the expected behaviour. There are 5 chunks. Each chunk contains and corresponds to a single solution interval. The printed statistics are per-chunk, but in this case that corresponds to per-interval.
• Case 5 - (1,1) chunks, (1,1) solution intervals, T0F0 - T9F0 found. This is the expected behaviour. There are 10 chunks. Each chunk contains and corresponds to a single solution interval. The printed statistics are per-chunk, but in this case that corresponds to per-interval.

Please let me know if anything is unclear.

[Joshuaalbert]

Ah, I understand now. Thanks for the lesson. And good to see things are working as expected.