Improve shifting of apertures.

[StuartLittlefair]

From @trmrsh,

1) I now flag an error if an aperture moves beyond the search box (with the change I made yesterday, I was not not quite doing this)

2) If there are reference apertures in play, I now skip the search step on any non-reference apertures, and instead go straight to fitting based on the offset determined from the reference apertures. I think this is a much more robust approach (might be what I do in the ULTRACAM pipeline indeed). It obviates the need for separate half widths for reference and non-reference searches, so I have changed genred and reduce to have just a single value now for the search width (new version number! old reduce files will not work with new version). I have also improved things now so that genred allows you to set the search half and the profile fit half widths separately. There is in fact nothing wrong with using a search half width smaller than the profile fit half width if guiding is good, or vice versa, so these should not have been set to be the same anyway. Even though the non-reference targets are not specifically searched, just fitted, I will still abort the extraction for them if their position moves outside the search box.

[StuartLittlefair]

Hi @trmrsh, testing this out as requested on the MAXI1820 dataset:

Won't start running at all at the moment, with the following error:

Traceback (most recent call last):
  File "/Users/sl/anaconda3/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/Users/sl/anaconda3/lib/python3.6/multiprocessing/pool.py", line 47, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/Users/sl/anaconda3/lib/python3.6/site-packages/hipercam/scripts/reduce.py", line 3048, in ccdproc
    moveApers(cnam, ccd, flat, rflat, ccdaper, ccdwins, rfile, store)
  File "/Users/sl/anaconda3/lib/python3.6/site-packages/hipercam/scripts/reduce.py", line 1796, in moveApers
    apsec['fit_thresh'], apsec['fit_ndiv']
  File "/Users/sl/anaconda3/lib/python3.6/site-packages/hipercam/fitting.py", line 110, in combFit
    read, gain, thresh, ndiv
  File "/Users/sl/anaconda3/lib/python3.6/site-packages/hipercam/fitting.py", line 1083, in fitGaussian
    full_output=True
  File "/Users/sl/anaconda3/lib/python3.6/site-packages/scipy/optimize/minpack.py", line 380, in leastsq
    raise TypeError('Improper input: N=%s must not exceed M=%s' % (n, m))
TypeError: Improper input: N=5 must not exceed M=4
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/sl/anaconda3/bin/reduce", line 11, in <module>
    load_entry_point('hipercam==0.8.dev136+g89c65c9.d20180608', 'console_scripts', 'reduce')()
  File "/Users/sl/anaconda3/lib/python3.6/site-packages/hipercam/scripts/reduce.py", line 755, in reduce
    allres = pool.starmap(ccdproc, arglist)
  File "/Users/sl/anaconda3/lib/python3.6/multiprocessing/pool.py", line 274, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/Users/sl/anaconda3/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
TypeError: Improper input: N=5 must not exceed M=4

[trmrsh]

OK, I know the problem here.

[StuartLittlefair]

Thanks Tom,

There's a new problem now, sadly:

Frame 1: 2018-06-08T00:18:41.7355610 [OK]
CCD 2, reference aperture 1, fit failed, error = Fitted position (1882.3,26.5) too close to or beyond edge of search window = Winhead(llx=1871, lly=15, nx=11, ny=11, xbin=2, ybin=2, head='NCYCLE  =                    1                                                  \nREFLECT =                    T                                                  \nDSTATUS =                    T                                                  \nMJDUTC  =    58277.01350388381                                                  \nGOODTIME=                    T                                                  \nEXPTIME =                 90.0                                                  ')
CCD 2: no reference aperture fit was successful; skipping
 *** WARNING: CCD 2: no measured FWHM to re-size apertures or carry out optimal extraction; no extraction possible
CCD 3, mean x,y shift from reference aperture(s) = -1.26, -10.32
CCD 4, reference aperture 1, fit failed, error = Fitted position (1878.1,7.7) too close to or beyond edge of search window = Winhead(llx=1869, lly=3, nx=11, ny=11, xbin=2, ybin=2, head='')
CCD 4: no reference aperture fit was successful; skipping
 *** WARNING: CCD 4: no measured FWHM to re-size apertures or carry out optimal extraction; no extraction possible
CCD 5, reference aperture 1, fit failed, error = Fitted position (1857.6,19.3) too close to or beyond edge of search window = Winhead(llx=1845, lly=11, nx=11, ny=11, xbin=2, ybin=2, head='NCYCLE  =                    1                                                  \nREFLECT =                    T                                                  \nDSTATUS =                    T                                                  \nMJDUTC  =    58277.01350388381                                                  \nGOODTIME=                    T                                                  \nEXPTIME =                 90.0                                                  ')
CCD 5: no reference aperture fit was successful; skipping
 *** WARNING: CCD 5: no measured FWHM to re-size apertures or carry out optimal extraction; no extraction possible
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/sl/anaconda3/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/Users/sl/anaconda3/lib/python3.6/multiprocessing/pool.py", line 47, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/Users/sl/anaconda3/lib/python3.6/site-packages/hipercam/scripts/reduce.py", line 3051, in ccdproc
    moveApers(cnam, ccd, flat, rflat, ccdaper, ccdwins, rfile, store)
  File "/Users/sl/anaconda3/lib/python3.6/site-packages/hipercam/scripts/reduce.py", line 1766, in moveApers
    peak = swdata.data[int(round(swdata.y_pixel(y))),int(round(swdata.y_pixel(x)))]
IndexError: index 73 is out of bounds for axis 1 with size 11
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/sl/anaconda3/bin/reduce", line 11, in <module>
    load_entry_point('hipercam==0.8.dev139+g11b9f0b', 'console_scripts', 'reduce')()
  File "/Users/sl/anaconda3/lib/python3.6/site-packages/hipercam/scripts/reduce.py", line 755, in reduce
    allres = pool.starmap(ccdproc, arglist)
  File "/Users/sl/anaconda3/lib/python3.6/multiprocessing/pool.py", line 274, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/Users/sl/anaconda3/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
IndexError: index 73 is out of bounds for axis 1 with size 11

[StuartLittlefair]

note also the large numbers of failures in reference aperture fitting. These are 96x72 unbinned pixel windows, with llx=113 and lrx=1825, so for CCD2, reference aperture 1 is pretty close to the centre of the window, but it fails your 'too close' test...

[trmrsh]

Arrggh! Sorry, can you update and try again? A problem is that the error report had a bug so I can't seee what is going on. The position reported is clearly inside the window, but that is the fit window and it should have been the search window. It will go wrong again for sure but the error should be more useful

[StuartLittlefair]

new traceback:

CCD 2, reference aperture 1, fit failed, error = Fitted position (1882.3,26.5) too close to or beyond edge of search window = Winhead(llx=1861, lly=13, nx=11, ny=11, xbin=2, ybin=2, head='NCYCLE  =                    1                                                  \nREFLECT =                    T                                                  \nDSTATUS =                    T                                                  \nMJDUTC  =    58277.01350388381                                                  \nGOODTIME=                    T                                                  \nEXPTIME =                 90.0                                                  ')
CCD 2: no reference aperture fit was successful; skipping
 *** WARNING: CCD 2: no measured FWHM to re-size apertures or carry out optimal extraction; no extraction possible
CCD 3, mean x,y shift from reference aperture(s) = -1.26, -10.32
CCD 4, reference aperture 1, fit failed, error = Fitted position (1878.1,7.7) too close to or beyond edge of search window = Winhead(llx=1859, lly=13, nx=11, ny=11, xbin=2, ybin=2, head='')
CCD 4: no reference aperture fit was successful; skipping
 *** WARNING: CCD 4: no measured FWHM to re-size apertures or carry out optimal extraction; no extraction possible
CCD 5, reference aperture 1, fit failed, error = Fitted position (1857.6,19.3) too close to or beyond edge of search window = Winhead(llx=1855, lly=21, nx=11, ny=11, xbin=2, ybin=2, head='NCYCLE  =                    1                                                  \nREFLECT =                    T                                                  \nDSTATUS =                    T                                                  \nMJDUTC  =    58277.01350388381                                                  \nGOODTIME=                    T                                                  \nEXPTIME =                 90.0                                                  ')
CCD 5: no reference aperture fit was successful; skipping
 *** WARNING: CCD 5: no measured FWHM to re-size apertures or carry out optimal extraction; no extraction possible
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/sl/anaconda3/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/Users/sl/anaconda3/lib/python3.6/multiprocessing/pool.py", line 47, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/Users/sl/anaconda3/lib/python3.6/site-packages/hipercam/scripts/reduce.py", line 3051, in ccdproc
    moveApers(cnam, ccd, flat, rflat, ccdaper, ccdwins, rfile, store)
  File "/Users/sl/anaconda3/lib/python3.6/site-packages/hipercam/scripts/reduce.py", line 1766, in moveApers
    peak = swdata.data[int(round(swdata.y_pixel(y))),int(round(swdata.y_pixel(x)))]
IndexError: index 73 is out of bounds for axis 1 with size 11
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/sl/anaconda3/bin/reduce", line 11, in <module>
    load_entry_point('hipercam==0.8.dev140+g7802f1c', 'console_scripts', 'reduce')()
  File "/Users/sl/anaconda3/lib/python3.6/site-packages/hipercam/scripts/reduce.py", line 755, in reduce
    allres = pool.starmap(ccdproc, arglist)
  File "/Users/sl/anaconda3/lib/python3.6/multiprocessing/pool.py", line 274, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/Users/sl/anaconda3/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
IndexError: index 73 is out of bounds for axis 1 with size 11

[trmrsh]

Could you try just upping the search half width to 15 say? I am wondering if this is just an unlucky case with a genuine large jump early on, unlikely as it seems. The fitted position really does look far off at the start in the traceback, so the error is being reported correctly. It seems to have jumped by ~10 in x (but it could still be a bug so I will keep looking)

[trmrsh]

sorry, stupid bug spotted. Please update & try again. Pathetic.

[StuartLittlefair]

Getting there! This looks like the same issue as the very first bug, but on a different line?

Frame 1: 2018-06-07T04:33:08.5906723 [NOK]
Frame 2: 2018-06-07T04:33:08.5936000 [NOK]
Frame 3: 2018-06-07T04:33:08.5965278 [NOK]
Frame 4: 2018-06-07T04:33:08.5994556 [NOK]
Frame 5: 2018-06-07T04:33:08.6023833 [NOK]
CCD 2, mean x,y shift from reference aperture(s) = -0.43, -0.30
CCD 3, mean x,y shift from reference aperture(s) = -0.40, -0.56
CCD 4, mean x,y shift from reference aperture(s) = -0.53, -0.37
CCD 5, mean x,y shift from reference aperture(s) = -1.08, -0.56
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/sl/anaconda3/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/Users/sl/anaconda3/lib/python3.6/multiprocessing/pool.py", line 47, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/Users/sl/anaconda3/lib/python3.6/site-packages/hipercam/scripts/reduce.py", line 3051, in ccdproc
    moveApers(cnam, ccd, flat, rflat, ccdaper, ccdwins, rfile, store)
  File "/Users/sl/anaconda3/lib/python3.6/site-packages/hipercam/scripts/reduce.py", line 1799, in moveApers
    apsec['fit_thresh'], apsec['fit_ndiv']
  File "/Users/sl/anaconda3/lib/python3.6/site-packages/hipercam/fitting.py", line 110, in combFit
    read, gain, thresh, ndiv
  File "/Users/sl/anaconda3/lib/python3.6/site-packages/hipercam/fitting.py", line 1083, in fitGaussian
    full_output=True
  File "/Users/sl/anaconda3/lib/python3.6/site-packages/scipy/optimize/minpack.py", line 380, in leastsq
    raise TypeError('Improper input: N=%s must not exceed M=%s' % (n, m))
TypeError: Improper input: N=5 must not exceed M=4
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/sl/anaconda3/bin/reduce", line 11, in <module>
    load_entry_point('hipercam==0.8.dev141+g423f90a', 'console_scripts', 'reduce')()
  File "/Users/sl/anaconda3/lib/python3.6/site-packages/hipercam/scripts/reduce.py", line 755, in reduce
    allres = pool.starmap(ccdproc, arglist)
  File "/Users/sl/anaconda3/lib/python3.6/multiprocessing/pool.py", line 274, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/Users/sl/anaconda3/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
TypeError: Improper input: N=5 must not exceed M=4

[trmrsh]

ok, let me get my own reduction going then I can cut down on this thread. Should be a simple one but it will tedious if I keep going back to you for it.

[trmrsh]

seems to be working. Did you re-install / update?

[StuartLittlefair]

I did - something specific to the very small windows? I can grab some hcm files and send them over for you to try?

[trmrsh]

well, the error you had was just a screw-up of mine, at least the first time around, and I am pretty sure I fixed it, but I take it you are still encountering it?

[StuartLittlefair]

i am...

[StuartLittlefair]

did you try running it on the hcm files i sent?

[trmrsh]

i haven't yet because I was leeting it run on AR Sco here. Will copy them over

[trmrsh]

you probably better send me the reduce and aperture files you are using otherwise I probably won't get your error

tom

[trmrsh]

ok, in business, I get the same error.

[StuartLittlefair]

increasing the fit_width fixes the error, but nice to know why

[trmrsh]

Ah. I see the problem. You were sending it a window of size 2x2, and then trying to fit umpteen (well, 5) params to it. Need to think a bit about what to do.

[StuartLittlefair]

could you simply check the binned window size is sensible at the start of the reduction?

[StuartLittlefair]

p.s running a big long effort on the maxi1820 data

[trmrsh]

I guess the only thing I can do is raise a more informative error in this case. Do you actually want the FWHMs in this case? (Could save one parameter by fixing the FWHM). It will be trying to fit the x,y position, the sky level, the height and the FWHM, i.e. 5 params. Of course, I doubt a 4 param fit to a 2x2 window will help much, but I can print an error without actually giving up, or I can just stop at an early stage and say you have to increase fit_width. I will probably do the latter.

[StuartLittlefair]

yes, the latter sounds good. Seems to slow up as we go along - was much slower by frame 5000 than at start. Just checking the master branch to see if it's the same...

[StuartLittlefair]

yes, master branch is no different

[trmrsh]

OK, looks as though we can merge it back then. Do you agree?

Nice to fix this time issue of course, but it's something else. Did you apply the tkeep parameter?

[StuartLittlefair]

yep, we can merge this I think.

I'll raise a new issue for the timing issue,