search

askap-vast / vast-pipeline

This repository holds the code of the Radio Transient detection pipeline for the VAST project.
https://vast-survey.org/vast-pipeline/
MIT License
8 stars 3 forks source link

Error in forced photometry #203

Closed srggrs closed 4 years ago

srggrs commented 4 years ago

When running with the following config:

IMAGE_FILES = sorted(glob.glob("/data/pilot/EPOCH01/COMBINED/STOKESI_IMAGES/*.fits")) + \
        sorted(glob.glob("/data/pilot/EPOCH02/COMBINED/STOKESI_IMAGES/*.fits")) + \
        sorted(glob.glob("/data/pilot/EPOCH03x/COMBINED/STOKESI_IMAGES/*.fits")) + \
        sorted(glob.glob("/data/pilot/EPOCH04x/COMBINED/STOKESI_IMAGES/*.fits")) + \
        sorted(glob.glob("/data/pilot/EPOCH05x/COMBINED/STOKESI_IMAGES/*.fits"))

SELAVY_FILES = sorted(glob.glob("/data/pilot/EPOCH01/COMBINED/STOKESI_SELAVY/*.selavy.components.txt")) + \
        sorted(glob.glob("/data/pilot/EPOCH02/COMBINED/STOKESI_SELAVY/*.selavy.components.txt")) + \
        sorted(glob.glob("/data/pilot/EPOCH03x/COMBINED/STOKESI_SELAVY/*.selavy.components.txt")) + \
        sorted(glob.glob("/data/pilot/EPOCH04x/COMBINED/STOKESI_SELAVY/*.selavy.components.txt")) + \
        sorted(glob.glob("/data/pilot/EPOCH05x/COMBINED/STOKESI_SELAVY/*.selavy.components.txt"))

BACKGROUND_FILES = sorted(glob.glob("/data/pilot/EPOCH01/COMBINED/STOKESI_RMSMAPS/*_bkg.fits")) + \
        sorted(glob.glob("/data/pilot/EPOCH02/COMBINED/STOKESI_RMSMAPS/*_bkg.fits")) + \
        sorted(glob.glob("/data/pilot/EPOCH03x/COMBINED/STOKESI_RMSMAPS/*_bkg.fits")) + \
        sorted(glob.glob("/data/pilot/EPOCH04x/COMBINED/STOKESI_RMSMAPS/*_bkg.fits")) + \
        sorted(glob.glob("/data/pilot/EPOCH05x/COMBINED/STOKESI_RMSMAPS/*_bkg.fits"))

NOISE_FILES = sorted(glob.glob("/data/pilot/EPOCH01/COMBINED/STOKESI_RMSMAPS/*_rms.fits")) + \
        sorted(glob.glob("/data/pilot/EPOCH02/COMBINED/STOKESI_RMSMAPS/*_rms.fits")) + \
        sorted(glob.glob("/data/pilot/EPOCH03x/COMBINED/STOKESI_RMSMAPS/*_rms.fits")) + \
        sorted(glob.glob("/data/pilot/EPOCH04x/COMBINED/STOKESI_RMSMAPS/*_rms.fits")) + \
        sorted(glob.glob("/data/pilot/EPOCH05x/COMBINED/STOKESI_RMSMAPS/*_rms.fits"))

...
MONITOR = True
...
ASSOCIATION_METHOD = 'advanced'

Error stack trace:

2020-07-15 03:42:08,259 logger WARNING FITSFixedWarning: 'datfix' made the change 'Invalid parameter values: MJD-OBS and DATE-OBS are inconsistent.
Set MJD-BEG to 58784.593054 from DATE-BEG.
Set MJD-END to 58784.601463 from DATE-END'.
2020-07-15 03:42:40,962 runpipeline ERROR Processing error:
operands could not be broadcast together with shapes (45,36) (45,35) 

Traceback
---------
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/local.py", line 222, in execute_task
    result = _execute_task(task, data)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/optimization.py", line 991, in __call__
    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/core.py", line 151, in get
    result = _execute_task(task, cache)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/core.py", line 121, in <genexpr>
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/core.py", line 115, in _execute_task
    return [_execute_task(a, cache) for a in arg]
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/core.py", line 115, in <listcomp>
    return [_execute_task(a, cache) for a in arg]
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/utils.py", line 30, in apply
    return func(*args, **kwargs)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/dataframe/core.py", line 5135, in apply_and_enforce
    df = func(*args, **kwargs)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/dataframe/groupby.py", line 174, in _groupby_slice_apply
    return g.apply(func, *args, **kwargs)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/pandas/core/groupby/groupby.py", line 735, in apply
    result = self._python_apply_general(f)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/pandas/core/groupby/groupby.py", line 751, in _python_apply_general
    keys, values, mutated = self.grouper.apply(f, self._selected_obj, self.axis)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/pandas/core/groupby/ops.py", line 206, in apply
    res = f(group)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/pandas/core/groupby/groupby.py", line 719, in f
    return func(g, *args, **kwargs)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/pipeline/pipeline/forced_extraction.py", line 41, in extract_from_image
    cluster_threshold=3,
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/pipeline/pipeline/forced_phot.py", line 410, in measure
    pa[ii], stamps=stamps
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/pipeline/pipeline/forced_phot.py", line 740, in _measure_cluster
    out = fitter(g, xx, yy, d, weights=1.0 / n)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/astropy/modeling/fitting.py", line 176, in wrapper
    return func(self, model, x, y, z=z, **kwargs)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/astropy/modeling/fitting.py", line 885, in __call__
    xtol=acc, full_output=True)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/scipy/optimize/minpack.py", line 388, in leastsq
    shape, dtype = _check_func('leastsq', 'func', func, x0, args, n)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/scipy/optimize/minpack.py", line 26, in _check_func
    res = atleast_1d(thefunc(*((x0[:numinputs],) + args)))
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/astropy/modeling/fitting.py", line 826, in objective_function
    return np.ravel(weights * (model(*args[2: -1]) - meas))
Traceback (most recent call last):
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/pipeline/management/commands/runpipeline.py", line 86, in handle
    pipeline.process_pipeline(p_run)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/pipeline/pipeline/main.py", line 111, in process_pipeline
    self.config.MONITOR_MIN_SIGMA
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/pipeline/pipeline/forced_extraction.py", line 218, in forced_extraction
    extr_df = parallel_extraction(extr_df, images_df, sources_df, min_sigma)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/pipeline/pipeline/forced_extraction.py", line 147, in parallel_extraction
    .compute(num_workers=n_cpu, scheduler='processes')
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/base.py", line 166, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/base.py", line 437, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/multiprocessing.py", line 222, in get
    **kwargs
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/local.py", line 486, in get_async
    raise_exception(exc, tb)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/multiprocessing.py", line 118, in reraise
    raise exc
dask.multiprocessing.ValueError: operands could not be broadcast together with shapes (45,36) (45,35) 

Traceback
---------
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/local.py", line 222, in execute_task
    result = _execute_task(task, data)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/optimization.py", line 991, in __call__
    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/core.py", line 151, in get
    result = _execute_task(task, cache)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/core.py", line 121, in <genexpr>
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/core.py", line 115, in _execute_task
    return [_execute_task(a, cache) for a in arg]
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/core.py", line 115, in <listcomp>
    return [_execute_task(a, cache) for a in arg]
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/utils.py", line 30, in apply
    return func(*args, **kwargs)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/dataframe/core.py", line 5135, in apply_and_enforce
    df = func(*args, **kwargs)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/dask/dataframe/groupby.py", line 174, in _groupby_slice_apply
    return g.apply(func, *args, **kwargs)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/pandas/core/groupby/groupby.py", line 735, in apply
    result = self._python_apply_general(f)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/pandas/core/groupby/groupby.py", line 751, in _python_apply_general
    keys, values, mutated = self.grouper.apply(f, self._selected_obj, self.axis)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/pandas/core/groupby/ops.py", line 206, in apply
    res = f(group)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/pandas/core/groupby/groupby.py", line 719, in f
    return func(g, *args, **kwargs)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/pipeline/pipeline/forced_extraction.py", line 41, in extract_from_image
    cluster_threshold=3,
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/pipeline/pipeline/forced_phot.py", line 410, in measure
    pa[ii], stamps=stamps
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/pipeline/pipeline/forced_phot.py", line 740, in _measure_cluster
    out = fitter(g, xx, yy, d, weights=1.0 / n)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/astropy/modeling/fitting.py", line 176, in wrapper
    return func(self, model, x, y, z=z, **kwargs)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/astropy/modeling/fitting.py", line 885, in __call__
    xtol=acc, full_output=True)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/scipy/optimize/minpack.py", line 388, in leastsq
    shape, dtype = _check_func('leastsq', 'func', func, x0, args, n)
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/scipy/optimize/minpack.py", line 26, in _check_func
    res = atleast_1d(thefunc(*((x0[:numinputs],) + args)))
  File "/home/ubuntu/sergio-vast-pipeline-testing/vast-pipeline/venv/lib/python3.7/site-packages/astropy/modeling/fitting.py", line 826, in objective_function
    return np.ravel(weights * (model(*args[2: -1]) - meas))
ajstewart commented 4 years ago

This is the annoying edge case, tweak the buffer parameter that I implemented, try 1.5.

ajstewart commented 4 years ago

see #191.

srggrs commented 4 years ago

Okay the code was update up to #184, so your fixes were not there... let's see now that I will update it if it works with no breaks.

srggrs commented 4 years ago

Fixed by #195