search

svalenti / pessto

pessto pipeline
MIT License
9 stars 6 forks source link

SOFI astrometry fails because querycatalogue gets blank response using iraf.noao.astcat.agetcat #43

Closed genghisken closed 8 years ago

genghisken commented 8 years ago

When doing SOFI astrometry, for some images, the PESSTO code bypasses vizier and does a query via IRAF (iraf.noao.astcat.agetcat). The response from the IRAF catalogue server is blank (just contains a header), so the code fails. I added an extra line to print out the catalogue response. The code fails for any SOFI image (or indeed any image at all). I called the code that fails directly thus:

>>> from efoscastrodef import querycatalogue
>>> c = querycatalogue('2mass','/tmp/pessto/testdata/sofi/LARGE_FIELD_IMAGING/SOFI.2013-01-28T17:09:34.393.fits')
# BEGIN CATALOG HEADER
# catdb /Users/kws/nttpipeline226/lib/python2.7/site-packages/ntt/standard/cat/catalogue.dat
# catname twomass@noao
# nquery 4
#     ra 19:39:22.19 hours
#     dec -23:17:24.4 degrees
#     radius 7.5 minutes
#     qsystem J2000.0 INDEF
# type stext
# nheader 1
#     csystem J2000.0
# nfields 7
#     id 1 0 c INDEF %14s
#     ra 2 0 d degrees %11.6f
#     dec 3 0 d degrees %11.6f
#     mag1 4 0 r INDEF %7.3f
#     mag2 5 0 r INDEF %7.3f
#     mag3 6 0 r INDEF %7.3f
#     dist 7 0 r minutes %6.2f
# END CATALOG HEADER
#
      body {
        background-color: #fff;
      }
      .message_content_header {
        background-color: #b7cdee;
        color: #000;
      }
      .message_content_body {
        color: #000;
      }
      .message_content_end {
        background-color: #b7cdee;
      }
    </style>
  </head>
  <body>
    <table border="0" width="100%">
      <tbody>
        <tr>
          <td class="message_content_header">
            <b>Not Found</b>
          </td>
        </tr>
      </tbody>
    </table>
    <blockquote class="message_content_body">
      <h1>The page you were looking for doesn't exist.</h1>
      You may have mistyped the address or the page may have moved.
      <p>
    </blockquote>
    <table width="100%" cellpadding="0" cellspacing="0">
      <tr>
        <td class="message_content_end">
          <img alt="" width="1" height="4">
        </td>
      </tr>
    </table>
  </body>
</html>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "efoscastrodef.py", line 1248, in querycatalogue
    stdcoo[j].append(i.split()[column[j] - 1])
IndexError: list index out of range
>>>
genghisken commented 8 years ago

FYI here is the message from the actual reduction code:

Traceback (most recent call last):
  File "/Users/eskank/tools/nttpipeline/bin/PESSTOSOFIPHOT", line 5, in <module>
    pkg_resources.run_script('ntt==2.2.9', 'PESSTOSOFIPHOT')
  File "/Users/eskank/tools/Ureka/python/lib/python2.7/site-packages/pkg_resources.py", line 528, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/Users/eskank/tools/Ureka/python/lib/python2.7/site-packages/pkg_resources.py", line 1394, in run_script
    execfile(script_filename, namespace, namespace)
  File "/Users/eskank/tools/nttpipeline/lib/python2.7/site-packages/ntt-2.2.9-py2.7.egg/EGG-INFO/scripts/PESSTOSOFIPHOT", line 98, in <module>
    _interactive, _regi, _verbose, method)
  File "/Users/eskank/tools/nttpipeline/lib/python2.7/site-packages/ntt-2.2.9-py2.7.egg/ntt/sofiphotredudef.py", line 1059, in sofireduction
    result = ntt.efoscastrodef.zeropoint(nameobjnew, '2mass', False, False)
  File "/Users/eskank/tools/nttpipeline/lib/python2.7/site-packages/ntt-2.2.9-py2.7.egg/ntt/efoscastrodef.py", line 293, in zeropoint
    stdcoo = querycatalogue('2mass', img)
  File "/Users/eskank/tools/nttpipeline/lib/python2.7/site-packages/ntt-2.2.9-py2.7.egg/ntt/efoscastrodef.py", line 1245, in querycatalogue
    stdcoo[j].append(i.split()[column[j] - 1])
IndexError: list index out of range
svalenti commented 8 years ago

please collect in a tar file a set of images for which this happen and I will take a look

thespacedoctor commented 8 years ago

I think I've fixed this issue. It was occurring in PESSTOSOFIPHOT within the querycatalogue function of the efoscastrodef.py module.

Whenever not enough catalogued sources are found in the image (<10 I think) as returned from Vizier's copy of 2MASS, the pipeline turns to another remote 2MASS catalogue source. I'm not entirely sure why a different version of 2MASS would return more sources? Is there a magnitude cut in what is returned form Vizier Stefano? Here is an example image where only 7 sources are matched with the Vizier catalogue:

Anyway, the remote catalogue the pipeline was defaulting to (archive.tuc.noao.edu) is now redundant, hence the pipeline was falling over.

I've replaced the catalogue source with the version at IPAC and adapted the code to parse the version of the catalogue from IPAC correctly.

I've only run a couple of tests, but PESSTOSOFIPHOT now seems to succeed at the astrometry phase.

Erkki can you please test with a few images and then either close this issue or report back with any other related failures? Fixes in v2.2.10

ekankare commented 8 years ago

I tried a few different fields and the v2.2.10 seems to be working. No crashing with PESSTOSOFIPHOT. With couple of fields the final astrometry was not correct, but can be corrected with PESSTOASTRO. Unless someone tells me otherwise I will start SOFI image reduction with this version.

I also get the following error both with v2.2.9 and v2.2.10 when the pipeline is running sextractor in PESSTOSOFIPHOT. It does not crash the pipeline, so I wonder if this is something to actually worry about it. The astrometry can still look fine in the final product.

/Users/eskank/tools/Ureka/python/lib/python2.7/site-packages/numpy/core/_methods.py:55: RuntimeWarning: Mean of empty slice. warnings.warn("Mean of empty slice.", RuntimeWarning) /Users/eskank/tools/Ureka/python/lib/python2.7/site-packages/numpy/core/_methods.py:67: RuntimeWarning: invalid value encountered in double_scalars ret = ret.dtype.type(ret / rcount) /Users/eskank/tools/Ureka/python/lib/python2.7/site-packages/numpy/core/_methods.py:79: RuntimeWarning: Degrees of freedom <= 0 for slice warnings.warn("Degrees of freedom <= 0 for slice", RuntimeWarning) /Users/eskank/tools/Ureka/python/lib/python2.7/site-packages/numpy/core/_methods.py:91: RuntimeWarning: invalid value encountered in true_divide arrmean, rcount, out=arrmean, casting='unsafe', subok=False) /Users/eskank/tools/Ureka/python/lib/python2.7/site-packages/numpy/core/_methods.py:113: RuntimeWarning: invalid value encountered in double_scalars ret = ret.dtype.type(ret / rcount)

I think it happens frequently, but this e.g. was done with PESSTOESO154-G10 files (final astrometry ok): SOFI.2014-12-15T06:35:05.599.fits SOFI.2014-12-15T06:35:53.857.fits SOFI.2014-12-15T06:36:42.401.fits SOFI.2014-12-15T06:37:30.607.fits

svalenti commented 8 years ago

I do not think these are killing warnings. If the astrometry is correct, I would proceed.

ekankare commented 8 years ago

I reduced the SOFI imaging with pipeline version 2.2.10. Closing this issue.

svalenti commented 8 years ago

great !

well done erkki !

On 8/25/16 4:47 AM, Erkki Kankare wrote:

I reduced the SOFI imaging with pipeline version 2.2.10. Closing this issue.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/svalenti/pessto/issues/43#issuecomment-242358626, or mute the thread https://github.com/notifications/unsubscribe-auth/AHcAnZHN5puDr6dNOenELYXKzCgMZ8Cmks5qjYDSgaJpZM4JmVjo.