Adding some example/demo script

[mtewes]

Including a small FITS image to play with. I'll do so, to test #3.

[reikonakajima]

I ran the minimal demo script, and got this error:

 $ python minimal_demo.py 
No handlers could be found for logger "sewpy.sewpy"
Traceback (most recent call last):
  File "minimal_demo.py", line 18, in <module>
    out = sew("image.fits")
  File "/vol/aibn61/aibn61_2/reiko/2code/sewpy/sewpy/sewpy.py", line 472, in __call__
    format="ascii.sextractor")
  File "/users/reiko/.local/lib/python2.7/site-packages/astropy-0.4.2-py2.7-linux-x86_64.egg/astropy/table/table.py", line 1729, in read
    return io_registry.read(cls, *args, **kwargs)
  File "/users/reiko/.local/lib/python2.7/site-packages/astropy-0.4.2-py2.7-linux-x86_64.egg/astropy/io/registry.py", line 330, in read
    data = reader(*args, **kwargs)
  File "/users/reiko/.local/lib/python2.7/site-packages/astropy-0.4.2-py2.7-linux-x86_64.egg/astropy/io/ascii/connect.py", line 36, in io_read
    return read(filename, format=format, **kwargs)
  File "/users/reiko/.local/lib/python2.7/site-packages/astropy-0.4.2-py2.7-linux-x86_64.egg/astropy/io/ascii/ui.py", line 132, in read
    dat = _guess(table, new_kwargs)
  File "/users/reiko/.local/lib/python2.7/site-packages/astropy-0.4.2-py2.7-linux-x86_64.egg/astropy/io/ascii/ui.py", line 174, in _guess
    dat = reader.read(table)
  File "/users/reiko/.local/lib/python2.7/site-packages/astropy-0.4.2-py2.7-linux-x86_64.egg/astropy/io/ascii/sextractor.py", line 55, in read
    output = core.BaseReader.read(self, table)
  File "/users/reiko/.local/lib/python2.7/site-packages/astropy-0.4.2-py2.7-linux-x86_64.egg/astropy/io/ascii/core.py", line 856, in read
    self.lines = self.inputter.get_lines(table)
  File "/users/reiko/.local/lib/python2.7/site-packages/astropy-0.4.2-py2.7-linux-x86_64.egg/astropy/io/ascii/core.py", line 146, in get_lines
    with get_readable_fileobj(table) as file_obj:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/users/reiko/.local/lib/python2.7/site-packages/astropy-0.4.2-py2.7-linux-x86_64.egg/astropy/utils/data.py", line 199, in get_readable_fileobj
    fileobj = open(name_or_obj, 'rb')
IOError: [Errno 2] No such file or directory: '/tmp/sextractor_dot_py_workdir_dTInaJ/image.cat.txt'

[reikonakajima]

The content of my tmp file is

 $ ls /tmp/sextractor_dot_py_workdir_dTInaJ/
config.txt  conv.txt  image.log.txt  params.txt

[mtewes]

OK, that's good to know. Probably sextractor did not run (whatever the reason), and this should lead to a warning or an exception raised before we try to access the catalog file. I'll break my SExtractor and see what happens to improve this.

[mtewes]

Is there anything interesting (or anything at all) in the log ? The path is /tmp/sextractor_dot_py_workdir_dTInaJ/image.log.txt

[reikonakajima]

I re-ran, and here is the error message:

 $ python minimal_demo.py
INFO: sewpy.sewpy(__init__): SExtractor version is 2.19.5
INFO: sewpy.sewpy(__call__): Preparing to run SExtractor on image.fits...
DEBUG: sewpy.sewpy(__call__): Using imgname image...
DEBUG: sewpy.sewpy(_write_default_config): Wrote /tmp/sewpy_workdir_5CRDeq/config.txt
DEBUG: sewpy.sewpy(_write_params): Wrote /tmp/sewpy_workdir_5CRDeq/params.txt
DEBUG: sewpy.sewpy(_write_default_conv): Wrote /tmp/sewpy_workdir_5CRDeq/conv.txt
INFO: sewpy.sewpy(__call__): Starting SExtractor now, with niceness None...
DEBUG: sewpy.sewpy(__call__): Running with command ['sex', 'image.fits', '-c', '/tmp/sewpy_workdir_5CRDeq/config.txt', '-CATALOG_NAME', '/tmp/sewpy_workdir_5CRDeq/image.cat.txt', '-PARAMETERS_NAME', '/tmp/sewpy_workdir_5CRDeq/params.txt', '-FILTER_NAME', '/tmp/sewpy_workdir_5CRDeq/conv.txt', '-PHOT_APERTURES', '5, 10, 20', '-DETECT_MINAREA', '10']...
INFO: sewpy.sewpy(__call__): SExtractor stderr:
INFO: sewpy.sewpy(__call__): 
> *Error*: Too many input images: 20

CRITICAL: sewpy.sewpy(__call__): Ouch, something seems wrong, check SExtractor log
INFO: sewpy.sewpy(__call__): Running SExtractor done, it took 0.04 seconds.
Traceback (most recent call last):
  File "minimal_demo.py", line 23, in <module>
    out = sew("image.fits")
  File "/vol/aibn61/aibn61_2/reiko/2code/sewpy/sewpy/sewpy.py", line 474, in __call__
    raise RuntimeError("It seems that SExtractor did not write the file '%s', check log." % (self._get_cat_filepath(imgname)))
RuntimeError: It seems that SExtractor did not write the file '/tmp/sewpy_workdir_5CRDeq/image.cat.txt', check log.

[reikonakajima]

...and the log file is

 $ cat /tmp/sewpy_workdir_5CRDeq/image.log.txt 
SExtractor was called with :
sex image.fits -c /tmp/sewpy_workdir_5CRDeq/config.txt -CATALOG_NAME /tmp/sewpy_workdir_5CRDeq/image.cat.txt -PARAMETERS_NAME /tmp/sewpy_workdir_5CRDeq/params.txt -FILTER_NAME /tmp/sewpy_workdir_5CRDeq/conv.txt -PHOT_APERTURES 5, 10, 20 -DETECT_MINAREA 10

A nicer view of the config:
                  CATALOG_NAME : /tmp/sewpy_workdir_5CRDeq/image.cat.txt
               PARAMETERS_NAME : /tmp/sewpy_workdir_5CRDeq/params.txt
                   FILTER_NAME : /tmp/sewpy_workdir_5CRDeq/conv.txt
                PHOT_APERTURES :                      5, 10, 20
                DETECT_MINAREA :                             10

####### stdout #######

####### stderr #######

> *Error*: Too many input images: 20

which is similar to the log file from the first run.

[reikonakajima]

Thanks for looking into this.

[mtewes]

Woah, this is strange. I've never seen SExtractor complaining about "too many input images" before. At least it crashes quite nicely now...

I'm also using SExtractor 2.19.5. Maybe the image file got corrupted ? This is not urgent, don't worry if you have no time for it this week!

[reikonakajima]

So I found something interesting---I deleted image.fits and it still runs, complaining the there are too many input images. I'll take a look...

[reikonakajima]

If I run the sex command as stated in the log file, I get the "too many input images" error. If I run sex image.fits, then (after specifying params and conv.txt) it complains that it cannot find image.fits.

[reikonakajima]

Aha! I found it. There is a space where there shouldn't be in the following command:

 sex image.fits -c /tmp/sewpy_workdir_hFkifu/config.txt -CATALOG_NAME /tmp/sewpy_workdir_hFkifu/image.cat.txt -PARAMETERS_NAME /tmp/sewpy_workdir_hFkifu/params.txt -FILTER_NAME /tmp/sewpy_workdir_hFkifu/conv.txt -PHOT_APERTURES 5, 10, 20 -DETECT_MINAREA 10

It's the space between "5, 10, 20" that SExtractor gets confused about. We shall remove this space in the command line, and see what happens.

[reikonakajima]

Yes, works fine without the space.

[kuntzer]

Nice!

On 16. 10. 14 11:23, Reiko Nakajima wrote:

Yes, works fine without the space.

— Reply to this email directly or view it on GitHub https://github.com/megalut/sewpy/issues/4#issuecomment-59335589.

Thibault Kuntzer Laboratoire d'astrophysique Ecole Polytechnique Fédérale de Lausanne (EPFL) Observatoire de Sauverny CH-1290 Versoix Switzerland

[reikonakajima]

I've updated the demo script (the only change necessary). Should we merge this branch to master, then close the issue?

[mtewes]

Cool, thanks indeed. Maybe we can fix this by writing "5, 10, 20" instead of 5, 10, 20 ?

[reikonakajima]

oh, let me try that.

[reikonakajima]

No, the following didn't work:

sew = sewpy.SEW(
                params=["X_IMAGE", "Y_IMAGE", "FLUX_APER(3)", "FLAGS"],
                config={"DETECT_MINAREA":10, "PHOT_APERTURES":'"5, 10, 20"'},
                sexpath="sex"
        )

[mtewes]

OK I will investigate as well if we can somehow "escape" some delimiters here. It would be good to have the wrapper take care of this. The user should still be allowed to just give the string exactly as it would have to be written in default.sex

[reikonakajima]

Thanks!

[mtewes]

Still haven't reproduced this (no access to the mentioned machine). @reikonakajima, can we try to find a solution for this today ?

[reikonakajima]

Sure. What can I do to help?

[reikonakajima]

Actually, I think you might want to get an euclid machine account. No? :)

[reikonakajima]

Hi Malte,
I tried various things for the multiple parameter case, and none of them work, as long as I have space in between them.

Even if I put a quote (or quotes) around it, it seems to think that it either has no quotes (> *Error*: Too many input images: 20), or that it has the wrong quote (> *Error*: Too many input images: 20').

I can put quotes around it, and as long as there are no spaces in between the 5,10,20, they will run fine (with quotes around the other parameters as well).

The Extractor Manual says "separated by a comma". Although it doesn't necessarily say "no spaces", none of the examples have spaces in them. I don't think it's supposed to work with spaces in between!

[reikonakajima]

So as we have discussed, I will simply remove any whitespace in any of the parameter items at sewpy.py. This should work.

[reikonakajima]

The simple fix with

            popencmd.append(str(value).replace(' ',''))

on line 441 of sewpy.py works.

[reikonakajima]

Pushed.

[reikonakajima]

Since the pull request for this one is closed, I have merged this branch to master. I think we can now close this issue.