How to run contrib scripts in python?

[hamogu]

I'm trying to use "download_chandra_obsid" from a python session (using the CIAO provided python), but I can't import it:

from ciao_contrib.runtool import download_chandra_obsid
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-14-b080b19ffad8> in <module>()
      1 import os
----> 2 from ciao_contrib.runtool import download_chandra_obsid
      3 
      4 downdir = '/melkor/d1/guenther/downdata/Chandra/grades/'

ImportError: cannot import name 'download_chandra_obsid'

I guess that's because download_chandra_obsid is not a CIAO tool, but a contributed script. Is there another way to get it from runtool? (Don't tell me about subprocess, I know I can do that, but I'm wondering if it is or should be available from runtool).

Most users don't know the difference between CIAO tools and contributed scripts as they are downloaded together, installed together and documented together, so it's surprising that I can import one, but not the other.

[DougBurke]

runtool doesn't support download_chandra_obsid. For historical reasons, download_chandra_obsid doesn't use the CIAO parameter library, and runtool only supports those scripts with a parameter file interface (since it uses the parameter interface to build the Python interface).

[DougBurke]

You can run other contributed scripts - e.g. merge_obs - from runtool because they use the parameter interface.

[DougBurke]

If you want to download Chandra data from the archive from Python then the code that download_chandra_obsid uses is available from the ciao_contrib.cda.data module - try ahelp cda_data for more information.

[hamogu]

That's great. (I first started coding that up with python's ftplib, but ended up just using subprocess and call). Do you think this is a unusual case or should we find a way to make that easier to discover (e.g. add a a stub download_chandra_obsid to runtool that, when imported, just puts out the text that you had above)? Do you have an idea how many cases like that there are? Or is this the only outlier?

[DougBurke]

There are a number of scripts in bin/ which do not have a parameter file.

Some are "internal" which it doesn't make sense to expose - I'm primarily thinking about the ds9_* scripts but there may be others.

Here's the current list of files in bin/ which don't have a param/*.par file:

acis_clear_status_bits
check_ciao_caldb
check_ciao_version
convert_xspec_script
convert_xspec_user_model
download_chandra_obsid
ds9_aper.sh
ds9_functs
ds9_imgfit.sh
ds9_imgproc_wrapper
ds9_lut
ds9_plot_blt
ds9_radial_fitacis_clear_status_bits
check_ciao_caldb
check_ciao_version
convert_xspec_script
convert_xspec_user_model
download_chandra_obsid
ds9_aper.sh
ds9_functs
ds9_imgfit.sh
ds9_imgproc_wrapper
ds9_lut
ds9_plot_blt
ds9_radial_fit
ds9_snap2.sh
ds9_specfit.sh
ds9_tgcoord.sh
install_marx
splitroi
summarize_status_bits
tg_bkg

ds9_snap2.sh
ds9_specfit.sh
ds9_tgcoord.sh
install_marx
splitroi
summarize_status_bits
tg_bkg

I am reluctant to add "stub" modules to map from a parameter interface to whatever interface a particular script supports since it is too complex (and runtool is complex enough as is).

What might be better is to ask whether we can add a parameter interface to the scripts we would want to support. The download_chandra_obsid case is because this was originally written outside of the "ciao contrib" package for various political reasons, and could now be re-worked. The issue there is that it then changes the interface for the tool and so we'd need to work out individually a plan for each script [*], and I personally don't have the time or energy for this.

[*] since each script is different in its own special way