https://cxc merge issues

[kglotfelty]

Not sure if there will be problems with the https:// transistion and the scripts. I could see problems with

• [ ] search_csc uses the limiting sensitivity service, /csc/sens/sensmap.php
• [x] Both search_csc scripts access a lookup table, /ciao/threads/csccli/cscrel1_version_info.txt to lookup version numbers for csc1 files.
• [x] Do the check_ciao script(s) access version info via a http:// request? (addressed in #329)
• [x] install_marx check for wget logic (as per @DougBurke comment below)

Not sure if the magic curl fall-over will work here for free or if there's more that needs to be done.

[DougBurke]

I think install_marx needs to be looked at as well.

The check_ciao_version and check_ciao_caldb scripts will need work: they rely on ciao_contrib/_tools/versioninfo.py and ciao_contrib/caldb.py.

Looks like there's a message to the user in chandra_repro that will also need updating

From the ciao-4.11/contrib dir I get the following (most are just comments):

% find . -type f -exec grep 'http://cxc' {} /dev/null \; | grep -v share/doc/xml
./lib/python3.5/site-packages/lightcurves.py:  http://cxc.harvard.edu/ciao/threads/acisbackground/
./lib/python3.5/site-packages/lightcurves.py:  http://cxc.harvard.edu/ciao/threads/filter_ltcrv/
./lib/python3.5/site-packages/chips_contrib/scatter.py:    Model, described in http://cxc.harvard.edu/ciao/ahelp/dmascii.html
./lib/python3.5/site-packages/sherpa_contrib/xspec/xsusermodels.py:http://cxc.harvard.edu/helpdesk/ - if you have problems.
./lib/python3.5/site-packages/sherpa_contrib/xspec/xsconvolve.py:http://cxc.harvard.edu/helpdesk/
./lib/python3.5/site-packages/sherpa_contrib/xspec/xsmodels.py:http://cxc.harvard.edu/helpdesk/ - if you have problems.
./lib/python3.5/site-packages/sherpa_contrib/chart.py:`ChART version 2 <http://cxc.harvard.edu/ciao/PSFs/chart2/>`_.
./lib/python3.5/site-packages/sherpa_contrib/utils.py:<http://cxc.harvard.edu/ciao/threads/spectral_weights/>`_ thread
./lib/python3.5/site-packages/ciao_contrib/cda/data.py:http://cxc.harvard.edu/cda/ - supporting mirror sites under the
./lib/python3.5/site-packages/ciao_contrib/cda/search.py:    http://cxc.harvard.edu/cda/footprint/cdaview.html
./lib/python3.5/site-packages/ciao_contrib/cda/csccli.py:        tab = make_URL_request( "http://cxc.harvard.edu/ciao/threads/csccli/cscrel1_version_info.txt", {} )
./lib/python3.5/site-packages/ciao_contrib/cda/csccli.py:    resource = "http://cxc.cfa.harvard.edu/csc/sens/sensmap.php"
./lib/python3.5/site-packages/ciao_contrib/caldb.py:release_notes_url = "http://cxc.harvard.edu/caldb/downloads/releasenotes.html"
./lib/python3.5/site-packages/ciao_contrib/caldb.py:      http://cxc.harvard.edu/caldb/
./lib/python3.5/site-packages/ciao_contrib/caldb.py:    access the Chandra CALDB web site: http://cxc.harvard.edu/caldb/
./lib/python3.5/site-packages/ciao_contrib/caldb.py:        (as obtained from http://cxc.harvard.edu/caldb/). If not
./lib/python3.5/site-packages/ciao_contrib/runtool.py:parameter" or http://cxc.harvard.edu/ciao/ahelp/parameter.html for
./lib/python3.5/site-packages/ciao_contrib/proptools.py:  http://cxc.harvard.edu/ciao/ahelp/modelflux.html
./lib/python3.5/site-packages/ciao_contrib/proptools.py:  http://cxc.harvard.edu/ciao/ahelp/ciao_runtool.html
./lib/python3.5/site-packages/ciao_contrib/proptools.py:    See http://cxc.harvard.edu/ciao/ahelp/colden.html for more information
./lib/python3.5/site-packages/ciao_contrib/proptools.py:    See http://cxc.harvard.edu/ciao/ahelp/precess.html for more information
./lib/python3.5/site-packages/ciao_contrib/proptools.py:    http://cxc.harvard.edu/ciao/ahelp/prop-coords.html for coordinate systems.
./lib/python3.5/site-packages/ciao_contrib/proptools.py:    See http://cxc.harvard.edu/ciao/ahelp/dates.html for more information
./lib/python3.5/site-packages/ciao_contrib/proptools.py:    http://cxc.harvard.edu/ciao/ahelp/prop-time.html for the calendars.
./lib/python3.5/site-packages/ciao_contrib/ancillaryfiles.py:    .. [1] http://cxc.harvard.edu/ciao/data_products_guide/
./lib/python3.5/site-packages/ciao_contrib/_tools/versioninfo.py:    query pages at http://cxc.harvard.edu/ciao/download/.
./lib/python3.5/site-packages/ciao_contrib/_tools/versioninfo.py:    base_url = "http://cxc.harvard.edu/ciao/download/"
./bin/make_psf_asymmetry_region:  http://cxc.harvard.edu/ciao/caveats/psf_artifact.html
./bin/chandra_repro: http://cxc.harvard.edu/ciao/threads/data.html
./bin/chandra_repro:        v1("\nWARNING: Before reprocessing CC-mode data, users should check\n         the accuracy of the source coordinates (RA_TARG,DEC_TARG)\n         in the file header:\n         http://cxc.harvard.edu/ciao/why/ccmode.html#coord")
./bin/check_ciao_caldb:  site at http://cxc.harvard.edu/caldb/.
./bin/check_ciao_version:    print("    http://cxc.harvard.edu/ciao/download/")
./bin/fullgarf:#   http://cxc.harvard.edu/ciao/threads/
./bin/readout_bkg:# http://cxc.harvard.edu/contrib/maxim/make_readout_bg
./bin/acis_clear_status_bits:  http://cxc.harvard.edu/ciao/threads/createL2/
./bin/specextract:        # cf. http://cxc.cfa.harvard.edu/cal/Hrc/detailed_info.html#rmf for HRC RMF information
./bin/specextract:    ##    raise ValueError("Support for ptype=PHA has been removed from specextract. Please contact the CXC HelpDesk at http://cxc.harvard.edu/helpdesk/ if you need to use this option.")
./bin/install_marx:wget http://cxc.harvard.edu/ciao/index.html > /dev/null 2>&1 
./bin/install_marx:  lwp-download http://cxc.harvard.edu/ciao/index.html > /dev/null 2>&1 

[kglotfelty]

Confirm that csc scripts fall over nicely to using curl for the csc1 file retrieve bit.

However, search_csc fails to access the csc1 limiting sensitivity server.

[DougBurke]

Are the CDA services also transitioning to https at the same time then?

Is the search_csc failure just because it doesn't use the "wrapper" routine to access the URL, or something deeper?

[kglotfelty]

I haven't heard anything about CDA.

search_csc failure is 'deeper'.  For the csc1 sensitivity map search, it does fall through to use curl -- but -- the .php page doesn't accept GET requests, only POST requests so it fails. [This is independent of http vs. https.]

To get csc1 sens working in 4.11 would need to change the curl command to use --form's instead of a URL encoding.

The csc1 file-retrieve in both scripts -- also falls through to curl and works.

On 11/13/19 7:51 AM, Doug Burke wrote:

Are the CDA services also transitioning to https at the same time then?

Is the |search_csc| failure just because it doesn't use the "wrapper" routine to access the URL, or something deeper?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/cxcsds/ciao-contrib/issues/266?email_source=notifications&email_token=ABS2QIZZEFVBGK33UK6GTTTQTPZ6HA5CNFSM4IVJU6CKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOED6BHII#issuecomment-553391009, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABS2QIZ4QBCGGUAYM3KBMWDQTPZ6HANCNFSM4IVJU6CA.

--

Kenny Glotfelty Chandra X-ray Center Center for Astrophysics | Harvard & Smithsonian 617-496-7773 60 Garden Street | MS 21 | Cambridge MA 02138

[DougBurke]

Oh, the CSC1 sensitivity service is actually being served from cxc.harvard.edu and not cda.blah.blah. That's why there's an issue.

[kglotfelty]

:+1:

[DougBurke]

The check_ciao_xxx scripts access via http URLs (well, the master branch now has them using https). I've updated the code in check_ciao_version to use the "fall over to curl" routine just for funsies.

These two (check_ciao_version / check_ciao_caldb) need to be updated to work with conda, but that's a different issue.

[DougBurke]

@kglotfelty can this be closed now?

[kglotfelty]

yes.