JSON output for web scraping

[rubyvanrooyen]

Moving toward TOM Toolkit compatibility, the current web scripting scripts should accommodate JSON output as well as strings to file

This will allow the remote broker output to be ingested directly into the SALTVisibility evaluation as an object (dictionary) to search for on sky visibility period as well as possible overlap.

Current list of python scripts:

readCHANDRAwebdata.py
readGRBwebdata.py
readINTEGRALwebdata.py
readNICERwebdata.py
readNUSTARwebdata.py
readSWIFTwebdata.py
readXMMwebdata.py

[rubyvanrooyen]

Currently the main function holds all the implementation, example script readINTEGRALwebdata.py

With the output target strings generated for what appears to be 3 implementation scripts

# outputs for Steve Potters IDL routines
print(source[k],'\t',RAhrs[k],RAmins[k],int(float(RAsecs[k])),str(DECsign[k])+str(DECdeg[k]),DECarcmins[k],int(float(DECarcsecs[k])), file=listintegral)
...
# OUTPUT for SALTvisibility.py routine to determine SALT observability windows
print(source[k],'\t',RAhrs[k],RAmins[k],RAsecs[k],'\t',str(DECsign[k])+str(DECdeg[k]),DECarcmins[k],DECarcsecs[k],'\t','2000', file=targets)
...
# OUTPUT for GTIvsSALT.py routine to determine overlaps
print(source[k],'\t',str(startGTI[k]).replace('-',':').replace(' ',':'),'\t',str(endGTI[k]).replace('-',':').replace(' ',':'),'\t',duration[k],'\t','INTEGRAL', file=gtis)

Suggestion is to refractor the implementation into multiple modules:

• input (extract data from web page)
• build a unified JSON dictionary containing all relevant information (this can be called by another script acting as SALT local broker (remote broker)
• add the file output as a function (this will ensure the existing system continues to work and act as independent verification)

[rubyvanrooyen]

Current output from web scraping scripts

Output strings

• per target output for Steve Potters IDL routines print(f'{source} \t {RAhrs} {RAmins} {int(float(RAsecs))} {str(DECsign)}{str(DECdeg)} {DECarcmins} {int(float(DECarcsecs))}')
• per target output for SALTvisibility.py routine to determine SALT observability windows print(f'{source} \t {RAhrs} {RAmins} {RAsecs} {str(DECsign)}{str(DECdeg)} {DECarcmins} {DECarcsecs} \t 2000')
• per target output for GTIvsSALT.py routine to determine overlaps
  print(f'{source} \t {str(startGTI).replace('-',':').replace(' ',':')} \t {str(endGTI).replace('-',':').replace(' ',':')} \t {duration} \t INTEGRAL')

Suggested update to more general output from web scraping scripts

(See issue #5 for GTIvsSALT.py summary of output print statements) Required

name = f'{source}'
ra_str = f'{RAhrs} {RAmins} {int(float(RAsecs))}'
dec_str = f'{str(DECsign)}{str(DECdeg)} {DECarcmins} {int(float(DECarcsecs))}'
start_time = f'{str(startGTI).replace('-',':').replace(' ',':')}'
end_time = f'{str(endGTI).replace('-',':').replace(' ',':')}'
duration = duration

Optional or TBD

epoch = '2000'
salt_win_start = f'{str(SALTstart).replace('-',':').replace(' ',':')}'
salt_win_end = f'{str(SALTend).replace('-',':').replace(' ',':')}'
satellite = satellite

[Option 1]: Output strings

• per target outputs for Steve Potters IDL routines
  print(f'{name} \t {ra_str} {dec_str}')
• per target output for SALTvisibility.py routine to determine SALT observability windows print(f'{name} \t {ra_str} {dec_str} \t {epoch}')
• per target output for GTIvsSALT.py routine to determine overlaps
  print(f'{name} \t {start_time} \t {end_time} \t {duration} \t INTEGRAL')

[Option 2]: Suggested JSON arguments:

[
  {
    "name":<str>, (default = name)
    "id":<int>, (default = iteration)
    "ra":<str>, ( default = ra_str)
    "dec":<str>, (default = dec_str)
    "epoch":<str>, (default = epoch )
    "salt_win_start":<str>, (default = salt_win_start)
    "salt_win_end":<str>, (default = salt_win_end)
    "start_time":<str>, (default = start_time)
    "end_time":<str>, (default = end_time)
    "duration":<float> (default = overlaplength)
    "satellite":<str> (optional, TBD)
  },
  ...
]

[rubyvanrooyen]

Implements SALT visibility evaluation IDL function (issue #4) Implements SALT vs GLI visibility overlap evaluation to obtain "start_time", "end_time", "duration" for JSON output (issue #5)