parsotat
commented
6 months ago Hi Maxime,
I am looking into the data download issue as this may be related to the swifttools python package that we utilize. As for the lack of GTI’s I am finding something different. If I run:
queryargs = dict(time="2016-12-19 .. 2016-12-21", fields='All', resultmax=0)
object_location = swiftbat.simbadlocation("Crab")
object_batsource = swiftbat.source(ra=object_location[0], dec=object_location[1], name=object_name)
table_everything = ba.from_heasarc(**queryargs)
minexposure = 1000 # cm^2 after cos adjust
exposures = np.array([object_batsource.exposure(\
ra=row['RA'], dec=row['DEC'], roll=row['ROLL_ANGLE'])[0] for row in table_everything])
table_exposed = table_everything[exposures > minexposure]
print(f"Finding everything finds {len(table_everything)} observations, of which {len(table_exposed)} have more than {minexposure:0} cm^2 coded")Finding everything finds 116 observations, of which 9 have more than 1000 cm^2 coded
obs_ids=[i for i in table_exposed['OBSID'] if result[i]['success']] input_dict=dict(cleansnr=6,cleanexpr='ALWAYS_CLEAN==T') batsurvey_obs=ba.parallel.batsurvey_analysis(obs_ids, input_dict=input_dict, patt_noise_dir=noise_map_dir, nprocs=4)
Working on Obsid 00034843001
Working on Obsid 00087071007
Working on Obsid 00034856001
Working on Obsid 00092352073
The observation ID folder needs to contain the bat/survey/ and auxil/ subdirectories in order to analyze BAT survey data. One or both of these folders are missing.
Done with Obsid 00092352073
Working on Obsid 00087254001
WARNING: HSPDeprecationWarning: heasoftpy.swifttime is being deprecated and will be removed. Use heasoftpy.swift.swifttime instead [batanalysis.batlib]
WARNING: HSPDeprecationWarning: heasoftpy.batsurvey is being deprecated and will be removed. Use heasoftpy.swift.batsurvey instead [batanalysis.bat_survey]
WARNING: HSPDeprecationWarning: heasoftpy.batsurvey is being deprecated and will be removed. Use heasoftpy.swift.batsurvey instead [batanalysis.bat_survey]
WARNING: HSPDeprecationWarning: heasoftpy.batsurvey is being deprecated and will be removed. Use heasoftpy.swift.batsurvey instead [batanalysis.bat_survey]
could not convert string to float: '$( )'
Done with Obsid 00034856001
Working on Obsid 00046362008
WARNING: HSPDeprecationWarning: heasoftpy.batsurvey is being deprecated and will be removed. Use heasoftpy.swift.batsurvey instead [batanalysis.bat_survey]
A save file has been written to /Home/eud/tparsota/debug_BatAnalysis/00087254001_surveyresult/batsurvey.pickle.
Done with Obsid 00087254001
Working on Obsid 00087077002
A save file has been written to /Home/eud/tparsota/debug_BatAnalysis/00046362008_surveyresult/batsurvey.pickle.
Done with Obsid 00046362008
Working on Obsid 00055763032
A save file has been written to /Home/eud/tparsota/debug_BatAnalysis/00055763032_surveyresult/batsurvey.pickle.
The results for each pointing of observation ID 00055763032 is:
There were no GTI intervals found for this observation ID 00055763032
Done with Obsid 00055763032
Working on Obsid 00031511027
A save file has been written to /Home/eud/tparsota/debug_BatAnalysis/00031511027_surveyresult/batsurvey.pickle.
The results for each pointing of observation ID 00031511027 is:
B 6.16 00031511027 point_20163551959 0 expo_small 503956743 503956829.84146 64.8837261665482 55.9534256090162 332.121696933384 89 0 0 8 0 0 0 0 0 0 0 0
Done with Obsid 00031511027 A save file has been written to /Home/eud/tparsota/debug_BatAnalysis/00087077002_surveyresult/batsurvey.pickle. Done with Obsid 00087077002 A save file has been written to /Home/eud/tparsota/debug_BatAnalysis/00087071007_surveyresult/batsurvey.pickle. Done with Obsid 00087071007 A save file has been written to /Home/eud/tparsota/debug_BatAnalysis/00034843001_surveyresult/batsurvey.pickle. Done with Obsid 00034843001
I actually get a different number of observations and end up with len(batsurvey_obs) = 5. Can you specify what version of heasoftpy, BatAnalysis, & swifttools you are using? For me the versions are 1.4.1 for heasoftpy, 1.0.6 for BatAnalysis, and 1.2.32 for swifttools.
From: Maxime P. @.> Date: Monday, April 15, 2024 at 9:29 AM To: parsotat/BatAnalysis @.> Cc: Subscribed @.***> Subject: [EXTERNAL] [BULK] [parsotat/BatAnalysis] Problem with download_swiftdata on old observations and batsurvey_obs on new observations (Issue #17) CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC.
Hi, I'm trying to make the bat_analysis tools work and am facing some issues. When trying to reproduce what is done in e.g. the trial_detection_Crab or trial_NGC2992 notebooks ex:
newdir = Path(os.getcwd())
ba.datadir(newdir, mkdir=True)
object_name='Crab_Nebula_Pulsar'
queryargs = dict(time="2004-12-15 .. 2006-10-27", fields="All", resultmax=0)
edited to test with only a part of the dataset
queryargs = dict(time="2004-12-19 .. 2004-12-21", fields='All', resultmax=0)
object_location = swiftbat.simbadlocation("Crab")
object_batsource = swiftbat.source(ra=object_location[0], dec=object_location[1], name=object_name)
table_everything = ba.from_heasarc(**queryargs)
minexposure = 1000 # cm^2 after cos adjust
exposures = np.array([object_batsource.exposure(\
ra=row['RA'], dec=row['DEC'], roll=row['ROLL_ANGLE'])[0] for row in table_everything])table_exposed = table_everything[exposures > minexposure]
print(f"Finding everything finds {len(table_everything)} observations, of which {len(table_exposed)} have more than {minexposure:0} cm^2 coded")
Gives me the output
Finding everything finds 39 observations, of which 12 have more than 1000 cm^2 coded
Then, when trying to download with
result = ba.download_swiftdata(table_exposed)
I get :
UserWarning: Did not download X data found for observation ID X No data found for X.
With X being every single of the 12 observations. I get the same behavior when trying to download a part of the NGC2992 survey dataset.
Now, when replacing with more recent data e.g.
queryargs = dict(time="2016-12-19 .. 2016-12-21", fields='All', resultmax=0)
The data does download perfectly, but then trying to reduce it with
obs_ids=[i for i in table_exposed['OBSID'] if result[i]['success']]
input_dict=dict(cleansnr=6,cleanexpr='ALWAYS_CLEAN==T')
noise_map_dir=Path('/home/parrama/Soft/Swift-BAT/pattern_maps/')
batsurvey_obs=ba.parallel.batsurvey_analysis(obs_ids, input_dict=input_dict, patt_noise_dir=noise_map_dir, nprocs=4)
I always obtain outputs similar to this
Working on Obsid 00034856001
Working on Obsid 00087071007
Working on Obsid 00092352073
The observation ID folder needs to contain the bat/survey/ and auxil/ subdirectories in order to analyze BAT survey data. One or both of these folders are missing.
Done with Obsid 00092352073
Working on Obsid 00087254001
A save file has been written to /home/parrama/Documents/Observ/copy_SSD/BHLMXB/Swift_4U/BAT_analysis/00087254001_surveyresult/batsurvey.pickle.
The results for each pointing of observation ID 00087254001 is:
There were no GTI intervals found for this observation ID 00087254001
Done with Obsid 00087254001
Working on Obsid 00046362008
A save file has been written to /home/parrama/Documents/Observ/copy_SSD/BHLMXB/Swift_4U/BAT_analysis/00087071007_surveyresult/batsurvey.pickle.
The results for each pointing of observation ID 00087071007 is:
There were no GTI intervals found for this observation ID 00087071007
Done with Obsid 00087071007
Working on Obsid 00087077002
A save file has been written to /home/parrama/Documents/Observ/copy_SSD/BHLMXB/Swift_4U/BAT_analysis/00034856001_surveyresult/batsurvey.pickle.
The results for each pointing of observation ID 00034856001 is:
There were no GTI intervals found for this observation ID 00034856001
and batsurvey_obs ends up empty
And there are never any gti intervals found. This issue is the most problematic as I get the same behavior for the source I want to analyze (4U1630-47), namely the data downloads but no GTIs are ever detected.
Am I missing something?
— Reply to this email directly, view it on GitHubhttps://github.com/parsotat/BatAnalysis/issues/17, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGCJ6LA5NN6S33VHU7OEFHLY5PI23AVCNFSM6AAAAABGHL2J3WVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI2DGNRZGE2TGMI. You are receiving this because you are subscribed to this thread.Message ID: @.***>
Parrazyte
Hi, I'm trying to make the bat_analysis tools work and am facing some issues. When trying to reproduce what is done in e.g. the trial_detection_Crab or trial_NGC2992 notebooks ex:
Gives me the output
Then, when trying to download with
I get :
With X being every single of the 12 observations. I get the same behavior when trying to download a part of the NGC2992 survey dataset.
Now, when replacing with more recent data e.g.
The data does download perfectly, but then trying to reduce it with
I always obtain outputs similar to this
Batsurvey_obs ends up empty as there are never any gti intervals found. This issue is the most problematic as I get the same behavior for the source I want to analyze (4U1630-47), namely the data downloads but no GTIs are ever detected.
Am I missing something?