Closed Parrazyte closed 5 months ago
Hi Maxime,
I am looking into the data download issue as this may be related to the swifttools python package that we utilize. As for the lack of GTI’s I am finding something different. If I run:
queryargs = dict(time="2016-12-19 .. 2016-12-21", fields='All', resultmax=0)
object_location = swiftbat.simbadlocation("Crab")
object_batsource = swiftbat.source(ra=object_location[0], dec=object_location[1], name=object_name)
table_everything = ba.from_heasarc(**queryargs)
minexposure = 1000 # cm^2 after cos adjust
exposures = np.array([object_batsource.exposure(\
ra=row['RA'], dec=row['DEC'], roll=row['ROLL_ANGLE'])[0] for row in table_everything])
table_exposed = table_everything[exposures > minexposure]
print(f"Finding everything finds {len(table_everything)} observations, of which {len(table_exposed)} have more than {minexposure:0} cm^2 coded")
Finding everything finds 116 observations, of which 9 have more than 1000 cm^2 coded
obs_ids=[i for i in table_exposed['OBSID'] if result[i]['success']] input_dict=dict(cleansnr=6,cleanexpr='ALWAYS_CLEAN==T') batsurvey_obs=ba.parallel.batsurvey_analysis(obs_ids, input_dict=input_dict, patt_noise_dir=noise_map_dir, nprocs=4)
Working on Obsid 00034843001
Working on Obsid 00087071007
Working on Obsid 00034856001
Working on Obsid 00092352073
The observation ID folder needs to contain the bat/survey/ and auxil/ subdirectories in order to analyze BAT survey data. One or both of these folders are missing.
Done with Obsid 00092352073
Working on Obsid 00087254001
WARNING: HSPDeprecationWarning: heasoftpy.swifttime is being deprecated and will be removed. Use heasoftpy.swift.swifttime
instead [batanalysis.batlib]
WARNING: HSPDeprecationWarning: heasoftpy.batsurvey is being deprecated and will be removed. Use heasoftpy.swift.batsurvey
instead [batanalysis.bat_survey]
WARNING: HSPDeprecationWarning: heasoftpy.batsurvey is being deprecated and will be removed. Use heasoftpy.swift.batsurvey
instead [batanalysis.bat_survey]
WARNING: HSPDeprecationWarning: heasoftpy.batsurvey is being deprecated and will be removed. Use heasoftpy.swift.batsurvey
instead [batanalysis.bat_survey]
could not convert string to float: '$( )'
Done with Obsid 00034856001
Working on Obsid 00046362008
WARNING: HSPDeprecationWarning: heasoftpy.batsurvey is being deprecated and will be removed. Use heasoftpy.swift.batsurvey
instead [batanalysis.bat_survey]
A save file has been written to /Home/eud/tparsota/debug_BatAnalysis/00087254001_surveyresult/batsurvey.pickle.
Done with Obsid 00087254001
Working on Obsid 00087077002
A save file has been written to /Home/eud/tparsota/debug_BatAnalysis/00046362008_surveyresult/batsurvey.pickle.
Done with Obsid 00046362008
Working on Obsid 00055763032
A save file has been written to /Home/eud/tparsota/debug_BatAnalysis/00055763032_surveyresult/batsurvey.pickle.
The results for each pointing of observation ID 00055763032 is:
There were no GTI intervals found for this observation ID 00055763032
Done with Obsid 00055763032
Working on Obsid 00031511027
A save file has been written to /Home/eud/tparsota/debug_BatAnalysis/00031511027_surveyresult/batsurvey.pickle.
The results for each pointing of observation ID 00031511027 is:
B 6.16 00031511027 point_20163551959 0 expo_small 503956743 503956829.84146 64.8837261665482 55.9534256090162 332.121696933384 89 0 0 8 0 0 0 0 0 0 0 0
Done with Obsid 00031511027 A save file has been written to /Home/eud/tparsota/debug_BatAnalysis/00087077002_surveyresult/batsurvey.pickle. Done with Obsid 00087077002 A save file has been written to /Home/eud/tparsota/debug_BatAnalysis/00087071007_surveyresult/batsurvey.pickle. Done with Obsid 00087071007 A save file has been written to /Home/eud/tparsota/debug_BatAnalysis/00034843001_surveyresult/batsurvey.pickle. Done with Obsid 00034843001
I actually get a different number of observations and end up with len(batsurvey_obs) = 5. Can you specify what version of heasoftpy, BatAnalysis, & swifttools you are using? For me the versions are 1.4.1 for heasoftpy, 1.0.6 for BatAnalysis, and 1.2.32 for swifttools.
From: Maxime P. @.> Date: Monday, April 15, 2024 at 9:29 AM To: parsotat/BatAnalysis @.> Cc: Subscribed @.***> Subject: [EXTERNAL] [BULK] [parsotat/BatAnalysis] Problem with download_swiftdata on old observations and batsurvey_obs on new observations (Issue #17) CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC.
Hi, I'm trying to make the bat_analysis tools work and am facing some issues. When trying to reproduce what is done in e.g. the trial_detection_Crab or trial_NGC2992 notebooks ex:
newdir = Path(os.getcwd())
ba.datadir(newdir, mkdir=True)
object_name='Crab_Nebula_Pulsar'
queryargs = dict(time="2004-12-19 .. 2004-12-21", fields='All', resultmax=0)
object_location = swiftbat.simbadlocation("Crab")
object_batsource = swiftbat.source(ra=object_location[0], dec=object_location[1], name=object_name)
table_everything = ba.from_heasarc(**queryargs)
minexposure = 1000 # cm^2 after cos adjust
exposures = np.array([object_batsource.exposure(\
ra=row['RA'], dec=row['DEC'], roll=row['ROLL_ANGLE'])[0] for row in table_everything])
table_exposed = table_everything[exposures > minexposure]
print(f"Finding everything finds {len(table_everything)} observations, of which {len(table_exposed)} have more than {minexposure:0} cm^2 coded")
Gives me the output
Finding everything finds 39 observations, of which 12 have more than 1000 cm^2 coded
Then, when trying to download with
result = ba.download_swiftdata(table_exposed)
I get :
UserWarning: Did not download X data found for observation ID X No data found for X.
With X being every single of the 12 observations. I get the same behavior when trying to download a part of the NGC2992 survey dataset.
Now, when replacing with more recent data e.g.
queryargs = dict(time="2016-12-19 .. 2016-12-21", fields='All', resultmax=0)
The data does download perfectly, but then trying to reduce it with
obs_ids=[i for i in table_exposed['OBSID'] if result[i]['success']]
input_dict=dict(cleansnr=6,cleanexpr='ALWAYS_CLEAN==T')
noise_map_dir=Path('/home/parrama/Soft/Swift-BAT/pattern_maps/')
batsurvey_obs=ba.parallel.batsurvey_analysis(obs_ids, input_dict=input_dict, patt_noise_dir=noise_map_dir, nprocs=4)
I always obtain outputs similar to this
Working on Obsid 00034856001
Working on Obsid 00087071007
Working on Obsid 00092352073
The observation ID folder needs to contain the bat/survey/ and auxil/ subdirectories in order to analyze BAT survey data. One or both of these folders are missing.
Done with Obsid 00092352073
Working on Obsid 00087254001
A save file has been written to /home/parrama/Documents/Observ/copy_SSD/BHLMXB/Swift_4U/BAT_analysis/00087254001_surveyresult/batsurvey.pickle.
The results for each pointing of observation ID 00087254001 is:
There were no GTI intervals found for this observation ID 00087254001
Done with Obsid 00087254001
Working on Obsid 00046362008
A save file has been written to /home/parrama/Documents/Observ/copy_SSD/BHLMXB/Swift_4U/BAT_analysis/00087071007_surveyresult/batsurvey.pickle.
The results for each pointing of observation ID 00087071007 is:
There were no GTI intervals found for this observation ID 00087071007
Done with Obsid 00087071007
Working on Obsid 00087077002
A save file has been written to /home/parrama/Documents/Observ/copy_SSD/BHLMXB/Swift_4U/BAT_analysis/00034856001_surveyresult/batsurvey.pickle.
The results for each pointing of observation ID 00034856001 is:
There were no GTI intervals found for this observation ID 00034856001
and batsurvey_obs ends up empty
And there are never any gti intervals found. This issue is the most problematic as I get the same behavior for the source I want to analyze (4U1630-47), namely the data downloads but no GTIs are ever detected.
Am I missing something?
— Reply to this email directly, view it on GitHubhttps://github.com/parsotat/BatAnalysis/issues/17, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGCJ6LA5NN6S33VHU7OEFHLY5PI23AVCNFSM6AAAAABGHL2J3WVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI2DGNRZGE2TGMI. You are receiving this because you are subscribed to this thread.Message ID: @.***>
Hi Tyler, I'm using heasoftpy 1.4.1 batanalysis 1.0.6 swifttools says 3.0.21, but when looking at swift_too specifically it is v1.2.32
So not so much change there it seems.
The pattern map files I have are from October 2022 (I don't know maybe the issue comes from there ?)
Best, Maxime
To address your data finding problem for 2004 data, you can do:
result = ba.download_swiftdata(table_exposed, uksdc=True)
and it will find the data and download it accordingly.
Can you let me know what your astroquery & astropy versions are as well? I don't believe that the pattern map files are the problem here since they are applied after the GTI filtering.
Thanks for the info.
My astroquery is 0.4.6, my astropy is 5.3.2 (I tested things on python 3.9.19 if that matters)
@Parrazyte to help narrow down the issue. Let's first try to identify why we may be finding different observation IDs. Can you run this chunk of code and post the observation IDs that you obtain:
import batanalysis as ba
import swiftbat
import numpy as np
ba.datadir()
queryargs = dict(time="2016-12-19 .. 2016-12-21", fields='All', resultmax=0)
object_name='Crab_Nebula_Pulsar'
object_location = swiftbat.simbadlocation("Crab")
object_batsource = swiftbat.source(ra=object_location[0], dec=object_location[1], name=object_name)
table_everything = ba.from_heasarc(**queryargs)
minexposure = 1000 # cm^2 after cos adjust
exposures = np.array([object_batsource.exposure(\
ra=row['RA'], dec=row['DEC'], roll=row['ROLL_ANGLE'])[0] for row in table_everything])
table_exposed = table_everything[exposures > minexposure]
print(f"Finding everything finds {len(table_everything)} observations, of which {len(table_exposed)} have more than {minexposure:0} cm^2 coded")
minexposure = 1000 # cm^2 after cos adjust
exposures = np.array([object_batsource.exposure(\
ra=row['RA'], dec=row['DEC'], roll=row['ROLL_ANGLE'])[0] for row in table_everything])
table_exposed = table_everything[exposures > minexposure]
print(f"Finding everything finds {len(table_everything)} observations, of which {len(table_exposed)} have more than {minexposure:0} cm^2 coded")
print(table_exposed)
My output is:
Finding everything finds 116 observations, of which 9 have more than 1000 cm^2 coded
NAME OBSID RA DEC START_TIME PROCESSING_DATE ... UVOT_EXPO_WH XRT_EXPO_IM XRT_EXPO_LR XRT_EXPO_PC XRT_EXPO_PU XRT_EXPO_WT
deg deg mjd mjd ... s s s s s s
-------------------------------------------------------------------------------- ----------- --------- --------- ---------------- --------------- ... ------------ ----------- ----------- ----------- ----------- -----------
G181.1+9.5 00034843001 96.80003 32.64433 57740.7242824074 57750 ... 0.00000 0.00000 0.00000 4938.94200 0.00000 0.00000
V1261Ori 00034856001 80.54824 -8.62813 57742.3013541667 57752 ... 0.00000 0.00000 0.00000 3559.57800 0.00000 0.26900
SwiftJ0310.7+3917 00087071007 47.61033 39.27723 57740.3853935185 57750 ... 0.00000 0.00000 0.00000 1957.05100 0.00000 0.00200
3C120 00092352073 68.26092 5.37224 57741.7749768519 57751 ... 0.00000 0.00000 0.00000 0.00000 0.00000 614.71100
COBRA081508.9+270347 00087254001 123.78717 27.07105 57741.1812268519 57751 ... 0.00000 0.00000 0.00000 588.03800 0.00000 0.00000
SDSSJ075551.44+352549.8 00046362008 118.96250 35.40127 57742.6451041667 57752 ... 0.00000 0.00000 0.00000 498.19900 0.00000 0.00000
SwiftJ0450.6+3015 00087077002 72.61883 30.20253 57742.6388773148 57752 ... 0.00000 0.00000 0.00000 413.69800 0.00000 0.00000
SA95-42 00055763032 58.41562 -0.05077 57742.0346990741 57752 ... 0.00000 0.00000 0.00000 0.00000 0.00000 125.63500
CICam 00031511027 64.88366 55.95340 57742.8305324074 57752 ... 0.00000 0.00000 0.00000 80.22400 0.00000 1.98500
Alright !
For now everything sounds very similar in term of sources. except that directly printing the table only gives me the name:
Finding everything finds 116 observations, of which 9 have more than 1000 cm^2 coded
NAME ...
...
-------------------------------------------------------------------------------- ...
G181.1+9.5 ...
V1261Ori ...
SwiftJ0310.7+3917 ...
3C120 ...
COBRA081508.9+270347 ...
SDSSJ075551.44+352549.8 ...
SwiftJ0450.6+3015 ...
SA95-42 ...
CICam ...
But if I manually print the other columns I get similar results, e.g. with table_exposed['OBSID']:
<Column name='OBSID' dtype='bytes11' length=9>
00034843001
00034856001
00087071007
00092352073
00087254001
00046362008
00087077002
00055763032
00031511027
ok, so we've got the same data obtained from our query. Can you download the data and verify that all 9 are downloaded?
I do get 9 directories with a total of 207 items (counting the 9 obsid folders) for 45.7 MB
great, we have the same starting point for processing the data now. With the data downloaded, please run this chunk of code:
#the below line can be run after redownloading the data or you can just create a list of the downloaded obsids
obs_ids=[i for i in table_exposed['OBSID'] if result[i]['success']]
#or
obs_ids=[i["OBS_ID"] for i in table_exposed]
input_dict=dict(cleansnr=6,cleanexpr='ALWAYS_CLEAN==T')
noise_map_dir=#fill this in here
#now we will process the survey data without parallelization so we can better compare results. This may take a bit to run. so just post your results when it completes for you. I will do the same when this completes for me.
batsurvey_obs=ba.parallel.batsurvey_analysis(obs_ids, input_dict=input_dict, patt_noise_dir=noise_map_dir, nprocs=1)
print(len(batsurvey_obs))
please paste the total output that gets printed to the screen. I will also post what gets printed out on my end when the run completes.
For me there is no "OBS_ID" column in table_exposed, only an "OBSID" column.
So I set obs_ids with this column, which gives me:
obs_ids
['00034843001',
'00034856001',
'00087071007',
'00092352073',
'00087254001',
'00046362008',
'00087077002',
'00055763032',
'00031511027']
then
input_dict=dict(cleansnr=6,cleanexpr='ALWAYS_CLEAN==T')
noise_map_dir='/home/parrama/Soft/Swift-BAT/pattern_maps/'
then
#now we will process the survey data without parallelization so we can better compare results. This may take a bit to run. so just post your results when it completes for you. I will do the same when this completes for me.
batsurvey_obs=ba.parallel.batsurvey_analysis(obs_ids, input_dict=input_dict, patt_noise_dir=noise_map_dir, nprocs=1)
print(len(batsurvey_obs))
which gives the following output:
Working on Obsid 00034843001
The observation ID folder needs to contain the bat/survey/ and auxil/ subdirectories in order to analyze BAT survey data. One or both of these folders are missing.
Done with Obsid 00034843001
Working on Obsid 00034856001
The observation ID folder needs to contain the bat/survey/ and auxil/ subdirectories in order to analyze BAT survey data. One or both of these folders are missing.
Done with Obsid 00034856001
Working on Obsid 00087071007
The observation ID folder needs to contain the bat/survey/ and auxil/ subdirectories in order to analyze BAT survey data. One or both of these folders are missing.
Done with Obsid 00087071007
Working on Obsid 00092352073
The observation ID folder needs to contain the bat/survey/ and auxil/ subdirectories in order to analyze BAT survey data. One or both of these folders are missing.
Done with Obsid 00092352073
Working on Obsid 00087254001
WARNING: HSPDeprecationWarning: heasoftpy.batsurvey is being deprecated and will be removed. Use ``heasoftpy.swift.batsurvey`` instead [batanalysis.bat_survey]
A save file has been written to /media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test/00087254001_surveyresult/batsurvey.pickle.
The results for each pointing of observation ID 00087254001 is:
There were no GTI intervals found for this observation ID 00087254001
Done with Obsid 00087254001
Working on Obsid 00046362008
A save file has been written to /media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test/00046362008_surveyresult/batsurvey.pickle.
The results for each pointing of observation ID 00046362008 is:
There were no GTI intervals found for this observation ID 00046362008
Done with Obsid 00046362008
Working on Obsid 00087077002
The observation ID folder needs to contain the bat/survey/ and auxil/ subdirectories in order to analyze BAT survey data. One or both of these folders are missing.
Done with Obsid 00087077002
Working on Obsid 00055763032
A save file has been written to /media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test/00055763032_surveyresult/batsurvey.pickle.
The results for each pointing of observation ID 00055763032 is:
There were no GTI intervals found for this observation ID 00055763032
Done with Obsid 00055763032
Working on Obsid 00031511027
A save file has been written to /media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test/00031511027_surveyresult/batsurvey.pickle.
The results for each pointing of observation ID 00031511027 is:
There were no GTI intervals found for this observation ID 00031511027
Done with Obsid 00031511027
0
Here is the output of my run:
In [11]: batsurvey_obs=ba.parallel.batsurvey_analysis(obs_ids, input_dict=input_dict, patt_noise_dir=noise_map_dir, nprocs=1)
Working on Obsid 00034843001
WARNING: HSPDeprecationWarning: heasoftpy.batsurvey is being deprecated and will be removed. Use ``heasoftpy.swift.batsurvey`` instead [batanalysis.bat_survey]
A save file has been written to /Home/eud/tparsota/debug_BatAnalysis/00034843001_surveyresult/batsurvey.pickle.
Done with Obsid 00034843001
Working on Obsid 00034856001
A save file has been written to /Home/eud/tparsota/debug_BatAnalysis/00034856001_surveyresult/batsurvey.pickle.
Done with Obsid 00034856001
Working on Obsid 00087071007
A save file has been written to /Home/eud/tparsota/debug_BatAnalysis/00087071007_surveyresult/batsurvey.pickle.
Done with Obsid 00087071007
Working on Obsid 00092352073
The observation ID folder needs to contain the bat/survey/ and auxil/ subdirectories in order to analyze BAT survey data. One or both of these folders are missing.
Done with Obsid 00092352073
Working on Obsid 00087254001
A save file has been written to /Home/eud/tparsota/debug_BatAnalysis/00087254001_surveyresult/batsurvey.pickle.
Done with Obsid 00087254001
Working on Obsid 00046362008
A save file has been written to /Home/eud/tparsota/debug_BatAnalysis/00046362008_surveyresult/batsurvey.pickle.
Done with Obsid 00046362008
Working on Obsid 00087077002
A save file has been written to /Home/eud/tparsota/debug_BatAnalysis/00087077002_surveyresult/batsurvey.pickle.
Done with Obsid 00087077002
Working on Obsid 00055763032
A save file has been written to /Home/eud/tparsota/debug_BatAnalysis/00055763032_surveyresult/batsurvey.pickle.
The results for each pointing of observation ID 00055763032 is:
There were no GTI intervals found for this observation ID 00055763032
Done with Obsid 00055763032
Working on Obsid 00031511027
A save file has been written to /Home/eud/tparsota/debug_BatAnalysis/00031511027_surveyresult/batsurvey.pickle.
The results for each pointing of observation ID 00031511027 is:
B 6.16 00031511027 point_20163551959 0 expo_small 503956743 503956829.84146 64.8837261665482 55.9534256090162 332.121696933384 89 0 0 8 0 0 0 0 0 0 0 0
Done with Obsid 00031511027
In [12]: len(batsurvey_obs)
Out[12]: 6
It seems as though your data download is not downloading the required folders "bat/" and auxil/" for each observation id.
Can you verify that you have something similar to my ls below:
In [22]: ls -lrth 00034843001
total 8.0K
drwxr-xr-x. 2 me group 4.0K Apr 30 11:53 auxil/
drwxr-xr-x. 7 me group 4.0K Apr 30 11:53 bat/
In [23]: ls -lrthd 00034843001
drwxr-xr-x. 4 me group 4.0K Apr 30 12:02 00034843001/
In [25]: ls -lrth 00034843001/*
00034843001/auxil:
total 11M
-rw-r--r--. 1 me group 114K Apr 30 11:53 SWIFT_TLE_ARCHIVE.txt.16361.85726095.gz
-rw-r--r--. 1 me group 128K Apr 30 11:53 sw00034843001pat.fits.gz
-rw-r--r--. 1 me group 2.4K Apr 30 11:53 sw00034843001pjb.par.gz
-rw-r--r--. 1 me group 4.8K Apr 30 11:53 sw00034843001pob.cat.gz
-rw-r--r--. 1 me group 1.8K Apr 30 11:53 sw00034843001ppr.par.gz
-rw-r--r--. 1 me group 348K Apr 30 11:53 sw00034843001s.mkf.gz
-rw-r--r--. 1 me group 8.6M Apr 30 11:53 sw00034843001sao.fits.gz
-rw-r--r--. 1 me group 133K Apr 30 11:53 sw00034843001sat.fits.gz
-rw-r--r--. 1 me group 1.5M Apr 30 11:53 sw00034843001sen.hk.gz
-rw-r--r--. 1 me group 3.8K Apr 30 11:53 sw00034843001sti.fits.gz
-rw-r--r--. 1 me group 93K Apr 30 11:53 sw00034843001uat.fits.gz
00034843001/bat:
total 20K
drwxr-xr-x. 2 me group 4.0K Apr 30 11:53 event/
drwxr-xr-x. 2 me group 4.0K Apr 30 11:53 hk/
drwxr-xr-x. 2 me group 4.0K Apr 30 11:53 masktag/
drwxr-xr-x. 2 me group 4.0K Apr 30 11:53 rate/
drwxr-xr-x. 2 me group 4.0K Apr 30 11:53 survey/
Hi Tyler, it seems here there's indeed a difference:
ls -lrth 00034843001
total 8,0K
drwxrwxr-x 2 parrama parrama 4,0K avril 30 12:27 auxil/
drwxrwxr-x 3 parrama parrama 4,0K avril 30 12:27 bat/
ls -lrthd 00034843001
drwxrwxr-x 5 parrama parrama 4,0K avril 30 18:14 00034843001/
ls -lrth 00034843001\/*
00034843001/auxil:
total 11M
-rw-rw-r-- 1 parrama parrama 114K avril 30 12:25 SWIFT_TLE_ARCHIVE.txt.16361.85726095.gz
-rw-rw-r-- 1 parrama parrama 128K avril 30 12:25 sw00034843001pat.fits.gz
-rw-rw-r-- 1 parrama parrama 2,4K avril 30 12:25 sw00034843001pjb.par.gz
-rw-rw-r-- 1 parrama parrama 4,8K avril 30 12:25 sw00034843001pob.cat.gz
-rw-rw-r-- 1 parrama parrama 1,8K avril 30 12:25 sw00034843001ppr.par.gz
-rw-rw-r-- 1 parrama parrama 348K avril 30 12:26 sw00034843001s.mkf.gz
-rw-rw-r-- 1 parrama parrama 8,6M avril 30 12:26 sw00034843001sao.fits.gz
-rw-rw-r-- 1 parrama parrama 133K avril 30 12:26 sw00034843001sat.fits.gz
-rw-rw-r-- 1 parrama parrama 1,5M avril 30 12:27 sw00034843001sen.hk.gz
-rw-rw-r-- 1 parrama parrama 3,8K avril 30 12:27 sw00034843001sti.fits.gz
-rw-rw-r-- 1 parrama parrama 93K avril 30 12:27 sw00034843001uat.fits.gz
00034843001/bat:
total 4,0K
drwxrwxr-x 2 parrama parrama 4,0K avril 30 12:27 event/
It seems I'm indeed missing subfolders for some reason...
One issue that may be occurring here is that your system is running out of space and some of the observation IDs get downloaded without the full set of files.
Can you list out all the observation ID directories to see what the subdirectories are? Then, can you also delete all the observation ID directories (we will want to start from a clean slate) and redownload them following this code below which will download the data sequentially instead of in parallel:
import batanalysis as ba
import swiftbat
import numpy as np
ba.datadir()
queryargs = dict(time="2016-12-19 .. 2016-12-21", fields='All', resultmax=0)
object_name='Crab_Nebula_Pulsar'
object_location = swiftbat.simbadlocation("Crab")
object_batsource = swiftbat.source(ra=object_location[0], dec=object_location[1], name=object_name)
table_everything = ba.from_heasarc(**queryargs)
minexposure = 1000 # cm^2 after cos adjust
exposures = np.array([object_batsource.exposure(\
ra=row['RA'], dec=row['DEC'], roll=row['ROLL_ANGLE'])[0] for row in table_everything])
table_exposed = table_everything[exposures > minexposure]
print(f"Finding everything finds {len(table_everything)} observations, of which {len(table_exposed)} have more than {minexposure:0} cm^2 coded")
minexposure = 1000 # cm^2 after cos adjust
exposures = np.array([object_batsource.exposure(\
ra=row['RA'], dec=row['DEC'], roll=row['ROLL_ANGLE'])[0] for row in table_everything])
table_exposed = table_everything[exposures > minexposure]
print(f"Finding everything finds {len(table_everything)} observations, of which {len(table_exposed)} have more than {minexposure:0} cm^2 coded")
print(table_exposed)
result = ba.download_swiftdata(table_exposed, jobs=1)
print(result)
Please copy and paste the output of print(result)
and then we can once again list all the directories to make sure that all the data has been downloaded correctly.
Here is the output of my print(result)
:
{'00034843001': {'obsid': '00034843001', 'success': True, 'obsoutdir': PosixPath('/Home/eud/tparsota/debug_BatAnalysis/00034843001'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00034843001',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00034856001': {'obsid': '00034856001', 'success': True, 'obsoutdir': PosixPath('/Home/eud/tparsota/debug_BatAnalysis/00034856001'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00034856001',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00087071007': {'obsid': '00087071007', 'success': True, 'obsoutdir': PosixPath('/Home/eud/tparsota/debug_BatAnalysis/00087071007'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00087071007',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00092352073': {'obsid': '00092352073', 'success': True, 'obsoutdir': PosixPath('/Home/eud/tparsota/debug_BatAnalysis/00092352073'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00092352073',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00087254001': {'obsid': '00087254001', 'success': True, 'obsoutdir': PosixPath('/Home/eud/tparsota/debug_BatAnalysis/00087254001'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00087254001',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00046362008': {'obsid': '00046362008', 'success': True, 'obsoutdir': PosixPath('/Home/eud/tparsota/debug_BatAnalysis/00046362008'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00046362008',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00087077002': {'obsid': '00087077002', 'success': True, 'obsoutdir': PosixPath('/Home/eud/tparsota/debug_BatAnalysis/00087077002'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00087077002',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00055763032': {'obsid': '00055763032', 'success': True, 'obsoutdir': PosixPath('/Home/eud/tparsota/debug_BatAnalysis/00055763032'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00055763032',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00031511027': {'obsid': '00031511027', 'success': True, 'obsoutdir': PosixPath('/Home/eud/tparsota/debug_BatAnalysis/00031511027'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00031511027',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}}
I think the 2.3 free TB on my SSD are enough for few Swift observations...
when listing the current directories
parrama@ipag-9049:/media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test$ ls -R
.:
00031511027 00034843001 00046362008 00055763032 00087071007 00087254001 00092352073
00031511027_surveyresult 00034856001 00046362008_surveyresult 00055763032_surveyresult 00087077002 00087254001_surveyresult
./00031511027:
auxil bat
./00031511027/auxil:
sw00031511027pat.fits.gz sw00031511027ppr.par.gz sw00031511027sen.hk.gz sw00031511027uat.fits.gz
sw00031511027pjb.par.gz sw00031511027sao.fits.gz sw00031511027s.mkf.gz SWIFT_TLE_ARCHIVE.txt.16363.91429006.gz
sw00031511027pob.cat.gz sw00031511027sat.fits.gz sw00031511027sti.fits.gz
./00031511027/bat:
hk masktag rate survey
./00031511027/bat/hk:
sw00031511027bdecb.hk.gz sw00031511027ben.hk.gz sw00031511027bgocb.hk.gz sw00031511027bhd.hk.gz
./00031511027/bat/masktag:
sw00031511027bmt00010009.lc.gz sw00031511027bmt00010009_rw.lc.gz sw00031511027bmt00010500.lc.gz sw00031511027bmt00010500_rw.lc.gz
./00031511027/bat/rate:
sw00031511027brt1s.lc.gz sw00031511027brtmc.lc.gz sw00031511027brtms.lc.gz sw00031511027brtqd.lc.gz
./00031511027/bat/survey:
sw00031511027bsvpbo0b13g041e.dph.gz
./00031511027_surveyresult:
auxil batsurvey.pickle dph dpi gti lc scratch stats_obs.dat
./00031511027_surveyresult/auxil:
./00031511027_surveyresult/dph:
./00031511027_surveyresult/dpi:
./00031511027_surveyresult/gti:
./00031511027_surveyresult/lc:
./00031511027_surveyresult/scratch:
survey.lis
./00034843001:
auxil bat
./00034843001/auxil:
sw00034843001pat.fits.gz sw00034843001ppr.par.gz sw00034843001sen.hk.gz sw00034843001uat.fits.gz
sw00034843001pjb.par.gz sw00034843001sao.fits.gz sw00034843001s.mkf.gz SWIFT_TLE_ARCHIVE.txt.16361.85726095.gz
sw00034843001pob.cat.gz sw00034843001sat.fits.gz sw00034843001sti.fits.gz
./00034843001/bat:
event
./00034843001/bat/event:
./00034856001:
auxil bat
./00034856001/auxil:
sw00034856001pat.fits.gz sw00034856001ppr.par.gz sw00034856001sen.hk.gz sw00034856001uat.fits.gz
sw00034856001pjb.par.gz sw00034856001sao.fits.gz sw00034856001s.mkf.gz SWIFT_TLE_ARCHIVE.txt.16363.91429006.gz
sw00034856001pob.cat.gz sw00034856001sat.fits.gz sw00034856001sti.fits.gz
./00034856001/bat:
event
./00034856001/bat/event:
sw00034856001bevshpo_uf.evt.gz
./00046362008:
auxil bat
./00046362008/auxil:
sw00046362008pat.fits.gz sw00046362008ppr.par.gz sw00046362008sen.hk.gz sw00046362008uat.fits.gz
sw00046362008pjb.par.gz sw00046362008sao.fits.gz sw00046362008s.mkf.gz SWIFT_TLE_ARCHIVE.txt.16363.91429006.gz
sw00046362008pob.cat.gz sw00046362008sat.fits.gz sw00046362008sti.fits.gz
./00046362008/bat:
hk masktag rate survey
./00046362008/bat/hk:
sw00046362008bdecb.hk.gz sw00046362008bdp.hk.gz sw00046362008ben.hk.gz sw00046362008bgocb.hk.gz sw00046362008bhd.hk.gz
./00046362008/bat/masktag:
sw00046362008bmt00010009.lc.gz sw00046362008bmt00010009_rw.lc.gz sw00046362008bmt00010500.lc.gz sw00046362008bmt00010500_rw.lc.gz
./00046362008/bat/rate:
sw00046362008brt1s.lc.gz sw00046362008brtmc.lc.gz sw00046362008brtms.lc.gz sw00046362008brtqd.lc.gz
./00046362008/bat/survey:
sw00046362008bsvpbo0b10g041d.dph.gz
./00046362008_surveyresult:
auxil batsurvey.pickle dph dpi gti lc scratch stats_obs.dat
./00046362008_surveyresult/auxil:
./00046362008_surveyresult/dph:
./00046362008_surveyresult/dpi:
./00046362008_surveyresult/gti:
./00046362008_surveyresult/lc:
./00046362008_surveyresult/scratch:
survey.lis
./00055763032:
auxil bat
./00055763032/auxil:
sw00055763032pat.fits.gz sw00055763032pob.cat.gz sw00055763032sao.fits.gz sw00055763032sen.hk.gz sw00055763032sti.fits.gz
sw00055763032pjb.par.gz sw00055763032ppr.par.gz sw00055763032sat.fits.gz sw00055763032s.mkf.gz SWIFT_TLE_ARCHIVE.txt.16363.91429006.gz
./00055763032/bat:
hk masktag rate survey
./00055763032/bat/hk:
sw00055763032bdecb.hk.gz sw00055763032bdp.hk.gz sw00055763032ben.hk.gz sw00055763032bgocb.hk.gz sw00055763032bhd.hk.gz
./00055763032/bat/masktag:
sw00055763032bmt00010002_rw.lc.gz sw00055763032bmt00010004_rw.lc.gz sw00055763032bmt00010009.lc.gz sw00055763032bmt00010009_rw.lc.gz
./00055763032/bat/rate:
sw00055763032brt1s.lc.gz sw00055763032brtmc.lc.gz sw00055763032brtms.lc.gz sw00055763032brtqd.lc.gz
./00055763032/bat/survey:
sw00055763032bsvpbo0b07g041a.dph.gz
./00055763032_surveyresult:
auxil batsurvey.pickle dph dpi gti lc scratch stats_obs.dat
./00055763032_surveyresult/auxil:
./00055763032_surveyresult/dph:
./00055763032_surveyresult/dpi:
./00055763032_surveyresult/gti:
./00055763032_surveyresult/lc:
./00055763032_surveyresult/scratch:
survey.lis
./00087071007:
auxil bat
./00087071007/auxil:
sw00087071007pat.fits.gz sw00087071007ppr.par.gz sw00087071007sen.hk.gz sw00087071007uat.fits.gz
sw00087071007pjb.par.gz sw00087071007sao.fits.gz sw00087071007s.mkf.gz SWIFT_TLE_ARCHIVE.txt.16361.85726095.gz
sw00087071007pob.cat.gz sw00087071007sat.fits.gz sw00087071007sti.fits.gz
./00087071007/bat:
event
./00087071007/bat/event:
sw00087071007bevshpo_uf.evt.gz
./00087077002:
auxil bat
./00087077002/auxil:
sw00087077002pat.fits.gz sw00087077002pob.cat.gz sw00087077002sao.fits.gz sw00087077002sen.hk.gz sw00087077002sti.fits.gz
sw00087077002pjb.par.gz sw00087077002ppr.par.gz sw00087077002sat.fits.gz sw00087077002s.mkf.gz SWIFT_TLE_ARCHIVE.txt.16363.91429006.gz
./00087077002/bat:
event hk
./00087077002/bat/event:
sw00087077002bevshpo_uf.evt.gz sw00087077002bevshsl_uf.evt.gz
./00087077002/bat/hk:
sw00087077002bdecb.hk.gz sw00087077002bdp.hk.gz sw00087077002bdqcb.hk.gz sw00087077002ben.hk.gz
./00087254001:
auxil bat
./00087254001/auxil:
sw00087254001pat.fits.gz sw00087254001ppr.par.gz sw00087254001sen.hk.gz sw00087254001uat.fits.gz
sw00087254001pjb.par.gz sw00087254001sao.fits.gz sw00087254001s.mkf.gz SWIFT_TLE_ARCHIVE.txt.16363.05166484.gz
sw00087254001pob.cat.gz sw00087254001sat.fits.gz sw00087254001sti.fits.gz
./00087254001/bat:
hk masktag rate survey
./00087254001/bat/hk:
sw00087254001bdecb.hk.gz sw00087254001bdp.hk.gz sw00087254001ben.hk.gz sw00087254001bgocb.hk.gz sw00087254001bhd.hk.gz
./00087254001/bat/masktag:
sw00087254001bmt00010009.lc.gz sw00087254001bmt00010009_rw.lc.gz sw00087254001bmt00010500.lc.gz sw00087254001bmt00010500_rw.lc.gz
./00087254001/bat/rate:
sw00087254001brt1s.lc.gz sw00087254001brtmc.lc.gz sw00087254001brtms.lc.gz sw00087254001brtqd.lc.gz
./00087254001/bat/survey:
sw00087254001bsvpbo0afag0415.dph.gz
./00087254001_surveyresult:
auxil batsurvey.pickle dph dpi gti lc scratch stats_obs.dat
./00087254001_surveyresult/auxil:
./00087254001_surveyresult/dph:
./00087254001_surveyresult/dpi:
./00087254001_surveyresult/gti:
./00087254001_surveyresult/lc:
./00087254001_surveyresult/scratch:
survey.lis
./00092352073:
auxil
./00092352073/auxil:
sw00092352073pat.fits.gz sw00092352073ppr.par.gz sw00092352073sen.hk.gz sw00092352073uat.fits.gz
sw00092352073pjb.par.gz sw00092352073sao.fits.gz sw00092352073s.mkf.gz SWIFT_TLE_ARCHIVE.txt.16363.05166484.gz
sw00092352073pob.cat.gz sw00092352073sat.fits.gz sw00092352073sti.fits.gz
And now if I download things again (in another main folder, for good measure):
import batanalysis as ba
import swiftbat
import numpy as np
ba.datadir()
queryargs = dict(time="2016-12-19 .. 2016-12-21", fields='All', resultmax=0)
object_name='Crab_Nebula_Pulsar'
object_location = swiftbat.simbadlocation("Crab")
object_batsource = swiftbat.source(ra=object_location[0], dec=object_location[1], name=object_name)
table_everything = ba.from_heasarc(**queryargs)
minexposure = 1000 # cm^2 after cos adjust
exposures = np.array([object_batsource.exposure(\
ra=row['RA'], dec=row['DEC'], roll=row['ROLL_ANGLE'])[0] for row in table_everything])
table_exposed = table_everything[exposures > minexposure]
print(f"Finding everything finds {len(table_everything)} observations, of which {len(table_exposed)} have more than {minexposure:0} cm^2 coded")
minexposure = 1000 # cm^2 after cos adjust
exposures = np.array([object_batsource.exposure(\
ra=row['RA'], dec=row['DEC'], roll=row['ROLL_ANGLE'])[0] for row in table_everything])
table_exposed = table_everything[exposures > minexposure]
print(f"Finding everything finds {len(table_everything)} observations, of which {len(table_exposed)} have more than {minexposure:0} cm^2 coded")
print(table_exposed)
result = ba.download_swiftdata(table_exposed, jobs=1)
print(result)
/home/parrama/Documents/Work/PhD/Scripts/Python/observations/venv/lib/python3.9/site-packages/batanalysis/batlib.py:113: UserWarning: Saving data in current directory /media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test_nopa
Finding everything finds 116 observations, of which 9 have more than 1000 cm^2 coded
Finding everything finds 116 observations, of which 9 have more than 1000 cm^2 coded
NAME ...
...
-------------------------------------------------------------------------------- ...
G181.1+9.5 ...
V1261Ori ...
SwiftJ0310.7+3917 ...
3C120 ...
COBRA081508.9+270347 ...
SDSSJ075551.44+352549.8 ...
SwiftJ0450.6+3015 ...
SA95-42 ...
CICam ...
Downloading files: 100%|██████████| 34/34 [01:25<00:00, 2.52s/files]
Downloading files: 100%|██████████| 36/36 [01:17<00:00, 2.14s/files]
Downloading files: 100%|██████████| 32/32 [01:15<00:00, 2.36s/files]
Downloading files: 100%|██████████| 11/11 [00:06<00:00, 1.69files/s]
Downloading files: 100%|██████████| 25/25 [00:17<00:00, 1.44files/s]
Downloading files: 100%|██████████| 25/25 [00:16<00:00, 1.48files/s]
Downloading files: 100%|██████████| 29/29 [00:38<00:00, 1.33s/files]
Downloading files: 100%|██████████| 24/24 [00:15<00:00, 1.58files/s]
Downloading files: 100%|██████████| 24/24 [00:15<00:00, 1.54files/s]
{'00034843001': {'obsid': '00034843001', 'success': True, 'obsoutdir': PosixPath('/media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test_nopa/00034843001'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00034843001',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00034856001': {'obsid': '00034856001', 'success': True, 'obsoutdir': PosixPath('/media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test_nopa/00034856001'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00034856001',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00087071007': {'obsid': '00087071007', 'success': True, 'obsoutdir': PosixPath('/media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test_nopa/00087071007'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00087071007',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00092352073': {'obsid': '00092352073', 'success': True, 'obsoutdir': PosixPath('/media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test_nopa/00092352073'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00092352073',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00087254001': {'obsid': '00087254001', 'success': True, 'obsoutdir': PosixPath('/media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test_nopa/00087254001'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00087254001',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00046362008': {'obsid': '00046362008', 'success': True, 'obsoutdir': PosixPath('/media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test_nopa/00046362008'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00046362008',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00087077002': {'obsid': '00087077002', 'success': True, 'obsoutdir': PosixPath('/media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test_nopa/00087077002'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00087077002',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00055763032': {'obsid': '00055763032', 'success': True, 'obsoutdir': PosixPath('/media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test_nopa/00055763032'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00055763032',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00031511027': {'obsid': '00031511027', 'success': True, 'obsoutdir': PosixPath('/media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test_nopa/00031511027'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00031511027',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}}
And here is the result of a new ls -R:
/media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test_nopa$ ls -R
.:
00031511027 00034856001 00055763032 00087077002 00092352073
00034843001 00046362008 00087071007 00087254001
./00031511027:
auxil bat
./00031511027/auxil:
sw00031511027pat.fits.gz sw00031511027sen.hk.gz
sw00031511027pjb.par.gz sw00031511027s.mkf.gz
sw00031511027pob.cat.gz sw00031511027sti.fits.gz
sw00031511027ppr.par.gz sw00031511027uat.fits.gz
sw00031511027sao.fits.gz SWIFT_TLE_ARCHIVE.txt.16363.91429006.gz
sw00031511027sat.fits.gz
./00031511027/bat:
hk masktag rate survey
./00031511027/bat/hk:
sw00031511027bdecb.hk.gz sw00031511027bgocb.hk.gz
sw00031511027ben.hk.gz sw00031511027bhd.hk.gz
./00031511027/bat/masktag:
sw00031511027bmt00010009.lc.gz sw00031511027bmt00010500.lc.gz
sw00031511027bmt00010009_rw.lc.gz sw00031511027bmt00010500_rw.lc.gz
./00031511027/bat/rate:
sw00031511027brt1s.lc.gz sw00031511027brtms.lc.gz
sw00031511027brtmc.lc.gz sw00031511027brtqd.lc.gz
./00031511027/bat/survey:
sw00031511027bsvpbo0b13g041e.dph.gz
./00034843001:
auxil bat
./00034843001/auxil:
sw00034843001pat.fits.gz sw00034843001sen.hk.gz
sw00034843001pjb.par.gz sw00034843001s.mkf.gz
sw00034843001pob.cat.gz sw00034843001sti.fits.gz
sw00034843001ppr.par.gz sw00034843001uat.fits.gz
sw00034843001sao.fits.gz SWIFT_TLE_ARCHIVE.txt.16361.85726095.gz
sw00034843001sat.fits.gz
./00034843001/bat:
event hk masktag rate survey
./00034843001/bat/event:
sw00034843001bevshpo_uf.evt.gz sw00034843001bevshsl_uf.evt.gz
./00034843001/bat/hk:
sw00034843001bdecb.hk.gz sw00034843001ben.hk.gz sw00034843001bgocb.hk.gz
sw00034843001bdp.hk.gz sw00034843001bevtlsp.hk.gz sw00034843001bhd.hk.gz
sw00034843001bdqcb.hk.gz sw00034843001bevtssp.hk.gz
./00034843001/bat/masktag:
sw00034843001bmt00010009.lc.gz sw00034843001bmt00010500.lc.gz
sw00034843001bmt00010009_rw.lc.gz sw00034843001bmt00010500_rw.lc.gz
./00034843001/bat/rate:
sw00034843001brt1s.lc.gz sw00034843001brtms.lc.gz
sw00034843001brtmc.lc.gz sw00034843001brtqd.lc.gz
./00034843001/bat/survey:
sw00034843001bsvpbo0af3g0412.dph.gz sw00034843001bsvpbo0af6g0413.dph.gz
sw00034843001bsvpbo0af4g0413.dph.gz sw00034843001bsvpbo0b06g0419.dph.gz
sw00034843001bsvpbo0af5g0413.dph.gz
./00034856001:
auxil bat
./00034856001/auxil:
sw00034856001pat.fits.gz sw00034856001sen.hk.gz
sw00034856001pjb.par.gz sw00034856001s.mkf.gz
sw00034856001pob.cat.gz sw00034856001sti.fits.gz
sw00034856001ppr.par.gz sw00034856001uat.fits.gz
sw00034856001sao.fits.gz SWIFT_TLE_ARCHIVE.txt.16363.91429006.gz
sw00034856001sat.fits.gz
./00034856001/bat:
event hk masktag rate survey
./00034856001/bat/event:
sw00034856001bevshpo_uf.evt.gz sw00034856001bevshsl_uf.evt.gz
./00034856001/bat/hk:
sw00034856001bdecb.hk.gz sw00034856001ben.hk.gz sw00034856001bgocb.hk.gz
sw00034856001bdp.hk.gz sw00034856001bevtlsp.hk.gz sw00034856001bhd.hk.gz
sw00034856001bdqcb.hk.gz sw00034856001bevtssp.hk.gz
./00034856001/bat/masktag:
sw00034856001bmt00010005.lc.gz sw00034856001bmt00010009_rw.lc.gz
sw00034856001bmt00010005_rw.lc.gz sw00034856001bmt00010603.lc.gz
sw00034856001bmt00010009.lc.gz sw00034856001bmt00010603_rw.lc.gz
./00034856001/bat/rate:
sw00034856001brt1s.lc.gz sw00034856001brtms.lc.gz
sw00034856001brtmc.lc.gz sw00034856001brtqd.lc.gz
./00034856001/bat/survey:
sw00034856001bsvpbo0b0bg041b.dph.gz sw00034856001bsvpbo0b0eg041c.dph.gz
sw00034856001bsvpbo0b0cg041b.dph.gz sw00034856001bsvpbo0b11g041d.dph.gz
sw00034856001bsvpbo0b0dg041c.dph.gz
./00046362008:
auxil bat
./00046362008/auxil:
sw00046362008pat.fits.gz sw00046362008sen.hk.gz
sw00046362008pjb.par.gz sw00046362008s.mkf.gz
sw00046362008pob.cat.gz sw00046362008sti.fits.gz
sw00046362008ppr.par.gz sw00046362008uat.fits.gz
sw00046362008sao.fits.gz SWIFT_TLE_ARCHIVE.txt.16363.91429006.gz
sw00046362008sat.fits.gz
./00046362008/bat:
hk masktag rate survey
./00046362008/bat/hk:
sw00046362008bdecb.hk.gz sw00046362008ben.hk.gz sw00046362008bhd.hk.gz
sw00046362008bdp.hk.gz sw00046362008bgocb.hk.gz
./00046362008/bat/masktag:
sw00046362008bmt00010009.lc.gz sw00046362008bmt00010500.lc.gz
sw00046362008bmt00010009_rw.lc.gz sw00046362008bmt00010500_rw.lc.gz
./00046362008/bat/rate:
sw00046362008brt1s.lc.gz sw00046362008brtms.lc.gz
sw00046362008brtmc.lc.gz sw00046362008brtqd.lc.gz
./00046362008/bat/survey:
sw00046362008bsvpbo0b10g041d.dph.gz
./00055763032:
auxil bat
./00055763032/auxil:
sw00055763032pat.fits.gz sw00055763032sat.fits.gz
sw00055763032pjb.par.gz sw00055763032sen.hk.gz
sw00055763032pob.cat.gz sw00055763032s.mkf.gz
sw00055763032ppr.par.gz sw00055763032sti.fits.gz
sw00055763032sao.fits.gz SWIFT_TLE_ARCHIVE.txt.16363.91429006.gz
./00055763032/bat:
hk masktag rate survey
./00055763032/bat/hk:
sw00055763032bdecb.hk.gz sw00055763032ben.hk.gz sw00055763032bhd.hk.gz
sw00055763032bdp.hk.gz sw00055763032bgocb.hk.gz
./00055763032/bat/masktag:
sw00055763032bmt00010002_rw.lc.gz sw00055763032bmt00010009.lc.gz
sw00055763032bmt00010004_rw.lc.gz sw00055763032bmt00010009_rw.lc.gz
./00055763032/bat/rate:
sw00055763032brt1s.lc.gz sw00055763032brtms.lc.gz
sw00055763032brtmc.lc.gz sw00055763032brtqd.lc.gz
./00055763032/bat/survey:
sw00055763032bsvpbo0b07g041a.dph.gz
./00087071007:
auxil bat
./00087071007/auxil:
sw00087071007pat.fits.gz sw00087071007sen.hk.gz
sw00087071007pjb.par.gz sw00087071007s.mkf.gz
sw00087071007pob.cat.gz sw00087071007sti.fits.gz
sw00087071007ppr.par.gz sw00087071007uat.fits.gz
sw00087071007sao.fits.gz SWIFT_TLE_ARCHIVE.txt.16361.85726095.gz
sw00087071007sat.fits.gz
./00087071007/bat:
event hk masktag rate survey
./00087071007/bat/event:
sw00087071007bevshpo_uf.evt.gz sw00087071007bevshsl_uf.evt.gz
./00087071007/bat/hk:
sw00087071007bdecb.hk.gz sw00087071007ben.hk.gz sw00087071007bgocb.hk.gz
sw00087071007bdp.hk.gz sw00087071007bevtlsp.hk.gz sw00087071007bhd.hk.gz
sw00087071007bdqcb.hk.gz sw00087071007bevtssp.hk.gz
./00087071007/bat/masktag:
sw00087071007bmt00010009.lc.gz sw00087071007bmt00010500.lc.gz
sw00087071007bmt00010009_rw.lc.gz sw00087071007bmt00010500_rw.lc.gz
./00087071007/bat/rate:
sw00087071007brt1s.lc.gz sw00087071007brtms.lc.gz
sw00087071007brtmc.lc.gz sw00087071007brtqd.lc.gz
./00087071007/bat/survey:
sw00087071007bsvpbo0aeeg0410.dph.gz sw00087071007bsvpbo0b02g0418.dph.gz
sw00087071007bsvpbo0afdg0416.dph.gz
./00087077002:
auxil bat
./00087077002/auxil:
sw00087077002pat.fits.gz sw00087077002sat.fits.gz
sw00087077002pjb.par.gz sw00087077002sen.hk.gz
sw00087077002pob.cat.gz sw00087077002s.mkf.gz
sw00087077002ppr.par.gz sw00087077002sti.fits.gz
sw00087077002sao.fits.gz SWIFT_TLE_ARCHIVE.txt.16363.91429006.gz
./00087077002/bat:
event hk masktag rate survey
./00087077002/bat/event:
sw00087077002bevshpo_uf.evt.gz sw00087077002bevshsl_uf.evt.gz
./00087077002/bat/hk:
sw00087077002bdecb.hk.gz sw00087077002ben.hk.gz sw00087077002bgocb.hk.gz
sw00087077002bdp.hk.gz sw00087077002bevtlsp.hk.gz sw00087077002bhd.hk.gz
sw00087077002bdqcb.hk.gz sw00087077002bevtssp.hk.gz
./00087077002/bat/masktag:
sw00087077002bmt00010009.lc.gz sw00087077002bmt00010500.lc.gz
sw00087077002bmt00010009_rw.lc.gz sw00087077002bmt00010500_rw.lc.gz
./00087077002/bat/rate:
sw00087077002brt1s.lc.gz sw00087077002brtms.lc.gz
sw00087077002brtmc.lc.gz sw00087077002brtqd.lc.gz
./00087077002/bat/survey:
sw00087077002bsvpbo0b10g041d.dph.gz
./00087254001:
auxil bat
./00087254001/auxil:
sw00087254001pat.fits.gz sw00087254001sen.hk.gz
sw00087254001pjb.par.gz sw00087254001s.mkf.gz
sw00087254001pob.cat.gz sw00087254001sti.fits.gz
sw00087254001ppr.par.gz sw00087254001uat.fits.gz
sw00087254001sao.fits.gz SWIFT_TLE_ARCHIVE.txt.16363.05166484.gz
sw00087254001sat.fits.gz
./00087254001/bat:
hk masktag rate survey
./00087254001/bat/hk:
sw00087254001bdecb.hk.gz sw00087254001ben.hk.gz sw00087254001bhd.hk.gz
sw00087254001bdp.hk.gz sw00087254001bgocb.hk.gz
./00087254001/bat/masktag:
sw00087254001bmt00010009.lc.gz sw00087254001bmt00010500.lc.gz
sw00087254001bmt00010009_rw.lc.gz sw00087254001bmt00010500_rw.lc.gz
./00087254001/bat/rate:
sw00087254001brt1s.lc.gz sw00087254001brtms.lc.gz
sw00087254001brtmc.lc.gz sw00087254001brtqd.lc.gz
./00087254001/bat/survey:
sw00087254001bsvpbo0afag0415.dph.gz
./00092352073:
auxil
./00092352073/auxil:
sw00092352073pat.fits.gz sw00092352073sen.hk.gz
sw00092352073pjb.par.gz sw00092352073s.mkf.gz
sw00092352073pob.cat.gz sw00092352073sti.fits.gz
sw00092352073ppr.par.gz sw00092352073uat.fits.gz
sw00092352073sao.fits.gz SWIFT_TLE_ARCHIVE.txt.16363.05166484.gz
sw00092352073sat.fits.gz
HI @Parrazyte, redownloading the data in your new directory worked fine. Eg the obsid 00034843001 has the bat/survey subdirectory and it has various .dph files in it which is what is needed. In your other directory for that obsid, there is no bat/survey subdirectory.
Can I ask that you try to delete all the observation ID folders in your new directory and try to redownload those observations with result = ba.download_swiftdata(table_exposed, jobs=2)
? This will test whether the parallelization was the cause of the "data missing" issue for you.
this is what I get with jobs=2:
{'00034843001': {'obsid': '00034843001', 'success': True, 'obsoutdir': PosixPath('/media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test_pa2/00034843001'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00034843001',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00034856001': {'obsid': '00034856001', 'success': True, 'obsoutdir': PosixPath('/media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test_pa2/00034856001'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00034856001',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00087071007': {'obsid': '00087071007', 'success': True, 'obsoutdir': PosixPath('/media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test_pa2/00087071007'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00087071007',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00092352073': {'obsid': '00092352073', 'success': True, 'obsoutdir': PosixPath('/media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test_pa2/00092352073'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00092352073',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00087254001': {'obsid': '00087254001', 'success': True, 'obsoutdir': PosixPath('/media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test_pa2/00087254001'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00087254001',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00046362008': {'obsid': '00046362008', 'success': True, 'obsoutdir': PosixPath('/media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test_pa2/00046362008'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00046362008',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00087077002': {'obsid': '00087077002', 'success': True, 'obsoutdir': PosixPath('/media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test_pa2/00087077002'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00087077002',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00055763032': {'obsid': '00055763032', 'success': True, 'obsoutdir': PosixPath('/media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test_pa2/00055763032'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00055763032',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}, '00031511027': {'obsid': '00031511027', 'success': True, 'obsoutdir': PosixPath('/media/parrama/crucial_SSD/Observ/BHLMXB/Swift/BAT_analysis/test_pa2/00031511027'), 'quicklook': False, 'data': Swift_Data(username='anonymous',obsid='00031511027',quicklook='False',auxil='True',bat='True',xrt='False',uvot='False',log='False',tdrss='True')}}
looking at the folder structures, it seems the result is the same than with jobs=1 (302 items for 178Mb). I also retested without the jobs parameters, and now I get the same thing (aka, much more files than previously). I have to say I'm very confused.
As I still didn't have anything when running the analysis with these new downloads, I did more digging by running a single _create_BatSurvey with a breakpoint, and it turns out that the issue was simply a lack of CALDB initalization before running my python console.
Now with this solved, I do get 6 successful analysis out of the 9 folders. I know it is very dumb but I would suggest testing for a $CALDB or the likes in os.environ before executing the scripts.
I was able to run batspectrum_analysis too, and I have a few questions:
-Is it possible to run batspectrum_analysis without a model (as I just want to create the spectra) ?
-How does the spectrum structure work ? I see that there are pha and rsp created, but also upper limits. I'm a bit confused about them being 5 sigma despite the xspec errors being 1 sigma usually ? Also, how are these upper limit files created? Can the upper limit change?
-I tried computing spectra for mosaics but while outventory_file=ba.merge_outventory(batsurvey_obs)
works, I have issues with the time bins, e.g.:
time_bins = ba.group_outventory(
outventory_file, np.timedelta64(1, "D"), end_datetime=Time("2016-12-21"))
Traceback (most recent call last):
File "/home/parrama/Documents/Work/PhD/Scripts/Python/observations/venv/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3526, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-18-ad6b0738b9e4>", line 1, in <module>
time_bins = ba.group_outventory(
File "/home/parrama/Documents/Work/PhD/Scripts/Python/observations/venv/lib/python3.9/site-packages/batanalysis/mosaic.py", line 368, in group_outventory
if binning_timedelta==np.timedelta64(1,'M'):
TypeError: Cannot get a common metadata divisor for Numpy datetime metadata [D] and [M] because they have incompatible nonlinear base time units.
and
time_bins = ba.group_outventory(
outventory_file, np.timedelta64(1, "M"), end_datetime=Time("2016-12-21"),recalc=True)
time_bins
Out[22]: array(['2016-12'], dtype='datetime64[M]')
works, but then I get the following issue with batmosaic_analysis
mosaic_list, total_mosaic = ba.parallel.batmosaic_analysis(
batsurvey_obs, outventory_file, time_bins, nprocs=8
)
Traceback (most recent call last):
File "/home/parrama/Documents/Work/PhD/Scripts/Python/observations/venv/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3526, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-24-5a1a677db8d7>", line 1, in <module>
mosaic_list, total_mosaic = ba.parallel.batmosaic_analysis(
File "/home/parrama/Documents/Work/PhD/Scripts/Python/observations/venv/lib/python3.9/site-packages/batanalysis/parallel.py", line 282, in batmosaic_analysis
total_mosaic_savedir=intermediate_mosaic_dir_list[0].parent.joinpath("total_mosaic")
IndexError: list index out of range
@Parrazyte My prediction is that there must have been something preventing the proper data download when you initially tried downloading the data and the download function got stuck thinking that all the necessary data was downloaded in subsequent runs of the data download function. But, I am glad that it is working now. If this issue pops up again, I think that deleting all of the downloaded observation ID directories and redownloading all data will help with this
Adding a check for $CALDB is a good idea, would you be able to open a separate GitHub issue for that so we can individually test such an implementation?
As for your spectra related questions, you can get more detailed information in the Example_Survey_Data_Analysis.ipynb notebook and in the published paper but I will try to quickly answer the questions here as well:
Thanks a lot!
I'll open a CALDB issue and look more to the notebooks then.
sounds good. I will close this issue since it has technically been resolved. Feel free to open others if things go awry/arent clear.
Thanks for using the BatAnalysis tool @Parrazyte.
Hi, I'm trying to make the bat_analysis tools work and am facing some issues. When trying to reproduce what is done in e.g. the trial_detection_Crab or trial_NGC2992 notebooks ex:
Gives me the output
Then, when trying to download with
I get :
With X being every single of the 12 observations. I get the same behavior when trying to download a part of the NGC2992 survey dataset.
Now, when replacing with more recent data e.g.
The data does download perfectly, but then trying to reduce it with
I always obtain outputs similar to this
Batsurvey_obs ends up empty as there are never any gti intervals found. This issue is the most problematic as I get the same behavior for the source I want to analyze (4U1630-47), namely the data downloads but no GTIs are ever detected.
Am I missing something?