Open micheledoro opened 2 years ago
We discussed it's better to keep the data outside the github, so I should write a small script to download data from elsewhere and decide where to store them
@giacomodamico24 @ibatkovi ok, I made it, please check the script https://github.com/arqus-gamma/arqus_school_2022/blob/main/download_data.sh. It should work on your machine and download for the moment the crab_nebula data
@ibatkovi, as discussed earlier, we would like to have different data samples, one is the crab nebula already downloaded, the other ones discussed are (possibly) NGC1275, Mrk421, M87. You should already have NGC1275 data in DL3 format. I would like to have maybe two sets, those of the flare (1 jan, to grant easy detection) and anothe dataset (not to many data). Can you put share those data?
Hi Michele,
Of course, I planned to put that data up. I was also checking some datasets from IC310 (I have 6 runs from the flaring state) and M87 that I could produce if needed (flare of 2010 and 2018 or some quiescent nights from those years). If we process the Mrk421 data, we could have that too. On the other hand, I checked some releases from the HESS and there are a few runs of PKS2155+304 and CrabNebula that we could use if we want.
Let me know what you think about this, Cheers, Ivana
On Mon, 11 Jul 2022 at 14:33, Michele Doro @.***> wrote:
@ibatkovi https://github.com/ibatkovi, as discussed earlier, we would like to have different data samples, one is the crab nebula already downloaded, the other ones discussed are (possibly) NGC1275, Mrk421, M87. You should already have NGC1275 data in DL3 format. I would like to have maybe two sets, those of the flare (1 jan, to grant easy detection) and anothe dataset (not to many data). Can you put share those data?
— Reply to this email directly, view it on GitHub https://github.com/arqus-gamma/arqus_school_2022/issues/1#issuecomment-1180353885, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASRO4AW2UEYLUQN2R4TMUHTVTQIBBANCNFSM5ZGHBVTA . You are receiving this because you were mentioned.Message ID: @.***>
Where are the PKS data?
On Mon, 11 Jul 2022 at 14:47, Ivana Batković @.***> wrote:
Hi Michele,
Of course, I planned to put that data up. I was also checking some datasets from IC310 (I have 6 runs from the flaring state) and M87 that I could produce if needed (flare of 2010 and 2018 or some quiescent nights from those years). If we process the Mrk421 data, we could have that too. On the other hand, I checked some releases from the HESS and there are a few runs of PKS2155+304 and CrabNebula that we could use if we want.
Let me know what you think about this, Cheers, Ivana
On Mon, 11 Jul 2022 at 14:33, Michele Doro @.***> wrote:
@ibatkovi https://github.com/ibatkovi, as discussed earlier, we would like to have different data samples, one is the crab nebula already downloaded, the other ones discussed are (possibly) NGC1275, Mrk421, M87. You should already have NGC1275 data in DL3 format. I would like to have maybe two sets, those of the flare (1 jan, to grant easy detection) and anothe dataset (not to many data). Can you put share those data?
— Reply to this email directly, view it on GitHub < https://github.com/arqus-gamma/arqus_school_2022/issues/1#issuecomment-1180353885 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/ASRO4AW2UEYLUQN2R4TMUHTVTQIBBANCNFSM5ZGHBVTA
. You are receiving this because you were mentioned.Message ID: @.***>
— Reply to this email directly, view it on GitHub https://github.com/arqus-gamma/arqus_school_2022/issues/1#issuecomment-1180367008, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZ3U4ZXISDDLDAUFGWOMOLVTQJTVANCNFSM5ZGHBVTA . You are receiving this because you authored the thread.Message ID: @.***>
You can see their release from 2018 https://arxiv.org/pdf/1810.04516.pdf and the folder is with the dl3 data is available for download here: https://www.mpi-hd.mpg.de/hfm/HESS/pages/dl3-dr1/. We still need to understand if we can use it because it says dl3_dl1, which I didn't find what it means in their release.
On Mon, 11 Jul 2022 at 14:49, Michele Doro @.***> wrote:
Where are the PKS data?
On Mon, 11 Jul 2022 at 14:47, Ivana Batković @.***> wrote:
Hi Michele,
Of course, I planned to put that data up. I was also checking some datasets from IC310 (I have 6 runs from the flaring state) and M87 that I could produce if needed (flare of 2010 and 2018 or some quiescent nights from those years). If we process the Mrk421 data, we could have that too. On the other hand, I checked some releases from the HESS and there are a few runs of PKS2155+304 and CrabNebula that we could use if we want.
Let me know what you think about this, Cheers, Ivana
On Mon, 11 Jul 2022 at 14:33, Michele Doro @.***> wrote:
@ibatkovi https://github.com/ibatkovi, as discussed earlier, we would like to have different data samples, one is the crab nebula already downloaded, the other ones discussed are (possibly) NGC1275, Mrk421, M87. You should already have NGC1275 data in DL3 format. I would like to have maybe two sets, those of the flare (1 jan, to grant easy detection) and anothe dataset (not to many data). Can you put share those data?
— Reply to this email directly, view it on GitHub <
https://github.com/arqus-gamma/arqus_school_2022/issues/1#issuecomment-1180353885
, or unsubscribe <
https://github.com/notifications/unsubscribe-auth/ASRO4AW2UEYLUQN2R4TMUHTVTQIBBANCNFSM5ZGHBVTA
. You are receiving this because you were mentioned.Message ID: @.***>
— Reply to this email directly, view it on GitHub < https://github.com/arqus-gamma/arqus_school_2022/issues/1#issuecomment-1180367008 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAZ3U4ZXISDDLDAUFGWOMOLVTQJTVANCNFSM5ZGHBVTA
. You are receiving this because you authored the thread.Message ID: @.***>
--
Michele Doro Associate Professor at Department of Physics and Astronomy G. Galilei DFA, University of Padova, via Marzolo 8, I-35131, Padova, Italy Email: @., @. Homepage: www.pd.infn.it/~mdoro Office: +39.049.827.5909 Fax: +39 049.827.7201 ORCID: https://orcid.org/0000-0001-9104-3214
— Reply to this email directly, view it on GitHub https://github.com/arqus-gamma/arqus_school_2022/issues/1#issuecomment-1180369119, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASRO4AXGTXOGPR7BSOWTQGDVTQJ35ANCNFSM5ZGHBVTA . You are receiving this because you were mentioned.Message ID: @.***>
This is the file list at the moment
|-- arqus_filelist.txt
|-- crab_nebula
| |-- fact
| | |-- 20131103_103_dl3.fits
| | |-- 20131103_104_dl3.fits
| | |-- 20131103_105_dl3.fits
| | |-- 20131103_106_dl3.fits
| | |-- 20131103_109_dl3.fits
| | |-- 20131103_110_dl3.fits
| | |-- 20131103_111_dl3.fits
| | |-- 20131103_112_dl3.fits
| | |-- 20131103_115_dl3.fits
| | |-- 20131103_116_dl3.fits
| | |-- 20131103_117_dl3.fits
| | |-- 20131103_118_dl3.fits
| | |-- 20131103_128_dl3.fits
| | |-- 20131103_129_dl3.fits
| | |-- 20131103_130_dl3.fits
| | |-- 20131103_131_dl3.fits
| | |-- 20131103_134_dl3.fits
| | |-- 20131103_135_dl3.fits
| | |-- 20131103_136_dl3.fits
| | |-- 20131103_137_dl3.fits
| | |-- 20131103_140_dl3.fits
| | |-- 20131103_141_dl3.fits
| | |-- 20131103_142_dl3.fits
| | |-- 20131103_143_dl3.fits
| | |-- 20131103_153_dl3.fits
| | |-- 20131103_154_dl3.fits
| | |-- 20131103_155_dl3.fits
| | |-- 20131103_156_dl3.fits
| | |-- 20131103_159_dl3.fits
| | |-- 20131103_160_dl3.fits
| | |-- 20131103_161_dl3.fits
| | |-- 20131103_162_dl3.fits
| | |-- 20131103_165_dl3.fits
| | |-- 20131103_166_dl3.fits
| | |-- 20131103_167_dl3.fits
| | |-- 20131103_168_dl3.fits
| | |-- 20131103_178_dl3.fits
| | |-- 20131103_179_dl3.fits
| | |-- 20131103_180_dl3.fits
| | |-- 20131103_181_dl3.fits
| | |-- 20131103_184_dl3.fits
| | |-- 20131103_93_dl3.fits
| | |-- 20131104_162_dl3.fits
| | |-- 20131104_163_dl3.fits
| | |-- 20131104_164_dl3.fits
| | |-- 20131104_165_dl3.fits
| | |-- 20131104_168_dl3.fits
| | |-- 20131104_169_dl3.fits
| | |-- 20131104_170_dl3.fits
| | |-- 20131104_171_dl3.fits
| | |-- 20131104_181_dl3.fits
| | |-- 20131104_182_dl3.fits
| | |-- 20131104_183_dl3.fits
| | |-- 20131104_184_dl3.fits
| | |-- 20131104_187_dl3.fits
| | |-- 20131104_188_dl3.fits
| | |-- 20131104_189_dl3.fits
| | |-- 20131104_190_dl3.fits
| | |-- 20131104_193_dl3.fits
| | |-- 20131104_194_dl3.fits
| | |-- 20131104_195_dl3.fits
| | |-- 20131104_206_dl3.fits
| | |-- 20131104_207_dl3.fits
| | |-- 20131104_208_dl3.fits
| | |-- 20131104_209_dl3.fits
| | |-- 20131104_212_dl3.fits
| | |-- 20131104_213_dl3.fits
| | |-- 20131104_214_dl3.fits
| | |-- 20131104_215_dl3.fits
| | |-- 20131104_218_dl3.fits
| | |-- 20131104_219_dl3.fits
| | |-- 20131104_220_dl3.fits
| | |-- 20131104_221_dl3.fits
| | |-- 20131104_231_dl3.fits
| | |-- 20131104_232_dl3.fits
| | |-- 20131104_233_dl3.fits
| | |-- 20131104_234_dl3.fits
| | |-- 20131104_237_dl3.fits
| | |-- 20131104_238_dl3.fits
| | |-- 20131104_239_dl3.fits
| | |-- 20131104_240_dl3.fits
| | |-- 20131104_243_dl3.fits
| | |-- 20131104_244_dl3.fits
| | |-- 20131104_245_dl3.fits
| | |-- 20131104_246_dl3.fits
| | |-- 20131105_152_dl3.fits
| | |-- 20131105_155_dl3.fits
| | |-- 20131105_156_dl3.fits
| | |-- 20131105_157_dl3.fits
| | |-- 20131105_158_dl3.fits
| | |-- 20131105_161_dl3.fits
| | |-- 20131105_162_dl3.fits
| | |-- 20131105_163_dl3.fits
| | |-- 20131105_164_dl3.fits
| | |-- 20131105_174_dl3.fits
| | |-- 20131105_177_dl3.fits
| | |-- 20131105_180_dl3.fits
| | |-- 20131105_181_dl3.fits
| | |-- 20131105_182_dl3.fits
| | |-- 20131105_183_dl3.fits
| | |-- 20131105_186_dl3.fits
| | |-- 20131105_187_dl3.fits
| | |-- 20131105_188_dl3.fits
| | |-- 20131105_189_dl3.fits
| | |-- 20131105_205_dl3.fits
| | |-- 20131105_206_dl3.fits
| | |-- 20131105_207_dl3.fits
| | |-- 20131105_208_dl3.fits
| | |-- 20131105_209_dl3.fits
| | |-- 20131105_212_dl3.fits
| | |-- 20131105_213_dl3.fits
| | |-- 20131105_214_dl3.fits
| | |-- 20131105_215_dl3.fits
| | |-- 20131105_225_dl3.fits
| | |-- 20131105_226_dl3.fits
| | |-- 20131105_227_dl3.fits
| | |-- 20131105_228_dl3.fits
| | |-- 20131105_231_dl3.fits
| | |-- 20131105_232_dl3.fits
| | |-- 20131105_233_dl3.fits
| | |-- 20131105_234_dl3.fits
| | |-- 20131105_237_dl3.fits
| | |-- 20131105_238_dl3.fits
| | |-- 20131105_239_dl3.fits
| | |-- 20131105_240_dl3.fits
| | |-- README.md
| | |-- fact_irf.fits
| | |-- hdu-index.fits.gz
| | `-- obs-index.fits.gz
| |-- fermi
| | |-- README.md
| | |-- events.fits.gz
| | |-- exposure_cube.fits.gz
| | |-- make.py
| | `-- psf.fits.gz
| |-- hess
| | |-- README.md
| | |-- data
| | | |-- hess_dl3_dr1_obs_id_023523.fits.gz
| | | |-- hess_dl3_dr1_obs_id_023526.fits.gz
| | | |-- hess_dl3_dr1_obs_id_023559.fits.gz
| | | `-- hess_dl3_dr1_obs_id_023592.fits.gz
| | |-- hdu-index.fits.gz
| | |-- make.py
| | `-- obs-index.fits.gz
| |-- magic
| | |-- README.md
| | |-- hdu-index.fits.gz
| | |-- obs-index.fits.gz
| | |-- run_05029747_DL3.fits
| | `-- run_05029748_DL3.fits
| |-- other
| | |-- README.md
| | `-- crab_mwl.fits.gz
| `-- veritas
| |-- 54809_DL3.fits
| |-- 57993_DL3.fits
| |-- README.md
| |-- hdu-index.fits.gz
| `-- obs-index.fits.gz
|-- ic310
| `-- magic
| |-- 20121113_05020910_DL3_NGC1275-W0.40+238.fits
| |-- 20121113_05020911_DL3_NGC1275-W0.40+157.fits
| |-- 20121113_05020915_DL3_NGC1275-W0.40+157.fits
| |-- 20121113_05020918_DL3_NGC1275-W0.40+238.fits
| |-- 20121113_05020919_DL3_NGC1275-W0.40+157.fits
| |-- 20121113_05020922_DL3_NGC1275-W0.40+238.fits
| `-- magicDL3_ST0301_IC310_04offset.rc
|-- ngc1275
| `-- magic
| |-- 20170101_05059487_DL3_NGC1275-W0.40+157.fits
| |-- 20170101_05059488_DL3_NGC1275-W0.40+337.fits
| |-- 20170101_05059489_DL3_NGC1275-W0.40+058.fits
| |-- 20170101_05059490_DL3_NGC1275-W0.40+238.fits
| |-- 20170101_05059497_DL3_NGC1275-W0.40+238.fits
| |-- 20170101_05059498_DL3_NGC1275-W0.40+157.fits
| |-- 20170101_05059499_DL3_NGC1275-W0.40+337.fits
| |-- 20170101_05059500_DL3_NGC1275-W0.40+337.fits
| |-- hdu-index.fits.gz
| `-- obs-index.fits.gz
`-- pks2155
`-- hess
|-- hess_dl3_dr1_obs_id_033787.fits.gz
|-- hess_dl3_dr1_obs_id_033788.fits.gz
|-- hess_dl3_dr1_obs_id_033789.fits.gz
|-- hess_dl3_dr1_obs_id_033790.fits.gz
|-- hess_dl3_dr1_obs_id_033791.fits.gz
|-- hess_dl3_dr1_obs_id_033792.fits.gz
|-- hess_dl3_dr1_obs_id_033793.fits.gz
|-- hess_dl3_dr1_obs_id_033794.fits.gz
|-- hess_dl3_dr1_obs_id_033795.fits.gz
|-- hess_dl3_dr1_obs_id_033796.fits.gz
|-- hess_dl3_dr1_obs_id_033797.fits.gz
|-- hess_dl3_dr1_obs_id_033798.fits.gz
|-- hess_dl3_dr1_obs_id_033799.fits.gz
|-- hess_dl3_dr1_obs_id_033800.fits.gz
`-- hess_dl3_dr1_obs_id_033801.fits.gz
14 directories, 187 files
Maybe we can remove FACT CRAB data and possibly other CRAB data not from MAGIC HESS
I run the script download_data.sh
and it works fine on my Mac.
What about instead those students that will come with a PC with Windows? Will they be able to download the data?
I have added a notebook to show how to load fits files using astropy
Please check if it works also for you
It works!
I have edited the README.me files in the folder. The only thing missing for the moment is
Hi I have check the Mrk421 data. I asked Juan Bernete about it, and he has Melibea files of Mrk421 from 2020_10_25 to 2021_06_07, meaning that, in case if he could give me the melibea MC flies too, and I could do the conversion.
Other possibility is to use the LST Mrk421 dl3 files that we have from Chaitanya, or the ones that you Michele converted for the ALPs+blazars analysis.
I have added Mrk421 data and updated the tree file list. Now all data for the school should be there.
Hi, I selected PKS2155 data and put them in the arqus data folder. Ivana, I guess DR1 means data release 1
MD
On Mon, 11 Jul 2022 at 15:03, Ivana Batković @.***> wrote:
You can see their release from 2018 https://arxiv.org/pdf/1810.04516.pdf and the folder is with the dl3 data is available for download here: https://www.mpi-hd.mpg.de/hfm/HESS/pages/dl3-dr1/. We still need to understand if we can use it because it says dl3_dl1, which I didn't find what it means in their release.
On Mon, 11 Jul 2022 at 14:49, Michele Doro @.***> wrote:
Where are the PKS data?
On Mon, 11 Jul 2022 at 14:47, Ivana Batković @.***> wrote:
Hi Michele,
Of course, I planned to put that data up. I was also checking some datasets from IC310 (I have 6 runs from the flaring state) and M87 that I could produce if needed (flare of 2010 and 2018 or some quiescent nights from those years). If we process the Mrk421 data, we could have that too. On the other hand, I checked some releases from the HESS and there are a few runs of PKS2155+304 and CrabNebula that we could use if we want.
Let me know what you think about this, Cheers, Ivana
On Mon, 11 Jul 2022 at 14:33, Michele Doro @.***> wrote:
@ibatkovi https://github.com/ibatkovi, as discussed earlier, we would like to have different data samples, one is the crab nebula already downloaded, the other ones discussed are (possibly) NGC1275, Mrk421, M87. You should already have NGC1275 data in DL3 format. I would like to have maybe two sets, those of the flare (1 jan, to grant easy detection) and anothe dataset (not to many data). Can you put share those data?
— Reply to this email directly, view it on GitHub <
https://github.com/arqus-gamma/arqus_school_2022/issues/1#issuecomment-1180353885
, or unsubscribe <
https://github.com/notifications/unsubscribe-auth/ASRO4AW2UEYLUQN2R4TMUHTVTQIBBANCNFSM5ZGHBVTA
. You are receiving this because you were mentioned.Message ID: @.***>
— Reply to this email directly, view it on GitHub <
https://github.com/arqus-gamma/arqus_school_2022/issues/1#issuecomment-1180367008
, or unsubscribe <
https://github.com/notifications/unsubscribe-auth/AAZ3U4ZXISDDLDAUFGWOMOLVTQJTVANCNFSM5ZGHBVTA
. You are receiving this because you authored the thread.Message ID: @.***>
--
Michele Doro Associate Professor at Department of Physics and Astronomy G. Galilei DFA, University of Padova, via Marzolo 8, I-35131, Padova, Italy Email: @., @. Homepage: www.pd.infn.it/~mdoro Office: +39.049.827.5909 Fax: +39 049.827.7201 ORCID: https://orcid.org/0000-0001-9104-3214
— Reply to this email directly, view it on GitHub < https://github.com/arqus-gamma/arqus_school_2022/issues/1#issuecomment-1180369119 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/ASRO4AXGTXOGPR7BSOWTQGDVTQJ35ANCNFSM5ZGHBVTA
. You are receiving this because you were mentioned.Message ID: @.***>
— Reply to this email directly, view it on GitHub https://github.com/arqus-gamma/arqus_school_2022/issues/1#issuecomment-1180383738, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZ3U4ZMD76APNZOGNXO66LVTQLSJANCNFSM5ZGHBVTA . You are receiving this because you authored the thread.Message ID: @.***>
Hi Michele,
I am not 100% sure since this it HESS data, but I assume yes. The data is for sure level 3 (dl3), but the release is probably numbered as 1, yes.
Cheers, Ivana
On Tue, 11 Oct 2022 at 11:04, Michele Doro @.***> wrote:
Hi, I selected PKS2155 data and put them in the arqus data folder. Ivana, I guess DR1 means data release 1
MD
On Mon, 11 Jul 2022 at 15:03, Ivana Batković @.***> wrote:
You can see their release from 2018 https://arxiv.org/pdf/1810.04516.pdf and the folder is with the dl3 data is available for download here: https://www.mpi-hd.mpg.de/hfm/HESS/pages/dl3-dr1/. We still need to understand if we can use it because it says dl3_dl1, which I didn't find what it means in their release.
On Mon, 11 Jul 2022 at 14:49, Michele Doro @.***> wrote:
Where are the PKS data?
On Mon, 11 Jul 2022 at 14:47, Ivana Batković @.***> wrote:
Hi Michele,
Of course, I planned to put that data up. I was also checking some datasets from IC310 (I have 6 runs from the flaring state) and M87 that I could produce if needed (flare of 2010 and 2018 or some quiescent nights from those years). If we process the Mrk421 data, we could have that too. On the other hand, I checked some releases from the HESS and there are a few runs of PKS2155+304 and CrabNebula that we could use if we want.
Let me know what you think about this, Cheers, Ivana
On Mon, 11 Jul 2022 at 14:33, Michele Doro @.***> wrote:
@ibatkovi https://github.com/ibatkovi, as discussed earlier, we would like to have different data samples, one is the crab nebula already downloaded, the other ones discussed are (possibly) NGC1275, Mrk421, M87. You should already have NGC1275 data in DL3 format. I would like to have maybe two sets, those of the flare (1 jan, to grant easy detection) and anothe dataset (not to many data). Can you put share those data?
— Reply to this email directly, view it on GitHub <
https://github.com/arqus-gamma/arqus_school_2022/issues/1#issuecomment-1180353885
, or unsubscribe <
https://github.com/notifications/unsubscribe-auth/ASRO4AW2UEYLUQN2R4TMUHTVTQIBBANCNFSM5ZGHBVTA
. You are receiving this because you were mentioned.Message ID: @.***>
— Reply to this email directly, view it on GitHub <
https://github.com/arqus-gamma/arqus_school_2022/issues/1#issuecomment-1180367008
, or unsubscribe <
https://github.com/notifications/unsubscribe-auth/AAZ3U4ZXISDDLDAUFGWOMOLVTQJTVANCNFSM5ZGHBVTA
. You are receiving this because you authored the thread.Message ID: @.***>
--
Michele Doro Associate Professor at Department of Physics and Astronomy G. Galilei DFA, University of Padova, via Marzolo 8, I-35131, Padova, Italy Email: @., @. Homepage: www.pd.infn.it/~mdoro Office: +39.049.827.5909 Fax: +39 049.827.7201 ORCID: https://orcid.org/0000-0001-9104-3214
— Reply to this email directly, view it on GitHub <
https://github.com/arqus-gamma/arqus_school_2022/issues/1#issuecomment-1180369119
, or unsubscribe <
https://github.com/notifications/unsubscribe-auth/ASRO4AXGTXOGPR7BSOWTQGDVTQJ35ANCNFSM5ZGHBVTA
. You are receiving this because you were mentioned.Message ID: @.***>
— Reply to this email directly, view it on GitHub < https://github.com/arqus-gamma/arqus_school_2022/issues/1#issuecomment-1180383738 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAZ3U4ZMD76APNZOGNXO66LVTQLSJANCNFSM5ZGHBVTA
. You are receiving this because you authored the thread.Message ID: @.***>
--
Michele Doro Associate Professor at Department of Physics and Astronomy G. Galilei DFA, University of Padova, via Marzolo 8, I-35131, Padova, Italy Email: @., @. Homepage: www.pd.infn.it/~mdoro Office: +39.049.827.5909 Fax: +39 049.827.7201 ORCID: https://orcid.org/0000-0001-9104-3214
— Reply to this email directly, view it on GitHub https://github.com/arqus-gamma/arqus_school_2022/issues/1#issuecomment-1274356921, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASRO4AWPER2MO64GX3H6BRLWCUUQRANCNFSM5ZGHBVTA . You are receiving this because you were mentioned.Message ID: @.***>
@giacomodamico24 I have uploaded the joint_crab dataset and updated the README.md.