search

gammapy / enrico

Enrico helps you with your Fermi data analysis
http://enrico.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
22 stars 26 forks source link

General Query #169

Open ajaykhoj opened 2 years ago

ajaykhoj commented 2 years ago

Hello

This is not an issue. I just want to know that when we prepare to analyze a source we should keep in mind what we want to do and set different parameters accordingly. Suppose I want to analyze a source and for that I have downloaded the required fits files from FSSC archive with 20 degree search radius. Now I have fixed the ROI equal to 20 degrees and want to accept all sources with TS equal to 25 and no sources with TS less than 25, lie within the ROI. Now when I ran command "enrico_sed" for unbinned likelihood analysis, I got some error while computing/creating the exposure map.

****

*** 7 gtexpmap --- Make an exposure map

****

time -p gtexpmap evfile=/home/ajay/enrico/5S0716+714_Full_roi_20_degree/S50716+714_TestofS50716+714_FRONTBACK_MkTime.fits evtable="EVENTS" scfile=/home/ajay/enrico/5S0716+714_Full_roi_20_degree/L220516023160FD19599823_SC00.fits sctable="SC_DATA" expcube=/home/ajay/enrico/5S0716+714_Full_roi_20_degree/S50716+714_TestofS50716+714_FRONTBACK_ltCube.fits outfile=/home/ajay/enrico/5S0716+714_Full_roi_20_degree/S50716+714_TestofS50716+714_FRONTBACK_ExpMap.fits irfs="P8R3_SOURCE_V3" evtype=3 srcrad=35.0 nlong=120 nlat=120 nenergies=35 submap=no nlongmin=0 nlongmax=0 nlatmin=0 nlatmax=0 chatter=2 clobber=yes debug=no gui=no mode="ql" The exposure maps generated by this tool are meant to be used for unbinned likelihood analysis only. Do not use them for binned analyses. Using evtype=3 (i.e., FRONT/BACK irfs) Caught St13runtime_error at the top level: RoiCuts::sortCuts: [Message] Warning: An error ocurred, retrying ... time -p gtexpmap evfile=/home/ajay/enrico/5S0716+714_Full_roi_20_degree/S50716+714_TestofS50716+714_FRONTBACK_MkTime.fits evtable="EVENTS" scfile=/home/ajay/enrico/5S0716+714_Full_roi_20_degree/L220516023160FD19599823_SC00.fits sctable="SC_DATA" expcube=/home/ajay/enrico/5S0716+714_Full_roi_20_degree/S50716+714_TestofS50716+714_FRONTBACK_ltCube.fits outfile=/home/ajay/enrico/5S0716+714_Full_roi_20_degree/S50716+714_TestofS50716+714_FRONTBACK_ExpMap.fits irfs="P8R3_SOURCE_V3" evtype=3 srcrad=35.0 nlong=120 nlat=120 nenergies=35 submap=no nlongmin=0 nlongmax=0 nlatmin=0 nlatmax=0 chatter=2 clobber=yes debug=no gui=no mode="ql" The exposure maps generated by this tool are meant to be used for unbinned likelihood analysis only. Do not use them for binned analyses. Using evtype=3 (i.e., FRONT/BACK irfs) Caught St13runtime_error at the top level: RoiCuts::sortCuts: [Message] Warning: An error ocurred, retrying ... time -p gtexpmap evfile=/home/ajay/enrico/5S0716+714_Full_roi_20_degree/S50716+714_TestofS50716+714_FRONTBACK_MkTime.fits evtable="EVENTS" scfile=/home/ajay/enrico/5S0716+714_Full_roi_20_degree/L220516023160FD19599823_SC00.fits sctable="SC_DATA" expcube=/home/ajay/enrico/5S0716+714_Full_roi_20_degree/S50716+714_TestofS50716+714_FRONTBACK_ltCube.fits outfile=/home/ajay/enrico/5S0716+714_Full_roi_20_degree/S50716+714_TestofS50716+714_FRONTBACK_ExpMap.fits irfs="P8R3_SOURCE_V3" evtype=3 srcrad=35.0 nlong=120 nlat=120 nenergies=35 submap=no nlongmin=0 nlongmax=0 nlatmin=0 nlatmax=0 chatter=2 clobber=yes debug=no gui=no mode="ql" The exposure maps generated by this tool are meant to be used for unbinned likelihood analysis only. Do not use them for binned analyses. Using evtype=3 (i.e., FRONT/BACK irfs) Caught St13runtime_error at the top level: RoiCuts::sortCuts: [Message] Warning: An error ocurred, retrying ... time -p gtexpmap evfile=/home/ajay/enrico/5S0716+714_Full_roi_20_degree/S50716+714_TestofS50716+714_FRONTBACK_MkTime.fits evtable="EVENTS" scfile=/home/ajay/enrico/5S0716+714_Full_roi_20_degree/L220516023160FD19599823_SC00.fits sctable="SC_DATA" expcube=/home/ajay/enrico/5S0716+714_Full_roi_20_degree/S50716+714_TestofS50716+714_FRONTBACK_ltCube.fits outfile=/home/ajay/enrico/5S0716+714_Full_roi_20_degree/S50716+714_TestofS50716+714_FRONTBACK_ExpMap.fits irfs="P8R3_SOURCE_V3" evtype=3 srcrad=35.0 nlong=120 nlat=120 nenergies=35 submap=no nlongmin=0 nlongmax=0 nlatmin=0 nlatmax=0 chatter=2 clobber=yes debug=no gui=no mode="ql" The exposure maps generated by this tool are meant to be used for unbinned likelihood analysis only. Do not use them for binned analyses. Using evtype=3 (i.e., FRONT/BACK irfs) Caught St13runtime_error at the top level: RoiCuts::sortCuts: [Message] Warning: An error ocurred, retrying ... time -p gtexpmap evfile=/home/ajay/enrico/5S0716+714_Full_roi_20_degree/S50716+714_TestofS50716+714_FRONTBACK_MkTime.fits evtable="EVENTS" scfile=/home/ajay/enrico/5S0716+714_Full_roi_20_degree/L220516023160FD19599823_SC00.fits sctable="SC_DATA" expcube=/home/ajay/enrico/5S0716+714_Full_roi_20_degree/S50716+714_TestofS50716+714_FRONTBACK_ltCube.fits outfile=/home/ajay/enrico/5S0716+714_Full_roi_20_degree/S50716+714_TestofS50716+714_FRONTBACK_ExpMap.fits irfs="P8R3_SOURCE_V3" evtype=3 srcrad=35.0 nlong=120 nlat=120 nenergies=35 submap=no nlongmin=0 nlongmax=0 nlatmin=0 nlatmax=0 chatter=2 clobber=yes debug=no gui=no mode="ql" The exposure maps generated by this tool are meant to be used for unbinned likelihood analysis only. Do not use them for binned analyses. Using evtype=3 (i.e., FRONT/BACK irfs) Caught St13runtime_error at the top level: RoiCuts::sortCuts: [Message] Warning: An error ocurred, retrying ...

Error [Message]: An error ocurred and could not be recovered. Exiting!

================================================================================================== Now I need your help to solve this. And can you also explain , when i want to set ROI =20 degree, what should be the value of parameters in Model section of config file and should i set "roicut=yes" in analysis section of config file to eliminate this error.

can you explain what does mean of "max_radius" in model section of .conf file ?

.conf file is :-

Default config and validation file for Enrico.

#

Besides providing default options it is also used

to check the user config file.

Options without default options here must be

specified in the user config file

Folder where the output will be stored

out = /home/ajay/enrico/5S0716+714_Full_roi_20_degree

verbosity

verbose = yes

Overwrite existing output files. Use with caution

clobber = yes

Submit the job to a cluster?

Submit = no

[target] name = S50716+714 ra = 110.473 dec = 71.3434 redshift = 0.0 spectrum = PowerLaw2 ebl_model = 4

0=Kneiske, 1=Primack05, 2=Kneiske_HighUV, 3=Stecker05, 4=Franceschini, 5=Finke, 6=Gilmore, 10=Dominguez11

# full list http://www-glast.stanford.edu/cgi-bin/viewcvs/celestialSources/eblAtten/eblAtten/EblAtten.h?view=markup
fit_tau = no

[space] xref = 110.473 yref = 71.3434 rad = 20.0 binsz = 0.1 coordsys = CEL proj = AIT phibins = 0

[file] spacecraft = /home/ajay/enrico/5S0716+714_Full_roi_20_degree/L220516023160FD19599823_SC00.fits event = /home/ajay/enrico/5S0716+714_Full_roi_20_degree/data.list xml = /home/ajay/enrico/5S0716+714_Full_roi_20_degree/S50716+714_PowerLaw2_model.xml tag = TestofS50716+714

keep track of the parent config during nested operations (e.g. energy/time bins)

parent_config = ""

[time] tmin = 239557417.0 tmax = 271093418.0 file = "" type = MET

[energy] emin = 100.0 emax = 300000.0 enumbins_per_decade = 10 decorrelation_energy = no

[event] irfs = CALDB evclass = 128 evtype = 3

[analysis] zmax = 90

General analysis options

likelihood = unbinned
# Correct energy dispersion?
EnergyDispersion = yes
ComputeDiffrsp = yes
roicut = no
# cuts applied during the first coarse bin selection:
evtroicuts = yes
evttimecuts = yes
filter = (DATA_QUAL>0)&&(LAT_CONFIG==1)
# should we keep the srcmaps for all sources in the srcMap file?
# this should speed up future fits, at the cost of disk space.
keep_all_srcmaps = no

[environ]

Analysis environment configuration

# Can also be done via shell environment variables
FERMI_DATA_DIR = ""
FERMI_CATALOG_DIR = ""
FERMI_CATALOG = ""
FERMI_DIFFUSE_DIR = ""
FERMI_PREPROCESSED_DIR = ""

[fitting] optimizer = NEWMINUIT ftol = 1e-06

if source_ts < min_source_TS in the fit, the source is removed from the model

min_source_TS = 1.0

[model]

The following options determine the xml model

diffuse_gal_dir = ""
diffuse_iso_dir = ""
diffuse_gal = ""
diffuse_iso = ""
# user points sources for diffuse catalog sources
# freeze spectral parameters for weak and far away sources:
min_significance_catalog = 4.0
min_significance_free = 25.0
max_radius = 5.0
# Add all source in a radius of analyse ROI + this radius
max_roi = 20.0
# Add gaussian noise to the spectral parameters of free sources (may help during the fit)
parameters_noise = 0.0

[Spectrum]

Generates fits files or not?

FitsGeneration = yes
#Generates plots (SED, model map)
ResultPlots = yes
#Plot (update) the parent config related plots during energy bin production
ResultParentPlots = no
#Freeze the spectral index of the source. Has no implication if 0 (Left free to vary)
FrozenSpectralIndex = 0.0
#Write covariance matrix to csv file
WriteCovMatrix = yes
#Write residuals to csv file
WriteResiduals = yes

[UpperLimit]

Assumed Spectral index

SpectralIndex = 2.0
# UL method could be Profile or Integral (provided by the fermi collaboration).
#Poisson is base on the Feldman-Cousins method for low signal. Only 95 % yet available
Method = Profile
envelope = no
#Compute an UL if the TS of the sources is <TSlimit
TSlimit = 25.0
# Confidence level for the Ul computation
cl = 0.95

[LightCurve]

Generates fits files or not?

FitsGeneration = yes
#Number of points for the LC, bin size is 7 days
NLCbin = 53
#Index for the power law. Left free to vary if 0
SpectralIndex = 2.0
MakeConfFile = yes
#Bayesian blocks
BayesianBlocks = no
#Compute Variability index as in the 2FGL.
ComputeVarIndex = yes
#Compute an UL if the TS of the sources is <TSLightCurve
TSLightCurve = 25.0
#Generates control plots
DiagnosticPlots = yes
# Plot (update) the parent config related plots during time bin production
DiagnosticParentPlots = no

[BayesianBlocks]

Generates fits files or not?

FitsGeneration = yes
#False alarm probability
p0 = 0.01
#Index for the power law. Left free to vary if 0
SpectralIndex = 2.0
MakeConfFile = yes
#Compute an UL if the TS of the sources is <TSLightCurve
TSLightCurve = 25.0
#Generates control plots
DiagnosticPlots = yes
# Plot (update) the parent config related plots during time bin production
DiagnosticParentPlots = no

[FoldedLC]

Number of bins for the orbitally folded LC

NLCbin = 10
#Epoch of phase=0 in MJD, equal to tmin is 0
epoch = 0.0
#Orbital period in days
Period = 0.0

[AppLC]

Generates fits files or not?

FitsGeneration = yes
#Spectral index for the exposure calculation
index = 2.0
#Apperture radius
rad = 1.0
#Number of bins
NLCbin = 10
#bin form data or frozen bin size
binsFromData = no
#minimum energy for the apperture lightcurve, if -1 use the one define in energy section
emin = -1.0
emax = -1.0

[Ebin]

Generates fits files or not?

FitsGeneration = yes
NumEnergyBins = 15
#Compute an UL if the TS of the sources is <TSEnergyBins
TSEnergyBins = 9.0
# How to generate the Ebins
# - logE: log scaled bins
# - TS: similar TS per bin
# - mix: half-way between log and TS.
# - [ E1, E2, E3, E4, ..., EN ]: custom binning. 
# - default is logE
DistEbins = logE

[TSMap]

Re-fit before computing the TS map

Re-Fit = no
#Numbers of pixel in x and y
npix = 10
#Remove or not the target from the model
RemoveTarget = yes
#Generate the TS map pixel by pixel or by grouping the pixels by row.
#(reduce the numbers of jobs but each job is longer)
method = row

[findsrc]

Generates fits files or not?

FitsGeneration = yes
#Reoptimize before
Refit = yes

[srcprob]

Generates fits files or not?

FitsGeneration = yes
#radius for the computation
rad = 1.0
# list of sources
srclist = ""
# number of photons to print
numberPhoton = 10

[Contours] parname1 = Prefactor parname2 = Index

[ComponentAnalysis]

EUnBinned to the energy (MeV) where

# binned and unbinned analysis should be split
# disabled if -1
EUnBinned = -1.0
# Divide the analysis in FrontBack classes
FrontBack = no
# Divide the analysis in PSF classes
PSF = no
# Divide the analysis in EDISP classes
EDISP = no
# Divide the analysis in EDISP classes
FGL4 = no

[ObservationSimulation] infile = Simmodel.xml srclist = source_names.txt

ajaykhoj commented 2 years ago

Hi,

I tried to solve this problem, i couldn't. First i downloaded the data with 30 degree radius and then choose 20 degree as region of interest (ROI). Then i ran the programme(enrico_sed ...) , but it killed in the middle, i don't know, why it has been stopped. C an someone help? Any help is much appreciated. Thanks

what i did, i'm mentioning here -> Data(with 30 degree radius) -> Choose ROI 20 degree (set "rad=20" in Space section of .conf file ) -> what should be the value of "min_significance_catalog, min_significance_free, max_radius, max_roi " ? in model section [model]

The following options determine the xml model

    diffuse_gal_dir = ""
    diffuse_iso_dir = ""
    diffuse_gal = ""
    diffuse_iso = ""
    # user points sources for diffuse catalog sources
    # freeze spectral parameters for weak and far away sources:
    min_significance_catalog = 4.0
    min_significance_free = 25.0
    max_radius = 5.0
    # Add all source in a radius of analyse ROI + this radius
    max_roi = 10.0
    # Add gaussian noise to the spectral parameters of free sources (may help during the fit)
    parameters_noise = 0.0
ajaykhoj commented 1 year ago

Hello,

I am getting an error while doing unbinned likelihood analysis.

time -p gtmodel srcmaps=/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_PowerLaw2_srcMap.fits srcmdl=/home/ajay/enrico/PKS0346-27/PKS0346-27_PowerLaw2_TestofPKS_out.xml outfile=/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_PowerLaw2_ModelMap.fits irfs="P8R3_SOURCE_V3" evtype="INDEF" expcube=/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_ltCube.fits bexpmap=/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_BinnedMap.fits convol=yes resample=yes rfactor=2 outtype="CMAP" psfcorr=yes phased_expmap=none edisp_bins=0 chatter=2 clobber=yes debug=no gui=no mode="ql" Parameter srcmaps: file "/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_PowerLaw2_srcMap.fits" is not accessible in mode r. Caught N5hoops12ApeExceptionE at the top level: Exception while querying for file name parameter srcmaps for component gtmodel; Ape exception code 43 (at /usr/share/miniconda/conda-bld/fermitools_1611158415999/work/hoops/build/debianstretch/sid-x86_64-64bit-gcc75-Optimized/src/hoops_ape.cxx: 565) [Message] Warning: An error ocurred, retrying ... time -p gtmodel srcmaps=/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_PowerLaw2_srcMap.fits srcmdl=/home/ajay/enrico/PKS0346-27/PKS0346-27_PowerLaw2_TestofPKS_out.xml outfile=/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_PowerLaw2_ModelMap.fits irfs="P8R3_SOURCE_V3" evtype="INDEF" expcube=/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_ltCube.fits bexpmap=/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_BinnedMap.fits convol=yes resample=yes rfactor=2 outtype="CMAP" psfcorr=yes phased_expmap=none edisp_bins=0 chatter=2 clobber=yes debug=no gui=no mode="ql" Parameter srcmaps: file "/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_PowerLaw2_srcMap.fits" is not accessible in mode r. Caught N5hoops12ApeExceptionE at the top level: Exception while querying for file name parameter srcmaps for component gtmodel; Ape exception code 43 (at /usr/share/miniconda/conda-bld/fermitools_1611158415999/work/hoops/build/debianstretch/sid-x86_64-64bit-gcc75-Optimized/src/hoops_ape.cxx: 565) [Message] Warning: An error ocurred, retrying ... time -p gtmodel srcmaps=/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_PowerLaw2_srcMap.fits srcmdl=/home/ajay/enrico/PKS0346-27/PKS0346-27_PowerLaw2_TestofPKS_out.xml outfile=/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_PowerLaw2_ModelMap.fits irfs="P8R3_SOURCE_V3" evtype="INDEF" expcube=/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_ltCube.fits bexpmap=/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_BinnedMap.fits convol=yes resample=yes rfactor=2 outtype="CMAP" psfcorr=yes phased_expmap=none edisp_bins=0 chatter=2 clobber=yes debug=no gui=no mode="ql" Parameter srcmaps: file "/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_PowerLaw2_srcMap.fits" is not accessible in mode r. Caught N5hoops12ApeExceptionE at the top level: Exception while querying for file name parameter srcmaps for component gtmodel; Ape exception code 43 (at /usr/share/miniconda/conda-bld/fermitools_1611158415999/work/hoops/build/debianstretch/sid-x86_64-64bit-gcc75-Optimized/src/hoops_ape.cxx: 565) [Message] Warning: An error ocurred, retrying ... time -p gtmodel srcmaps=/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_PowerLaw2_srcMap.fits srcmdl=/home/ajay/enrico/PKS0346-27/PKS0346-27_PowerLaw2_TestofPKS_out.xml outfile=/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_PowerLaw2_ModelMap.fits irfs="P8R3_SOURCE_V3" evtype="INDEF" expcube=/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_ltCube.fits bexpmap=/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_BinnedMap.fits convol=yes resample=yes rfactor=2 outtype="CMAP" psfcorr=yes phased_expmap=none edisp_bins=0 chatter=2 clobber=yes debug=no gui=no mode="ql" Parameter srcmaps: file "/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_PowerLaw2_srcMap.fits" is not accessible in mode r. Caught N5hoops12ApeExceptionE at the top level: Exception while querying for file name parameter srcmaps for component gtmodel; Ape exception code 43 (at /usr/share/miniconda/conda-bld/fermitools_1611158415999/work/hoops/build/debianstretch/sid-x86_64-64bit-gcc75-Optimized/src/hoops_ape.cxx: 565) [Message] Warning: An error ocurred, retrying ... time -p gtmodel srcmaps=/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_PowerLaw2_srcMap.fits srcmdl=/home/ajay/enrico/PKS0346-27/PKS0346-27_PowerLaw2_TestofPKS_out.xml outfile=/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_PowerLaw2_ModelMap.fits irfs="P8R3_SOURCE_V3" evtype="INDEF" expcube=/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_ltCube.fits bexpmap=/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_BinnedMap.fits convol=yes resample=yes rfactor=2 outtype="CMAP" psfcorr=yes phased_expmap=none edisp_bins=0 chatter=2 clobber=yes debug=no gui=no mode="ql" Parameter srcmaps: file "/home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_PowerLaw2_srcMap.fits" is not accessible in mode r. Caught N5hoops12ApeExceptionE at the top level: Exception while querying for file name parameter srcmaps for component gtmodel; Ape exception code 43 (at /usr/share/miniconda/conda-bld/fermitools_1611158415999/work/hoops/build/debianstretch/sid-x86_64-64bit-gcc75-Optimized/src/hoops_ape.cxx: 565) [Message] Warning: An error ocurred, retrying ...

Error [Message]: An error ocurred and could not be recovered. Exiting!

(fermi2.0.8) ajay@ajay:~/enrico/PKS0346-27$ enrico_sed PKS.conf

any help would be much appreciated.

mireianievas commented 1 year ago

Does this exist? /home/ajay/enrico/PKS0346-27/PKS0346-27_TestofPKS_FRONTBACK_PowerLaw2_srcMap.fits ?