search

fermiPy / fermipy

Fermi-LAT Python Analysis Framework
http://fermipy.readthedocs.org/
BSD 3-Clause "New" or "Revised" License
52 stars 53 forks source link

Error in generating srcmaps #23

Closed astro-fermi closed 8 years ago

astro-fermi commented 8 years ago

Hello,

I am trying to perform a basic analysis with fermipy. When running 'gta.setup()', I get the following error

2016-03-29 15:38:28 INFO GTBinnedAnalysis.run_gtapp(): Running gtsrcmaps 2016-03-29 15:38:28 INFO GTBinnedAnalysis.run_gtapp(): time -p /Users/vaidehi/Softwares/ScienceTools-v10r0p5-fssc-patch-20150518-x86_64-apple-darwin14.4.0-without-root/x86_64-apple-darwin14.4.0/bin/gtsrcmaps scfile=ft2.fits sctable="SC_DATA" expcube=/Users/vaidehi/Downloads/FermiPy_Trial/output/ltcube_00.fits cmap=/Users/vaidehi/Downloads/FermiPy_Trial/output/ccube_00.fits srcmdl=/Users/vaidehi/Downloads/FermiPy_Trial/output/srcmdl_00.xml bexpmap=/Users/vaidehi/Downloads/FermiPy_Trial/output/bexpmap_00.fits outfile=/Users/vaidehi/Downloads/FermiPy_Trial/output/srcmap_00.fits irfs="P8R2_SOURCE_V6" evtype=4 convol=yes resample=yes rfactor=2 minbinsz=0.05 ptsrc=yes psfcorr=yes emapbnds=no copyall=no chatter=4 clobber=yes debug=no gui=no mode="ql" 2016-03-29 15:38:50 INFO GTBinnedAnalysis.run_gtapp(): This is gtsrcmaps version ScienceTools-v10r0p5-fssc-20150819 ...... many lines .... 2016-03-29 15:38:50 INFO GTBinnedAnalysis.run_gtapp(): Generating SourceMap for galdiffMapBase::readFitsFile: creating WcsMap2 object 2016-03-29 15:38:50 INFO GTBinnedAnalysis.run_gtapp(): 2016-03-29 15:38:50 INFO GTBinnedAnalysis.run_gtapp(): resampling factor: 2 2016-03-29 15:38:50 INFO GTBinnedAnalysis.run_gtapp(): mapsize: 680 2016-03-29 15:38:50 INFO GTBinnedAnalysis.run_gtapp(): Caught St13runtime_error at the top level: WcsMap2: Requested energy index is out-of-range.

The sample config file is attached. sample_config.yaml.txt

)

Please help.

woodmd commented 8 years ago

The problem is that the lower energy bound of your analysis (selection.emin=50) is less than the first bin edge of the standard galactic diffuse model (~58 MeV). The STs do not support using mapcube components that do not fully encompass the analysis energy range. To fix this issue you should make sure that your emin/emax fall within the energy range of the IEM (~58 MeV to ~580 GeV). For the minimum energy I would actually recommend using 100 MeV unless you have prior experience doing LAT analysis at low energies. You can find the matrix of analysis recommendations provided by the FSSC on this page.

astro-fermi commented 8 years ago

Thanks for the help. That problem got solved and I was able to proceed. As you would have noticed that I am trying to run component analysis (with 4 PSFs). The tool was running without any issue for first 3 analysis components, however, in the last step, while generating ltcube, it showed the following error

Traceback (most recent call last): File "", line 1, in File "/Users/vaidehi/Softwares/ScienceTools-v10r0p5-fssc-patch-20150518-x86_64-apple-darwin14.4.0-without-root/x86_64-apple-darwin14.4.0/lib/python2.7/site-packages/fermipy/gtanalysis.py", line 687, in setup c.setup(overwrite=overwrite) File "/Users/vaidehi/Softwares/ScienceTools-v10r0p5-fssc-patch-20150518-x86_64-apple-darwin14.4.0-without-root/x86_64-apple-darwin14.4.0/lib/python2.7/site-packages/fermipy/gtanalysis.py", line 3842, in setup self.energies) File "/Users/vaidehi/Softwares/ScienceTools-v10r0p5-fssc-patch-20150518-x86_64-apple-darwin14.4.0-without-root/x86_64-apple-darwin14.4.0/lib/python2.7/site-packages/fermipy/irfs.py", line 46, in init self._dtheta,egy,cth_min) File "/Users/vaidehi/Softwares/ScienceTools-v10r0p5-fssc-patch-20150518-x86_64-apple-darwin14.4.0-without-root/x86_64-apple-darwin14.4.0/lib/python2.7/site-packages/fermipy/irfs.py", line 86, in create_average_psf for et in event_types: TypeError: 'NoneType' object is not iterable

I am interested in understanding this component wise PSF analysis. Please help.

Thanks.

astro-fermi commented 8 years ago

I think I got the mistake. Thanks.

woodmd commented 8 years ago

Ok good. I guess you figured it out but the problem was that your configuration was creating 8 components instead of 4. Each element of components should be a dictionary with the same hierarchy as the root configuration. So in your example the components section should look like:

components:
  - { selection : { evtype : 4  }, model: {isodiff: $FERMI_DIR/refdata/fermi/galdiffuse/iso_P8R2_SOURCE_V6_PSF0_v06.txt} } # PSF0
  - { selection : { evtype : 8  }, model: {isodiff: $FERMI_DIR/refdata/fermi/galdiffuse/iso_P8R2_SOURCE_V6_PSF1_v06.txt} } # PSF1
  - { selection : { evtype : 16 }, model: {isodiff: $FERMI_DIR/refdata/fermi/galdiffuse/iso_P8R2_SOURCE_V6_PSF2_v06.txt} } # PSF2
  - { selection : { evtype : 32 }, model: {isodiff: $FERMI_DIR/refdata/fermi/galdiffuse/iso_P8R2_SOURCE_V6_PSF3_v06.txt} } # PSF3
astro-fermi commented 8 years ago

I followed your suggestions but now there is a new issue. It is generating all the files associated with 4 components but then I face the following error ... many lines ... 2016-03-30 22:11:07 INFO GTAnalysis.setup(): Initializing source properties 2016-03-30 22:11:07 DEBUG GTAnalysis.get_src_model(): Generating source dict for 3FGL J0434.6+0921 2016-03-30 22:11:07 DEBUG GTAnalysis.get_src_model(): Generating source dict for 3FGL J0440.3+1444 2016-03-30 22:11:07 DEBUG GTAnalysis.get_src_model(): Generating source dict for 3FGL J0449.0+1121 2016-03-30 22:11:07 DEBUG GTAnalysis.get_src_model(): Generating source dict for 3FGL J0457.0+0643 2016-03-30 22:11:07 DEBUG GTAnalysis.get_src_model(): Generating source dict for galdiff Traceback (most recent call last): File "", line 1, in File "/Users/vaidehi/Softwares/ScienceTools-v10r0p5-fssc-patch-20150518-x86_64-apple-darwin14.4.0-without-root/x86_64-apple-darwin14.4.0/lib/python2.7/site-packages/fermipy/gtanalysis.py", line 699, in setup self._init_source(name) File "/Users/vaidehi/Softwares/ScienceTools-v10r0p5-fssc-patch-20150518-x86_64-apple-darwin14.4.0-without-root/x86_64-apple-darwin14.4.0/lib/python2.7/site-packages/fermipy/gtanalysis.py", line 745, in _init_source src = self.roi.get_source_by_name(name, True) File "/Users/vaidehi/Softwares/ScienceTools-v10r0p5-fssc-patch-20150518-x86_64-apple-darwin14.4.0-without-root/x86_64-apple-darwin14.4.0/lib/python2.7/site-packages/fermipy/roi_model.py", line 1722, in get_source_by_name raise Exception('No source matching name: ' + name) Exception: No source matching name: isodiff

I am attaching the latest yaml and log files here. Thanks. sample_config.yaml.txt LOG.txt

woodmd commented 8 years ago

This was a bug. I just uploaded a new 0.6.8 release of fermipy to pypi that includes a patch to fix this issue. You should be able to install it with: pip install fermipy --upgrade --no-deps Let me know if you run into any other issues.

astro-fermi commented 8 years ago

I was able to perform the fitting but when running 'gta.write_roi('fit_model')' command, suddenly python got crashed and it threw me out. See the attached screenshot. Has it something to do with my Mac? screen shot 2016-03-31 at 11 31 26 am

woodmd commented 8 years ago

That's not an error that I've seen before. Since it happened in write_roi it might be an issue with the graphical backend. Could you try running matplotlib.use('agg') at the top of your script?

astro-fermi commented 8 years ago

Yes. When I used matplotlib.use('agg'), it finished without any error. Since the analysis is over, I want to see the final TS, flux, and index values of the source of interest. I found two files 'fit_model.npy' and 'fit_model.yaml'. But they contains lots of output. Please tell me where I should look (or what command I shall give) to get the parameters of interest. Thanks.

astro-fermi commented 8 years ago

I Googled little bit and understood about how to print a dictionary value. Thanks.

woodmd commented 8 years ago

Sorry for not being more responsive about this. Creating some documentation for the output file has been on my to-do list. If you're already familiar with LAT analysis then most of the dictionary values should be pretty self-explanatory. The follow example shows how you would extract some basic information about a source from the output file. One thing to note is that some entries contain both the value and its 1-sigma error in a 2-element numpy array. Examples of this are all of the flux and differential flux quantities (flux, eflux, dfde, etc.).

o = np.load('data.npy').flat[0]
src = o['sources']['MySource']
print 'Test Statistic: ', src['ts']
print 'Predicted Counts: ', src['Npred']
print 'Energy Flux Value/Error: ', src['eflux']
print 'Flux Value/Error: ', src['flux']
print 'Fit Parameters: ', src['params']