search

gammapy / enrico

Enrico helps you with your Fermi data analysis
http://enrico.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
22 stars 26 forks source link

New tool: enrico_getdata? #9

Open cdeil opened 12 years ago

cdeil commented 12 years ago

We should make a tool that gets the data for the ROI, energy range, event class the user wants from one of these locations:

Making this a separate tool from enrico_like gives more flexibility for the user how to get started with the analysis, i.e. where to get the input photon and spacecraft files.

cdeil commented 11 years ago

Making Fermi data server queries from Python was implemented in astroquery.fermi by @keflavich .

The implementation and documentation is not finished (e.g. https://github.com/astropy/astroquery/pull/181), but once it is we should integrate it into Enrico.

If astropy and astroquery installation works with the upcoming Fermi LAT ScienceTools software (@kialio and @davidsanchez could maybe check?) we can simply use astroquery as a dependency for Enrico ... if not I'll try to extract a copy of astroquery.fermi and put it into Enrico.

keflavich commented 11 years ago

Let me know how the install goes; the latest PR made astroquery compatible with astropy < 0.3.

kialio commented 11 years ago

The data server won't change when the new data release happens. Should be exactly the same but it will be serving P7Rep instead of P7.

One thing to note: We've had a few cases in the past year of users not thinking through the scripting process and submitting 100's (and one time 1000's) of queries in less than a minute. This pretty much looks like a dds attack and overwhelms our servers. We've implemented checks for this and block IPs that do it. I'm not sure if you can implement something in enrico that prohibits this kind of behavior but if you can, that would be good. I'll cross-post this to astroquery as well.

In the end, if a user wants to submit 1000's of queries, it would be more efficient for the user to grab the weekly files instead. These are updated as fast as the data in the data server. ie. the most recent weekly file might not be a full week, it just includes the most recent data.

davidsanchez commented 11 years ago

To get the data, I use enrico_download which download the weekly file and I think this is quite useful.

2013/9/13 Jeremy Perkins notifications@github.com

The data server won't change when the new data release happens. Should be exactly the same but it will be serving P7Rep instead of P7.

One thing to note: We've had a few cases in the past year of users not thinking through the scripting process and submitting 100's (and one time 1000's) of queries in less than a minute. This pretty much looks like a dds attack and overwhelms our servers. We've implemented checks for this and block IPs that do it. I'm not sure if you can implement something in enrico that prohibits this kind of behavior but if you can, that would be good. I'll cross-post this to astroquery as well.

In the end, if a user wants to submit 1000's of queries, it would be more efficient for the user to grab the weekly files instead. These are updated as fast as the data in the data server. ie. the most recent weekly file might not be a full week, it just includes the most recent data.

— Reply to this email directly or view it on GitHubhttps://github.com/gammapy/enrico/issues/9#issuecomment-24394036 .

David Sanchez