search

vtsuperdarn / davitpy

DEPRECATED The DaViT Python project
http://vtsuperdarn.github.com/davitpy/
GNU General Public License v3.0
37 stars 59 forks source link

Bugfix rbsp website update #237

Closed ksterne closed 8 years ago

ksterne commented 8 years ago

Found the website information for accessing Van Allen probe orbit data outdated in #209. This pull request fixes this bug by updating the URL as well as adjusting to the new returned data format from the JHU/APL server.

You can confirm that the old URL does not work by using:

# Get all the FPs for 2/Jul/2016 from 0 to 12 UT
from datetime import datetime
import rbsp

    print ""
    print "Testing footprint collection and apogee calculation..."
    print ""
    print "Expected results for orbits on July 2, 2012 between"
    print "00:00 and 12:00 utc:"
    print "    01:45 UT, A: ( 68.47 N, 94.93 E)     (-51.43 N, 106.23 E)"
    print "    01:55 UT, B: ( 68.44 N, 92.27 E)     (-51.56 N, 104.03 E)"
    print ""
    print "Calculated results:"
    print ""
    sTime = datetime(2016, 7, 2, 0)
    eTime = datetime(2016, 7, 2, 12)
    fps = rbsp.rbspFp(sTime, eTime)
    # Pretty print the apogees in that period
    print fps

I figured out what was going on before. Before I had an example way in the past which the code goes and accesses a database we generate at VT. So, I've set the date here to go out in the future which we haven't written values to the database.

On the master/develop branch, I expect you'll get a "connection timed out" type of error. On the proposed branch here, it should work fairly well. I've also added a 'force_web_read' parameter that can be set to force the code to re-read the data from the JHU/APL website instead of relying on the VT database.

Also added in a unit test for the module to hopefully start conforming to a file being able to test itself. The code above is largely the unit test for the file, so when using this branch, you can just run python rbsp.py in that directory.

I'm thinking I've got this somewhat down now, but please test away!

Shirling-VT commented 8 years ago

Hi Kevin,

I just tested it with the examples you provided. The old branch does give a "connection time out" error. And the branch you proposed is working well in my new computer.

Best wishes, Xueling

On Sat, Apr 16, 2016 at 12:47 AM, Kevin Sterne notifications@github.com wrote:

Found the website information for accessing Van Allen probe orbit data outdated in #209 https://github.com/vtsuperdarn/davitpy/issues/209. This pull request fixes this bug by updating the URL as well as adjusting to the new returned data format from the JHU/APL server.

You can confirm that the old URL does not work by using:

Get all the FPs for 2/Jul/2016 from 0 to 12 UT

from datetime import datetime import rbsp

print ""
print "Testing footprint collection and apogee calculation..."
print ""
print "Expected results for orbits on July 2, 2012 between"
print "00:00 and 12:00 utc:"
print "    01:45 UT, A: ( 68.47 N, 94.93 E)     (-51.43 N, 106.23 E)"
print "    01:55 UT, B: ( 68.44 N, 92.27 E)     (-51.56 N, 104.03 E)"
print ""
print "Calculated results:"
print ""
sTime = datetime(2016, 7, 2, 0)
eTime = datetime(2016, 7, 2, 12)
fps = rbsp.rbspFp(sTime, eTime)
# Pretty print the apogees in that period
print fps

I figured out what was going on before. Before I had an example way in the past which the code goes and accesses a database we generate at VT. So, I've set the date here to go out in the future which we haven't written values to the database.

On the master/develop branch, I expect you'll get a "connection timed out" type of error. On the proposed branch here, it should work fairly well. I've also added a 'force_web_read' parameter that can be set to force the code to re-read the data from the JHU/APL website instead of relying on the VT database.

Also added in a unit test for the module to hopefully start conforming to a file being able to test itself. The code above is largely the unit test for the file, so when using this branch, you can just run python rbsp.py in that directory.

I'm thinking I've got this somewhat down now, but please test away!

You can view, comment on, or merge this pull request online at:

https://github.com/vtsuperdarn/davitpy/pull/237 Commit Summary

  • Spacecraft input check, start to redoURL
  • Adding new URL, start of file processing
  • Finishing change to post-web-fetch processing
  • Removing old variable
  • Removing debugging bits
  • Start of PEP8 style edits
  • Finishing PEP8 style edits
  • Start of unit test
  • Removed force debugging line
  • Fixed line continuation errors
  • Cleanup of debug line in getOrbit
  • Using suggested example
  • Undoing debugging statements
  • Problem format string fixed
  • Cleaning up and adding notes for unit test
  • PEP8 re-edits
  • Adding override database data ability
  • Working on marking apogees in __getOrbit
  • Continuing to debug, found possible problem
  • Converting arrays to numpy arrays, cleaning up debugging
  • Adding length of request debug statement
  • Change repr loop index, self.time doesn't need filtered by apogees
  • use calendar.timegm instead of time.mktime for epoch conversion
  • epoch strings have changed using calendar
  • Add force trace to getTrace, mark start of B trace as False
  • Adding more to self test
  • More pep8 edits, cleaning up some

File Changes

Patch Links:

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/vtsuperdarn/davitpy/pull/237

Best wishes, Xueling

MuhammadVT commented 8 years ago

Hey Kevin,

I got the expected <urlopen error [Errno 110] Connection timed out> error in the current develop branch as Shirling has described. This pull request fixes the "connectin timed out " error. But I am seeing the following two lines of message with a great number of repetition after continuing on to the second example in rbsp.py. If you run fps.map() and plt.show(), the generated map is exmpty. Is it how it is supposed to be?

**\ RECALC_08 WARNS: YEAR IS OUT OF INTERVAL 1965-2015: IYEAR=2016 CALCULATIONS WILL BE DONE FOR IYEAR=2015

Thanks

On Sat, Apr 16, 2016 at 12:47 AM, Kevin Sterne notifications@github.com wrote:

Found the website information for accessing Van Allen probe orbit data outdated in #209 https://github.com/vtsuperdarn/davitpy/issues/209. This pull request fixes this bug by updating the URL as well as adjusting to the new returned data format from the JHU/APL server.

You can confirm that the old URL does not work by using:

Get all the FPs for 2/Jul/2016 from 0 to 12 UT

from datetime import datetime import rbsp

print ""
print "Testing footprint collection and apogee calculation..."
print ""
print "Expected results for orbits on July 2, 2012 between"
print "00:00 and 12:00 utc:"
print "    01:45 UT, A: ( 68.47 N, 94.93 E)     (-51.43 N, 106.23 E)"
print "    01:55 UT, B: ( 68.44 N, 92.27 E)     (-51.56 N, 104.03 E)"
print ""
print "Calculated results:"
print ""
sTime = datetime(2016, 7, 2, 0)
eTime = datetime(2016, 7, 2, 12)
fps = rbsp.rbspFp(sTime, eTime)
# Pretty print the apogees in that period
print fps

I figured out what was going on before. Before I had an example way in the past which the code goes and accesses a database we generate at VT. So, I've set the date here to go out in the future which we haven't written values to the database.

On the master/develop branch, I expect you'll get a "connection timed out" type of error. On the proposed branch here, it should work fairly well. I've also added a 'force_web_read' parameter that can be set to force the code to re-read the data from the JHU/APL website instead of relying on the VT database.

Also added in a unit test for the module to hopefully start conforming to a file being able to test itself. The code above is largely the unit test for the file, so when using this branch, you can just run python rbsp.py in that directory.

I'm thinking I've got this somewhat down now, but please test away!

You can view, comment on, or merge this pull request online at:

https://github.com/vtsuperdarn/davitpy/pull/237 Commit Summary

  • Spacecraft input check, start to redoURL
  • Adding new URL, start of file processing
  • Finishing change to post-web-fetch processing
  • Removing old variable
  • Removing debugging bits
  • Start of PEP8 style edits
  • Finishing PEP8 style edits
  • Start of unit test
  • Removed force debugging line
  • Fixed line continuation errors
  • Cleanup of debug line in getOrbit
  • Using suggested example
  • Undoing debugging statements
  • Problem format string fixed
  • Cleaning up and adding notes for unit test
  • PEP8 re-edits
  • Adding override database data ability
  • Working on marking apogees in __getOrbit
  • Continuing to debug, found possible problem
  • Converting arrays to numpy arrays, cleaning up debugging
  • Adding length of request debug statement
  • Change repr loop index, self.time doesn't need filtered by apogees
  • use calendar.timegm instead of time.mktime for epoch conversion
  • epoch strings have changed using calendar
  • Add force trace to getTrace, mark start of B trace as False
  • Adding more to self test
  • More pep8 edits, cleaning up some

File Changes

Patch Links:

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/vtsuperdarn/davitpy/pull/237

ksterne commented 8 years ago

Hey @MuhammadVT,

I did originally try to look at the fps.map() function, but I'm not sure about how it functions. Here, I just wanted to fix the URL being out of date.

As well, I think the recalc warning comes from the Tsyganenko model when the satellite position is traced along the magnetic field line. I'm not sure if a database needs to be updated or a function needs to be updated.

MuhammadVT commented 8 years ago

Ok. Since this pull request targets the issue of outdated URL and it does fix it, I will go ahead and merge it then.

On Tue, Apr 19, 2016 at 11:40 AM, Kevin Sterne notifications@github.com wrote:

Hey @MuhammadVT https://github.com/MuhammadVT,

I did originally try to look at the fps.map() function, but I'm not sure about how it functions. Here, I just wanted to fix the URL being out of date.

As well, I think the recalc warning comes from the Tsyganenko model when the satellite position is traced along the magnetic field line. I'm not sure if a database needs to be updated or a function needs to be updated.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/vtsuperdarn/davitpy/pull/237#issuecomment-211984819

asreimer commented 8 years ago

Hey @ksterne and @MuhammadVT

That error that you saw is just a warning message. The model only has magnetic field coefficients for certain years and gets updated once ever 5 years or something like that. We don't have coefficients for beyond 2015, so the code spits out a warning to the user. Nothing we can do until the new coefficient files are out (maybe they already are?).