search

rochaporto / soaring

5 stars 2 forks source link

crawler for onlinecontest flights #2

Open rochaporto opened 13 years ago

rochaporto commented 13 years ago

Similar to the netcoupe crawler, it should:

  1. Go through the flight listing at http://www.onlinecontest.org
    • It needs to do it in an incremental manner, probably caching the last flight ID as new ones are always incremental
  2. Download both the flight metadata and the flight igc file
  3. Parse the flight using the usual parser library
  4. Optimize the flight using the different available optimizers
  5. Store the flight metadata, generated metadata and original flight igc data in the flights fusion table

We probably need some additional information for visualization performance, such as a pregenerated kml.

rochaporto commented 13 years ago

This should be done as a appengine cronjob.