search

core2062 / CORE-Scouting-Server

the server module for the CORE Scouting Database (CSD)
GNU General Public License v3.0
2 stars 2 forks source link

write scrapers for match data from previous years #13

Open notslang opened 11 years ago

notslang commented 11 years ago

the blue alliance code doesn't include the scrapers used to get old data (2003/2004/2005 matches), so our current scrapers should be modified to get this data... should be pretty easy. new member project?

nightpool commented 11 years ago

Is it formatted differently? On Nov 27, 2012 10:26 PM, "Sean Lang" notifications@github.com wrote:

the blue alliance code doesn't include the scrapers used to get old data, so our current scrapers should be modified to get this data... should be pretty easy. freshmen project?

— Reply to this email directly or view it on GitHubhttps://github.com/2062/CORE-Scouting-Server/issues/13.

notslang commented 11 years ago

yeah... it's those awful pages that were exported from MS-Word

notslang commented 11 years ago

wait... actually, it might not have been those ones that were exported from MS-Word... but they definitely have different formatting. here are the example links:

also, 2005 might be the same as all the others... but i wrote myself a note that it was different, so I still need to go back and take a look.

nightpool commented 11 years ago

2005's doesn't even give me any results cause your session expired...

On Tue, Nov 27, 2012 at 11:35 PM, Sean Lang notifications@github.comwrote:

wait... actually, it might not have been those ones that were exported from MS-Word... but they definitely have different formatting. here are the example links:

also, 2005 might be the same as all the others... but i wrote myself a note that it was different, so I still need to go back and take a look.

— Reply to this email directly or view it on GitHubhttps://github.com/2062/CORE-Scouting-Server/issues/13#issuecomment-10791368.

notslang commented 11 years ago

oh yeah... forgot about that expiring session key * cough* * cough* bad design cough \ lasso look at this page: http://www.usfirst.org/roboticsprograms/frc/2005-frc-regional-and-championship-results

notslang commented 11 years ago

the blue alliance seems to have an export of all of the data in CSV... rather than dealing with FIRST's old html pages, i'll use those and just make a quick script to reformat the data into our schema

https://github.com/gregmarra/the-blue-alliance/blob/master/test_data/

nightpool commented 10 years ago

Works? Do we have data pre-2005?