search

calpoly-csai / csai-scraping

Web scraping for Nimbus
4 stars 4 forks source link

Run all scrapers through Sustainer #5

Closed austinsilveria closed 4 years ago

austinsilveria commented 4 years ago

Currently each scraper is run independently and saves data in various ways.

Update the Sustainer to import all scrapers, run them sequentially, and aggregate all data in a JSON object such as the following:

scrapedData {
    schedules: ",Name,Alias,Title,Phone,Office,Office Hours,Course,Type,Days,Start,End,Location,Department\n0,"Meagher, James M.",jmeagher,Instr Fac AY,+1.805.756.2132,192-0309,,,,,,,,CENG-Advancement\n1,"King, Carolyn E.",cking29,Admin Support Coord 12 Mo,+1.805.756.5374,192-0310,,,,,,,,CENG-Advancement\n2,"Spikes, Amy Blosser",spikes,Asst Dean of Dev & Ext,+1.805.756.2163,192-0302,,,,,,,,CENG-Advancement"
    courses: ...
    clubs: ...
    faculty: ...
    curriculum: ...
}