[story] Scraper

[a-camarillo]

Build Scraper And Associated Functionality

Edit(Josh): Just changing this to "story" for tracking purposes

Notes:

• I imagine the bulk of the work will come from figuring out how to parse each respective job post since each site has a different structure to its posts
• Ideally we want a uniform structure of data to store in the database in order to avoid having a database table per company which wouldn't be terrible for now but likely won't scale. This means sending the data to the database might only be one task.

Tasks:

Meta:

• [ ] #68
• [ ] Structure data
• [ ] Send data to database

Amazon:

• [ ] #69
• [ ] Structure data
• [ ] Send data to database

Netflix:

• [ ] #70
• [ ] Structure data
• [ ] Send data to database

Google:

• [x] #71
• [ ] Structure data
• [ ] Send data to database

Apple:

• [x] #72
• [ ] Structure data
• [ ] Send data to database

To Discuss:

• How often do we expect this scraper to run? Can we determine the frequency internships can be posted?
• Is there data being expected that we cannot scrape/deliver?

[jwu910]

@lilyxmeng FYI, here's the story @a-camarillo set up to break up the scraping, we can use this to focus on the key parts for each company one at a time

[jwu910]

@kpxwinter One of these doesn't fit 🤔 did scope increase to scrape 6 sites instead of 5--or did we miss one while planning?

[kpxwinter]

Note: We're not doing Netflix so the goal is still to scrape 5 sites total