Web scraper upgrade to Playwright (Initial simple features) - Githubissues

jamesgeer / conf-twitter-bot

A Twitter Bot to post about academic papers

MIT License

4 stars 0 forks source link

Web scraper upgrade to Playwright (Initial simple features) #91

Closed EdwardChristian closed 1 year ago

EdwardChristian commented 1 year ago

Upgraded the web scraper to use playwright. Main changes:

Using Playwright now
Backend on Ubuntu instead of Alpine because of dependency issue
Created table for Papers
Papers gets scraped from dl.acm (just the data from proceedings atm)
Papers get added to database and can be viewed on the frontend
Can scrape multiple pages at once