researchseminar scraper

[AlainRossier]

Scraping

• [x] Get title, description of the agora
• [x] Get list of future talks with title, speaker, description
• [x] Construct start and finish dates
• [x] Get speaker website
• [x] Login to access some of the links
• [x] Get links to future talks
• [x] Get video and slides from past talks
• [x] Log in
• [x] driver close at the end of the scraping

Parsing

• [x] Create agora
• [x] Create all talks (past and future)

Frontend

• [x] Ask the user for the link to his seminar series on RS
• [x] Ask the user to categorize his agora
• [x] Ask audience level
• [x] Ask whether he wants his talks to require registration
• [x] Ask who he wants to automatically accept
• [x] OnSubmit, open another overlay with progress bar (important as it's taking around 1s/seminar, so big series can take a long time

Test

• [x] Connect everything
• [x] Test a few series

[AlainRossier]

There were two problems with the previous version 1) Flask stalling after 30s: solved it using async/await in the frontend 2) Login requires to restart gunicorn (idk why for now): As the login is there to parse the link, it is important but not essential for now. Our goal is to have a way to migrate all the seminars first.

[AlainRossier]

Did a neat trick to get the event links: I made the assumption that it was the same for every talk, so I parse it once at the agora creation (where the login is still active and possible) and copy it for each talk. I also added a logout.