researchseminars.org scraping

agora-inc / platform

Full code for mora.stream, the "Twitch.tv for academic researchers". The streaming platform served several thousands of concurrent viewers. This has now been acquired and absorbed by Cassyni.

https://cassyni.com/mora

4 stars 0 forks source link

researchseminars.org scraping #432

Closed AlainRossier closed 2 years ago

AlainRossier commented 3 years ago

Scraping

[x] Get title, description of the agora
[x] Get list of future talks with title, speaker, description
[x] Construct start and finish dates
[x] Get speaker website
[x] Login to access some of the links
[x] Get links to future talks
[x] Get video and slides from past talks

Parsing

[x] Create agora
[x] Create all talks (past and future)

Frontend

[x] Ask the user for the link to his seminar series on RS
[x] Ask the user to categorize his agora
[x] Ask audience level
[x] Ask whether he wants his talks to require registration
[x] Ask who he wants to automatically accept
[ ] OnSubmit, open another overlay with progress bar (important as it's taking around 1s/seminar, so big series can take a long time

Test

[x] Connect everything
[x] Test a few series

argoopjmc commented 3 years ago

you mean seminar series?

AlainRossier commented 3 years ago

Yeah, the goal is that the user inputs the link to his seminar series, like

https://researchseminars.org/seminar/NYC-NCG

and we parse all the past and future talks with all the info

AlainRossier commented 3 years ago

Ok, I actually have a few issues before this can be fully merged

The scraping randomly fails after a few talks being parsed, with the following error message

Screenshot from 2021-09-13 17-52-28

If I restrict the number of talks being parsed to 10, there is no problem.

I have to restart the backend everytime, otherwise it doesn't work

Screenshot from 2021-09-13 17-50-21

I don't know if these two problems are there because I'm on localhost. Locally, I don't have any problem with the scraping, so the scraper is not the issue.

Also

[ ] Video and slide links are being parsed, but not stored.

argoopjmc commented 3 years ago

This method is not used anymore.

I realised that , and have added the header to the new methods ...

AlainRossier commented 2 years ago

This remote branch has unrelated commits, created another unpolluted branch