search

Athons / events

Making the events.yaml file from the main wiki + tools to maintain it
MIT License
1 stars 1 forks source link

Scraping to find new events #2

Open bahorn opened 3 years ago

bahorn commented 3 years ago

As mentioned in the readme, I'm interested in maintaining more of an effort to scrape sites and such to discover new events.

One approach that can use the existing data we are collecting is the tech society Github list.

As github provides atom feeds, we can track changes across all their repositories to see when their sites change by just appending a ".atom" to the url.

Also sneaky and we can catch things before publicly announced :)

bahorn commented 3 years ago

So, I setup a channel called events in the gitter community, which is where notifications from the bot should be posted.

bahorn commented 3 years ago

Ok, so I attempted to get the approach where I append .atom to organization URLs to get updates.

That doesn't work because apparently they do it from when it was last generated? You can see my attempt in the current revision 8f6f0a44e5839a67874f6cee86051b46596dcb02

Best approach now would be to just use an API token to list all organization repos, check if anything new turned up (like a 2021 event repo, etc) or changes to known website repositories.

bahorn commented 3 years ago

https://api.github.com/users/{org}/events

Will list all recent events that organization, which I just pushed code to use. Just needs scheduling now. Will probably aim to run on a wednesday at 4am. Should capture most new stuff then.

bahorn commented 3 years ago

Now scheduled, and it seems to work!

I'll keep this issue open for progress on the other methods of scraping.