search

tighten / symposium

Management of proposals, bios, photos, etc. for conference speakers.
https://symposiumapp.com/
MIT License
178 stars 66 forks source link

Make some sort of connection with CFPLand? #289

Closed mattstauffer closed 2 years ago

mattstauffer commented 4 years ago

https://www.cfpland.com/

@karllhughes

karllhughes commented 4 years ago

I'd love to explore sharing data or some kind of centralized data hub with us and Symposium.

I'd also loop in @heiglandreas of callingallpapers.com: https://github.com/joindin/callingallpapers.com

Tim Bourguignon of seecfp.com might also be interested in collaborating.

I currently gather data from callingallpapers, confs.tech, twitter searches, and user submissions then add info about what perks are covered. I wrote about the data collection here: https://www.cfpland.com/blog/data/

Anyway, if we had a central data model and API that we could all read and write from, that would be great.

The part that seems hardest to automate is deduplication and adding perks. Right now I'm doing it manually and it takes about 15 minutes per week.

mattstauffer commented 4 years ago

Karl and I talked earlier and I was saying how it seems like a lot of people are duplicating the effort of scraping all sorts of tech conference lists and normalizing them.

He pointed out that CallingAllPapers, with the lovely @heiglandreas, does this. I had forgotten until he said that, but we actually recently updated Symposium to pull our data from CallingAllPapers instead of joind.in.

I'm wondering--since Andreas is a friend, and his project is open source, would it be possible for us to imagine a suite of modifications that we could make to callingallpapers that Andreas likes that would make it function as the single source of truth for all of our platforms?

And, also, as always, can we at Symposium help any of y'all in any way? Also going to tag @timothep.

Just to note this publicly:

We've constantly seen folks try to enter the space of targeting speakers, CFP software, direct competition to Symposium, etc. and we just sort of go "hey, if you're passionate about this, go for it!" Symposium is a small labor of love and I know it's never going to be monetized. So I don't want to step on anyone's toes, but I'm also happy to be a part of the solution to everyone's needs, if such a thing exists, and I'd be happy to work with Andreas on updates to his project, if he's interested and we find it could be helpful to all of us.

heiglandreas commented 4 years ago

Thanks for looping me in, @karllhughes.

The idea of callingallpapers was that of a kind-of-central-datahub :-)

And I know that we are far from reaching it. But perhaps joint efforts can help here.

So far we didn't tackle the user-submission part as it involves a lot of manual labour in verifying all the stuff (we're doing that at joind.in and php.ug and know what we're talking about here :wink:) but the automation part would already be possible.

Currently we're grabbing data from some websites in an automated way. That's at the time of writing confs.tech, papercall.io, sessionize and joind.in (where it all started).

The hard part is indeed the deduplication which we try to solve by fetching the final website of the event (so going through all the DNS-redirects to reach a final site) and generate a hash from that which we then use as primary key. That way the CFP for a conference should be uniquely identified. This works in most of the cases rather well.

As CallingAllPapers is already built around an API and the information is gathered by crawlers that send the data to the API it is no problem to add new crawlers or to pass in new data via other means.

More interesting currently is the challenge of which description do we keep as each platform can have a different description and currently on each run from one of the websites the entry is updated, so descriptions from one site are overwritten by those from the next site. Just this morning I was thinking of using the actual Event-Websites Meta-Data for pulling in the description instead of using that of the CFP-Aggregation-site. But that is a feature to come.

Likewise I was thinking about a certain meta-tag for Event-Sites that gives informations about a possible CFP like explained in a blog-post I wrote some time back (wow. Indeed some time back...). That could also be used to pull in data.

All in all: The CallingAllPapers project is happy to have changes that allow it to work as a central source for people that are looking for CFPs! That's what it is about.

Let's have a chat about that!

Timothep commented 4 years ago

Hi *, thanks for the tagging. Indeed very interested in this discussion.

karllhughes commented 4 years ago

I also had a conversation with @bendechrai from Auth0 about conference CFP aggregation.

They've built their own in-house system for deduplication and use CFP Land and other data sources to do aggregation too.

bendechrai commented 4 years ago

Thanks for tagging me, @karllhughes - and hi everyone. A single point CFP aggregation has been a desire of mine for many years now - I'd love to help out!

And yes, Auth0 have their own internal CFP tracking tool for devrel to measure things (success rate, post-event reports linked to talk topics, etc). I know many other companies do too.

Having a single repo of all CFPs with good taxonomy would be something many companies would pay for a subscription to. It doesn't make sens for everyone to dedupe everyone's lists :rofl: