Rio 2016

[valeriosouza]

Any idea reactivate for 2016?

[wkoffel]

I don't have immediate plans to, Valerio. I could probably bring it back up fairly easily, though, once the new 2016 olympics site official goes live and I can set up the scraper. Would you use it if it were online? What for?

[valeriosouza]

Yes, I create a WordPress plugin to help websites to make available the information

[dcohenb]

I would use it also

[Jack-Collins]

I would also be interested in using it to feed some information into an olympics chatroom

[wkoffel]

If anyone has a good source for medals data, that'd be great. I think Google is going to be the best bet. Check out https://www.google.com/search?q=rio&aqs=chrome..69i57j69i64.3104j0j4&sourceid=chrome&ie=UTF-8#mie=oly%2C%5B%22%2Fm%2F03tnk7%22%2C1%2C%22m%22%2C1%5D

I originally used the Kimono Labs service, but they have since shut down, so will have to roll my own.

The challenge is that page isn't really scraper friendly (probably on purpose). There are definitely good ways to do this, but I'm not sure I'll have time this week to update the API, unless I'm properly inspired.

[dcohenb]

I think i can work out a scrapper and create a reliable API i will work on it tomorrow.

[ghost]

I'd be interested in using an API like this. Might build a scraper myself if I get the time. @dcohenb - keep us updated on how you go!

[devinvb]

I'd be interested in the API too. @dcohenb looking forward to your work.

[neliocastro]

I'd be interested in the API too. @dcohenb looking forward to your work too.

[fschaal]

I'd be interested in the API too. @dcohenb looking forward to your work as well.

[acrogenesis]

Nice! Thanks

[dcohenb]

Hi everyone! it is here: http://olympics.atelerix.co/medals I've created a scrapper that runs on my personal home server & a backup one in the cloud. Currently the update interval is once every hour. i will shorten this interval as we go further into the Olympics as I don't want to be blocked by google.

Enjoy!

[fschaal]

@dcohenb Realy nice! Thank you!

[dcohenb]

If there is interest in the rest of the data there let me know and i'll see what i can do

[fschaal]

@dcohenb Maybe a list of the categories including the bronze / silver and gold winners.

[dcohenb]

@fschaal Not really sure where i can find this info..

I found that each discipline has it's own medals page: https://www.google.com/search?q=rio&aqs=chrome..69i57j69i64.3104j0j4&sourceid=chrome&ie=UTF-8#mie=oly%2C%5B%22%2Fm%2F01191277%22%2C5%2C%22m%22%5D But I don't know what the DOM structure will be so i can't build it at the moment..

[devinvb]

@dcohenb Very nice work, thanks a lot!

[fschaal]

@dcohenb Ah yea true. I was thinking of structure like this: Day -> categories -> winners

The day and events could be fetched from this page: https://www.google.com/search?q=rio&aqs=chrome..69i57j69i64.3104j0j4&sourceid=chrome&ie=UTF-8#q=rio%202016&mie=oly%2C%5B%22%2Fm%2F03tnk7%22%2C1%2C%22r%22%2Cnull%2Cnull%2Cnull%2Cnull%2Cnull%2Cnull%2Cnull%2Cnull%2Cnull%2Cnull%2Cnull%2Cnull%2Cnull%2Cnull%2C%222016-08-04%22%5D

Not sure if the structure is any workable. Winners are not included I think might be when there are actually winners available.

But none the less the current version is already super nice :)

[wkoffel]

Thanks @dcohenb . Did you do it as a fork of this, or a separate repo? Happy to take a pull request, give you credit on this README or update other pointers to a new repo, I'm sure others will be looking around for this.

Excited to the games to start tomorrow!

[dcohenb]

@wkoffel Noop this is not a fork as I don't know ruby, built it using Node.js. it is currently on a private repo, I'll try to put it on Github later, just need to remove all my private credentials from the code first. Thank you for the recognition :)

[theneedyguy]

http://wowappprd.rio2016.com/json/medals/OG2016_medalsList.json < This is the official API used in the Rio 2016 Android app. This will show medals by country. I am not sure what it will look like once they fill it with data. Make of it what you want :)

EDIT: If you want more data you might as well capture packets from the Rio App with a capturing app to get all the URLs they are using.

[LinuxSDA]

@theneedyguy I guess this closes the issue.

[wkoffel]

Thanks @theneedyguy !

I knew there must be a more official solution to this. I wish Google, NBC, or Rio would publish a proper public API, but leveraging that is the next best thing. If you already have sniffing setup, it'd be cool to see a more comprehensive set of endpoints and parameters.

Closing this issue, between @dcohenb work and @theneedyguy discovery, I don't plan to update this olympics-api project for Rio2016.

[lucasstark]

@dcohenb I turned it into a slack command :-), http://d.pr/i/TQGs

[dcohenb]

@lucasstark that is awesome! i've added SSL support BTW: https://olympics.atelerix.co/medals

[Js41637]

Mmm I added it to my slack bot as well as using the official API.

[dcohenb]

@Js41637 @lucasstark The API i created pulls the data from the official API only it enhances the data (adds flag links & country names) I've added NOC cods toady and uploaded it to Github It is available at: https://github.com/dcohenb/olympics-scrapper

[Js41637]

Yes I use your API for the medals list as the official one doesn't seem to be as reliable but I use a different official api to fetch the medals for each sport.

[dcohenb]

@Js41637 can you share that API endpoint with me? i need something like that

[Js41637]

The official one that returns medals for a country is http://wowappprd.rio2016.com/countries/countriesMedals?lang_code=ENG&os_kind=ANDROID&competition_code=OG2016&noc_code=${NOCCODE} From my testing, the API has to be structured like that or it won't respond but these APIs seem to behave randomly

The hard part is parsing the data, they only return the IDs of the sports. Had to do a lot of packet sniffing and stuff to get all the info so I could use the data. If you're interested you can copy the data I have stored for my bot here; https://github.com/luigiplr/sloth-bot/blob/master/src/plugins/olympics/utils/olympics.js

Should note this that like the medal list OG2016 means Olympic Games, as such this data doesn't apply to the Paralympic Games (PG2016) which has different data that I didn't bother getting.

[dcohenb]

Great work you've done there.. the data seems to correspond the ODF documentation. more info can be found here: http://odf.olympictech.org/2016-Rio/rio_2016_OG.htm

For example all the discipline codes are listed here: http://odf.olympictech.org/2016-Rio/codes/HTML/odf_codes_OG/odf_codes.htm

The ODF is a complete mess due so good luck parsing anything with it... I'll try to enhance what I have, Thanks!

[Js41637]

Oh I didn't even use those, I stole all my data from a DB that I found on the API, http://wowappprd.rio2016.com/contents/SQLite_DB/result_client_20160809080708313.zip

[dcohenb]

Yes. you can use their tool "ODF Document Downloader" to download all those files (and the ref database you got) http://odf.olympictech.org/tools/index_tools.htm

[dcohenb]

@Js41637 Hi, where did you find the URL to the db? I think it's being constantly updated with new values...

[Js41637]

Oh right sorry. http://wowappprd.rio2016.com/intro/masterDB

[dcohenb]

@wkoffel Sorry we hijacked the issue, if it's annoying will get out.

@Js41637 Thanks, Using the database you got and parsing the ODF xml's I've linked before, I managed to create an endpoint that lists all the medals of a country together with Who's the athlete/team and the event if Somebody needs it: https://olympics.atelerix.co/medals/USA https://olympics.atelerix.co/medals/CAN

Also available is a full list of countries with flags: https://olympics.atelerix.co/countries

Thanks for you help!