estiens / world_cup_json

Rails backend for a scraper that outputs World Cup data as JSON
928 stars 107 forks source link

World Cup 2022 #184

Open fduartecastro opened 2 years ago

fduartecastro commented 2 years ago

Hi, this is not an issue :-)

Do you have plans to make the API available for the World Cup 2022?

Thanks

guilhermecgs commented 2 years ago

+1!

Raghav-Maheshwari commented 1 year ago

Also +1!

estiens commented 1 year ago

If we've reached another World Cup without an accessible API.....

As I did last time around I'll see what I can do...last world cup required a major rewrite to try to parse the new SPA but eventually got lucky with a hidden JSON API...

I've seen quite a few APIs pop up over the past few years, but haven't judged their trustworthiness or saw if they charge or not.

Perhaps someone will drop a few better resources in this issue!

This remains one of the first things I ever coded, and it's originally based around web scraping the FIFA site 8 years ago, so it's getting a little long in the tooth :D Not even sure the dependencies install at this point...

guilhermecgs commented 1 year ago

dude, the world is asking :-) answer to the call hahahaha

https://github.com/raminmr/free-api-worldcup2022 is full of bugs

estiens commented 1 year ago

I'm honestly suprised FIFA hasn't opened up some sort of highly rate limited free API (hell in 2018 this ended up using their non documented api for their mobile apps just from looking at requests) - make the companies that want to build gambling apps or news feeds or whatever pay for it, but release the basic API for free

s-stefanov commented 1 year ago

+1

I'm also searching for an API for basic personal project. Will be great if they open one, but I have no success for now too.

FelipeZNascimento commented 1 year ago

I'm also looking for an API for a personal project. It's a Bet'Em platform to play with friends and I'd love to have the results in real time.

Here's the link to the platform if you wanna check it out (still in development but almost there): https://bolaocopa22.omegafox.me/

If you need a hand regarding this API, I'd be happy to help :)

estiens commented 1 year ago

I'm also looking for an API for a personal project. It's a Bet'Em platform to play with friends and I'd love to have the results in real time.

Here's the link to the platform if you wanna check it out (still in development but almost there): https://bolaocopa22.omegafox.me/

If you need a hand regarding this API, I'd be happy to help :)

PRs always welcome! Work is absolute crunch time right now, so not sure I'll have time to look into stuff - if anyone runs into any bugs getting setup /before/ things start might be able to help get the app running, but as for finding FIFA feeds or looking at new match results pages CSS ????

estiens commented 1 year ago

FWIW WWC pages still have an endpoint at

https://api.fifa.com/api/v1/timelines/103/278513/278523/300438262?language=en

no guarantee will be up for this one, but probably likely

estiens commented 1 year ago

probably most up to date response, from last FIFA comp

https://api.fifa.com/api/v3/calendar/matches?idCompetition=2000001035

estiens commented 1 year ago

note change from v1 to v3 :D

estiens commented 1 year ago

and a match timeline/event request

https://api.fifa.com/api/v3/live/football/2000001032/400243454/8tug861ibh72j04p3ge06e80k/ehdoklsljkbftwk88wc0nlus4?language=en

FelipeZNascimento commented 1 year ago

So... following your tips, I went here (World Cup opening) and found this API call: https://api.fifa.com/api/v3/live/football/17/255711/285063/400128082?language=en

All World Cups are ID'd as 17 at FIFA's database.

Then I got into a match, cleaned up some calls, and found out that this endpoint returns info from all World Cup matches at the selected dates (just change "from" or "to" accordingly): https://api.fifa.com/api/v3/calendar/matches?from=2022-11-19T00%3A00%3A00Z&to=2022-11-21T23%3A59%3A59Z&language=en&count=500&idCompetition=17

We'll have to wait and see if it'll be live updated (and if it'll remain open until then) but I think that's it.

estiens commented 1 year ago

Nice one! I guess that means I can at least get the app running and change the urls and check the JSON structure we are parsing in the code :)

estiens commented 1 year ago

Alright, I guess we're doing this - got some stuff working behind the scenes tonight - will try to get deployed and readme updated, etc

FelipeZNascimento commented 1 year ago

One more lol

For live events, this is the API call for the 1st World Cup Match: https://api.fifa.com/api/v3/live/football/17/255711/285063/400128082?language=en

This is how the endpoint is formed: https://api.fifa.com/api/v3/live/football/{IdCompetition}/{IdSeason]/{IdMatch}

So far as I tested, all the URL params are mandatory. You can find the specific ones (idGroup and idMatch) by performing this API call I mentioned on my previous comment

I think that wraps it all up. Here's hoping that this API remains open and untouched haha

fduartecastro commented 1 year ago

Felipe, are you from Brazil? sometime ago I got all matches from Globo.com: _for($i=3929;$i<=3936;$i++){ for($rodada=1;$rodada<=3;$rodada++){ $jogo = @file_getcontents("https://api.globoesporte.globo.com/tabela/b5ff9c28-476e-4816-a699-7645acc94cd0/fase/fase-grupos-copa-do-mundo-2022/rodada/$rodada/grupo/$i/jogos/");

I liked your tips to use FIFA API.

I have some code from last World Cup that are ready to get data from worldcup.sfg.io to monitor matches. I will wait ...

FelipeZNascimento commented 1 year ago

Felipe, are you from Brazil? sometime ago I got all matches from Globo.com: _for($i=3929;$i<=3936;$i++){ for($rodada=1;$rodada<=3;$rodada++){ $jogo = @file_getcontents("https://api.globoesporte.globo.com/tabela/b5ff9c28-476e-4816-a699-7645acc94cd0/fase/fase-grupos-copa-do-mundo-2022/rodada/$rodada/grupo/$i/jogos/");

I liked your tips to use FIFA API.

I have some code from last World Cup that are ready to get data from worldcup.sfg.io to monitor matches. I will wait ...

Yes, I'm from Brazil. Nice to have this option from Globoesporte as well... cool findings and thanks for sharing!

Do you know if this API also has live events like goals, cards, subs?

fduartecastro commented 1 year ago

Do you know if this API also has live events like goals, cards, subs? I think not. An example from "Brazilian Football Championship": only score, data, local https://api.globoesporte.globo.com/tabela/d1a37fa4-e948-43a6-ba53-ab24ab3a45b1/fase/fase-unica-campeonato-brasileiro-2022/rodada/30/jogos/

estiens commented 1 year ago

One more lol

For live events, this is the API call for the 1st World Cup Match: https://api.fifa.com/api/v3/live/football/17/255711/285063/400128082?language=en

This is how the endpoint is formed: [https://api.fifa.com/api/v3/live/football/{IdCompetition}/{IdSeason]/{IdGroup}/{IdMatch}](https://api.fifa.com/api/v3/live/football/%7BIdCompetition%7D/%7BIdSeason%5D/%7BIdGroup%7D/%7BIdMatch%7D)

So far as I tested, all the URL params are mandatory. You can find the specific ones (idGroup and idMatch) by performing this API call I mentioned on my previous comment

I think that wraps it all up. Here's hoping that this API remains open and untouched haha

Got it and will get it it all running soon, pretty much the same way they were constructed 4 years ago for the v1 API so shouldn't be a problem to pull them in - FIFA prob considers this site a load balancer :D May even make the whole extended JSON payload of a match avail by request, but basic endpoints will work the same schema they always have for match info and match stats

estiens commented 1 year ago

Working and up so far - may have to adjust some things, but considering we have historical v3 API JSON to test against from say, last year, think it should be fine!

new url: https://world-cup-json-2022.fly.dev

will get all the fixups/refactoring from this year merged sometime this week and then PRs welcome

estiens commented 1 year ago

Mind the URL change (though old one will still work) wanting to stick cloudflare on the FE cause, traffic -

We are good to go for 2022 minus a few tweaks to making sure events get written correctly and not duplicated, and whatever edge cases show up when we are live...

Weird november but I'll take it!

estiens commented 1 year ago

now at:

https://worldcupjson.net/matches

romain25 commented 1 year ago

Hello and thank for your job. I have a question how do you define end of match from fifa ?

When I see this link : I don't find end of match, period, first periode, second etc ?

https://api.fifa.com/api/v3/live/football/400171208/5j4ocw1ewxnjz8n913emc249g/5jgl49ttd05rkhm1ommp09o2c/db1o4it73j711p5oobazd6qs4?language=fr

Thanks and have a nice day.

Romain

estiens commented 1 year ago

We will have to figure out what they are using when the first game goes live exactly. But I believe we can use a combination of status and isUpdateable - which is better than what we used to have to do of waiting for the CSS to change :D

romain25 commented 1 year ago

Ok thanks I am analyse live match now on match center if this can help you :

https://api.fifa.com/api/v3/live/football/cesdwwnxbc5fmajgroc0hqzy2/40sncpbsyexdrmedcwjz1j0gk/5w0vi7wp50objhjfn51o5ck5w/di8rjv25gardtihqqg6jidwyc?language=fr

Property Period value: 0 - Match not start 3 - Match live first half time 4 - Half time 5 - Second half time 10 - Match is finish

When Halftime property MatchTime => 0'

MatchStatus 1 - Match does not start 3 - Match in live 0 - Match Finish

ResultType : 1 - Victory Normal ResultType : 2 - Victory - shoot on goal

OfficialityStatus 0 - When match not start and not in live 2 - Match ok and confirm

I hope this can help

estiens commented 1 year ago

Excellent that helps a ton!

For our purposes, when a match is finished it moves to status == 'completed' and a winner is determined

estiens commented 1 year ago

Ok thanks I am analyse live match now on match center if this can help you :

https://api.fifa.com/api/v3/live/football/cesdwwnxbc5fmajgroc0hqzy2/40sncpbsyexdrmedcwjz1j0gk/5w0vi7wp50objhjfn51o5ck5w/di8rjv25gardtihqqg6jidwyc?language=fr

Property Period value: 0 - Match not start 3 - Match live first half time 4 - Half time 5 - Second half time 10 - Match is finish

When Halftime property MatchTime => 0'

MatchStatus 1 - Match does not start 3 - Match in live 0 - Match Finish

ResultType : 1 - Victory Normal ResultType : 2 - Victory - shoot on goal

OfficialityStatus 0 - When match not start and not in live 2 - Match ok and confirm

I hope this can help

hmmm, some of this are different even from matches a year ago with the same "version" of the API! But I think period == 10 is a good finish stat and 3 is a good live stat (or any positive number in time) - thanks!

LucasHayashi commented 1 year ago

Please update the points for the Netherlands in Group A of the 2022 World Cup: The Netherlands has earned 7 points, not 10.

{
  "country": "NED",
  "name": "Netherlands",
  "group_letter": "A",
  "group_points": 10,
  "wins": 3,
  "draws": 1,
  "losses": 1,
  "games_played": 5,
  "goals_for": 10,
  "goals_against": 4,
  "goal_differential": 6
}

API: https://worldcupjson.net/teams