estiens / world_cup_json

Rails backend for a scraper that outputs World Cup data as JSON
928 stars 107 forks source link

Data is scraped in "latest_json" column but do not fill the other columns in the database #255

Open zacharylr opened 1 month ago

zacharylr commented 1 month ago

Hey!

First of all, I want to say that I used this API for a personal web project in 2022 and 2018 for the World Cup and it was a life saver. It worked really well and I liked how the info from multiple endpoints from the FIFA API was regrouped into a single endpoint (matches?details=true for example).

Now, I want to use and modify this code to scrape the FIFA API and generate standardized "offline" json files for different World Cup (previous editions). I basically want to create a version of /matches?details=true for different World cups I started by just trying to install the project locally and I get to a certain point but I'm stumped (I have very limited knowledge of ruby).

I followed the instructions in the readme to call Setup2022.setup_teams, Setup2022.setup_groups and AllMatchesService.setup. I have 64 matches so I guess that's good. The data gets scraped in the latest_json column in the matches table, but not every column in the table are filled with the data that is present in latest_json . A lot of them stays null (home_team_score, away_team_score, status, stage_name, home_team_events, away_team_events, etc). The events table is still completely empty.

Here is a comparison between the online version and my local version: worldcupJson

I'm trying to see which jobs should I call or what steps should I follow to at least have all the info needed to populate the database and the different endpoints.

Thanks!