br-g / openf1

OpenF1 API - Real-time and historical Formula 1 data
Other
398 stars 22 forks source link

Fixed duplicate and missing database items #55

Closed PrestonHager closed 1 month ago

PrestonHager commented 7 months ago

Fixes #42, fixes #43.

The data for the ingestor is gathered via the JSON Stream's on F1's static site.

I've fixed the historical ingestor by including separate logic for updating streams. For example, the driver's list can have multiple updates pushed to it. This can be seen in the JSON Stream for the Sazuka Race where line 1 is basic driver data, then later followed by headshots on line 3.

1   00:00:02.035{"1":{"RacingNumber":"1","BroadcastName":"M VERSTAPPEN","FullName":"Max VERSTAPPEN","Tla":"VER","Line":1}, ... }
...
3   00:02:10.211{"1":{"HeadshotUrl":"https://media.formula1.com/d_driver_fallback_image.png/content/dam/fom-website/drivers/M/MAXVER01_Max_Verstappen/maxver01.png.transform/1col/image.png"}, ... }

Compared to the sessions that have complete data, for example the Melborne Qualifying, all of the driver data is on one line.

1   00:00:06.017{"1":{"RacingNumber":"1","BroadcastName":"M VERSTAPPEN","FullName":"Max VERSTAPPEN","Tla":"VER","Line":2,"TeamName":"Red Bull Racing","TeamColour":"3671C6","FirstName":"Max","LastName":"Verstappen","Reference":"MAXVER01","HeadshotUrl":"https://media.formula1.com/d_driver_fallback_image.png/content/dam/fom-website/drivers/M/MAXVER01_Max_Verstappen/maxver01.png.transform/1col/image.png","CountryCode":"NED"}, ... }

This is a draft as there is one issue that isn't fixed with this PR. The real time ingestor still has this bug. I've written code to fix this, however, I can't test it at the moment. The code can be viewed on the dev-driver-data-fix-patch branch.

We can merge this PR without fixing the real time ingestor, but it will only fix already processed history. Additionally, it does not delete previous database entries so I had to scrap my local database and rerun the ingestor.

PrestonHager commented 1 month ago

No longer needed. Fixed with release of v1.0 I think