search

rogers1000 / cyclingchaos

Cycling Data Package
6 stars 0 forks source link

UCI Road Race - 2024 Data #21

Open rogers1000 opened 5 months ago

rogers1000 commented 5 months ago

Need to start process of ingesting 2024 data.

rogers1000 commented 5 months ago

calendar data ingestion, still need to do transform.

Need to edit results function to look if the race has happened or in the future.

rogers1000 commented 5 months ago

calendar data transformed.

Botch job 2024 race results data done. Need to look at putting in long term code solution to filter only races that doesn't reingest files that have already been ingested.

rogers1000 commented 5 months ago

Botch job done. Will need to look into more sustainable, scalable approach.

rogers1000 commented 4 months ago

added extra layer to botch job to make the ingestions from all of 2024 to just races since last ingestion.

Still is a manual process but step in the right direction.

rogers1000 commented 3 months ago

Data ingested as of 25th Feb

rogers1000 commented 3 months ago

Races changing the amount of stages might be a big problem when trying to automate the ETL process...

Would need to constantly check if the pipeline worked and could require lots of manual intervention as not sure how else to do this in a sustainable way while wanting to be able to look at the future calendar.

rogers1000 commented 3 months ago

Data Ingested from 16th March