search

HSLdevcom / hfp-analytics

Repository for HFP (High-frequency positioning) analytic tools. More info about HFP https://digitransit.fi/en/developers/apis/4-realtime-api/vehicle-positions/.
MIT License
2 stars 0 forks source link

Fix data import automation #46

Closed Divinare closed 1 year ago

Divinare commented 2 years ago

HFP import seems to be slow for some reason --> importing gets stuck and dev db doens't get HFP data for that reason.

jhanninen commented 1 year ago

Looks like the issue is now resolved. The main problem was too small database storage. There were also one missing foreign key constraint and timescaledb wasn't initialized for hfp data.

We need to monitor the database closely until we have all 14 days stored. The estimate for required storage is somewhere between 200-250 GB. We probably 512 GB data disk, since 256 GB could have too little spare space.