Currently the Azure locking system to enforce concurrency, while functional, is a bit of a mess.
When testing out, we ran into problems where prometheus would scrape the metrics endpoint after the machine population interval had expired, and if it scraped it again within the window where the machines were being queried from the API, we'd run into a deadlocking situation, exacerbated by the readiness checks trying the locks.
We should re-write it to use channels as they're much easier to work through the concurrent problems.
The structure of the pricing map and machine map seems to be sound, but may need to be touched as well.
closing as we have sufficiently solved this problem with the current locking implementation. Channels won't get us anything except slightly more idiomatic go code, in my opinion
Currently the Azure locking system to enforce concurrency, while functional, is a bit of a mess.
When testing out, we ran into problems where prometheus would scrape the metrics endpoint after the machine population interval had expired, and if it scraped it again within the window where the machines were being queried from the API, we'd run into a deadlocking situation, exacerbated by the readiness checks trying the locks.
We should re-write it to use channels as they're much easier to work through the concurrent problems.
The structure of the pricing map and machine map seems to be sound, but may need to be touched as well.