search

contao / contao-manager

Contao Manager
GNU Lesser General Public License v3.0
85 stars 33 forks source link

Resolver Cloud Issues #600

Closed Toflar closed 4 years ago

Toflar commented 4 years ago

There have been stability issues with the https://composer-resolver.cloud over the last two weeks which were all caused by multiple different problems.

First of all, we've released Contao Manager 1.3.0 which contained an issue that caused too many requests to the Cloud Resolver. Combined with the fact that we've released security releases for Contao, the demand was so high that it reached around 250 requests per second (!). A number the Cloud was simply not able to cope with.

The number of requests have since been greatly reduced, thanks to @aschempp who released new versions of the Contao Manager. On top of that, I've added caching headers and put the Cloud behind KeyCDN to reduce the number of requests actually reaching the Cloud.

However, there was another issue introduced with autoscaling which caused loads of jobs to be dropped and thus seemed like they were running endlessly. Today, I've disabled autoscaling and fixed the number of workers for Contao to 7 so this issue should be gone for the moment. Due to technical reasons we cannot increase the number to more than 7 for now.

@leofeyer and I are actively working on moving the Composer Resolver Cloud away from Google Cloud and we'll go back to some good old regular, powerful VPS instead of some fancy Kubernetes autoscaling stuff. This is a bigger task and might well take another month or two to complete, though.

Moreover, I've been actively working on improving the performance of Composer v2 which will hopefully allow us to run even more workers with the same amount of resources or in best case, even forgo the Resolver Cloud completely.

I'll post here once migration is done. Until then, please bear with the lower number of workers.

Toflar commented 4 years ago

Because a few people were asking (Numbers on the left represent the total number of jobs since we started counting in March 2019):

graph
Toflar commented 4 years ago

Well, we were pretty quick and worked hard to get this issue resolved asap. Hosting provider migration will take place next Tuesday: https://composer-resolver-cloud.statuspage.io/incidents/pd93jbwqxzdj

Contao will be back at 10 workers ready for your projects.

Next up will be Composer 2.0 preparation work where I expect us being able to double the workers to 20. Future Composer 2 versions will likely become even more memory efficient and thus allowing us to increase that number even more. I know that because I work on it myself :P

Toflar commented 4 years ago

Migration's been completed successfully. Goodbye Google and k8s for now 👋