search

crosscloudci / cross-cloud

Cross-Cloud - multi-cloud K8s provisioner for CNCF CI Project
https://cncf.ci
Apache License 2.0
167 stars 59 forks source link

Packet: failed deployments related to CoreOS issues on c1.small #171

Open vielmetti opened 6 years ago

vielmetti commented 6 years ago

Please see https://github.com/coreos/bugs/issues/2508 for an issue we are tracking at Packet regarding elevated failure rates for provisioning c1.small instances on CoreOS Stable. The symptom is provisions time out after 25 minutes; if you dig into the problem, and examine a booting system, you'll find that the error message "bond0: link status down for interface enp1s0f0, disabling it in 200 ms" is repeated on the console.

vielmetti commented 6 years ago

Problem update:

This issue manifests itself twofold; one in the CoreOS "installer" stage where CoreOS writes bits to disk, and one in the CoreOS runtime when the newly installed system boots or reboots.

We have repaired the problem in our "installer" part, which runs for every install; that would appear to have decreased the frequency of problems. It hasn't eliminated them yet because the runtime for CoreOS beta and stable both still exhibit this issue. CoreOS alpha should be set (it doesn't use the same kernel as beta and stable).

The issue is not specific to c1.small so switching to a different machine type or location will not change the behavior.

vielmetti commented 5 years ago

Based on #415 being resolved, I think this and #175 can be closed.