search

cloudfoundry / eirini

Pluggable container orchestration for Cloud Foundry, and a Kubernetes backend
Apache License 2.0
115 stars 30 forks source link

Failed to create Statefulset : guid already exists Error #92

Closed KesavanKing closed 4 years ago

KesavanKing commented 4 years ago

Description

We deployed cf-for-k8s 0.1.0 version and performed scalability tests . We pushed source code based apps and reached till 800 apps. During the course of tests at we could see the below errors. 

{"timestamp":"2020-04-24T14:46:28.275329914Z","level":"error","source":"handler","message":"handler.desire-app.bifrost-failed","data":{"error":"failed to desire: failed to create statefulset: statefulsets.apps \"app-2-6-light-2-space-2-dedc6afb16\" already exists","guid":"22875703-d484-48c0-8e35-e9e50f30ca72-25926a2f-476d-4650-b904-a7e45134bdb0","session":"38768"}}
{"timestamp":"2020-04-24T14:46:28.469625717Z","level":"error","source":"handler","message":"handler.desire-app.bifrost-failed","data":{"error":"failed to desire: failed to create statefulset: statefulsets.apps \"app-2-1-light-21-space-2-3ab4ad2579\" already exists","guid":"5d94793f-de02-4294-98e0-fd97db282f87-d6e2c77c-c911-4fb7-9d64-9fb330fba0f4","session":"38769"}}
{"timestamp":"2020-04-24T14:46:28.708660473Z","level":"error","source":"handler","message":"handler.desire-app.bifrost-failed","data":{"error":"failed to desire: failed to create statefulset: statefulsets.apps \"app-2-5-light-15-space-2-85ef1faa94\" already exists","guid":"68558507-37cc-4cc6-81ee-de4585b9120e-d46e4837-f455-47e0-aa66-4a065e47e9e3","session":"38770"}}

We are sure that we didn't push with same appname.

Steps to reproduce

Deploy cf-for-k8s and push source code based apps beyond 500.

cf-gitbot commented 4 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/172528293

The labels on this github issue will be updated when the story is started.

mnitchev commented 4 years ago

Hello @KesavanKing,

This is actually a desired behaviour in Eirini (see here) and not a bug. Long story short, we believe the Cloud Controller is requesting the app twice through the process_sync and the desire_app_handler, so it sounds more like an issue in CAPI.

Best Regards, @mnitchev && @yulianedyalkova

mnitchev commented 4 years ago

Hi @KesavanKing,

We actually reconsidered this and decided to make our api idempotent in this regard and not fail when the app is requested a second time. It will just log it when it happens.

Regards, @mnitchev && @herrjulz