search

cloudfoundry / cf-for-k8s

The open source deployment manifest for Cloud Foundry on Kubernetes
Apache License 2.0
301 stars 115 forks source link

Failing to push applications on cf-for-k8s v5.4.3 #695

Open aveiga opened 2 years ago

aveiga commented 2 years ago

Describe the bug

We're seeing multiple, random failures when deploying applications (via cf cli v7) on one of our cf-for-k8s clusters. The cf push command fails rather quickly (around 4 seconds). This is what we can gather from our logs:

{
    "error_code"=>"UnknownError",
    "description"=>"An unknown error occurred.",
    "code"=>10001,
    "test_mode_info"=>{
        "description"=>"undefined method `spec' for nil:NilClass",
        "error_code"=>"CF-spec",
        "backtrace"=>[
            "/workspace/lib/kubernetes/route_resource_manager.rb: 51:in `block in update_destinations'",
            "/workspace/lib/kubernetes/update_reapply_client.rb: 18:in `block in apply_route_update'",
            "/workspace/lib/kubernetes/update_reapply_client.rb: 46:in `block in retry_on_conflict'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/retryable-3.0.5/lib/retryable.rb: 72:in `retryable'",
            "/workspace/lib/kubernetes/update_reapply_client.rb: 44:in `retry_on_conflict'",
            "/workspace/lib/kubernetes/update_reapply_client.rb: 17:in `apply_route_update'",
            "/workspace/lib/kubernetes/route_resource_manager.rb: 50:in `update_destinations'",
            "/workspace/app/actions/update_route_destinations.rb: 106:in `update'",
            "/workspace/app/actions/update_route_destinations.rb: 27:in `add'",
            "/workspace/app/actions/manifest_route_update.rb: 39:in `block in update'",
            "/workspace/app/actions/manifest_route_update.rb: 36:in `each'",
            "/workspace/app/actions/manifest_route_update.rb: 36:in `update'",
            "/workspace/app/actions/app_apply_manifest.rb: 98:in `update_routes'",
            "/workspace/app/actions/app_apply_manifest.rb: 63:in `apply'",
            "/workspace/app/jobs/space_apply_manifest_action_job.rb: 16:in `block in perform'",
            "/workspace/app/jobs/space_apply_manifest_action_job.rb: 15:in `each'",
            "/workspace/app/jobs/space_apply_manifest_action_job.rb: 15:in `perform'",
            "/workspace/app/jobs/wrapping_job.rb: 13:in `perform'",
            "/workspace/app/jobs/wrapping_job.rb: 13:in `perform'",
            "/workspace/app/jobs/timeout_job.rb: 13:in `block in perform'",
            "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/timeout.rb: 95:in `block in timeout'",
            "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/timeout.rb: 33:in `block in catch'",
            "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/timeout.rb: 33:in `catch'",
            "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/timeout.rb: 33:in `catch'",
            "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/timeout.rb: 110:in `timeout'",
            "/workspace/app/jobs/timeout_job.rb: 12:in `perform'",
            "/workspace/app/jobs/wrapping_job.rb: 13:in `perform'",
            "/workspace/app/jobs/logging_context_job.rb: 17:in `block in perform'",
            "/workspace/app/jobs/logging_context_job.rb: 67:in `with_request_id_set'",
            "/workspace/app/jobs/logging_context_job.rb: 15:in `perform'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/backend/base.rb: 81:in `block in invoke_job'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 61:in `block in initialize'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 66:in `execute'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 40:in `run_callbacks'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/backend/base.rb: 78:in `invoke_job'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 230:in `block (2 levels) in run'",
            "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/timeout.rb: 95:in `block in timeout'",
            "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/timeout.rb: 105:in `timeout'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 230:in `block in run'",
            "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/benchmark.rb: 308:in `realtime'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 229:in `run'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 312:in `block in reserve_and_run_one_job'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 61:in `block in initialize'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 66:in `execute'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 40:in `run_callbacks'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 312:in `reserve_and_run_one_job'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 213:in `block in work_off'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 212:in `times'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 212:in `work_off'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 175:in `block (4 levels) in start'",
            "/layers/paketo-buildpacks_mri/mri/lib/ruby/2.7.0/benchmark.rb: 308:in `realtime'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 174:in `block (3 levels) in start'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 61:in `block in initialize'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 66:in `execute'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 40:in `run_callbacks'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 173:in `block (2 levels) in start'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 172:in `loop'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 172:in `block in start'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/plugins/clear_locks.rb: 7:in `block (2 levels) in <class:ClearLocks>'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 79:in `block (2 levels) in add'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 61:in `block in initialize'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 79:in `block in add'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 66:in `execute'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb: 40:in `run_callbacks'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb: 171:in `start'",
            "/workspace/lib/tasks/jobs.rake: 76:in `start_working'",
            "/workspace/lib/tasks/jobs.rake: 49:in `block (2 levels) in <top (required)>'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb: 281:in `block in execute'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb: 281:in `each'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb: 281:in `execute'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/honeycomb-beeline-2.7.0/lib/honeycomb/integrations/rake.rb: 14:in `execute'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb: 219:in `block in invoke_with_call_chain'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb: 199:in `synchronize'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb: 199:in `invoke_with_call_chain'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb: 188:in `invoke'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb: 160:in `invoke_task'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb: 116:in `block (2 levels) in top_level'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb: 116:in `each'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb: 116:in `block in top_level'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb: 125:in `run_with_threads'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb: 110:in `top_level'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb: 83:in `block in run'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb: 186:in `standard_exception_handling'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb: 80:in `run'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/gems/rake-13.0.6/exe/rake: 27:in `<top (required)>'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/bin/rake: 23:in `load'",
            "/layers/paketo-buildpacks_bundle-install/launch-gems/ruby/2.7.0/bin/rake: 23:in `<top (required)>'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/cli/exec.rb: 58:in `load'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/cli/exec.rb: 58:in `kernel_load'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/cli/exec.rb: 23:in `run'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/cli.rb: 477:in `exec'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/vendor/thor/lib/thor/command.rb: 27:in `run'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/vendor/thor/lib/thor/invocation.rb: 127:in `invoke_command'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/vendor/thor/lib/thor.rb: 392:in `dispatch'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/cli.rb: 31:in `dispatch'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/vendor/thor/lib/thor/base.rb: 485:in `start'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/cli.rb: 25:in `start'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/exe/bundle: 49:in `block in <top (required)>'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/lib/bundler/friendly_errors.rb: 128:in `with_friendly_errors'",
            "/layers/paketo-buildpacks_bundler/bundler/gems/bundler-2.2.26/exe/bundle: 37:in `<top (required)>'",
            "/layers/paketo-buildpacks_bundler/bundler/bin/_bundle: 23:in `load'",
            "/layers/paketo-buildpacks_bundler/bundler/bin/_bundle: 23:in `<main>'"
        ]
    }
}
{
    "reconciler group": "networking.cloudfoundry.org",
    "reconciler kind": "Route",
    "name": "0fbcc8c6-4e56-46e8-b3ab-64976a92dd71",
    "namespace": "cf-workloads",
    "error": "Operation cannot be fulfilled on routes.networking.cloudfoundry.org \"0fbcc8c6-4e56-46e8-b3ab-64976a92dd71\": StorageError: invalid object, Code: 4, Key: /registry/networking.cloudfoundry.org/routes/cf-workloads/0fbcc8c6-4e56-46e8-b3ab-64976a92dd71, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 8df8a849-fa61-43f7-8aa7-bfc72561cbb8, UID in object meta: "
}

To Reproduce

Steps to reproduce the behavior:

  1. On a cluster with 7 CF spaces, each populated with ~50 Spring Boot applications, push multiple applications in sequence

Expected behavior

All applications are successfully pushed

Additional context

cf-for-k8s SHA

v5.4.3

Cluster information

On premises, running cf-for-k8s on top of a Rancher Labs' RKE2 cluster running on VMWare vSphere.

CLI versions

paste output of the following commands

  1. ytt --version: ytt version 0.31.0
  2. kapp --version: kapp version 0.35.0
  3. kubectl version: 
    Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:28:09Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.7+rke2r2", GitCommit:"132a687512d7fb058d0f5890f07d4121b3f0a2e2", GitTreeState:"clean", BuildDate:"2021-05-26T04:47:27Z", GoVersion:"go1.15.8b5", Compiler:"gc", Platform:"linux/amd64"}
  4. cf version: cf version 7.2.0+be4a5ce2b.2020-12-10
cf-gitbot commented 2 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/181345931

The labels on this github issue will be updated when the story is started.