search

cloudfoundry / cloud_controller_ng

Cloud Foundry Cloud Controller
Apache License 2.0
191 stars 357 forks source link

Nokogiri::XML::SyntaxError - 1:1: FATAL: Extra content at the end of the document #1532

Closed kramerul closed 4 years ago

kramerul commented 4 years ago

Thanks for submitting an issue to cloud_controller_ng. We are always trying to improve! To help us, please fill out the following template.

Issue

We tried to run the cloud_controller_ng in a quarks environment using the capi-k8s-release. After several steps we ended up with the following problem when running cf push:

Error with blobstore: Nokogiri::XML::SyntaxError - 1:1: FATAL: Extra content at the end of the document

Context

We are using the following configuration

local_route: 0.0.0.0

external_port: 9022
tls_port: 9023
internal_service_hostname: cloud_controller_ng.kubecf.svc.cluster.local

pid_filename: /cloud_controller_ng/cloud_controller_ng.pid
newrelic_enabled: false
development_mode: false

external_protocol: https
external_domain: api.cf.ingress.kronos.istio.shoot.canary.k8s-hana.ondemand.com
temporary_disable_deployments: false
temporary_use_logcache: true

system_domain_organization: system
system_domain: cf.ingress.kronos.istio.shoot.canary.k8s-hana.ondemand.com
app_domains: [cf.ingress.kronos.istio.shoot.canary.k8s-hana.ondemand.com]

reserved_private_domains: "/dev/null"
disable_private_domain_cross_space_context_path_route_sharing: false

jobs:
  global:
    timeout_in_seconds: 14400

default_app_memory: 1024
default_app_disk_in_mb: 1024
maximum_app_disk_in_mb: 2048

instance_file_descriptor_limit: 16384

request_timeout_in_seconds: 900

bulk_api:
  auth_user: TODO
  auth_password: TODO

internal_api:
  auth_user: TODO
  auth_password: TODO

nginx:
  use_nginx: true
  instance_socket: "/data/cloud_controller_ng/cloud_controller.sock"

# TODO: index????
index: 0
name: TODO
route_services_enabled: true
volume_services_enabled: false

info:
  name: ""
  build: ""
  version: 0
  support_address: ""
  description: ""
  app_ssh_endpoint: TODO.TODO
  app_ssh_host_key_fingerprint: "placeholder"
  app_ssh_oauth_client: "placeholder"
  min_cli_version: ""
  min_recommended_cli_version: ""

directories:
  tmpdir: /tmp/
  diagnostics: /cloud_controller_ng/diagnostics/

logging:
  file: /dev/stdout
  syslog: vcap.cloud_controller_ng
  level: "debug"
  max_retries: 1

logcache:
  host: log-cache.cf.ingress.kronos.istio.shoot.canary.k8s-hana.ondemand.com
  port: 8080

logcache_tls:
  key_file: "/dev/null"
  cert_file: "/dev/null"
  ca_file: "/dev/null"
  subject_name: log_cache

loggregator:
  router: 127.0.0.1:3457
  internal_url: https://log-api.cf.ingress.kronos.istio.shoot.canary.k8s-hana.ondemand.com:8081

log_stream:
  url: https://log-stream.cf.ingress.kronos.istio.shoot.canary.k8s-hana.ondemand.com

doppler:
  url: wss://doppler.cf.ingress.kronos.istio.shoot.canary.k8s-hana.ondemand.com:443

db: &db
  database:
    adapter: mysql
    host: kubecf-database.kubecf
    port: 3306
    user: cloud_controller
    password: ***
    database: cloud_controller
  max_connections: 25
  pool_timeout: 10
  log_level: "debug2"
  log_db_queries: false
  ssl_verify_hostname: false
  read_timeout: 3600
  connection_validation_timeout: 3600
  # TODO: secure db connection?
  #ca_cert_path: "/dev/null"

telemetry_log_path: "/dev/null"
log_cache:
  url: "TODO.TODO"
threadpool_size: 20
internal_route_vip_range: ""

login:
  url: https://login.cf.ingress.kronos.istio.shoot.canary.k8s-hana.ondemand.com
# TODO: change the UAA's Kube DNS name when the service is named correctly later
uaa:
  url: https://uaa.cf.ingress.kronos.istio.shoot.canary.k8s-hana.ondemand.com
  internal_url: https://uaa.service.cf.internal:8443
  resource_id: cloud_controller,cloud_controller_service_permissions
  client_timeout: 60
  ca_file: /config/uaa/certs/uaa.crt

routing_api:
  url: https://api.cf.ingress.kronos.istio.shoot.canary.k8s-hana.ondemand.com/routing
  routing_client_name: "TODO"
  routing_client_secret: "TODO"

credential_references:
  interpolate_service_bindings: true

# App staging parameters
staging:
  timeout_in_seconds: 900
  minimum_staging_memory_mb: 1024
  minimum_staging_disk_mb: 4096
  minimum_staging_file_descriptor_limit: 16384
  auth:
    user: TODO
    password: TODO

default_health_check_timeout: 60
maximum_health_check_timeout: 180

runtimes_file: "/dev/null"
stacks_file: "/cloud_controller_ng/config/stacks.yml"

shared_isolation_segment_name: shared

quota_definitions: {"default":{"memory_limit":102400,"non_basic_services_allowed":true,"total_routes":1000,"total_services":-1,"total_reserved_route_ports":100}}
default_quota_definition: default

resource_pool:
  resource_directory_key: cc-resources
  blobstore_type: fog
  fog_connection:
    provider: AWS
    endpoint: https://apps-hyperion.s3-eu-central-1.amazonaws.com/
    aws_access_key_id: ***
    aws_secret_access_key: ***
    aws_signature_version: "4"
    region: "eu-central-1"
    path_style: true
  minimum_size: 65536
  maximum_size: 536870912

  cdn:
    uri:
    key_pair_id:
    private_key: ""

  fog_aws_storage_options: {}

packages:
  app_package_directory_key: cc-packages
  blobstore_type: fog
  fog_connection:
    provider: AWS
    endpoint: https://apps-hyperion.s3-eu-central-1.amazonaws.com/
    aws_access_key_id: ***
    aws_secret_access_key: ***
    aws_signature_version: "4"
    region: "eu-central-1"
    path_style: true
  max_valid_packages_stored: 5
  max_package_size: 1073741824

  cdn:
    uri:
    key_pair_id:
    private_key: ""

  fog_aws_storage_options: {}

droplets:
  droplet_directory_key: cc-droplets
  blobstore_type: fog
  fog_connection:
    provider: AWS
    endpoint: https://apps-hyperion.s3-eu-central-1.amazonaws.com/
    aws_access_key_id: ***
    aws_secret_access_key: ***
    aws_signature_version: "4"
    region: "eu-central-1"
    path_style: true

  cdn:
    uri:
    key_pair_id:
    private_key: ""

  fog_aws_storage_options: {}
  max_staged_droplets_stored: 5

buildpacks:
  buildpack_directory_key: cc-buildpacks
  blobstore_type: fog
  fog_connection:
    provider: AWS
    endpoint: https://apps-hyperion.s3-eu-central-1.amazonaws.com/
    aws_access_key_id: ***
    aws_secret_access_key: ***
    aws_signature_version: "4"
    region: "eu-central-1"
    path_style: true

  cdn:
    uri:
    key_pair_id:
    private_key: ""

  fog_aws_storage_options: {}

db_encryption_key: TODO

database_encryption:
  keys: {"encryption_key_0":"TODO"}
  current_key_label: "encryption_key_0"
  pbkdf2_hmac_iterations: 2048

disable_custom_buildpacks: false

broker_client_timeout_seconds: 60
broker_client_default_async_poll_interval_seconds: 60
broker_client_max_async_poll_duration_minutes: 10080
renderer:
  max_results_per_page: 100
  default_results_per_page: 50
  max_inline_relations_depth: 2

uaa_client_name: "cc-service-dashboards"
uaa_client_secret: TODO
uaa_client_scope: openid,cloud_controller_service_permissions.read

cloud_controller_username_lookup_client_name: "cloud_controller_username_lookup"
cloud_controller_username_lookup_client_secret: ****

cc_service_key_client_name: "cc_service_key_client"
cc_service_key_client_secret: TODO

allow_app_ssh_access: true
default_app_ssh_access: true

skip_cert_verify: true

install_buildpacks: []

app_bits_upload_grace_period_in_seconds: 1200

security_group_definitions: [{"name":"public_networks","rules":[{"destination":"0.0.0.0-9.255.255.255","protocol":"all"},{"destination":"11.0.0.0-169.253.255.255","protocol":"all"},{"destination":"169.255.0.0-172.15.255.255","protocol":"all"},{"destination":"172.32.0.0-192.167.255.255","protocol":"all"},{"destination":"192.169.0.0-255.255.255.255","protocol":"all"}]},{"name":"dns","rules":[{"destination":"0.0.0.0/0","ports":"53","protocol":"tcp"},{"destination":"0.0.0.0/0","ports":"53","protocol":"udp"}]}]
default_running_security_groups: ["public_networks","dns"]
default_staging_security_groups: ["public_networks","dns"]

allowed_cors_domains: []

statsd_host: 127.0.0.1
statsd_port: 8125

security_event_logging:
  enabled: false
  file: "/dev/null"

system_hostnames: ["api", "uaa", "login", "doppler", "loggregator", "hm9000", "credhub"]

bits_service:
  enabled: false
  public_endpoint: https://bits.cf.ingress.kronos.istio.shoot.canary.k8s-hana.ondemand.com
  private_endpoint: https://TODO.TODO
  username: TODO
  password: TODO

  ca_cert_path: "/dev/null"

rate_limiter:
  enabled: false
  general_limit: 2000
  unauthenticated_limit: 100
  reset_interval_in_minutes: 60

diego:
  bbs:
    url: https://TODO.TODO
    key_file: "/dev/null"
    cert_file: "/dev/null"
    ca_file: "/dev/null"
    connect_timeout: 10
    send_timeout: 10
    receive_timeout: 10
  cc_uploader_url: https://TODO.TODO
  docker_staging_stack: cflinuxfs3
  file_server_url: https://TODO.TODO
  insecure_docker_registry_list: []
  lifecycle_bundles: {"buildpack/cflinuxfs3":"buildpack_app_lifecycle/buildpack_app_lifecycle.tgz","buildpack/windows":"buildpack_app_lifecycle/buildpack_app_lifecycle.tgz","buildpack/windows2012R2":"windows_app_lifecycle/windows_app_lifecycle.tgz","buildpack/windows2016":"buildpack_app_lifecycle/buildpack_app_lifecycle.tgz","docker":"docker_app_lifecycle/docker_app_lifecycle.tgz"}
  droplet_destinations: {"cflinuxfs3":"/home/vcap","windows":"/Users/vcap","windows2012R2":"/","windows2016":"/Users/vcap"}
  pid_limit: 1024
  temporary_oci_buildpack_mode:
  enable_declarative_asset_downloads: false
  use_privileged_containers_for_running: false
  use_privileged_containers_for_staging: false

opi:
  url: "https://eirini.kubecf.svc.cluster.local:8085"
  opi_staging: true
  enabled: true
  cc_uploader_url: "https://TODO.TODO"
  ca_file: /config/opi/certs/opi.ca
  client_cert_file: /config/opi/certs/opi.crt
  client_key_file: /config/opi/certs/opi.key

perm:
  enabled: false

max_labels_per_resource: 50
max_annotations_per_resource: 200

# worker property
perform_blob_cleanup: true 

# clock properties
app_usage_events:
  cutoff_age_in_days: 31
audit_events: 
  cutoff_age_in_days: 31
failed_jobs:
  cutoff_age_in_days: 31
completed_tasks:
  cutoff_age_in_days: 31
service_usage_events:
  cutoff_age_in_days: 31
max_retained_deployments_per_app: 100
max_retained_builds_per_app: 100
max_retained_revisions_per_app: 100
diego_sync:
  frequency_in_seconds: 0
pending_builds:
  frequency_in_seconds: 300
  expiration_in_seconds: 42
pending_droplets:
  frequency_in_seconds: 300
  expiration_in_seconds: 42

# deployment updater properties
deployment_updater:
  update_frequency_in_seconds: 1

Steps to Reproduce

Expected result

No Nokogiri::XML::SyntaxError

Current result

Error with blobstore: Nokogiri::XML::SyntaxError - 1:1: FATAL: Extra content at the end of the document 
/usr/local/lib/ruby/gems/2.5.0/gems/nokogiri-1.10.5/lib/nokogiri/xml/sax/push_parser.rb:47:in `native_write' 
/usr/local/lib/ruby/gems/2.5.0/gems/nokogiri-1.10.5/lib/nokogiri/xml/sax/push_parser.rb:47:in `write' 
/usr/local/lib/ruby/gems/2.5.0/gems/nokogiri-1.10.5/lib/nokogiri/xml/sax/push_parser.rb:55:in `finish' 
/usr/local/lib/ruby/gems/2.5.0/gems/fog-xml-0.1.3/lib/fog/xml/sax_parser_connection.rb:39:in `request' 
/usr/local/lib/ruby/gems/2.5.0/gems/fog-xml-0.1.3/lib/fog/xml/connection.rb:7:in `request' 
/usr/local/lib/ruby/gems/2.5.0/gems/fog-aws-2.0.1/lib/fog/aws/storage.rb:613:in `_request' 
/usr/local/lib/ruby/gems/2.5.0/gems/fog-aws-2.0.1/lib/fog/aws/storage.rb:608:in `request' 
/usr/local/lib/ruby/gems/2.5.0/gems/fog-aws-2.0.1/lib/fog/aws/requests/storage/get_bucket.rb:43:in `get_bucket' 
/usr/local/lib/ruby/gems/2.5.0/gems/fog-aws-2.0.1/lib/fog/aws/models/storage/directories.rb:21:in `get' 
/cloud_controller_ng/lib/cloud_controller/blobstore/fog/directory.rb:16:in `get' 
/cloud_controller_ng/lib/cloud_controller/blobstore/fog/idempotent_directory.rb:10:in `get_or_create' 
/cloud_controller_ng/lib/cloud_controller/blobstore/fog/fog_client.rb:180:in `dir' 
/cloud_controller_ng/lib/cloud_controller/blobstore/fog/fog_client.rb:130:in `files' 
/cloud_controller_ng/lib/cloud_controller/blobstore/fog/fog_client.rb:176:in `file' 
/cloud_controller_ng/lib/cloud_controller/blobstore/fog/fog_client.rb:40:in `exists?' 
/cloud_controller_ng/lib/cloud_controller/blobstore/retryable_client.rb:25:in `block in exists?' 
/cloud_controller_ng/lib/cloud_controller/blobstore/retryable_client.rb:139:in `with_retries' 
/cloud_controller_ng/lib/cloud_controller/blobstore/retryable_client.rb:20:in `exists?' 
/cloud_controller_ng/lib/cloud_controller/blobstore/fog/error_handling_client.rb:23:in `block in exists?' 
/cloud_controller_ng/lib/cloud_controller/blobstore/fog/error_handling_client.rb:61:in `error_handling' 
/cloud_controller_ng/lib/cloud_controller/blobstore/fog/error_handling_client.rb:23:in `exists?' 
/cloud_controller_ng/lib/cloud_controller/resource_pool.rb:98:in `resource_known?' 
/cloud_controller_ng/lib/cloud_controller/resource_match.rb:29:in `block (2 levels) in match_resources' 
/cloud_controller_ng/lib/cloud_controller/resource_match.rb:27:in `each' 
/cloud_controller_ng/lib/cloud_controller/resource_match.rb:27:in `block in match_resources' 
/cloud_controller_ng/lib/cloud_controller/resource_match.rb:26:in `each' 
/cloud_controller_ng/lib/cloud_controller/resource_match.rb:26:in `match_resources' 
/cloud_controller_ng/lib/cloud_controller/resource_pool.rb:36:in `match_resources' 
/cloud_controller_ng/lib/cloud_controller/resource_pool_wrapper.rb:11:in `call' 
/cloud_controller_ng/app/controllers/runtime/resource_matches_controller.rb:14:in `match' 
/cloud_controller_ng/app/controllers/base/base_controller.rb:84:in `dispatch' 
/cloud_controller_ng/lib/cloud_controller/rest_controller/routes.rb:16:in `block in define_route' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1634:in `call' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1634:in `block in compile!' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:992:in `block (3 levels) in route!' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1011:in `route_eval' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:992:in `block (2 levels) in route!' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1040:in `block in process_route' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1038:in `catch' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1038:in `process_route' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:990:in `block in route!' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:989:in `each' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:989:in `route!' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1097:in `block in dispatch!' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1076:in `block in invoke' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1076:in `catch' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1076:in `invoke' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1094:in `dispatch!' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:924:in `block in call!' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1076:in `block in invoke' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1076:in `catch' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1076:in `invoke' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:924:in `call!' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:913:in `call' 
/usr/local/lib/ruby/gems/2.5.0/gems/rack-protection-2.0.5/lib/rack/protection/xss_header.rb:18:in `call' 
/usr/local/lib/ruby/gems/2.5.0/gems/rack-protection-2.0.5/lib/rack/protection/path_traversal.rb:16:in `call' 
/usr/local/lib/ruby/gems/2.5.0/gems/rack-protection-2.0.5/lib/rack/protection/json_csrf.rb:26:in `call' 
/usr/local/lib/ruby/gems/2.5.0/gems/rack-protection-2.0.5/lib/rack/protection/base.rb:50:in `call' 
/usr/local/lib/ruby/gems/2.5.0/gems/rack-protection-2.0.5/lib/rack/protection/base.rb:50:in `call' 
/usr/local/lib/ruby/gems/2.5.0/gems/rack-protection-2.0.5/lib/rack/protection/frame_options.rb:31:in `call' 
/usr/local/lib/ruby/gems/2.5.0/gems/rack-2.0.8/lib/rack/null_logger.rb:9:in `call' 
/usr/local/lib/ruby/gems/2.5.0/gems/rack-2.0.8/lib/rack/head.rb:12:in `call' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:194:in `call' 
/usr/local/lib/ruby/gems/2.5.0/gems/sinatra-2.0.5/lib/sinatra/base.rb:1957:in `call' 
/usr/local/lib/ruby/gems/2.5.0/gems/rack-2.0.8/lib/rack/urlmap.rb:68:in `block in call' 
/usr/local/lib/ruby/gems/2.5.0/gems/rack-2.0.8/lib/rack/urlmap.rb:53:in `each' 
/usr/local/lib/ruby/gems/2.5.0/gems/rack-2.0.8/lib/rack/urlmap.rb:53:in `call' 
/cloud_controller_ng/middleware/request_logs.rb:22:in `call' 
/cloud_controller_ng/middleware/security_context_setter.rb:19:in `call' 
/cloud_controller_ng/middleware/vcap_request_id.rb:15:in `call' 
/cloud_controller_ng/middleware/cors.rb:49:in `call_app' 
/cloud_controller_ng/middleware/cors.rb:14:in `call' 
/cloud_controller_ng/middleware/request_metrics.rb:12:in `call' 
/usr/local/lib/ruby/gems/2.5.0/gems/rack-2.0.8/lib/rack/builder.rb:153:in `call' 
/usr/local/lib/ruby/gems/2.5.0/gems/thin-1.7.2/lib/thin/connection.rb:86:in `block in pre_process' 
/usr/local/lib/ruby/gems/2.5.0/gems/thin-1.7.2/lib/thin/connection.rb:84:in `catch' 
/usr/local/lib/ruby/gems/2.5.0/gems/thin-1.7.2/lib/thin/connection.rb:84:in `pre_process' 
/usr/local/lib/ruby/gems/2.5.0/gems/thin-1.7.2/lib/thin/connection.rb:50:in `block in process' 
/usr/local/lib/ruby/gems/2.5.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:1067:in `block in spawn_threadpool'","log_level":"error","source":"cc.error_handling_client","data":{"request_guid":"e8b492dc-ab51-4480-45c9-5ba9060b0c7c::704e0078-5046-4370-b59c-ea4a3db13f50"},"thread_id":47187291039320,"fiber_id":47187268195040,"process_id":1,"file":"
/cloud_controller_ng/lib/cloud_controller/blobstore/fog/error_handling_client.rb","lineno":63,"method":"rescue in error_handling"

Possible Fix

[not obligatory, but suggest fixes or reasons for the bug]

name of issue screenshot

[if relevant, include a screenshot]

cf-gitbot commented 4 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/170640405

The labels on this github issue will be updated when the story is started.

cwlbraa commented 4 years ago

Interesting... what blobstore are you using in this quarks environment?

It bears mentioning we ourselves have never got capi-k8s-release to complete a push.

How far through the push did you get before it errored? It seems likely that it would break around the package upload endpoint (POST /v2/.../bits) because the nginx upload module is not configured in that context.

I think the present plan is to try to move away from old school buildpacks and towards cloud native buildpacks. That means building in kpack support first. We're thinking that we might be able to remove our dependency on an object store [or at least make it an optional part to support old school buildpacks] and replace it with a container registry.

kramerul commented 4 years ago

We are using S3 as blobstore.

I have no idea how far the push did. When this error occurred, we gave up.

aashah commented 4 years ago

Hey @kramerul,

Given how fast some things are moving in capi-k8s-release, and this issue being related to the cf push experience, could you share whether you are still experiencing this after updating capi-k8s-release?

kramerul commented 4 years ago

in the meanwhile , we switch our deployment from kubecf to cf-for-k8s. cf push is working. But we are not using an external blob store. I think, this is currently not possible with cf-for-k8s. For the moment we are not able to reproduce this issue.

aashah commented 4 years ago

That's good to hear!

Please re-open or open a new issue if you come across other issues!