Question concerning the internal_address_config parameter on Envoy

[joel-vaz]

Title: Question concerning the internal_address_config parameter on Envoy

internal_address_config is not configured. The existing default behaviour will trust RFC1918 IP addresses, but this will be changed in next release. Please explictily config internal address config as the migration step.

Description: I got the following message about a configuration parameter of envoy missing from my bootstrap file: internal_address_config is not configured. The existing default behaviour will trust RFC1918 IP addresses, but this will be changed in next release. Please explictily config internal address config as the migration step..

Consul Version: 1.18.2 Envoy Version: 1.29.9

This is my bootstrap configuration file that I use to start Envoy as a proxy on my services:

{
  "admin": {
    "access_log": [
      {
        "name": "envoy.access_loggers.file",
        "typed_config": {
          "@type": "type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog",
          "path": "/dev/null"
        }
      }
    ],
    "address": {
      "socket_address": {
        "address": "<>",
        "port_value": <>
      }
    }
  },
  "node": {
    "cluster": "<>",
    "id": "<>",
    "metadata": {
      "namespace": "default",
      "partition": "default"
    }
  },
  "layered_runtime": {
    "layers": [
      {
        "name": "base",
        "static_layer": {
          "re2.max_program_size.error_level": <>
        }
      }
    ]
  },
  "static_resources": {
    "clusters": [
      {
        "name": "local_agent",
        "ignore_health_on_host_removal": false,
        "connect_timeout": "1s",
        "type": "STATIC",
        "transport_socket": {
          "name": "tls",
          "typed_config": {
            "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext",
            "common_tls_context": {
              "validation_context": {
                "trusted_ca": {
                  "inline_string": "<CERT>"
                }
              },
              "tls_params": {
                "tls_minimum_protocol_version": "TLSv1_3",
                "tls_maximum_protocol_version": "TLSv1_3"
              }
            }
          }
        },
        "typed_extension_protocol_options": {
          "envoy.extensions.upstreams.http.v3.HttpProtocolOptions": {
            "@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions",
            "explicit_http_config": {
              "http2_protocol_options": {}
            }
          }
        },
        "loadAssignment": {
          "clusterName": "<>",
          "endpoints": [
            {
              "lbEndpoints": [
                {
                  "endpoint": {
                    "address": {
                      "socket_address": {
                        "address": "<>",
                        "port_value": <>
                      }
                    }
                  }
                }
              ]
            }
          ]
        }
      },
      {
        "connect_timeout": "3.000s",
        "dns_lookup_family": "V4_ONLY",
        "lb_policy": "ROUND_ROBIN",
        "load_assignment": {
          "cluster_name": "<>",
          "endpoints": [
            {
              "lb_endpoints": [
                {
                  "endpoint": {
                    "address": {
                      "socket_address": {
                        "address": "<>",
                        "port_value": <>,
                        "protocol": "TCP"
                      }
                    }
                  }
                }
              ]
            }
          ]
        },
        "name": "<>",
        "type": "STRICT_DNS"
      },
      {
        "name": "<>",
        "ignore_health_on_host_removal": false,
        "connect_timeout": "5s",
        "type": "STATIC",
        "typed_extension_protocol_options": {
          "envoy.extensions.upstreams.http.v3.HttpProtocolOptions": {
            "@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions",
            "explicit_http_config": {
              "http_protocol_options": {}
            }
          }
        },
        "loadAssignment": {
          "clusterName": "self_admin",
          "endpoints": [
            {
              "lbEndpoints": [
                {
                  "endpoint": {
                    "address": {
                      "socket_address": {
                        "address": "<>",
                        "port_value": <>
                      }
                    }
                  }
                }
              ]
            }
          ]
        }
      }
    ],
    "listeners": [
      {
        "name": "envoy_metrics_listener",
        "address": {
          "socket_address": {
            "address": "<>",
            "port_value": <>
          }
        },
        "filter_chains": [
          {
            "filters": [
              {
                "name": "envoy.filters.network.http_connection_manager",
                "typedConfig": {
                  "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
                  "stat_prefix": "envoy_metrics",
                  "codec_type": "HTTP1",
                  "route_config": {
                    "name": "self_admin_route",
                    "virtual_hosts": [
                      {
                        "name": "<>",
                        "domains": [
                          "*"
                        ],
                        "routes": [
                          {
                            "match": {
                              "prefix": "/stats"
                            },
                            "route": {
                              "cluster": "<>",
                              "prefix_rewrite": "/stats"
                            }
                          },
                          {
                            "match": {
                              "prefix": "/"
                            },
                            "direct_response": {
                              "status": 404
                            }
                          }
                        ]
                      }
                    ]
                  },
                  "http_filters": [
                    {
                      "name": "envoy.filters.http.router",
                      "typedConfig": {
                        "@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
                      }
                    }
                  ],
                  "internal_address_config": {
                    "cidr_ranges": [
                      {
                        "address_prefix": "10.0.0.0",
                        "prefix_len": 8
                      },
                      {
                        "address_prefix": "127.0.0.1",
                        "prefix_len": 32
                      },
                      {
                        "address_prefix": "::10",
                        "prefix_len": 128
                      }
                    ]
                  }
                }
              }
            ]
          }
        ]
      }
    ]
  },
  "stats_config": {
    "stats_tags": [
      {
        "regex": "^cluster\\.(?:passthrough~)?((?:([^.]+)~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.destination.custom_hash"
      },
      {
        "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:([^.]+)\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.destination.service_subset"
      },
      {
        "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?([^.]+)\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.destination.service"
      },
      {
        "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.([^.]+)\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.destination.namespace"
      },
      {
        "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:([^.]+)\\.)?[^.]+\\.internal[^.]*\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.destination.partition"
      },
      {
        "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?([^.]+)\\.internal[^.]*\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.destination.datacenter"
      },
      {
        "regex": "^cluster\\.([^.]+\\.(?:[^.]+\\.)?([^.]+)\\.external\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.destination.peer"
      },
      {
        "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.([^.]+)\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.destination.routing_type"
      },
      {
        "regex": "^cluster\\.(?:passthrough~)?((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.([^.]+)\\.consul\\.)",
        "tag_name": "consul.destination.trust_domain"
      },
      {
        "regex": "^cluster\\.(?:passthrough~)?(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+)\\.[^.]+\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.destination.target"
      },
      {
        "regex": "^cluster\\.(?:passthrough~)?(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+)\\.consul\\.)",
        "tag_name": "consul.destination.full_target"
      },
      {
        "regex": "^(?:tcp|http)\\.upstream(?:_peered)?\\.(([^.]+)(?:\\.[^.]+)?(?:\\.[^.]+)?\\.[^.]+\\.)",
        "tag_name": "consul.upstream.service"
      },
      {
        "regex": "^(?:tcp|http)\\.upstream\\.([^.]+(?:\\.[^.]+)?(?:\\.[^.]+)?\\.([^.]+)\\.)",
        "tag_name": "consul.upstream.datacenter"
      },
      {
        "regex": "^(?:tcp|http)\\.upstream_peered\\.([^.]+(?:\\.[^.]+)?\\.([^.]+)\\.)",
        "tag_name": "consul.upstream.peer"
      },
      {
        "regex": "^(?:tcp|http)\\.upstream(?:_peered)?\\.([^.]+(?:\\.([^.]+))?(?:\\.[^.]+)?\\.[^.]+\\.)",
        "tag_name": "consul.upstream.namespace"
      },
      {
        "regex": "^(?:tcp|http)\\.upstream\\.([^.]+(?:\\.[^.]+)?(?:\\.([^.]+))?\\.[^.]+\\.)",
        "tag_name": "consul.upstream.partition"
      },
      {
        "regex": "^cluster\\.((?:([^.]+)~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.custom_hash"
      },
      {
        "regex": "^cluster\\.((?:[^.]+~)?(?:([^.]+)\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.service_subset"
      },
      {
        "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?([^.]+)\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.service"
      },
      {
        "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.([^.]+)\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.namespace"
      },
      {
        "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?([^.]+)\\.internal[^.]*\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.datacenter"
      },
      {
        "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.([^.]+)\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.routing_type"
      },
      {
        "regex": "^cluster\\.((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.([^.]+)\\.consul\\.)",
        "tag_name": "consul.trust_domain"
      },
      {
        "regex": "^cluster\\.(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+)\\.[^.]+\\.[^.]+\\.consul\\.)",
        "tag_name": "consul.target"
      },
      {
        "regex": "^cluster\\.(((?:[^.]+~)?(?:[^.]+\\.)?[^.]+\\.[^.]+\\.(?:[^.]+\\.)?[^.]+\\.[^.]+\\.[^.]+)\\.consul\\.)",
        "tag_name": "consul.full_target"
      },
      {
        "tag_name": "local_cluster",
        "fixed_value": "mesh-sb-test-appliA"
      },
      {
        "tag_name": "consul.source.service",
        "fixed_value": "mesh-sb-test-appliA"
      },
      {
        "tag_name": "consul.source.namespace",
        "fixed_value": "default"
      },
      {
        "tag_name": "consul.source.partition",
        "fixed_value": "default"
      },
      {
        "tag_name": "consul.source.datacenter",
        "fixed_value": "sb-test-sandbox-pdx"
      }
    ],
    "use_all_default_tags": true
  },
  "tracing": {
    "http": {
      "name": "envoy.tracers.datadog",
      "typedConfig": {
        "@type": "type.googleapis.com/envoy.config.trace.v3.DatadogConfig",
        "collector_cluster": "<>",
        "service_name": "<>"
      }
    }
  },
  "dynamic_resources": {
    "lds_config": {
      "ads": {},
      "initial_fetch_timeout": "0s",
      "resource_api_version": "V3"
    },
    "cds_config": {
      "ads": {},
      "initial_fetch_timeout": "0s",
      "resource_api_version": "V3"
    },
    "ads_config": {
      "api_type": "DELTA_GRPC",
      "transport_api_version": "V3",
      "grpc_services": {
        "initial_metadata": [
          {
            "key": "<>",
            "value": "<>"
          }
        ],
        "envoy_grpc": {
          "cluster_name": "<>"
        }
      }
    }
  }
}

Could you help me when to define the configuration parameter correctly?

[optional Relevant Links:] https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto

Full log snippet:

[2024-10-10 12:15:15.875][49][info][admin] [source/server/admin/admin.cc:66] admin address: <>
[2024-10-10 12:15:15.875][49][info][config] [source/server/configuration_impl.cc:160] loading tracing configuration
[2024-10-10 12:15:15.875][49][info][config] [source/server/configuration_impl.cc:171]   validating default server-wide tracing driver: envoy.tracers.datadog
[2024-10-10 12:15:15.875][49][info][config] [source/server/configuration_impl.cc:119] loading 0 static secret(s)
[2024-10-10 12:15:15.875][49][info][config] [source/server/configuration_impl.cc:125] loading 3 cluster(s)
[2024-10-10 12:15:15.943][49][info][config] [source/server/configuration_impl.cc:129] loading 1 listener(s)
[2024-10-10 12:15:15.946][49][info][config] [source/server/configuration_impl.cc:146] loading stats configuration
[2024-10-10 12:15:15.946][49][info][runtime] [source/common/runtime/runtime_impl.cc:584] RTDS has finished initialization
[2024-10-10 12:15:15.946][49][info][upstream] [source/common/upstream/cluster_manager_impl.cc:236] cm init: initializing cds
[2024-10-10 12:15:15.947][49][warning][main] [source/server/server.cc:910] There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.downstream_connections` resource monitor.
[2024-10-10 12:15:15.947][49][info][main] [source/server/server.cc:951] starting main dispatch loop
[2024-10-10 12:15:15.951][49][info][upstream] [source/common/upstream/cds_api_helper.cc:32] cds: add 2 cluster(s), remove 0 cluster(s)
[2024-10-10 12:15:16.032][49][info][upstream] [source/common/upstream/cds_api_helper.cc:71] cds: added/updated 2 cluster(s), skipped 0 unmodified cluster(s)
[2024-10-10 12:15:16.032][49][info][upstream] [source/common/upstream/cluster_manager_impl.cc:214] cm init: initializing secondary clusters
[2024-10-10 12:15:16.034][49][info][upstream] [source/common/upstream/cluster_manager_impl.cc:240] cm init: all clusters initialized
[2024-10-10 12:15:16.034][49][info][main] [source/server/server.cc:932] all clusters initialized. initializing init manager
[2024-10-10 12:15:16.039][49][info][upstream] [source/common/listener_manager/lds_api.cc:99] lds: add/update listener 'public_listener:<ip>:<port>'
[2024-10-10 12:15:16.043][49][warning][misc] [source/extensions/filters/network/http_connection_manager/config.cc:82] internal_address_config is not configured. The existing default behaviour will trust RFC1918 IP addresses, but this will be changed in next release. Please explictily config internal address config as the migration step.

[alyssawilk]

internal_address_config is a property of the HCM

https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#envoy-v3-api-msg-extensions-filters-network-http-connection-manager-v3-httpconnectionmanager-internaladdressconfig

you only need to set this if you want 127. and 192. addresses to remain trusted

[joel-vaz]

First of all, thank you for the quick reply 🙏

I added the list of CIDR ranges to the configuration:

                  "internal_address_config": {
                    "cidr_ranges": [
                      {
                        "address_prefix": "10.0.0.0",
                        "prefix_len": 8
                      },
                      {
                        "address_prefix": "127.0.0.1",
                        "prefix_len": 32
                      },
                      {
                        "address_prefix": "::10",
                        "prefix_len": 128
                      }
                    ]
                  }

But I was not able to suppress the log warning. My question is what I can do to resolve this warning during this migration? Since I'm a bit lost as to what this warning is referring to.

internal_address_config is not configured. The existing default behaviour will trust RFC1918 IP addresses, but this will be changed in next release. Please explictily config internal address config as the migration step.

[alyssawilk]

you should only get that warning if one or more HCMs doesn't have an internal address config. Are you perhaps missing one?

[joel-vaz]

Hello @alyssawilk

Is it possible that some listeners are configured at runtime after envoy was bootstrapped? 🤔 (Sorry if this question sounds naive)

After the bootstrap I got this listener that appears it was added after the envoy was bootstrapped with the configuration.

lds: add/update listener 'public_listener:<ip>:<port>'

In that case, this listener would be missing the internal_address_config parameter, but how can I configure this parameter on this type of scenarios?

Kind Regards,

[alyssawilk]

Yep, LDS means you're dynamically updating your listeners, so I think you need to track down the source of your lds config and update the internal address config there as well