search

spiffe / spire

The SPIFFE Runtime Environment
https://spiffe.io
Apache License 2.0
1.8k stars 475 forks source link

Spire do not federate between k8s clusters which implements Istio multi-primary on different networks #3503

Closed sudeeptoroy closed 2 years ago

sudeeptoroy commented 2 years ago

An application is make to federate between two trust domain using spire and Istio. The app being sleep app and helloworld app. The sleep application is able to discover the helloworld service in the other cluster. spire-agent is managing the SDS interface for envoy and its helping to mint the required certs for the envoy. spire is configured with auto attestation of workload.

A sleep app in cluster is able to reach and connect the helloworld app in the same cluster. however when the same sleep application is trying to connect to the helloworld app in the other cluster it fails with SSL error: CERTIFICATE_VERIFY_FAILED

Istio is configured for multi-primary with different network ( meaning there is no direct connection between the pods across cluster boundary). An additional east-west ( ew gw ) is installed. This ew gw has a public address for the clusters to reach and it does SNI pass-through for the traffic to directly reach the service hosted inside the cluster. The service is protected with envoy. Envoy validates all connection for mTLS. on successful validation it would allow the communication to get established.

In case of federation the mTLS will be between two different trust domain and with spire configured correctly it would perform the trust bundle exchange and make the federated CA available to the envoy sitting next to the service.

Expectation: As the CA and federated CA has made available to the helloworld and sleep service, the connection between the sleep application and helloworld should have gone through. While testing it fails and hence this issue has been raised.

Topology

note: entire configuration can be found here: https://github.com/sudeeptoroy/spirefed

  1. brought up two kind clusters: kind-aws-cluster and kind-google-cluster
  2. configured spire for these two clusters and put them in different trust domain: aws.com and google.com
  3. brought up istio on both clusters in multi-primary mode. i have followed this article to bringup istio: https://istio.io/latest/docs/setup/install/multicluster/multi-primary_multi-network/
  4. brought up sample app to rest the federation. 4a. sleep app on aws-cluster 4.b helloworld app on google-cluster 4.c from the sleep app execute curl to helloworld on the other cluster

here is pictorial representation of the topolgy.

image

Result: The curl fails with this error: kubectl exec --context=kind-aws-cluster -n sample -c sleep sleep-95d8696-bk822 -- curl -sS helloworld.sample:5000/hello upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

Steps to reproduce: Clone: https://github.com/sudeeptoroy/spirefed Follow the readme

Observation:

when i run the helloworld istio-proxy in trace mode i see the following logs:

2022-10-15T10:39:52.419167Z debug   envoy filter    original_dst: new connection accepted
2022-10-15T10:39:52.419262Z trace   envoy filter    original_dst: set destination to 10.241.1.8:5000
2022-10-15T10:39:52.419278Z debug   envoy filter    tls inspector: new connection accepted
2022-10-15T10:39:52.419291Z trace   envoy filter    tls inspector: recv: 517
2022-10-15T10:39:52.419313Z trace   envoy filter    tls:onALPN(), ALPN: istio-http/1.1,istio,http/1.1
2022-10-15T10:39:52.419330Z debug   envoy filter    tls:onServerName(), requestedServerName: outbound_.5000_._.helloworld.sample.svc.cluster.local
2022-10-15T10:39:52.419399Z trace   envoy misc  enableTimer called on 0x55e2b1ee4080 for 3600000ms, min is 3600000ms
2022-10-15T10:39:52.419448Z debug   envoy conn_handler  [C146] new connection from 10.241.1.6:58350
2022-10-15T10:39:52.419474Z trace   envoy connection    [C146] socket event: 3
2022-10-15T10:39:52.419486Z trace   envoy connection    [C146] write ready
2022-10-15T10:39:52.420578Z trace   envoy connection    [C146] ssl error occurred while read: WANT_READ
2022-10-15T10:39:52.420705Z trace   envoy connection    [C146] read ready. dispatch_buffered_data=0
2022-10-15T10:39:52.420744Z trace   envoy connection    [C146] ssl error occurred while read: WANT_READ
2022-10-15T10:39:52.424542Z trace   envoy connection    [C146] socket event: 3
2022-10-15T10:39:52.424707Z trace   envoy connection    [C146] write ready
2022-10-15T10:39:52.424750Z trace   envoy connection    [C146] ssl error occurred while read: SSL
2022-10-15T10:39:52.424767Z debug   envoy connection    [C146] TLS error: 268436502:SSL routines:**OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN**
2022-10-15T10:39:52.424780Z debug   envoy connection    [C146] closing socket: 0
2022-10-15T10:39:52.424824Z debug   envoy connection    [C146] TLS error: 268436502:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN
2022-10-15T10:39:52.424859Z trace   envoy connection    [C146] raising connection event 0
2022-10-15T10:39:52.424905Z trace   envoy conn_handler  [C146] connection on event 0
2022-10-15T10:39:52.424918Z debug   envoy conn_handler  [C146] adding to cleanup list
2022-10-15T10:39:52.424929Z trace   envoy main  item added to deferred deletion list (size=1)
2022-10-15T10:39:52.424941Z trace   envoy main  clearing deferred deletion list (size=1)
2022-10-15T10:39:52.432818Z debug   envoy filter    original_dst: new connection accepted
2022-10-15T10:39:52.432913Z trace   envoy filter    original_dst: set destination to 10.241.1.8:5000
2022-10-15T10:39:52.432929Z debug   envoy filter    tls inspector: new connection accepted
2022-10-15T10:39:52.432942Z trace   envoy filter    tls inspector: recv: 517
2022-10-15T10:39:52.432967Z trace   envoy filter    tls:onALPN(), ALPN: istio-http/1.1,istio,http/1.1
2022-10-15T10:39:52.432984Z debug   envoy filter    tls:onServerName(), requestedServerName: outbound_.5000_._.helloworld.sample.svc.cluster.local
2022-10-15T10:39:52.433065Z trace   envoy misc  enableTimer called on 0x55e2b1ee4080 for 3600000ms, min is 3600000ms
2022-10-15T10:39:52.433086Z debug   envoy conn_handler  [C147] new connection from 10.241.1.6:58354
2022-10-15T10:39:52.433166Z trace   envoy connection    [C147] socket event: 3
2022-10-15T10:39:52.433209Z trace   envoy connection    [C147] write ready
2022-10-15T10:39:52.433493Z trace   envoy connection    [C147] ssl error occurred while read: WANT_READ
2022-10-15T10:39:52.433679Z trace   envoy connection    [C147] read ready. dispatch_buffered_data=0
2022-10-15T10:39:52.433698Z trace   envoy connection    [C147] ssl error occurred while read: WANT_READ
2022-10-15T10:39:52.438301Z trace   envoy connection    [C147] socket event: 3
2022-10-15T10:39:52.438685Z trace   envoy connection    [C147] write ready
2022-10-15T10:39:52.438821Z trace   envoy connection    [C147] ssl error occurred while read: SSL
2022-10-15T10:39:52.438896Z debug   envoy connection    [C147] TLS error: 268436502:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN
2022-10-15T10:39:52.439010Z debug   envoy connection    [C147] closing socket: 0
2022-10-15T10:39:52.440473Z debug   envoy connection    [C147] TLS error: 268436502:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN
2022-10-15T10:39:52.440731Z trace   envoy connection    [C147] raising connection event 0
2022-10-15T10:39:52.440776Z trace   envoy conn_handler  [C147] connection on event 0
2022-10-15T10:39:52.440874Z debug   envoy conn_handler  [C147] adding to cleanup list
2022-10-15T10:39:52.440920Z trace   envoy main  item added to deferred deletion list (size=1)
2022-10-15T10:39:52.440939Z trace   envoy main  clearing deferred deletion list (size=1)
2022-10-15T10:39:52.464980Z debug   envoy filter    original_dst: new connection accepted
2022-10-15T10:39:52.465038Z trace   envoy filter    original_dst: set destination to 10.241.1.8:5000
2022-10-15T10:39:52.465045Z debug   envoy filter    tls inspector: new connection accepted
2022-10-15T10:39:52.465052Z trace   envoy filter    tls inspector: recv: 517
2022-10-15T10:39:52.465073Z trace   envoy filter    tls:onALPN(), ALPN: istio-http/1.1,istio,http/1.1
2022-10-15T10:39:52.465083Z debug   envoy filter    tls:onServerName(), requestedServerName: outbound_.5000_._.helloworld.sample.svc.cluster.local
2022-10-15T10:39:52.465158Z trace   envoy misc  enableTimer called on 0x55e2b19a1c80 for 3600000ms, min is 3600000ms
2022-10-15T10:39:52.465199Z debug   envoy conn_handler  [C148] new connection from 10.241.1.6:58360
2022-10-15T10:39:52.465220Z trace   envoy connection    [C148] socket event: 3
2022-10-15T10:39:52.465224Z trace   envoy connection    [C148] write ready
2022-10-15T10:39:52.465663Z trace   envoy connection    [C148] ssl error occurred while read: WANT_READ
2022-10-15T10:39:52.465673Z trace   envoy connection    [C148] read ready. dispatch_buffered_data=0
2022-10-15T10:39:52.465678Z trace   envoy connection    [C148] ssl error occurred while read: WANT_READ
2022-10-15T10:39:52.473615Z trace   envoy connection    [C148] socket event: 3
2022-10-15T10:39:52.473658Z trace   envoy connection    [C148] write ready
2022-10-15T10:39:52.473751Z trace   envoy connection    [C148] ssl error occurred while read: SSL
2022-10-15T10:39:52.473760Z debug   envoy connection    [C148] TLS error: 268436502:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN
2022-10-15T10:39:52.473765Z debug   envoy connection    [C148] closing socket: 0
2022-10-15T10:39:52.473908Z debug   envoy connection    [C148] TLS error: 268436502:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN
2022-10-15T10:39:52.473934Z trace   envoy connection    [C148] raising connection event 0
2022-10-15T10:39:52.473942Z trace   envoy conn_handler  [C148] connection on event 0
2022-10-15T10:39:52.473944Z debug   envoy conn_handler  [C148] adding to cleanup list

From the logs and envoy config at helloworld:

The first filter at envoy is "original_dst" where tls inspector should route it to "outbound|5000||helloworld.sample.svc.cluster.local". And for some reason this is not accepting the mTLS from the other domain "aws.com".

Listener dump: check the last section: original_dst

{
  "name": "virtualInbound",
  "active_state": {
    "version_info": "2022-10-15T10:34:48Z/11",
    "listener": {
      "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
      "name": "virtualInbound",
      "address": {
        "socket_address": {
          "address": "0.0.0.0",
          "port_value": 15006
        }
      },
      "filter_chains": [
        {
          "filter_chain_match": {
            "destination_port": 15006
          },
          "filters": [
            {
              "name": "istio.metadata_exchange",
              "typed_config": {
                "@type": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange",
                "protocol": "istio-peer-exchange"
              }
            },
            {
              "name": "istio.stats",
              "typed_config": {
                "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                "type_url": "type.googleapis.com/envoy.extensions.filters.network.wasm.v3.Wasm",
                "value": {
                  "config": {
                    "root_id": "stats_inbound",
                    "vm_config": {
                      "vm_id": "tcp_stats_inbound",
                      "runtime": "envoy.wasm.runtime.null",
                      "code": {
                        "local": {
                          "inline_string": "envoy.wasm.stats"
                        }
                      }
                    },
                    "configuration": {
                      "@type": "type.googleapis.com/google.protobuf.StringValue",
                      "value": "{\n  \"debug\": \"false\",\n  \"stat_prefix\": \"istio\",\n  \"metrics\": [\n    {\n      \"dimensions\": {\n        \"destination_cluster\": \"node.metadata['CLUSTER_ID']\",\n        \"source_cluster\": \"downstream_peer.cluster_id\"\n      }\n    }\n  ]\n}\n"
                    }
                  }
                }
              }
            },
            {
              "name": "envoy.filters.network.tcp_proxy",
              "typed_config": {
                "@type": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
                "stat_prefix": "BlackHoleCluster",
                "cluster": "BlackHoleCluster"
              }
            }
          ],
          "name": "virtualInbound-blackhole"
        },
        {
          "filter_chain_match": {
            "prefix_ranges": [
              {
                "address_prefix": "0.0.0.0",
                "prefix_len": 0
              }
            ],
            "transport_protocol": "tls",
            "application_protocols": [
              "istio-http/1.0",
              "istio-http/1.1",
              "istio-h2"
            ]
          },
          "filters": [
            {
              "name": "istio.metadata_exchange",
              "typed_config": {
                "@type": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange",
                "protocol": "istio-peer-exchange"
              }
            },
            {
              "name": "envoy.filters.network.http_connection_manager",
              "typed_config": {
                "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
                "stat_prefix": "InboundPassthroughClusterIpv4",
                "route_config": {
                  "name": "InboundPassthroughClusterIpv4",
                  "virtual_hosts": [
                    {
                      "name": "inbound|http|0",
                      "domains": [
                        "*"
                      ],
                      "routes": [
                        {
                          "match": {
                            "prefix": "/"
                          },
                          "route": {
                            "cluster": "InboundPassthroughClusterIpv4",
                            "timeout": "0s",
                            "max_stream_duration": {
                              "max_stream_duration": "0s",
                              "grpc_timeout_header_max": "0s"
                            }
                          },
                          "decorator": {
                            "operation": ":0/*"
                          },
                          "name": "default"
                        }
                      ]
                    }
                  ],
                  "validate_clusters": false
                },
                "http_filters": [
                  {
                    "name": "istio.metadata_exchange",
                    "typed_config": {
                      "@type": "type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm",
                      "config": {
                        "vm_config": {
                          "runtime": "envoy.wasm.runtime.null",
                          "code": {
                            "local": {
                              "inline_string": "envoy.wasm.metadata_exchange"
                            }
                          }
                        },
                        "configuration": {
                          "@type": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange"
                        }
                      }
                    }
                  },
                  {
                    "name": "envoy.filters.http.fault",
                    "typed_config": {
                      "@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
                    }
                  },
                  {
                    "name": "envoy.filters.http.cors",
                    "typed_config": {
                      "@type": "type.googleapis.com/envoy.extensions.filters.http.cors.v3.Cors"
                    }
                  },
                  {
                    "name": "istio.stats",
                    "typed_config": {
                      "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                      "type_url": "type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm",
                      "value": {
                        "config": {
                          "root_id": "stats_inbound",
                          "vm_config": {
                            "vm_id": "stats_inbound",
                            "runtime": "envoy.wasm.runtime.null",
                            "code": {
                              "local": {
                                "inline_string": "envoy.wasm.stats"
                              }
                            }
                          },
                          "configuration": {
                            "@type": "type.googleapis.com/google.protobuf.StringValue",
                            "value": "{\n  \"debug\": \"false\",\n  \"stat_prefix\": \"istio\",\n  \"disable_host_header_fallback\": true,\n  \"metrics\": [\n    {\n      \"dimensions\": {\n        \"destination_cluster\": \"node.metadata['CLUSTER_ID']\",\n        \"source_cluster\": \"downstream_peer.cluster_id\"\n      }\n    }\n  ]\n}\n"
                          }
                        }
                      }
                    }
                  },
                  {
                    "name": "envoy.filters.http.router",
                    "typed_config": {
                      "@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
                    }
                  }
                ],
                "tracing": {
                  "client_sampling": {
                    "value": 100
                  },
                  "random_sampling": {
                    "value": 1
                  },
                  "overall_sampling": {
                    "value": 100
                  },
                  "custom_tags": [
                    {
                      "tag": "istio.authorization.dry_run.allow_policy.name",
                      "metadata": {
                        "kind": {
                          "request": {}
                        },
                        "metadata_key": {
                          "key": "envoy.filters.http.rbac",
                          "path": [
                            {
                              "key": "istio_dry_run_allow_shadow_effective_policy_id"
                            }
                          ]
                        }
                      }
                    },
                    {
                      "tag": "istio.authorization.dry_run.allow_policy.result",
                      "metadata": {
                        "kind": {
                          "request": {}
                        },
                        "metadata_key": {
                          "key": "envoy.filters.http.rbac",
                          "path": [
                            {
                              "key": "istio_dry_run_allow_shadow_engine_result"
                            }
                          ]
                        }
                      }
                    },
                    {
                      "tag": "istio.authorization.dry_run.deny_policy.name",
                      "metadata": {
                        "kind": {
                          "request": {}
                        },
                        "metadata_key": {
                          "key": "envoy.filters.http.rbac",
                          "path": [
                            {
                              "key": "istio_dry_run_deny_shadow_effective_policy_id"
                            }
                          ]
                        }
                      }
                    },
                    {
                      "tag": "istio.authorization.dry_run.deny_policy.result",
                      "metadata": {
                        "kind": {
                          "request": {}
                        },
                        "metadata_key": {
                          "key": "envoy.filters.http.rbac",
                          "path": [
                            {
                              "key": "istio_dry_run_deny_shadow_engine_result"
                            }
                          ]
                        }
                      }
                    },
                    {
                      "tag": "istio.canonical_revision",
                      "literal": {
                        "value": "v2"
                      }
                    },
                    {
                      "tag": "istio.canonical_service",
                      "literal": {
                        "value": "helloworld"
                      }
                    },
                    {
                      "tag": "istio.mesh_id",
                      "literal": {
                        "value": "devup-mesh"
                      }
                    },
                    {
                      "tag": "istio.namespace",
                      "literal": {
                        "value": "sample"
                      }
                    }
                  ]
                },
                "server_name": "istio-envoy",
                "use_remote_address": false,
                "forward_client_cert_details": "APPEND_FORWARD",
                "set_current_client_cert_details": {
                  "subject": true,
                  "dns": true,
                  "uri": true
                },
                "upgrade_configs": [
                  {
                    "upgrade_type": "websocket"
                  }
                ],
                "stream_idle_timeout": "0s",
                "normalize_path": true,
                "request_id_extension": {
                  "typed_config": {
                    "@type": "type.googleapis.com/envoy.extensions.request_id.uuid.v3.UuidRequestIdConfig",
                    "use_request_id_for_trace_sampling": true
                  }
                },
                "path_with_escaped_slashes_action": "KEEP_UNCHANGED"
              }
            }
          ],
          "transport_socket": {
            "name": "envoy.transport_sockets.tls",
            "typed_config": {
              "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext",
              "common_tls_context": {
                "tls_params": {
                  "tls_minimum_protocol_version": "TLSv1_2",
                  "tls_maximum_protocol_version": "TLSv1_3",
                  "cipher_suites": [
                    "ECDHE-ECDSA-AES256-GCM-SHA384",
                    "ECDHE-RSA-AES256-GCM-SHA384",
                    "ECDHE-ECDSA-AES128-GCM-SHA256",
                    "ECDHE-RSA-AES128-GCM-SHA256",
                    "AES256-GCM-SHA384",
                    "AES128-GCM-SHA256"
                  ]
                },
                "alpn_protocols": [
                  "h2",
                  "http/1.1"
                ],
                "tls_certificate_sds_secret_configs": [
                  {
                    "name": "default",
                    "sds_config": {
                      "api_config_source": {
                        "api_type": "GRPC",
                        "grpc_services": [
                          {
                            "envoy_grpc": {
                              "cluster_name": "sds-grpc"
                            }
                          }
                        ],
                        "set_node_on_first_message_only": true,
                        "transport_api_version": "V3"
                      },
                      "initial_fetch_timeout": "0s",
                      "resource_api_version": "V3"
                    }
                  }
                ],
                "combined_validation_context": {
                  "default_validation_context": {
                    "match_subject_alt_names": [
                      {
                        "prefix": "spiffe://google.com/"
                      }
                    ]
                  },
                  "validation_context_sds_secret_config": {
                    "name": "ROOTCA",
                    "sds_config": {
                      "api_config_source": {
                        "api_type": "GRPC",
                        "grpc_services": [
                          {
                            "envoy_grpc": {
                              "cluster_name": "sds-grpc"
                            }
                          }
                        ],
                        "set_node_on_first_message_only": true,
                        "transport_api_version": "V3"
                      },
                      "initial_fetch_timeout": "0s",
                      "resource_api_version": "V3"
                    }
                  }
                }
              },
              "require_client_certificate": true
            }
          },
          "name": "virtualInbound-catchall-http"
        },
        {
          "filter_chain_match": {
            "prefix_ranges": [
              {
                "address_prefix": "0.0.0.0",
                "prefix_len": 0
              }
            ],
            "transport_protocol": "tls"
          },
          "filters": [
            {
              "name": "istio.metadata_exchange",
              "typed_config": {
                "@type": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange",
                "protocol": "istio-peer-exchange"
              }
            },
            {
              "name": "istio.stats",
              "typed_config": {
                "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                "type_url": "type.googleapis.com/envoy.extensions.filters.network.wasm.v3.Wasm",
                "value": {
                  "config": {
                    "root_id": "stats_inbound",
                    "vm_config": {
                      "vm_id": "tcp_stats_inbound",
                      "runtime": "envoy.wasm.runtime.null",
                      "code": {
                        "local": {
                          "inline_string": "envoy.wasm.stats"
                        }
                      }
                    },
                    "configuration": {
                      "@type": "type.googleapis.com/google.protobuf.StringValue",
                      "value": "{\n  \"debug\": \"false\",\n  \"stat_prefix\": \"istio\",\n  \"metrics\": [\n    {\n      \"dimensions\": {\n        \"destination_cluster\": \"node.metadata['CLUSTER_ID']\",\n        \"source_cluster\": \"downstream_peer.cluster_id\"\n      }\n    }\n  ]\n}\n"
                    }
                  }
                }
              }
            },
            {
              "name": "envoy.filters.network.tcp_proxy",
              "typed_config": {
                "@type": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
                "stat_prefix": "InboundPassthroughClusterIpv4",
                "cluster": "InboundPassthroughClusterIpv4"
              }
            }
          ],
          "transport_socket": {
            "name": "envoy.transport_sockets.tls",
            "typed_config": {
              "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext",
              "common_tls_context": {
                "tls_params": {
                  "tls_minimum_protocol_version": "TLSv1_2",
                  "tls_maximum_protocol_version": "TLSv1_3",
                  "cipher_suites": [
                    "ECDHE-ECDSA-AES256-GCM-SHA384",
                    "ECDHE-RSA-AES256-GCM-SHA384",
                    "ECDHE-ECDSA-AES128-GCM-SHA256",
                    "ECDHE-RSA-AES128-GCM-SHA256",
                    "AES256-GCM-SHA384",
                    "AES128-GCM-SHA256"
                  ]
                },
                "alpn_protocols": [
                  "istio-peer-exchange",
                  "h2",
                  "http/1.1"
                ],
                "tls_certificate_sds_secret_configs": [
                  {
                    "name": "default",
                    "sds_config": {
                      "api_config_source": {
                        "api_type": "GRPC",
                        "grpc_services": [
                          {
                            "envoy_grpc": {
                              "cluster_name": "sds-grpc"
                            }
                          }
                        ],
                        "set_node_on_first_message_only": true,
                        "transport_api_version": "V3"
                      },
                      "initial_fetch_timeout": "0s",
                      "resource_api_version": "V3"
                    }
                  }
                ],
                "combined_validation_context": {
                  "default_validation_context": {
                    "match_subject_alt_names": [
                      {
                        "prefix": "spiffe://google.com/"
                      }
                    ]
                  },
                  "validation_context_sds_secret_config": {
                    "name": "ROOTCA",
                    "sds_config": {
                      "api_config_source": {
                        "api_type": "GRPC",
                        "grpc_services": [
                          {
                            "envoy_grpc": {
                              "cluster_name": "sds-grpc"
                            }
                          }
                        ],
                        "set_node_on_first_message_only": true,
                        "transport_api_version": "V3"
                      },
                      "initial_fetch_timeout": "0s",
                      "resource_api_version": "V3"
                    }
                  }
                }
              },
              "require_client_certificate": true
            }
          },
          "name": "virtualInbound"
        },
        {
          "filter_chain_match": {
            "destination_port": 5000,
            "transport_protocol": "tls"
          },
          "filters": [
            {
              "name": "istio.metadata_exchange",
              "typed_config": {
                "@type": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange",
                "protocol": "istio-peer-exchange"
              }
            },
            {
              "name": "envoy.filters.network.http_connection_manager",
              "typed_config": {
                "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
                "stat_prefix": "inbound_0.0.0.0_5000",
                "route_config": {
                  "name": "inbound|5000||",
                  "virtual_hosts": [
                    {
                      "name": "inbound|http|5000",
                      "domains": [
                        "*"
                      ],
                      "routes": [
                        {
                          "match": {
                            "prefix": "/"
                          },
                          "route": {
                            "cluster": "inbound|5000||",
                            "timeout": "0s",
                            "max_stream_duration": {
                              "max_stream_duration": "0s",
                              "grpc_timeout_header_max": "0s"
                            }
                          },
                          "decorator": {
                            "operation": "helloworld.sample.svc.cluster.local:5000/*"
                          },
                          "name": "default"
                        }
                      ]
                    }
                  ],
                  "validate_clusters": false
                },
                "http_filters": [
                  {
                    "name": "istio.metadata_exchange",
                    "typed_config": {
                      "@type": "type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm",
                      "config": {
                        "vm_config": {
                          "runtime": "envoy.wasm.runtime.null",
                          "code": {
                            "local": {
                              "inline_string": "envoy.wasm.metadata_exchange"
                            }
                          }
                        },
                        "configuration": {
                          "@type": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange"
                        }
                      }
                    }
                  },
                  {
                    "name": "envoy.filters.http.fault",
                    "typed_config": {
                      "@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
                    }
                  },
                  {
                    "name": "envoy.filters.http.cors",
                    "typed_config": {
                      "@type": "type.googleapis.com/envoy.extensions.filters.http.cors.v3.Cors"
                    }
                  },
                  {
                    "name": "istio.stats",
                    "typed_config": {
                      "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                      "type_url": "type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm",
                      "value": {
                        "config": {
                          "root_id": "stats_inbound",
                          "vm_config": {
                            "vm_id": "stats_inbound",
                            "runtime": "envoy.wasm.runtime.null",
                            "code": {
                              "local": {
                                "inline_string": "envoy.wasm.stats"
                              }
                            }
                          },
                          "configuration": {
                            "@type": "type.googleapis.com/google.protobuf.StringValue",
                            "value": "{\n  \"debug\": \"false\",\n  \"stat_prefix\": \"istio\",\n  \"disable_host_header_fallback\": true,\n  \"metrics\": [\n    {\n      \"dimensions\": {\n        \"destination_cluster\": \"node.metadata['CLUSTER_ID']\",\n        \"source_cluster\": \"downstream_peer.cluster_id\"\n      }\n    }\n  ]\n}\n"
                          }
                        }
                      }
                    }
                  },
                  {
                    "name": "envoy.filters.http.router",
                    "typed_config": {
                      "@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
                    }
                  }
                ],
                "tracing": {
                  "client_sampling": {
                    "value": 100
                  },
                  "random_sampling": {
                    "value": 1
                  },
                  "overall_sampling": {
                    "value": 100
                  },
                  "custom_tags": [
                    {
                      "tag": "istio.authorization.dry_run.allow_policy.name",
                      "metadata": {
                        "kind": {
                          "request": {}
                        },
                        "metadata_key": {
                          "key": "envoy.filters.http.rbac",
                          "path": [
                            {
                              "key": "istio_dry_run_allow_shadow_effective_policy_id"
                            }
                          ]
                        }
                      }
                    },
                    {
                      "tag": "istio.authorization.dry_run.allow_policy.result",
                      "metadata": {
                        "kind": {
                          "request": {}
                        },
                        "metadata_key": {
                          "key": "envoy.filters.http.rbac",
                          "path": [
                            {
                              "key": "istio_dry_run_allow_shadow_engine_result"
                            }
                          ]
                        }
                      }
                    },
                    {
                      "tag": "istio.authorization.dry_run.deny_policy.name",
                      "metadata": {
                        "kind": {
                          "request": {}
                        },
                        "metadata_key": {
                          "key": "envoy.filters.http.rbac",
                          "path": [
                            {
                              "key": "istio_dry_run_deny_shadow_effective_policy_id"
                            }
                          ]
                        }
                      }
                    },
                    {
                      "tag": "istio.authorization.dry_run.deny_policy.result",
                      "metadata": {
                        "kind": {
                          "request": {}
                        },
                        "metadata_key": {
                          "key": "envoy.filters.http.rbac",
                          "path": [
                            {
                              "key": "istio_dry_run_deny_shadow_engine_result"
                            }
                          ]
                        }
                      }
                    },
                    {
                      "tag": "istio.canonical_revision",
                      "literal": {
                        "value": "v2"
                      }
                    },
                    {
                      "tag": "istio.canonical_service",
                      "literal": {
                        "value": "helloworld"
                      }
                    },
                    {
                      "tag": "istio.mesh_id",
                      "literal": {
                        "value": "devup-mesh"
                      }
                    },
                    {
                      "tag": "istio.namespace",
                      "literal": {
                        "value": "sample"
                      }
                    }
                  ]
                },
                "server_name": "istio-envoy",
                "use_remote_address": false,
                "forward_client_cert_details": "APPEND_FORWARD",
                "set_current_client_cert_details": {
                  "subject": true,
                  "dns": true,
                  "uri": true
                },
                "upgrade_configs": [
                  {
                    "upgrade_type": "websocket"
                  }
                ],
                "stream_idle_timeout": "0s",
                "normalize_path": true,
                "request_id_extension": {
                  "typed_config": {
                    "@type": "type.googleapis.com/envoy.extensions.request_id.uuid.v3.UuidRequestIdConfig",
                    "use_request_id_for_trace_sampling": true
                  }
                },
                "path_with_escaped_slashes_action": "KEEP_UNCHANGED"
              }
            }
          ],
          "transport_socket": {
            "name": "envoy.transport_sockets.tls",
            "typed_config": {
              "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext",
              "common_tls_context": {
                "tls_params": {
                  "tls_minimum_protocol_version": "TLSv1_2",
                  "tls_maximum_protocol_version": "TLSv1_3",
                  "cipher_suites": [
                    "ECDHE-ECDSA-AES256-GCM-SHA384",
                    "ECDHE-RSA-AES256-GCM-SHA384",
                    "ECDHE-ECDSA-AES128-GCM-SHA256",
                    "ECDHE-RSA-AES128-GCM-SHA256",
                    "AES256-GCM-SHA384",
                    "AES128-GCM-SHA256"
                  ]
                },
                "alpn_protocols": [
                  "h2",
                  "http/1.1"
                ],
                "tls_certificate_sds_secret_configs": [
                  {
                    "name": "default",
                    "sds_config": {
                      "api_config_source": {
                        "api_type": "GRPC",
                        "grpc_services": [
                          {
                            "envoy_grpc": {
                              "cluster_name": "sds-grpc"
                            }
                          }
                        ],
                        "set_node_on_first_message_only": true,
                        "transport_api_version": "V3"
                      },
                      "initial_fetch_timeout": "0s",
                      "resource_api_version": "V3"
                    }
                  }
                ],
                "combined_validation_context": {
                  "default_validation_context": {
                    "match_subject_alt_names": [
                      {
                        "prefix": "spiffe://google.com/"
                      }
                    ]
                  },
                  "validation_context_sds_secret_config": {
                    "name": "ROOTCA",
                    "sds_config": {
                      "api_config_source": {
                        "api_type": "GRPC",
                        "grpc_services": [
                          {
                            "envoy_grpc": {
                              "cluster_name": "sds-grpc"
                            }
                          }
                        ],
                        "set_node_on_first_message_only": true,
                        "transport_api_version": "V3"
                      },
                      "initial_fetch_timeout": "0s",
                      "resource_api_version": "V3"
                    }
                  }
                }
              },
              "require_client_certificate": true
            }
          },
          "name": "0.0.0.0_5000"
        }
      ],
      "listener_filters": [
        {
          "name": "envoy.filters.listener.original_dst",
          "typed_config": {
            "@type": "type.googleapis.com/envoy.extensions.filters.listener.original_dst.v3.OriginalDst"
          }
        },
        {
          "name": "envoy.filters.listener.tls_inspector",
          "typed_config": {
            "@type": "type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector"
          },
          "filter_disabled": {
            "destination_port_range": {
              "start": 15006,
              "end": 15007
            }
          }
        }
      ],
      "listener_filters_timeout": "0s",
      "traffic_direction": "INBOUND",
      "continue_on_listener_filters_timeout": true
    },
    "last_updated": "2022-10-15T10:35:30.395Z"
  }
}

Cluster dump: this is the cluster dump which shows that the SDS validation is with ROOTCA

"outbound|5000||helloworld.sample.svc.cluster.local"
{
  "@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
  "name": "outbound|5000||helloworld.sample.svc.cluster.local",
  "type": "EDS",
  "eds_cluster_config": {
    "eds_config": {
      "ads": {},
      "initial_fetch_timeout": "0s",
      "resource_api_version": "V3"
    },
    "service_name": "outbound|5000||helloworld.sample.svc.cluster.local"
  },
  "connect_timeout": "10s",
  "lb_policy": "LEAST_REQUEST",
  "circuit_breakers": {
    "thresholds": [
      {
        "max_connections": 4294967295,
        "max_pending_requests": 4294967295,
        "max_requests": 4294967295,
        "max_retries": 4294967295,
        "track_remaining": true
      }
    ]
  },
  "metadata": {
    "filter_metadata": {
      "istio": {
        "default_original_port": 5000,
        "services": [
          {
            "name": "helloworld",
            "host": "helloworld.sample.svc.cluster.local",
            "namespace": "sample"
          }
        ]
      }
    }
  },
  "common_lb_config": {
    "locality_weighted_lb_config": {}
  },
  "filters": [
    {
      "name": "istio.metadata_exchange",
      "typed_config": {
        "@type": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange",
        "protocol": "istio-peer-exchange"
      }
    }
  ],
  "transport_socket_matches": [
    {
      "name": "tlsMode-istio",
      "match": {
        "tlsMode": "istio"
      },
      "transport_socket": {
        "name": "envoy.transport_sockets.tls",
        "typed_config": {
          "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext",
          "common_tls_context": {
            "tls_params": {
              "tls_minimum_protocol_version": "TLSv1_2",
              "tls_maximum_protocol_version": "TLSv1_3"
            },
            "alpn_protocols": [
              "istio-peer-exchange",
              "istio"
            ],
            "tls_certificate_sds_secret_configs": [
              {
                "name": "default",
                "sds_config": {
                  "api_config_source": {
                    "api_type": "GRPC",
                    "grpc_services": [
                      {
                        "envoy_grpc": {
                          "cluster_name": "sds-grpc"
                        }
                      }
                    ],
                    "set_node_on_first_message_only": true,
                    "transport_api_version": "V3"
                  },
                  "initial_fetch_timeout": "0s",
                  "resource_api_version": "V3"
                }
              }
            ],
            "combined_validation_context": {
              "default_validation_context": {
                "match_subject_alt_names": [
                  {
                    "exact": "spiffe://google.com/ns/sample/sa/default"
                  }
                ]
              },
              "validation_context_sds_secret_config": {
                "name": "ROOTCA",
                "sds_config": {
                  "api_config_source": {
                    "api_type": "GRPC",
                    "grpc_services": [
                      {
                        "envoy_grpc": {
                          "cluster_name": "sds-grpc"
                        }
                      }
                    ],
                    "set_node_on_first_message_only": true,
                    "transport_api_version": "V3"
                  },
                  "initial_fetch_timeout": "0s",
                  "resource_api_version": "V3"
                }
              }
            }
          },
          "sni": "outbound_.5000_._.helloworld.sample.svc.cluster.local"
        }
      }
    },
    {
      "name": "tlsMode-disabled",
      "match": {},
      "transport_socket": {
        "name": "envoy.transport_sockets.raw_buffer",
        "typed_config": {
          "@type": "type.googleapis.com/envoy.extensions.transport_sockets.raw_buffer.v3.RawBuffer"
        }
      }
    }
  ]
}

Secret dump: secret dump which has both the CA configured. upon inspecting you would see that they are minded by spire.


  {
   "@type": "type.googleapis.com/envoy.admin.v3.SecretsConfigDump",
   "dynamic_active_secrets": [
    {
     "name": "default",
     "version_info": "2",
     "last_updated": "2022-10-15T10:35:38.847Z",
     "secret": {
      "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret",
      "name": "default",
      "tls_certificate": {
       "certificate_chain": {
        "inline_bytes": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUREakNDQWZhZ0F3SUJBZ0lRR3EwOWliZTVYNDl4T2VZRklVby81akFOQmdrcWhraUc5dzBCQVFzRkFEQWUKTVFzd0NRWURWUVFHRXdKVlV6RVBNQTBHQTFVRUNoTUdVMUJKUmtaRk1CNFhEVEl5TVRBeE5URXdNelV5T0ZvWApEVEl5TVRBeE5URXhNelV6T0Zvd1JURUxNQWtHQTFVRUJoTUNWVk14RGpBTUJnTlZCQW9UQlZOUVNWSkZNU1l3CkpBWURWUVFERXgxb1pXeHNiM2R2Y214a0xYWXlMVFUyTldRM1pHUTNaaTFqY0RnMmFEQlpNQk1HQnlxR1NNNDkKQWdFR0NDcUdTTTQ5QXdFSEEwSUFCRDMreXU0T25oTDVrZW4vVGR5K0FCdWhQRTJvSnVWeGUzWXJtWkhEOXdPQgpyd1BsQjhBNHE0bjhHR1NPVm9qUDJrY3Y1TEJrZGFjeS9JeHNNOUhmbHZLamdlc3dnZWd3RGdZRFZSMFBBUUgvCkJBUURBZ09vTUIwR0ExVWRKUVFXTUJRR0NDc0dBUVVGQndNQkJnZ3JCZ0VGQlFjREFqQU1CZ05WSFJNQkFmOEUKQWpBQU1CMEdBMVVkRGdRV0JCUnlFNVFuZkFGUWVmMlU4enRkVEhDSlVOc0lvREFmQmdOVkhTTUVHREFXZ0JRVgpzK0pMN2dCTzVoZW1GcU11c1c1bnJKUDhEakJwQmdOVkhSRUVZakJnZ2gxb1pXeHNiM2R2Y214a0xYWXlMVFUyCk5XUTNaR1EzWmkxamNEZzJhSUlWYUdWc2JHOTNiM0pzWkM1ellXMXdiR1V1YzNaamhpaHpjR2xtWm1VNkx5OW4KYjI5bmJHVXVZMjl0TDI1ekwzTmhiWEJzWlM5ellTOWtaV1poZFd4ME1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQgpBUUJrRDFnaHVvOFZhTUVBOXJMQ3A0VHM0cnV4QmV5K0FnWnBQMTRvM1I4RHhJSStHWFJDOStHS3RXOGNrQUxWClBrYXRMT1E3NEJCSXk0UHJiaGdQRkVVQ0wvaE9Kb24wazVmS29LbnA0OEFZZTQ2Tk5nckxzZEp3YVhzZzhmdVcKRGE2bExndlZQMkN6ckJxN3FHeHIxbjdJUytzQk1VL3dCaU9BbHlFb3lCTFJIbFg3N3I5dkpIOXNaVW9IckVaMApEMnBIVCtwb3RhNlVMcWQ5OHRIbzN4bnY2R0JxdnZuNXRCUTR3bWVSSnFKTlRTbWZTQ2ZrdmljTjE2VUN3ckNQCm9BeWhWYWJRRUVzcTd4bnFTclZqK2dmcXFQNzlSMldreERra0FOYS9CTUNna0RtM0p5ajJSYmFTYTZCWllhLzgKNnF6YW9wNlFFejJ1RlhoYkxPWW1wYlRoCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
       },
       "private_key": {
        "inline_bytes": "W3JlZGFjdGVkXQ=="
       }
      }
     }
    },
    {
     "name": "ROOTCA",
     "version_info": "1",
     "last_updated": "2022-10-15T10:35:38.910Z",
     "secret": {
      "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret",
      "name": "ROOTCA",
      "validation_context": {
       "trusted_ca": {},
       "custom_validator_config": {
        "name": "envoy.tls.cert_validator.spiffe",
        "typed_config": {
         "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.SPIFFECertValidatorConfig",
         "trust_domains": [
          {
           "name": "aws.com",
           "trust_bundle": {
            "inline_bytes": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURKVENDQWcyZ0F3SUJBZ0lRQ1NqMGxNQ2IybVhnVUZOMDRiMFBWREFOQmdrcWhraUc5dzBCQVFzRkFEQWUKTVFzd0NRWURWUVFHRXdKVlV6RVBNQTBHQTFVRUNoTUdVMUJKUmtaRk1CNFhEVEl5TVRBeE5URXdNamt5TTFvWApEVEl5TVRBeE5qRXdNamt6TTFvd0hqRUxNQWtHQTFVRUJoTUNWVk14RHpBTkJnTlZCQW9UQmxOUVNVWkdSVENDCkFTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBSzZpUW5ucEsyQ3RXS3VrbTIvT0tJaDIKL1R2NGlTRGlWKzZROFZkWUJRRmQxcmVsUkEzaU0rRjRyQUgvZXFEOHFhSzJGajY5TkJTWGlWNytXdktvZHM4OQpHZW8xUGczV3o0cjl5aE56c0tXN0xuOGprbUFSMmY5RlgzUXAzR2h2QWZDR1V0aU02NkZHSXFsUUZNTmVWNVlCCm9DbzkwVG00TFpwYk1xTXFiZkkxcUlwYm5GUmFZclltU2t1NFB3ZzJVeWlxQUxvbUNEOThjcmQ3VUVUU1Rva1oKUFV5OCsrNXZwclBGSlhQQkg3VzlZdmRrelh5bUt4YXl5N1NPOUtma1JIc2NuWnNxLzhtb1owYndNaFhid2RjMwo2TjVhTFd2WXdDSytxL1ZoY0loRkh2MnJBUWYvcTNzVzh1OEkxdUZ2bzg5Mng3NVArNFpzdFBpMDZjT0Y0UXNDCkF3RUFBYU5mTUYwd0RnWURWUjBQQVFIL0JBUURBZ0VHTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME8KQkJZRUZEQXhST0R2bWNwQTA4UjVLV2RNRFQrczFWNFNNQnNHQTFVZEVRUVVNQktHRUhOd2FXWm1aVG92TDJGMwpjeTVqYjIwd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFFYWVSRytoNG1hUEo5Vm9UanpYSE1nM1dzMVBKMVNSCkF4bzZ2YmszNUhxc0pRbUVnQ1hiZzlsODkvTXBJYUMySzNmQXJBRGdweEhCSU1oRWplZlhJZGFQRnloZDRjYmEKODg3QTF1V0RaOXhFUjNJU0ttek50Y09aRlA2S2dPWTlrMXpLNGp2cTJtdHRYdko2cTZlL1RXRDkrTjhBVS91Uwp1eGw3V3pkVGwrRUlNQ2FyNkRCMlVidUtLY1hQWnBUenVYMWtxSXV4N002Q3kzT1lOTlJ4MzhSKzZ2WWZrdGtuCjF4MlNINnc5blJMeGhlQ3BGeVBhWHc0RFFqOE55RGUvcWpaci9ReFhmRG4zQysyUms0alRQUFdZWTlscExxMGsKNVRxQXlndVpUZ2VLbDNJdXpMcXlBMlJ0Y3p0S3ZoOHF0UmFES0lleHZCWEFTQ054YWRVVUR0Zz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo="
           }
          },
          {
           "name": "google.com",
           "trust_bundle": {
            "inline_bytes": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURLRENDQWhDZ0F3SUJBZ0lRUCtVVFh5SlhRVUFVMlhGU0dhSDBkekFOQmdrcWhraUc5dzBCQVFzRkFEQWUKTVFzd0NRWURWUVFHRXdKVlV6RVBNQTBHQTFVRUNoTUdVMUJKUmtaRk1CNFhEVEl5TVRBeE5URXdNamsxT1ZvWApEVEl5TVRBeE5qRXdNekF3T1Zvd0hqRUxNQWtHQTFVRUJoTUNWVk14RHpBTkJnTlZCQW9UQmxOUVNVWkdSVENDCkFTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTUc4UTBGYlhBUHBweW92UGNvVnlRRDMKdnh2OERpczlSQlc1WE9vSzRPd0FPMTlxYXFMbjAyWHRGUXBEblJ1bzhJVzhPLzJxbjU2Sytzc1BBS2JkTUtMbwpXYXJrZGdERVlTZFJzdTFlYmhlb2cxSmgzTVJ4YTJBUFJtVkpzeVJ6S1FxM284ZVBwM1ZPcEk5OGFSaE9LWXZCCmIwdERkTGtWU0NJaFJZZ0VaQXVoVHI1WlByZGdEYmJTSThtSmFnaTg2cldtR2xCYlR5VFh4ZTVkQitxQmtxc2kKUURpbHRCNU1xUTRBWEg4S2Mxbk5FZFlTL2RlSi9iVDZvM2cwZDhyUFI3bXJ4alVLN1Fhb1Q5eS8wQ1RjbU5ESQpFWVhMTGtGMHBtU3YwaWx6QjlUaU5NNDlyOER0QkxKdk1rWjg2ckpPWWEwQk1IMWJodDZTa1hob2RYRUpYRzhDCkF3RUFBYU5pTUdBd0RnWURWUjBQQVFIL0JBUURBZ0VHTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME8KQkJZRUZCV3o0a3Z1QUU3bUY2WVdveTZ4Ym1lc2svd09NQjRHQTFVZEVRUVhNQldHRTNOd2FXWm1aVG92TDJkdgpiMmRzWlM1amIyMHdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBS2UxT21LQ1pGejE3RklrYkROTUlxVExtNTRRCmt2VlRSZXdSR0FYWUQ0aXJlcDltR01PMjBXeTUxQ1cxSmFyN1JrenV0WENNQjZGYy9WYXdMb3k4bE9CL0xNc2wKRjlicGRJKzgzUmZrcWVsY1hvRWt3dEhvTXdBZHE0RG9yZVpKUVVRbGlZaDQvTDlHUkVIa1JyTDV1WWhmSURsaQpLK3pkenVRYkRzbzlZc1M0SVByYkc3QndsWmpaYVRBT1lmT0V6N0oxeTk2YWp4ZzhZeFdKSG1Kb3ViTTBtclRrCnJtN0lUbnBSZ1h1U3NySy9LSFpqMHdiNTgza1prQVVxL0lldFdiQkt0a3pITVNSK2dDT0FkbUR6b0wralVlWXkKUDFkelhqdmdMQ0RDVWJZSzArUG8zcWtjaTZGYTlKNGxQR2drc1djSFRrTkpuajlqWGU2L1FNbk5SdUE9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
           }
          }
         ]
        }
       }
      }
     }
    }
   ]
  }
 ]
}

attaching the entire envoy config of helloworld app

helloworld-v2-config-dump.json.zip

evan2645 commented 2 years ago

@maxlambrecht do you know if there's anything special that needs to be done to do federated auth with an Istio workload?

There's a lot of config here and it's hard for me to parse it all.

The curl fails with this error:

Curl isn't SPIFFE aware, so unless you will have a hard time seeing success with it unless you feed it a bunch of flags and the bundle etc. For example, I think it needs, at a minimum, configuration for a client key + cert.

I can't quite tell which end of the connection validation is failing on. Do you know? It might be useful to use a tool (e.g. openssl s_client to suck the certificate off of the server endpoint and compare it against what you're expecting.

maxlambrecht commented 2 years ago

@maxlambrecht do you know if there's anything special that needs to be done to do federated auth with an Istio workload?

In both clusters the Envoy proxies need to have the the SPIFFECertValidatorConfig with the trust bundle of the other trust domain. To achieve that, on the “aws.com” cluster, SPIRE server should have a registration entry for the sleep worklaod with a FederatesWith “google.com”. Analogously, on the “google.com” cluster, SPIRE server should have a registration entry for the helloworld workload with a FederatesWith “aws.com”.

@sudeeptoroy When I run the example from your repo, the verify.sh fails because the helloworld and sleep workloads don’t get their identities from SPIRE, because attestation fails. I couldn't figure out what's wrong.

Curl isn't SPIFFE aware, so unless you will have a hard time seeing success with it unless you feed it a bunch of flags and the bundle etc. For example, I think it needs, at a minimum, configuration for a client key + cert.

The curl is done through in the sleep pod, so the mTLS connection is handled by the Envoy sidecar.

sudeeptoroy commented 2 years ago

Hi @evan2645 and @maxlambrecht,

Thank you for your reply. As @maxlambrecht mentioned; curl is from the sleep pod to helloworld via the sidecar. The sidecar is going to upgrade the connection and use the cert as minted by the spire.

@maxlambrecht, when i inspect the identity certs for helloworld app i see that its minted by spire. Also i have not created any manual attestation rules so it should be auto attested. Did you see any logs which says attestation failed. 

echo "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUREakNDQWZhZ0F3SUJBZ0lRR3EwOWliZTVYNDl4T2VZRklVby81akFOQmdrcWhraUc5dzBCQVFzRkFEQWUKTVFzd0NRWURWUVFHRXdKVlV6RVBNQTBHQTFVRUNoTUdVMUJKUmtaRk1CNFhEVEl5TVRBeE5URXdNelV5T0ZvWApEVEl5TVRBeE5URXhNelV6T0Zvd1JURUxNQWtHQTFVRUJoTUNWVk14RGpBTUJnTlZCQW9UQlZOUVNWSkZNU1l3CkpBWURWUVFERXgxb1pXeHNiM2R2Y214a0xYWXlMVFUyTldRM1pHUTNaaTFqY0RnMmFEQlpNQk1HQnlxR1NNNDkKQWdFR0NDcUdTTTQ5QXdFSEEwSUFCRDMreXU0T25oTDVrZW4vVGR5K0FCdWhQRTJvSnVWeGUzWXJtWkhEOXdPQgpyd1BsQjhBNHE0bjhHR1NPVm9qUDJrY3Y1TEJrZGFjeS9JeHNNOUhmbHZLamdlc3dnZWd3RGdZRFZSMFBBUUgvCkJBUURBZ09vTUIwR0ExVWRKUVFXTUJRR0NDc0dBUVVGQndNQkJnZ3JCZ0VGQlFjREFqQU1CZ05WSFJNQkFmOEUKQWpBQU1CMEdBMVVkRGdRV0JCUnlFNVFuZkFGUWVmMlU4enRkVEhDSlVOc0lvREFmQmdOVkhTTUVHREFXZ0JRVgpzK0pMN2dCTzVoZW1GcU11c1c1bnJKUDhEakJwQmdOVkhSRUVZakJnZ2gxb1pXeHNiM2R2Y214a0xYWXlMVFUyCk5XUTNaR1EzWmkxamNEZzJhSUlWYUdWc2JHOTNiM0pzWkM1ellXMXdiR1V1YzNaamhpaHpjR2xtWm1VNkx5OW4KYjI5bmJHVXVZMjl0TDI1ekwzTmhiWEJzWlM5ellTOWtaV1poZFd4ME1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQgpBUUJrRDFnaHVvOFZhTUVBOXJMQ3A0VHM0cnV4QmV5K0FnWnBQMTRvM1I4RHhJSStHWFJDOStHS3RXOGNrQUxWClBrYXRMT1E3NEJCSXk0UHJiaGdQRkVVQ0wvaE9Kb24wazVmS29LbnA0OEFZZTQ2Tk5nckxzZEp3YVhzZzhmdVcKRGE2bExndlZQMkN6ckJxN3FHeHIxbjdJUytzQk1VL3dCaU9BbHlFb3lCTFJIbFg3N3I5dkpIOXNaVW9IckVaMApEMnBIVCtwb3RhNlVMcWQ5OHRIbzN4bnY2R0JxdnZuNXRCUTR3bWVSSnFKTlRTbWZTQ2ZrdmljTjE2VUN3ckNQCm9BeWhWYWJRRUVzcTd4bnFTclZqK2dmcXFQNzlSMldreERra0FOYS9CTUNna0RtM0p5ajJSYmFTYTZCWllhLzgKNnF6YW9wNlFFejJ1RlhoYkxPWW1wYlRoCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K" | base64 -d | openssl x509 -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            1a:ad:3d:89:b7:b9:5f:8f:71:39:e6:05:21:4a:3f:e6
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=SPIFFE
        Validity
            Not Before: Oct 15 10:35:28 2022 GMT
            Not After : Oct 15 11:35:38 2022 GMT
        Subject: C=US, O=SPIRE, CN=helloworld-v2-565d7dd7f-cp86h
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:3d:fe:ca:ee:0e:9e:12:f9:91:e9:ff:4d:dc:be:
                    00:1b:a1:3c:4d:a8:26:e5:71:7b:76:2b:99:91:c3:
                    f7:03:81:af:03:e5:07:c0:38:ab:89:fc:18:64:8e:
                    56:88:cf:da:47:2f:e4:b0:64:75:a7:32:fc:8c:6c:
                    33:d1:df:96:f2
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Key Agreement
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                72:13:94:27:7C:01:50:79:FD:94:F3:3B:5D:4C:70:89:50:DB:08:A0
            X509v3 Authority Key Identifier:
                keyid:15:B3:E2:4B:EE:00:4E:E6:17:A6:16:A3:2E:B1:6E:67:AC:93:FC:0E

            X509v3 Subject Alternative Name:
                DNS:helloworld-v2-565d7dd7f-cp86h, DNS:helloworld.sample.svc, URI:spiffe://google.com/ns/sample/sa/default

I have mentioned the annotations with the deployments for federating them across. 

https://github.com/sudeeptoroy/spirefed/blob/main/helloworld/sleep-aws.yaml#L52
https://github.com/sudeeptoroy/spirefed/blob/main/helloworld/helloworld.yaml#L67

if you are fine, i can recreate the issue and we can debug it together. Let me know.

sudeeptoroy commented 2 years ago

coping @nrjpoddar for your views wrt istio. @alexandrealvino for help and visibility.

azdagron commented 2 years ago

~Another thing I noticed is that the requested SDS resource for the validation context is ROOTCA. By default, this will only return the bundle for the trust domain of the workload. ALL is the resource name that will return all federated bundles.~

Nevermind, I see in your config that you are overriding those names such that ROOTCA is the resource name for all bundles.

azdagron commented 2 years ago

In order for the k8s-workload-registrar to register the workloads with federation, your pods need a spiffe.io/federatesWith annotation set to the trust domain name you want the workload to federation with. See https://github.com/spiffe/spire/tree/main/support/k8s/k8s-workload-registrar/mode-crd#federated-entry-registration.

maxlambrecht commented 2 years ago

In order for the k8s-workload-registrar to register the workloads with federation, your pods need a spiffe.io/federatesWith annotation set to the trust domain name you want the workload to federation with. See https://github.com/spiffe/spire/tree/main/support/k8s/k8s-workload-registrar/mode-crd#federated-entry-registration.

The example uses the spiffe.io/federatesWith in the pod configs.

I see something that is not necessary though, the helloworld workload has 2 versions, one is configured with spiffe.io/federatesWith: "aws.com" which is correct, the other has spiffe.io/federatesWith: "google.com", as the workload already runs in the google.com trust domain, it's not necessary and I don't know if it can cause an issue.

sudeeptoroy commented 2 years ago

Hi @maxlambrecht, yes there are two versions of helloworld v1 and v2. V1 runs in aws.com cluster and v2 runs in google.com. They are there so that from the sleep app i can get response from both the helloworld versions.

azdagron commented 2 years ago

What do you SPIRE registration entries look like? Can you confirm that they have the appropriate FederatesWith fields set?

sudeeptoroy commented 2 years ago

i recreated the setup again. I see the FederatesWith entry:

Aws.com ( marked with >>>>>>>>>> )

kubectl --context kind-aws-cluster get pod -n sample
NAME                  READY   STATUS    RESTARTS   AGE
sleep-95d8696-wncnx   2/2     Running   0          11m

kubectl --context kind-aws-cluster exec --stdin spire-server-0 -c spire-server -n spire  -- /opt/spire/bin/spire-server entry show  -socketPath /run/spire/sockets/server.sock
Found 11 entries
Entry ID         : adfa6e46-5aa7-40e0-a28a-016fb0a63763
SPIFFE ID        : spiffe://aws.com/k8s-workload-registrar/aws-cluster/node/aws-cluster-control-plane
Parent ID        : spiffe://aws.com/spire/server
Revision         : 0
TTL              : default
Selector         : k8s_psat:agent_node_uid:a1c8d15c-ef99-475a-a65b-fdfb4d6af44b
Selector         : k8s_psat:cluster:aws-cluster

Entry ID         : 20525459-a68c-4d4d-be68-b9f18dd1d54e
SPIFFE ID        : spiffe://aws.com/k8s-workload-registrar/aws-cluster/node/aws-cluster-worker
Parent ID        : spiffe://aws.com/spire/server
Revision         : 0
TTL              : default
Selector         : k8s_psat:agent_node_uid:01f4fd19-7cf1-4f73-9e5c-caacc89da913
Selector         : k8s_psat:cluster:aws-cluster

Entry ID         : d2b7db5b-503c-4af5-9fad-48683966ee67
SPIFFE ID        : spiffe://aws.com/ns/istio-system/sa/istio-eastwestgateway-service-account
Parent ID        : spiffe://aws.com/k8s-workload-registrar/aws-cluster/node/aws-cluster-worker
Revision         : 1
TTL              : default
Selector         : k8s:node-name:aws-cluster-worker
Selector         : k8s:ns:istio-system
Selector         : k8s:pod-uid:d68fb46d-340e-4f28-8ba1-02f991d4f0ec
FederatesWith    : google.com
DNS name         : istio-eastwestgateway-56fcddbf4-bfv9g
DNS name         : istio-eastwestgateway.istio-system.svc

Entry ID         : 083d1e23-89e6-4911-99a1-c03810f5081c
SPIFFE ID        : spiffe://aws.com/ns/istio-system/sa/istio-ingressgateway-service-account
Parent ID        : spiffe://aws.com/k8s-workload-registrar/aws-cluster/node/aws-cluster-worker
Revision         : 1
TTL              : default
Selector         : k8s:node-name:aws-cluster-worker
Selector         : k8s:ns:istio-system
Selector         : k8s:pod-uid:c7c7cc0d-b8a8-49c6-9d8b-037c82728f2f
FederatesWith    : google.com
DNS name         : istio-ingressgateway-548b4dc876-hzpd7
DNS name         : istio-ingressgateway.istio-system.svc

Entry ID         : 270e2292-d4d4-4eff-bd30-91a8ba01ef87
SPIFFE ID        : spiffe://aws.com/ns/istio-system/sa/istiod
Parent ID        : spiffe://aws.com/k8s-workload-registrar/aws-cluster/node/aws-cluster-worker
Revision         : 1
TTL              : default
Selector         : k8s:node-name:aws-cluster-worker
Selector         : k8s:ns:istio-system
Selector         : k8s:pod-uid:6341a288-9a19-4b1f-be0e-aa8d8bc87f92
DNS name         : istiod-86bbc56b97-4wj8p
DNS name         : istiod.istio-system.svc

Entry ID         : 430b47b7-df66-4baf-92d0-0227a20c5b20
SPIFFE ID        : spiffe://aws.com/ns/local-path-storage/sa/local-path-provisioner-service-account
Parent ID        : spiffe://aws.com/k8s-workload-registrar/aws-cluster/node/aws-cluster-control-plane
Revision         : 0
TTL              : default
Selector         : k8s:node-name:aws-cluster-control-plane
Selector         : k8s:ns:local-path-storage
Selector         : k8s:pod-uid:c0cb41c8-aec7-4de7-9708-d32cf1713c67
DNS name         : local-path-provisioner-9cd9bd544-qg4vn

Entry ID         : 23b3f8d3-f421-4411-b21b-e3cb12ef7f23
SPIFFE ID        : spiffe://aws.com/ns/metallb-system/sa/controller
Parent ID        : spiffe://aws.com/k8s-workload-registrar/aws-cluster/node/aws-cluster-worker
Revision         : 0
TTL              : default
Selector         : k8s:node-name:aws-cluster-worker
Selector         : k8s:ns:metallb-system
Selector         : k8s:pod-uid:990f64e3-a7c9-4da6-8655-092dd88ef932
DNS name         : controller-d9d8c78b6-2r78q

Entry ID         : 89f99fa3-7752-4d67-b4ec-7719624e1ead
SPIFFE ID        : spiffe://aws.com/ns/metallb-system/sa/speaker
Parent ID        : spiffe://aws.com/k8s-workload-registrar/aws-cluster/node/aws-cluster-worker
Revision         : 0
TTL              : default
Selector         : k8s:node-name:aws-cluster-worker
Selector         : k8s:ns:metallb-system
Selector         : k8s:pod-uid:6b39a2ed-0bd2-453e-8924-89da691bcbef
DNS name         : speaker-wlm2q

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Entry ID         : 4a73bd43-60f3-4576-95f2-9e59bc32b7e8
SPIFFE ID        : spiffe://aws.com/ns/sample/sa/sleep
Parent ID        : spiffe://aws.com/k8s-workload-registrar/aws-cluster/node/aws-cluster-worker
Revision         : 1
TTL              : default
Selector         : k8s:node-name:aws-cluster-worker
Selector         : k8s:ns:sample
Selector         : k8s:pod-uid:9862aaa8-668b-4cdf-91b2-0432a53314b3
FederatesWith    : google.com
DNS name         : sleep-95d8696-wncnx
DNS name         : sleep.sample.svc
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Entry ID         : 588630d7-620c-46bc-9ff1-b7852afa07b5
SPIFFE ID        : spiffe://aws.com/ns/spire/sa/spire-agent
Parent ID        : spiffe://aws.com/k8s-workload-registrar/aws-cluster/node/aws-cluster-worker
Revision         : 0
TTL              : default
Selector         : k8s:node-name:aws-cluster-worker
Selector         : k8s:ns:spire
Selector         : k8s:pod-uid:25eface8-5c33-4458-b39d-9e803265c2c7
DNS name         : spire-agent-p7vrg

Entry ID         : 08afe2d4-24fc-491a-b379-37f56feda140
SPIFFE ID        : spiffe://aws.com/ns/spire/sa/spire-server
Parent ID        : spiffe://aws.com/k8s-workload-registrar/aws-cluster/node/aws-cluster-worker
Revision         : 1
TTL              : default
Selector         : k8s:node-name:aws-cluster-worker
Selector         : k8s:ns:spire
Selector         : k8s:pod-uid:c51f0445-7d7e-4a7a-add8-6630295448b8
DNS name         : spire-server-0
DNS name         : spire-server.spire.svc

Google.com

k -n sample get pod
NAME                            READY   STATUS    RESTARTS   AGE
helloworld-v2-565d7dd7f-7wkg9   2/2     Running   0          11m
sleep-7b8db96f6b-jkhxn          2/2     Running   0          10m

kubectl --context kind-google-cluster exec --stdin spire-server-0 -c spire-server -n spire  -- /opt/spire/bin/spire-server entry show  -socketPath /run/spire/sockets/server.sock
Found 12 entries
Entry ID         : a854f009-9d45-4d4f-8cd9-966b2558edb5
SPIFFE ID        : spiffe://google.com/k8s-workload-registrar/google-cluster/node/google-cluster-control-plane
Parent ID        : spiffe://google.com/spire/server
Revision         : 0
TTL              : default
Selector         : k8s_psat:agent_node_uid:f7dbc74d-f5f1-42bd-9873-1f9547f2d371
Selector         : k8s_psat:cluster:google-cluster

Entry ID         : 5be3862c-9077-4947-a12e-38dd32cc370a
SPIFFE ID        : spiffe://google.com/k8s-workload-registrar/google-cluster/node/google-cluster-worker
Parent ID        : spiffe://google.com/spire/server
Revision         : 0
TTL              : default
Selector         : k8s_psat:agent_node_uid:2a47376f-dc74-40b9-b728-7cd04011a80c
Selector         : k8s_psat:cluster:google-cluster

Entry ID         : db9d739b-0af3-48bb-9b7a-a0d24e73e39f
SPIFFE ID        : spiffe://google.com/ns/istio-system/sa/istio-eastwestgateway-service-account
Parent ID        : spiffe://google.com/k8s-workload-registrar/google-cluster/node/google-cluster-worker
Revision         : 1
TTL              : default
Selector         : k8s:node-name:google-cluster-worker
Selector         : k8s:ns:istio-system
Selector         : k8s:pod-uid:1c76be39-419e-4b1b-87d7-58b09728ad95
FederatesWith    : aws.com
DNS name         : istio-eastwestgateway-78d5c8f996-jh7hc
DNS name         : istio-eastwestgateway.istio-system.svc

Entry ID         : 2977eec2-ed39-49b7-8e1b-dbbd73d812c6
SPIFFE ID        : spiffe://google.com/ns/istio-system/sa/istio-ingressgateway-service-account
Parent ID        : spiffe://google.com/k8s-workload-registrar/google-cluster/node/google-cluster-worker
Revision         : 1
TTL              : default
Selector         : k8s:node-name:google-cluster-worker
Selector         : k8s:ns:istio-system
Selector         : k8s:pod-uid:4769b0f5-72f6-4d7a-bbaf-385ff85b7078
FederatesWith    : aws.com
DNS name         : istio-ingressgateway-b4d65fcb-xc5pv
DNS name         : istio-ingressgateway.istio-system.svc

Entry ID         : cdb34340-c457-4566-a1e4-89aafab21504
SPIFFE ID        : spiffe://google.com/ns/istio-system/sa/istiod
Parent ID        : spiffe://google.com/k8s-workload-registrar/google-cluster/node/google-cluster-worker
Revision         : 1
TTL              : default
Selector         : k8s:node-name:google-cluster-worker
Selector         : k8s:ns:istio-system
Selector         : k8s:pod-uid:b49ee16e-52a8-4d7b-a1c0-f13852e26d0c
DNS name         : istiod-789d4868ff-t4bm7
DNS name         : istiod.istio-system.svc

Entry ID         : 37c97107-8814-4dab-84e1-93338357d61e
SPIFFE ID        : spiffe://google.com/ns/local-path-storage/sa/local-path-provisioner-service-account
Parent ID        : spiffe://google.com/k8s-workload-registrar/google-cluster/node/google-cluster-control-plane
Revision         : 0
TTL              : default
Selector         : k8s:node-name:google-cluster-control-plane
Selector         : k8s:ns:local-path-storage
Selector         : k8s:pod-uid:052b2c57-6002-42aa-9af3-1f362072d005
DNS name         : local-path-provisioner-9cd9bd544-jjkdk

Entry ID         : 26ca7c7f-faee-4e44-bcd7-9c846ef05559
SPIFFE ID        : spiffe://google.com/ns/metallb-system/sa/controller
Parent ID        : spiffe://google.com/k8s-workload-registrar/google-cluster/node/google-cluster-worker
Revision         : 0
TTL              : default
Selector         : k8s:node-name:google-cluster-worker
Selector         : k8s:ns:metallb-system
Selector         : k8s:pod-uid:653071a5-c4d3-4fde-8bc8-ef6914a15af4
DNS name         : controller-d9d8c78b6-vqfxd

Entry ID         : 2ac92a38-ceca-412d-81de-0dd3b5262b14
SPIFFE ID        : spiffe://google.com/ns/metallb-system/sa/speaker
Parent ID        : spiffe://google.com/k8s-workload-registrar/google-cluster/node/google-cluster-worker
Revision         : 0
TTL              : default
Selector         : k8s:node-name:google-cluster-worker
Selector         : k8s:ns:metallb-system
Selector         : k8s:pod-uid:75b1b31d-1faf-4f8d-b0d0-efa37c26068a
DNS name         : speaker-k8q9n

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Entry ID         : 21634ffe-7fe6-49ea-8d01-bef1d9792961
SPIFFE ID        : spiffe://google.com/ns/sample/sa/default
Parent ID        : spiffe://google.com/k8s-workload-registrar/google-cluster/node/google-cluster-worker
Revision         : 1
TTL              : default
Selector         : k8s:node-name:google-cluster-worker
Selector         : k8s:ns:sample
Selector         : k8s:pod-uid:8a4fa521-4ef2-4ccb-9961-ef6ceed66626
FederatesWith    : aws.com
DNS name         : helloworld-v2-565d7dd7f-7wkg9
DNS name         : helloworld.sample.svc
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Entry ID         : 949d3932-5383-459c-9464-e41bad60ab37
SPIFFE ID        : spiffe://google.com/ns/sample/sa/sleep
Parent ID        : spiffe://google.com/k8s-workload-registrar/google-cluster/node/google-cluster-worker
Revision         : 1
TTL              : default
Selector         : k8s:node-name:google-cluster-worker
Selector         : k8s:ns:sample
Selector         : k8s:pod-uid:3e809a90-85f1-44b1-b585-724009348921
FederatesWith    : aws.com
DNS name         : sleep-7b8db96f6b-jkhxn
DNS name         : sleep.sample.svc

Entry ID         : b5b8533f-853b-43f8-8564-37fd9bfdd920
SPIFFE ID        : spiffe://google.com/ns/spire/sa/spire-agent
Parent ID        : spiffe://google.com/k8s-workload-registrar/google-cluster/node/google-cluster-worker
Revision         : 0
TTL              : default
Selector         : k8s:node-name:google-cluster-worker
Selector         : k8s:ns:spire
Selector         : k8s:pod-uid:401cdb10-0045-40a1-aacb-85b68b0c7eec
DNS name         : spire-agent-lkvfs

Entry ID         : a871f759-cdb2-46df-9383-4510ba91c532
SPIFFE ID        : spiffe://google.com/ns/spire/sa/spire-server
Parent ID        : spiffe://google.com/k8s-workload-registrar/google-cluster/node/google-cluster-worker
Revision         : 0
TTL              : default
Selector         : k8s:node-name:google-cluster-worker
Selector         : k8s:ns:spire
Selector         : k8s:pod-uid:e994279b-3373-4e73-b670-9153a08dc537
DNS name         : spire-server-0
DNS name         : spire-server.spire.svc
azdagron commented 2 years ago

That looks correct. One option here is to run a simple tool inside the same pod as the workload to see what the SPIRE Agent is returning from SDS. I have such a tool but you'd need to build it and get it into a container: https://gist.github.com/azdagron/c93712195342e5a472f1d08a7eab13b7

You'd run it like so to see what is being returned for the validation context:

sds-test --resources ROOTCA
sudeeptoroy commented 2 years ago

Thank you @azdagron , I was able to run the executable in the istio-proxy container.

Here is output from the helloworld-v2 pod:

istio-proxy@helloworld-v2-565d7dd7f-ctvzq:/run/secrets/workload-spiffe-credentials$ ./sds --target unix:///var/run/secrets/workload-spiffe-uds/socket --resources ROOTCA,spiffe://google.com/ns/sample/sa/default
{
    "resources":  [
        {
            "@type":  "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret",
            "name":  "ROOTCA",
            "validationContext":  {
                "customValidatorConfig":  {
                    "name":  "envoy.tls.cert_validator.spiffe",
                    "typedConfig":  {
                        "@type":  "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.SPIFFECertValidatorConfig",
                        "trustDomains":  [
                            {
                                "name":  "aws.com",
                                "trustBundle":  {
                                    "inlineBytes":  "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURKVENDQWcyZ0F3SUJBZ0lRVkpaVEJKbzdVa2YyRW5PS2tRVjVHakFOQmdrcWhraUc5dzBCQVFzRkFEQWUKTVFzd0NRWURWUVFHRXdKVlV6RVBNQTBHQTFVRUNoTUdVMUJKUmtaRk1CNFhEVEl5TVRBeU1EQTJNRGcxTkZvWApEVEl5TVRBeU1UQTJNRGt3TkZvd0hqRUxNQWtHQTFVRUJoTUNWVk14RHpBTkJnTlZCQW9UQmxOUVNVWkdSVENDCkFTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTnlkNisvbFlPVTRrM0xZQnBDVkFOOXAKTU5jQm1tb3BTa0NwSVBDcXVXeUYyd1B2UVlweE14R21OWXVGeFRpMHM2Ukc1a05VU0daNVdrMmsyZ1ZEMVVzawpjam9MSm8vSS9yMFpEMmhZczdoU2lrVHBMdmFFcmswVC9Pa2VYeS9uRzhnMHFQbEYzOWppcDAwT2I3c0hkMTdqCmg2YzdlOTlkc21PekRIakZlaHE1c24zWDRCV2l2Rm1nS09ZNDNCeVZJUm5KSHZSNGxOVkpGVHl4Lzl4UFIwbEgKblo5Zk1yWUFqR0JiZU0wMXhkL0NUOGdPU2kxS2Zyd1I3eUdpWGZKYUVZZTV2SkxvcDdtSmd4OTZybW1CQkpIKwpvbEhqU25QRTgwVkZxRmtIRVFhVjhOZHB0QVBlWnduZFB1MDMwSDF6VUNpQVBSTFdVMUpSZU5FQWkyQzVxR2tDCkF3RUFBYU5mTUYwd0RnWURWUjBQQVFIL0JBUURBZ0VHTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME8KQkJZRUZPMjdEZXhqU1NoZUhkcGo3OXl1eWpWaGxmSzdNQnNHQTFVZEVRUVVNQktHRUhOd2FXWm1aVG92TDJGMwpjeTVqYjIwd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFGR3MxUUc0TktwN2RuZHRIUmg4cDU2bFZEODF0UDBCCjlVc2JaUnJiMytJNXkwNEc2TUNKd1NOWUg2YURlRUZSQUFTNEMxUVBCcVdDNjJLVTBqaGhKNG1TdXFDUzJXeG0KMHMwRTkzUFNXSSt4WFZqOWtONlp1NUhYRHkvTzJSa1RkMHQxZVp3M2g4Z1h6ZTNjN1FIUEQ4ZU8wZE93TzB2WAowMkJvTlNldlpUMjBJNTl1N1R6aUZsUXlJc2F3RkRSMFV4RHZjM3ZmWElCUnU5a0FRaFloWklMUnZuZS90QWh6CjVBRHZWMldWL3dZbjhHVUIzVTlKMWF3amNQYlZYc1MwdTVQbG9GQk9nY0ZrNlJzcFM4aHVRbjZyRW8wMFd1NVQKQzFaZUwwVHlXcVBGYnZwRDNjS1JrN0IzOVJicHd1cVQyT3RLckl2SllUZ0lSaFo3b3dlMGNsMD0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo="
                                }
                            },
                            {
                                "name":  "google.com",
                                "trustBundle":  {
                                    "inlineBytes":  "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURLRENDQWhDZ0F3SUJBZ0lRTGxIRFpYYmVaNEtGRElCM2N1aGJWekFOQmdrcWhraUc5dzBCQVFzRkFEQWUKTVFzd0NRWURWUVFHRXdKVlV6RVBNQTBHQTFVRUNoTUdVMUJKUmtaRk1CNFhEVEl5TVRBeU1EQTJNRGt6TjFvWApEVEl5TVRBeU1UQTJNRGswTjFvd0hqRUxNQWtHQTFVRUJoTUNWVk14RHpBTkJnTlZCQW9UQmxOUVNVWkdSVENDCkFTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTHhOV2I0NStoZUltM2dYRDVOejEvbUoKODFlOWtIcTVnbjFwTTA1R2RlM1dKSFpobFpKM2dTR2lUMVBHL1M5TDhmZVVrcklrS056Rzd5c2tTRUtmRTUzUwpXcFNMR2o4RlJFKzJZWHU3REZpaEJvZi9OdDNKR3d5M0RNdDhzdDlrZmRTclhMZTZ4bUdDWThWWW4zNFNJaWhtClRFRU52cE4rZzg0Sk5qM1BDejNLdkZBQm1sZjgxaFlpNjBadlk4RFhha29yT0FnWVBsWHlYa1RSbWl0bGJ3TXkKSHpFVllIY202bWRmR2hSdVlNS1NJR2NVUS9YU2N2Rys3ajMrUXJyZXg3aVF1Z3ZwL1ZnNkdRUkVrUG1NSk0rdgpCdWNXTGgwS3lqQUxPYTJ6emtrR1BvOE9PaUVIcml0ZUxZNnRMeWp0bnNRV1pxSnFEUEg1dUZVMjhvMDhJUHNDCkF3RUFBYU5pTUdBd0RnWURWUjBQQVFIL0JBUURBZ0VHTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME8KQkJZRUZBYUxDK1ZFekxXTDlpYkhoWDRGT24xVmh5OUJNQjRHQTFVZEVRUVhNQldHRTNOd2FXWm1aVG92TDJkdgpiMmRzWlM1amIyMHdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBSjIyYjRCRkxWNVdKdUxsdHJZM2xwWUd0UVg5Ckd3d2dycWhtUVBhWTV4QnJvVmdOT2lZN2czcmJ3QTVlYmthV000VWVhM1JNWEY0NWp3R3lTUG4vT29jTHkwNFQKQXRmWHZieWlHMnArdkZXQUlNZTJoWDVHVnB6dEVEdjhpNFVSUjBrdjA1L0ZyN0tmS1ZITUQ0ZVJHc1d6ZTUyUAo1MFlzbFNJK2l6L1FNZFpxdXRtaVZxVlNQVjBET1RzMkl4VFNRNlFqekZDSEpxRkE0dEN6ZnE5RTVNRDN4T2RqCnlYOTJQcmFMRzhCT20yYzhzK0c5THZNNnR2dmcyakZDaTdvTnZhc0ExR0ZFL0dyNy9jUkNzYkFPUElzTTJFcmkKWVR2WExjRDFGNUFmb3lIVmttL1NpQjhsa2x3R1lHZm40cDgwNW96d2lkb2lGWW01YmZlMWlqZ2tvYmM9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
                                }
                            }
                        ]
                    }
                }
            }
        },
        {
            "@type":  "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret",
            "name":  "spiffe://google.com/ns/sample/sa/default",
            "tlsCertificate":  {
                "certificateChain":  {
                    "inlineBytes":  "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUREekNDQWZlZ0F3SUJBZ0lSQU9CYWVBUHJBdUFrdUdzLzQzampDTDB3RFFZSktvWklodmNOQVFFTEJRQXcKSGpFTE1Ba0dBMVVFQmhNQ1ZWTXhEekFOQmdOVkJBb1RCbE5RU1VaR1JUQWVGdzB5TWpFd01qQXdOekl5TlRkYQpGdzB5TWpFd01qQXdPREl6TURkYU1FVXhDekFKQmdOVkJBWVRBbFZUTVE0d0RBWURWUVFLRXdWVFVFbFNSVEVtCk1DUUdBMVVFQXhNZGFHVnNiRzkzYjNKc1pDMTJNaTAxTmpWa04yUmtOMll0WTNSMmVuRXdXVEFUQmdjcWhrak8KUFFJQkJnZ3Foa2pPUFFNQkJ3TkNBQVJzNVBKMjNFMnhJeGxDdDVpNzdLQTFTUlNSUTZ0RDdoS05wTnBQUFgrUgpML0FGTHBuNGlsajdNNm5tT2JxSDU3cXBGd2xMdGRvZ2ZPYkY5Zk9tdFJZN280SHJNSUhvTUE0R0ExVWREd0VCCi93UUVBd0lEcURBZEJnTlZIU1VFRmpBVUJnZ3JCZ0VGQlFjREFRWUlLd1lCQlFVSEF3SXdEQVlEVlIwVEFRSC8KQkFJd0FEQWRCZ05WSFE0RUZnUVUzOENNNVVGRTBDUnN0WXZxNUR0VjNDMUxaWFF3SHdZRFZSMGpCQmd3Rm9BVQpCb3NMNVVUTXRZdjJKc2VGZmdVNmZWV0hMMEV3YVFZRFZSMFJCR0l3WUlJZGFHVnNiRzkzYjNKc1pDMTJNaTAxCk5qVmtOMlJrTjJZdFkzUjJlbkdDRldobGJHeHZkMjl5YkdRdWMyRnRjR3hsTG5OMlk0WW9jM0JwWm1abE9pOHYKWjI5dloyeGxMbU52YlM5dWN5OXpZVzF3YkdVdmMyRXZaR1ZtWVhWc2REQU5CZ2txaGtpRzl3MEJBUXNGQUFPQwpBUUVBaVdzQ0hVdXI1Ry9FaENITllGN205a05hQXk1VVErYUJIRUFFNnpKTEI5S3JiRUtSdFczMmsxMFo0YXN0CitOZ1NFVlU1VGUzSUUzSkZ5YzhBZ0dSdlFXM2lrRlJvdXFsdHdwMnZ2YVdhOTQ2cGMwcnZjQmlhVnZ4M0RBYkIKcks4OUNrUyt5YmxxWHYxK2duRFdBanNldnZZSVVxZ2wrd3NhK0dxYjlISncydGl4eDVxSXVtSmdjeXQzQWFvVQpvQk9PaDlZbW1MelRjTGt2K1gxUDR4VGhjeEVmUEQya2tvSjg1SDVaZGpLYTB5Q0lQL1pyVjlvRmFpNW94VjlkCmIrbUFXQ1VOSzIwK2xkb3ZNNGluZ2cxdkV0eW4yYmN1VVAwb0lMenMyaDJob1drNXRyaE1nK0Jld2dNWHA2b3QKbGZZTTdzV2NZaWlpZHBsVGlwZFJkRVVXVmc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="
                },
                "privateKey":  {
                    "inlineBytes":  "LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JR0hBZ0VBTUJNR0J5cUdTTTQ5QWdFR0NDcUdTTTQ5QXdFSEJHMHdhd0lCQVFRZ1hOYklKSXMzNXdGOUZyZUYKc0UwQWg1NVpmVDFvY2VsU3lkSEY3dDhiSjh5aFJBTkNBQVJzNVBKMjNFMnhJeGxDdDVpNzdLQTFTUlNSUTZ0RAo3aEtOcE5wUFBYK1JML0FGTHBuNGlsajdNNm5tT2JxSDU3cXBGd2xMdGRvZ2ZPYkY5Zk9tdFJZNwotLS0tLUVORCBQUklWQVRFIEtFWS0tLS0tCg=="
                }
            }
        }
    ]
}

and a similar output from the sleep pod from the other cluster:

istio-proxy@sleep-95d8696-jxjv6:/var/run/secrets/workload-spiffe-credentials$ ./sds --target unix:///var/run/secrets/workload-spiffe-uds/socket --resources ROOTCA,spiffe://aws.com/ns/sample/sa/sleep
{
    "resources":  [
        {
            "@type":  "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret",
            "name":  "ROOTCA",
            "validationContext":  {
                "customValidatorConfig":  {
                    "name":  "envoy.tls.cert_validator.spiffe",
                    "typedConfig":  {
                        "@type":  "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.SPIFFECertValidatorConfig",
                        "trustDomains":  [
                            {
                                "name":  "aws.com",
                                "trustBundle":  {
                                    "inlineBytes":  "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURKVENDQWcyZ0F3SUJBZ0lRVkpaVEJKbzdVa2YyRW5PS2tRVjVHakFOQmdrcWhraUc5dzBCQVFzRkFEQWUKTVFzd0NRWURWUVFHRXdKVlV6RVBNQTBHQTFVRUNoTUdVMUJKUmtaRk1CNFhEVEl5TVRBeU1EQTJNRGcxTkZvWApEVEl5TVRBeU1UQTJNRGt3TkZvd0hqRUxNQWtHQTFVRUJoTUNWVk14RHpBTkJnTlZCQW9UQmxOUVNVWkdSVENDCkFTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTnlkNisvbFlPVTRrM0xZQnBDVkFOOXAKTU5jQm1tb3BTa0NwSVBDcXVXeUYyd1B2UVlweE14R21OWXVGeFRpMHM2Ukc1a05VU0daNVdrMmsyZ1ZEMVVzawpjam9MSm8vSS9yMFpEMmhZczdoU2lrVHBMdmFFcmswVC9Pa2VYeS9uRzhnMHFQbEYzOWppcDAwT2I3c0hkMTdqCmg2YzdlOTlkc21PekRIakZlaHE1c24zWDRCV2l2Rm1nS09ZNDNCeVZJUm5KSHZSNGxOVkpGVHl4Lzl4UFIwbEgKblo5Zk1yWUFqR0JiZU0wMXhkL0NUOGdPU2kxS2Zyd1I3eUdpWGZKYUVZZTV2SkxvcDdtSmd4OTZybW1CQkpIKwpvbEhqU25QRTgwVkZxRmtIRVFhVjhOZHB0QVBlWnduZFB1MDMwSDF6VUNpQVBSTFdVMUpSZU5FQWkyQzVxR2tDCkF3RUFBYU5mTUYwd0RnWURWUjBQQVFIL0JBUURBZ0VHTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME8KQkJZRUZPMjdEZXhqU1NoZUhkcGo3OXl1eWpWaGxmSzdNQnNHQTFVZEVRUVVNQktHRUhOd2FXWm1aVG92TDJGMwpjeTVqYjIwd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFGR3MxUUc0TktwN2RuZHRIUmg4cDU2bFZEODF0UDBCCjlVc2JaUnJiMytJNXkwNEc2TUNKd1NOWUg2YURlRUZSQUFTNEMxUVBCcVdDNjJLVTBqaGhKNG1TdXFDUzJXeG0KMHMwRTkzUFNXSSt4WFZqOWtONlp1NUhYRHkvTzJSa1RkMHQxZVp3M2g4Z1h6ZTNjN1FIUEQ4ZU8wZE93TzB2WAowMkJvTlNldlpUMjBJNTl1N1R6aUZsUXlJc2F3RkRSMFV4RHZjM3ZmWElCUnU5a0FRaFloWklMUnZuZS90QWh6CjVBRHZWMldWL3dZbjhHVUIzVTlKMWF3amNQYlZYc1MwdTVQbG9GQk9nY0ZrNlJzcFM4aHVRbjZyRW8wMFd1NVQKQzFaZUwwVHlXcVBGYnZwRDNjS1JrN0IzOVJicHd1cVQyT3RLckl2SllUZ0lSaFo3b3dlMGNsMD0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo="
                                }
                            },
                            {
                                "name":  "google.com",
                                "trustBundle":  {
                                    "inlineBytes":  "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURLRENDQWhDZ0F3SUJBZ0lRTGxIRFpYYmVaNEtGRElCM2N1aGJWekFOQmdrcWhraUc5dzBCQVFzRkFEQWUKTVFzd0NRWURWUVFHRXdKVlV6RVBNQTBHQTFVRUNoTUdVMUJKUmtaRk1CNFhEVEl5TVRBeU1EQTJNRGt6TjFvWApEVEl5TVRBeU1UQTJNRGswTjFvd0hqRUxNQWtHQTFVRUJoTUNWVk14RHpBTkJnTlZCQW9UQmxOUVNVWkdSVENDCkFTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTHhOV2I0NStoZUltM2dYRDVOejEvbUoKODFlOWtIcTVnbjFwTTA1R2RlM1dKSFpobFpKM2dTR2lUMVBHL1M5TDhmZVVrcklrS056Rzd5c2tTRUtmRTUzUwpXcFNMR2o4RlJFKzJZWHU3REZpaEJvZi9OdDNKR3d5M0RNdDhzdDlrZmRTclhMZTZ4bUdDWThWWW4zNFNJaWhtClRFRU52cE4rZzg0Sk5qM1BDejNLdkZBQm1sZjgxaFlpNjBadlk4RFhha29yT0FnWVBsWHlYa1RSbWl0bGJ3TXkKSHpFVllIY202bWRmR2hSdVlNS1NJR2NVUS9YU2N2Rys3ajMrUXJyZXg3aVF1Z3ZwL1ZnNkdRUkVrUG1NSk0rdgpCdWNXTGgwS3lqQUxPYTJ6emtrR1BvOE9PaUVIcml0ZUxZNnRMeWp0bnNRV1pxSnFEUEg1dUZVMjhvMDhJUHNDCkF3RUFBYU5pTUdBd0RnWURWUjBQQVFIL0JBUURBZ0VHTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME8KQkJZRUZBYUxDK1ZFekxXTDlpYkhoWDRGT24xVmh5OUJNQjRHQTFVZEVRUVhNQldHRTNOd2FXWm1aVG92TDJkdgpiMmRzWlM1amIyMHdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBSjIyYjRCRkxWNVdKdUxsdHJZM2xwWUd0UVg5Ckd3d2dycWhtUVBhWTV4QnJvVmdOT2lZN2czcmJ3QTVlYmthV000VWVhM1JNWEY0NWp3R3lTUG4vT29jTHkwNFQKQXRmWHZieWlHMnArdkZXQUlNZTJoWDVHVnB6dEVEdjhpNFVSUjBrdjA1L0ZyN0tmS1ZITUQ0ZVJHc1d6ZTUyUAo1MFlzbFNJK2l6L1FNZFpxdXRtaVZxVlNQVjBET1RzMkl4VFNRNlFqekZDSEpxRkE0dEN6ZnE5RTVNRDN4T2RqCnlYOTJQcmFMRzhCT20yYzhzK0c5THZNNnR2dmcyakZDaTdvTnZhc0ExR0ZFL0dyNy9jUkNzYkFPUElzTTJFcmkKWVR2WExjRDFGNUFmb3lIVmttL1NpQjhsa2x3R1lHZm40cDgwNW96d2lkb2lGWW01YmZlMWlqZ2tvYmM9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
                                }
                            }
                        ]
                    }
                }
            }
        },
        {
            "@type":  "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret",
            "name":  "spiffe://aws.com/ns/sample/sa/sleep",
            "tlsCertificate":  {
                "certificateChain":  {
                    "inlineBytes":  "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM4RENDQWRpZ0F3SUJBZ0lRWDNGRDBUZFA3RFo1UnI2N1ZYSXdFREFOQmdrcWhraUc5dzBCQVFzRkFEQWUKTVFzd0NRWURWUVFHRXdKVlV6RVBNQTBHQTFVRUNoTUdVMUJKUmtaRk1CNFhEVEl5TVRBeU1EQTNOVE13T0ZvWApEVEl5TVRBeU1EQTROVE14T0Zvd096RUxNQWtHQTFVRUJoTUNWVk14RGpBTUJnTlZCQW9UQlZOUVNWSkZNUnd3CkdnWURWUVFERXhOemJHVmxjQzA1TldRNE5qazJMV3A0YW5ZMk1Ga3dFd1lIS29aSXpqMENBUVlJS29aSXpqMEQKQVFjRFFnQUVDY2JwVThJdVc3OGFiS0IrNk4rK3UxemRCY3JoWm1SSjQxdjFtL2RiM2NZVkQyYnBnQ3MyK2xxWQpFMkEwcjBGYVZpQmZ1Z0pONlcyeVIwUUtuekpaWHFPQjF6Q0IxREFPQmdOVkhROEJBZjhFQkFNQ0E2Z3dIUVlEClZSMGxCQll3RkFZSUt3WUJCUVVIQXdFR0NDc0dBUVVGQndNQ01Bd0dBMVVkRXdFQi93UUNNQUF3SFFZRFZSME8KQkJZRUZIbEdVc290Q0tsSDY0d1FJeUp1Nm53UmNuQ3pNQjhHQTFVZEl3UVlNQmFBRk8yN0RleGpTU2hlSGRwago3OXl1eWpWaGxmSzdNRlVHQTFVZEVRUk9NRXlDRTNOc1pXVndMVGsxWkRnMk9UWXRhbmhxZGphQ0VITnNaV1Z3CkxuTmhiWEJzWlM1emRtT0dJM053YVdabVpUb3ZMMkYzY3k1amIyMHZibk12YzJGdGNHeGxMM05oTDNOc1pXVncKTUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFDY09LcmJXRnlIalVOSTAwN3VIdEhoYXZicDJURzhuZEVGZGpQUwpoSEdHdlNWbXF2UVpkTFZPUTJlSU9mc2NMa1YxRDZRaDg3aWlRbGdvVVFaaVR0UWppOGNlYjRRN1ppZUZvT0cxCmhnV0ZKcEZlWTZRSlJaQUYxUHI1S01BSGdwVCt4UU1qNXZGL09RY2xXdHZjOTZ6eDJtclRuZCtva1pzRnF1UHMKWDdVdTdENkhYbHU1akFPRXY1SVYwL05tL3RDZHlFV0ttNEVmWjB0aXFSOWNyTXFWSDZIS2NsVEV0Q0pwTUxKWgpUVFRmS09IMk5YbitjQmVLbndmWm9QQ1FJaG9qRW45YzdtaVBrdHhWZUZWTWZkcytlSTRXemdNZHB2RDlWclZKCkF2WDBOQVR6Y1cvN3Q3KzJuSW9QMnJqdEFuaFhZTFZYcFNFYW1EcnIrMlpQOHFJVgotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="
                },
                "privateKey":  {
                    "inlineBytes":  "LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JR0hBZ0VBTUJNR0J5cUdTTTQ5QWdFR0NDcUdTTTQ5QXdFSEJHMHdhd0lCQVFRZ3EvSWhmVEI1VVIwR2FxWEEKUzE2N1dTZEtBU1k3VU9UWExkZVBqSmdNaGdpaFJBTkNBQVFKeHVsVHdpNWJ2eHBzb0g3bzM3NjdYTjBGeXVGbQpaRW5qVy9XYjkxdmR4aFVQWnVtQUt6YjZXcGdUWURTdlFWcFdJRis2QWszcGJiSkhSQXFmTWxsZQotLS0tLUVORCBQUklWQVRFIEtFWS0tLS0tCg=="
                }
            }
        }
    ]
}
sudeeptoroy commented 2 years ago

Additionally i did this experiment.

On cluster2 ( google.com ): I connected to the spire server and minted a cert. I brought up a debug pod and used the certs i minted from the above step. I was successful in connecting to the helloworld-v2 app.

On cluster 1 ( aws.com ): I bought up the same debug pod and used the same cert minted earlier to connect to helloworld-v2 app on the other cluster. It fails with the same SSL error (CERTIFICATE_VERIFY_FAILED).

If you feel that spire is minting the certs correctly, are there probabilities that envoy is not configured properly to use the cacerts ( ROOTCA ). shall we check the envoy configurations a little closer?

Check the validation context. It does not mention the other Trust domain. 

        "combined_validation_context": {
          "default_validation_context": {
            "match_subject_alt_names": [
              {
                "exact": "spiffe://google.com/ns/sample/sa/default"
              }
            ]
          }
"outbound|5000||helloworld.sample.svc.cluster.local"
{
  "@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
  "name": "outbound|5000||helloworld.sample.svc.cluster.local",
  "type": "EDS",
  "eds_cluster_config": {
    "eds_config": {
      "ads": {},
      "initial_fetch_timeout": "0s",
      "resource_api_version": "V3"
    },
    "service_name": "outbound|5000||helloworld.sample.svc.cluster.local"
  },
  "connect_timeout": "10s",
  "lb_policy": "LEAST_REQUEST",
  "circuit_breakers": {
    "thresholds": [
      {
        "max_connections": 4294967295,
        "max_pending_requests": 4294967295,
        "max_requests": 4294967295,
        "max_retries": 4294967295,
        "track_remaining": true
      }
    ]
  },
  "metadata": {
    "filter_metadata": {
      "istio": {
        "default_original_port": 5000,
        "services": [
          {
            "name": "helloworld",
            "host": "helloworld.sample.svc.cluster.local",
            "namespace": "sample"
          }
        ]
      }
    }
  },
  "common_lb_config": {
    "locality_weighted_lb_config": {}
  },
  "filters": [
    {
      "name": "istio.metadata_exchange",
      "typed_config": {
        "@type": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange",
        "protocol": "istio-peer-exchange"
      }
    }
  ],
  "transport_socket_matches": [
    {
      "name": "tlsMode-istio",
      "match": {
        "tlsMode": "istio"
      },
      "transport_socket": {
        "name": "envoy.transport_sockets.tls",
        "typed_config": {
          "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext",
          "common_tls_context": {
            "tls_params": {
              "tls_minimum_protocol_version": "TLSv1_2",
              "tls_maximum_protocol_version": "TLSv1_3"
            },
            "alpn_protocols": [
              "istio-peer-exchange",
              "istio"
            ],
            "tls_certificate_sds_secret_configs": [
              {
                "name": "default",
                "sds_config": {
                  "api_config_source": {
                    "api_type": "GRPC",
                    "grpc_services": [
                      {
                        "envoy_grpc": {
                          "cluster_name": "sds-grpc"
                        }
                      }
                    ],
                    "set_node_on_first_message_only": true,
                    "transport_api_version": "V3"
                  },
                  "initial_fetch_timeout": "0s",
                  "resource_api_version": "V3"
                }
              }
            ],
            "combined_validation_context": {
              "default_validation_context": {
                "match_subject_alt_names": [
                  {
                    "exact": "spiffe://google.com/ns/sample/sa/default"
                  }
                ]
              },
              "validation_context_sds_secret_config": {
                "name": "ROOTCA",
                "sds_config": {
                  "api_config_source": {
                    "api_type": "GRPC",
                    "grpc_services": [
                      {
                        "envoy_grpc": {
                          "cluster_name": "sds-grpc"
                        }
                      }
                    ],
                    "set_node_on_first_message_only": true,
                    "transport_api_version": "V3"
                  },
                  "initial_fetch_timeout": "0s",
                  "resource_api_version": "V3"
                }
              }
            }
          },
          "sni": "outbound_.5000_._.helloworld.sample.svc.cluster.local"
        }
      }
    },
    {
      "name": "tlsMode-disabled",
      "match": {},
      "transport_socket": {
        "name": "envoy.transport_sockets.raw_buffer",
        "typed_config": {
          "@type": "type.googleapis.com/envoy.extensions.transport_sockets.raw_buffer.v3.RawBuffer"
        }
      }
    }
  ]
}
azdagron commented 2 years ago

As an aside, you may want to redact private keys from the SDS output before pasting here.

It looks like your SDS configuration, specifically the match_subject_alt_names directives, is preventing the SVIDs from the other trust domain from being authorized. I don't know much about Istio or how to influence those directives, but for sure they are rejecting the URI SANs being presented in SVIDs from the other trust domain after those SVIDs are authenticated.

sudeeptoroy commented 2 years ago

Yeah, not sure how to validate this. Do you have any istio experts who can help? I pasted the private key as its selfsigned but yes, you are right i should practice to redact it whenever i copy paste them. 👍🏻

sudeeptoroy commented 2 years ago

additionally if you look at the server logs. It says :SSLV3_ALERT_CERTIFICATE_UNKNOWN

and if i look at the client logs it says CERTIFICATE_VERIFY_FAILED

client:

upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

Server:

2022-10-15T10:39:52.420744Z trace   envoy connection    [C146] ssl error occurred while read: WANT_READ
2022-10-15T10:39:52.424542Z trace   envoy connection    [C146] socket event: 3
2022-10-15T10:39:52.424707Z trace   envoy connection    [C146] write ready
2022-10-15T10:39:52.424750Z trace   envoy connection    [C146] ssl error occurred while read: SSL
2022-10-15T10:39:52.424767Z debug   envoy connection    [C146] TLS error: 268436502:SSL routines:**OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN**
2022-10-15T10:39:52.424780Z debug   envoy connection    [C146] closing socket: 0
2022-10-15T10:39:52.424824Z debug   envoy connection    [C146] TLS error: 268436502:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN
2022-10-15T10:39:52.424859Z trace   envoy connection    [C146] raising connection event 0
2022-10-15T10:39:52.424905Z trace   envoy conn_handler  [C146] connection on event 0
2022-10-15T10:39:52.424918Z debug   envoy conn_handler  [C146] adding to cleanup list
maxlambrecht commented 2 years ago

I'm not really sure because I didn't test it but I'd look into defining the helloworld service as a service entry in the client mesh an apply a DestinationRule with mTLS on it.

sudeeptoroy commented 2 years ago

Hi @maxlambrecht, were you able to reproduce this with your setup. Let me know if face any issue. I am ready to work with you incase you need any help.

azdagron commented 2 years ago

At this point I don't believe we have any evidence that SPIRE is at fault for this correct? @maxlambrecht @sudeeptoroy, can we close this issue while you folks sort out the Istio bits?

sudeeptoroy commented 2 years ago

Hi @azdagron i would suggest we don't close the issue as its still not clear. The code changes on Istio was done by this group and there are possibilities that istio was not correctly configured or this end to end mTLS use case was not tested when the feature was designed.

maxlambrecht commented 2 years ago

Hi @maxlambrecht, were you able to reproduce this with your setup. Let me know if face any issue. I am ready to work with you incase you need any help.

Yes, I could reproduce the issue, got the same error CERTIFICATE_VERIFY_FAILED, and all the evidence points to the URI exact validation in the Upstream Cluster config in Envoy.

At this point I don't believe we have any evidence that SPIRE is at fault for this correct? @maxlambrecht @sudeeptoroy, can we close this issue while you folks sort out the Istio bits?

I agree that it doesn't seem to be a SPIRE issue, based on the logs and Envoy configs, the Envoys are getting the all bundles in the proper SDS response from the SPIRE Agent.

Hi @azdagron i would suggest we don't close the issue as its still not clear. The code changes on Istio was done by this group and there are possibilities that istio was not correctly configured or this end to end mTLS use case was not tested when the feature was designed.

The Envoy configuration that is causing the issue is generated by Istio, SPIRE cannot influence how the UpstreamClusters with the exact URI validation are configured in the Envoy proxies. AFAIK this end to end mTLS federation use case wasn't tested and I think it's a good use case to bring to Istio to get some input.

sudeeptoroy commented 2 years ago

Thank you @maxlambrecht for the confirmation. How do you want to take it forward to the istio community. I would like to be a part of it.

maxlambrecht commented 2 years ago

Thank you @maxlambrecht for the confirmation. How do you want to take it forward to the istio community. I would like to be a part of it.

Would you be willing to open an issue in Istio? You could summarize what we found here, sharing your setup, pointing to what is causing the certificate validation error, the Envoy exact URI validation, asking how that could be solved through Istio configs.

azdagron commented 2 years ago

I'll go ahead and close this issue in lieu of the issue that will be opened against Istio.

nrjpoddar commented 2 years ago

@sudeeptoroy I mentioned this issue to my team and @linsun from my team will help you out here.

swamibluedata commented 2 years ago

@sudeeptoroy Though intended for trust domain migration, trustDomainAliases can be used to address this. That is the only way i was able to get this working.

One more note: For the east-west gateway configuration, federatesWith label is not required. It will do an mtls passthrough to the workload directly

sudeeptoroy commented 2 years ago

thanks @nrjpoddar

Hi @swamibluedata, thank you for testing my setup. Yes i had to add aliases to make it work. However aliases are used for a different use case and i am presently in conversation with Istio community to get their views. Also thanks for the note around "federatesWith" labels, its helpful, it would mean less changes to manage. Which is great.