Open Tzvonimir opened 3 years ago
lkysow
commented
3 years ago Hi @Tzvonimir, sorry to hear you're having this issue.
I just tried to get things set up but I think I'm missing things like your helm values file and the Docker image for the gateway (image: my.image:latest) and also the search and reservation services.
Can you supply us with simple step by step instructions on how to reproduce this ourselves? In addition can you supply the CORS error you're getting? Thanks!
Tzvonimir
commented
3 years ago @lkysow thank you for replying.
here is my consul helm chart values file:
global:
name: consul
enabled: true
datacenter: ${datacenter_name}
gossipEncryption:
secretName: "consul-gossip-key"
secretKey: "key"
tls:
enabled: true
enableAutoEncrypt: true
verify: true
# Vault chart not working with acls enabled
#acls:
#manageSystemACLs: true
server:
replicas: ${replicas}
bootstrapExpect: ${replicas}
disruptionBudget:
enabled: true
maxUnavailable: 0
extraConfig: |
{
"telemetry": {
"prometheus_retention_time": "10s"
},
"ui_config": {
"enabled": true,
"metrics_provider": "prometheus",
"metrics_proxy": {
"base_url": "http://prometheus-server"
}
}
}
client:
enabled: true
# enable gRPC on your client to support Consul service mesh
grpc: true
ui:
enabled: true
connectInject:
#aclBindingRuleSelector: ""
enabled: true
grpc: true
# inject an envoy sidecar into every new pod, except for those with annotations that prevent injection
default: true
# these settings enable L7 metrics collection and are new in 1.5
centralConfig:
enabled: true
# set the default protocol (can be overwritten with annotations)
defaultProtocol: 'http'
# proxyDefaults is a raw json string that will be applied to all Connect
# proxy sidecar pods that can include any valid configuration for the
# configured proxy.
proxyDefaults: |
{
"envoy_prometheus_bind_addr": "0.0.0.0:9102"
}
controller:
enabled: true
ingressGateways:
enabled: true
gateways:
- name: ingress-gateway
service:
type: LoadBalancer
More high level look into architecture:
AWS cluster on EKS running consul and vault (+prometheus and grafana)
After deploying the cluster I am deploying multiple Kubernetes microservices in private subnet (Most of them are written in go and only one is in java), but only one is important, and that one is a "gateway" service. Because everything has to pass through "gateway"
Ingress gateway is running in public subnet and is exposed to "external internet".
Currently, everything is deployed and working. iOS, Android, and backend developers can access it without a problem. They are making GRPC calls and getting response back.
But when we try to access it via browser using GRPC web we can't do it. (Request is blocked somewhere on ingress gateway, it is not able to reach our "gateway")
We tried to setup envoy proxy locally with custom CORS and HTTP to GRPC conversion and it is working perfectly.
Here is the local envoy config that we are using to test our GRPC-web calls.
admin:
access_log_path: /dev/stdout
address:
socket_address: { address: 0.0.0.0, port_value: 9901 }
static_resources:
listeners:
- name: listener_0
address:
socket_address: { address: 0.0.0.0, port_value: 8080 }
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match: { prefix: "/" }
route:
cluster: gateway_service
max_grpc_timeout: 0s
cors:
allow_origin_string_match:
- prefix: "*"
allow_methods: GET, PUT, DELETE, POST, OPTIONS
allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
max_age: "1728000"
expose_headers: custom-header-1,grpc-status,grpc-message
http_filters:
- name: envoy.filters.http.grpc_web
- name: envoy.filters.http.cors
- name: envoy.filters.http.router
clusters:
- name: gateway_service
connect_timeout: 0.25s
type: logical_dns
http2_protocol_options: {}
lb_policy: round_robin
load_assignment:
cluster_name: cluster_0
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: host.docker.internal
port_value: 9990
So for this question:
my.image:latest is an image hosted on a private repository (unfortunately I am not able to share access to that because of company policy. But you can use any GRPC server running an insecure connection for testing. If you do not want to lose your time to set up the GRPC server, I will create a simple gateway project with the GRPC server and deploy the docker image to the public hub (and update config accordingly, but again the problem is somewhere on ingress gateway, envoy config is probably not structured correctly).
apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
name: ingress-gateway
spec:
config:
protocol: grpc
envoy_extra_static_listeners_json: '{"name":"listener_0","address":{"socket_address":{"address":"my-host.com","port_value":8080}},"filter_chains":[{"filters":[{"name":"envoy.filters.network.http_connection_manager","typed_config":{"@type":"type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager","codec_type":"auto","stat_prefix":"ingress_http","route_config":{"name":"local_route","virtual_hosts":[{"name":"local_service","domains":["*"],"routes":[{"match":{"prefix":"/"},"route":{"cluster":"gateway_service","max_grpc_timeout":"0s"}}],"cors":{"allow_origin_string_match":[{"prefix":"*"}],"allow_methods":"GET, PUT, DELETE, POST, OPTIONS","allow_headers":"keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout","max_age":"1728000","expose_headers":"custom-header-1,grpc-status,grpc-message"}}]},"http_filters":[{"name":"envoy.filters.http.grpc_web"},{"name":"envoy.filters.http.cors"},{"name":"envoy.filters.http.router"}]}}]}]}'
CORS error:
Access to XMLHttpRequest at 'http://gateway.web.elude.co:8080/elude.proto.GatewayController/RunSearch' from origin 'http://localhost:8080' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.
When I disable CORS in the browser I get the following error:
upstream connect error or disconnect/reset before headers. reset reason: remote resetThank you for your time. I am grateful for all the help that you guys can give :)
HofmannZ
commented
3 years ago Might be related hashicorp/consul-helm#911
ndhanushkodi
commented
3 years ago Hi @Tzvonimir the ingress gateway does not currently support CORS configurations, which is why you're seeing the web-grpc clients unable to connect but other clients are able to. We are tracking CORS configuration as a feature request and I can add this as a +1 to that feature request. Sorry for the trouble!!!
david-yu
commented
3 years ago Reposting from hashicorp/consul-helm#946 from @Shige99011
Hi, I'm using Ingress gateway to make it access from an external service that is simple javascript application. When an API that is in consul service mesh is called via Ingress gateway, an error like below happens:
Access to fetch at 'http://192.168.202.199:31000/hello-service/hello' from origin 'http://localhost:9020' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
How should it be configured to enable Access-Control-Allow-Origin for Ingress gateway?
btw, the API responses correctly when it is called by using curl command like:
curl -H "Host: api-service.ingress.consul" "http://<ip address of the node>:31000/hello-service/hello"
My environment is below: OS: ubuntu 18.04 on hyper-v microk8s: v1.20.5 kubernetes: v1.20.5 consul: v1.9.4 Consul helm chart: v0.31.1 (Consul helm chart was installed by command "microk8s helm3 install -f config.yaml consul hashicorp/consul --version "0.31.1"") There is only one kubernetes node.
The description of my ingress gateway object:
Name: ingress-gateway
Namespace: default
Labels: <none>
Annotations: <none>
API Version: consul.hashicorp.com/v1alpha1
Kind: IngressGateway
Metadata:
Creation Timestamp: 2021-05-07T05:22:28Z
Finalizers:
finalizers.consul.hashicorp.com
Generation: 4
Managed Fields:
API Version: consul.hashicorp.com/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.:
v:"finalizers.consul.hashicorp.com":
f:spec:
f:tls:
.:
f:enabled:
f:status:
.:
f:conditions:
f:lastSyncedTime:
Manager: consul-k8s
Operation: Update
Time: 2021-05-07T05:22:28Z
API Version: consul.hashicorp.com/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:listeners:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2021-05-07T05:22:28Z
Resource Version: 2996629
Self Link: /apis/consul.hashicorp.com/v1alpha1/namespaces/default/ingressgateways/ingress-gateway
UID: 64506060-8217-4568-b74e-6221786ce80f
Spec:
Listeners:
Port: 7000
Protocol: http
Services:
Name: api-service
Tls:
Enabled: false
Status:
Conditions:
Last Transition Time: 2021-05-07T05:40:10Z
Status: True
Type: Synced
Last Synced Time: 2021-05-07T05:40:10Z
Events: <none>The configured values.yaml of Conul helm chart is
global:
name: consul
server:
replicas: 1
client:
enabled: true
connectInject:
enabled: true
controller:
enabled: true
terminatingGateways:
enabled: true
defaults:
replicas: 1
gateways:
- name: terminating-gateway
ingressGateways:
enabled: true
defaults:
replicas: 1
gateways:
- name: ingress-gateway
service:
type: NodePort
ports:
- port: 7000
nodePort: 31000Transferred the issue to hashicorp/consul so we can track CORS configuration support within Consul Core for the Ingress Gateway.
itsbibeksaini
commented
2 years ago Hi,
I've configuring Consul API Gateway and when I am trying to connect to gateway through my angular application(client) request get blocked due to CORS policy. However gateway is working fine when trying to reach it from other client like postman. Is there anyway by which I can make Consul API Gateway with CORS? I see CORS is not yeet supported in Consul is there any roadmap for supporting this?
Tzvonimir
commented
2 years ago @itsbibeksaini, I can share how I solved my problem. That might help you or at least put you on the right track.
Not sure how you exposed the gateway to the public subnet. But in my case, I am running everything on Kubernetes, so I exposed (via ingress gateway) an additional envoy proxy (in that envoy config I deal with the CORS issue). From there, I redirect all traffic to my API gateway.
Sample envoy config:
static_resources:
listeners:
- name: listener_0
address:
socket_address: { address: 0.0.0.0, port_value: 8060 }
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match: { prefix: "/" }
route:
cluster: gateway_service
max_grpc_timeout: 0s
cors:
allow_origin_string_match:
- prefix: "*"
allow_methods: GET, PUT, DELETE, POST, OPTIONS
allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,authorization,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
max_age: "1728000"
expose_headers: custom-header-1,grpc-status,grpc-message
http_filters:
- name: envoy.filters.http.grpc_web
- name: envoy.filters.http.cors
- name: envoy.filters.http.router
clusters:
- name: gateway_service
connect_timeout: 0.25s
type: logical_dns
http2_protocol_options: {}
lb_policy: round_robin
load_assignment:
cluster_name: cluster_0
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 9990Sample Dockerfile to build this envoy config:
FROM envoyproxy/envoy:v1.15.0
COPY envoy.yaml /etc/envoy/envoy.yaml
CMD /usr/local/bin/envoy --base-id 1 -c /etc/envoy/envoy.yaml -l trace --log-path /dev/stdoutSample Kubernetes files to deploy all of this:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: envoy
labels:
app: 'envoy'
spec:
selector:
matchLabels:
run: envoy
template:
metadata:
labels:
run: envoy
app: envoy
annotations:
consul.hashicorp.com/connect-inject: "true"
consul.hashicorp.com/connect-service-upstreams: "gateway-service:9990"
spec:
containers:
- name: envoy
image: <envoy-image-tag-and-name>
imagePullPolicy: "Always"
env:
- name: DEBUG
value: "true"
ports:
- containerPort: 8060
name: http
imagePullSecrets:
- name: gitlab-auth
- name: docker-io
serviceAccountName: envoy
---
apiVersion: v1
kind: Service
metadata:
name: envoy
spec:
type: ClusterIP
selector:
app: envoy
ports:
- name: http
port: 8060
targetPort: 8060
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: envoy
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
name: envoy
spec:
protocol: http
upstreamConfig:
defaults:
connectTimeoutMs: 10000
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceRouter
metadata:
name: envoy
spec:
routes:
- destination:
requestTimeout: "10s"
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: IngressGateway
metadata:
name: public-wildcard-ingress-gateway
spec:
listeners:
- port: 443
protocol: http
services:
- name: envoy
hosts: ['envoy.host.com']
---
andrewstucki
commented
2 years ago @itsbibeksaini thanks for the interest in the APi Gateway. Since the gateway itself is currently built on top of some custom ingress gateways and leverages Consul's internal RDS for route and filter chain definitions -- offloading CORS handling to the gateway isn't currently possible. There is some work we have scheduled to move towards doing more direct xDS generation to allow for addition of commonly requested gateway features such as CORS, but no solid timeframes or feature set I can give you at the moment.
Your best bet for the time being if you want to keep CORS logic out of your application code is to do something like what @Tzvonimir suggested (thanks!) and shim in an extra proxy to handle the CORS for you, or, if you can, just handle CORS preflight headers directly in your application code.
Hey everybody,
I have deployed consul on Kubernetes cluster via helm chart, and that part is working exactly as it is supposed to. On top of that, I have enabled Ingress Gateway with connect inject option and deployed my gRPC gateway behind that proxy. When I try to connect to that proxy, It works for everything except gRPC-web(https://github.com/grpc/grpc-web). I am facing a CORS issue now that I can't resolve. So it would be really helpful if someone has any idea how to resolve this issue.
I have configured my local envoy configuration, and It works fine. Here is the local envoy config:
Here is my gateway Kubernetes configuration:
ingress gateway:
pod:
service:
service account:
service defaults:
As much as I was able to understand from documentation and some other issues I have to add ProxyDefaults kind and
envoy_extra_static_listeners_json. And I have tried to do so, but nothing has worked for me, here is the example of config I tried:Thanks.