grs
commented
4 years ago Yes, that is certainly possible. Do you use OLM for managing operators?
Closed Kampe closed 4 years ago
grs
commented
4 years ago Yes, that is certainly possible. Do you use OLM for managing operators?
Kampe
commented
4 years ago We do not currently no - would be interested in moving towards it in the future.
Ideally we could just grab the yaml from github and apply it at boot, then follow up by laying down our skupper config
grs
commented
4 years ago The 0.3.0 release includes an optional site-controller, which behaves much like an operator and allows setup without using the skupper cli. You can deploy the site controller either to watch just the current namespace[1] (i.e. the namespace in which it is deployed) or to watch all namespaces[2].
Once deployed, the site-controller allows a namespace to be initialised simply be creating a configmap named skupper-site within it. That configmap can contain various settings (essentially the same options as can be passed to the CLI in skupper init)[3].
The site controller will also allow connection tokens to be simply applied as secrets, which will then trigger a connection to the specified skupper site. Note when using tokens generated by the cli, you will likely want/need to change the name of the secret to avoid collisions.
It will also generate tokens. To do so you create an empty secret with label skupper.io/type=connection-token-request, that secret will then have the necessary data inserted into it and can then be exported and transfered to the appropriate site where it can be applied with kubectl apply -f.
[1] https://raw.githubusercontent.com/skupperproject/skupper/0.3/cmd/site-controller/deploy-watch-current-ns.yaml [2] https://raw.githubusercontent.com/skupperproject/skupper/0.3/cmd/site-controller/deploy-watch-all-ns.yaml [3] https://github.com/skupperproject/skupper/blob/master/cmd/site-controller/README.md
Kampe
commented
4 years ago So I just apply the connection token as a secret and the site manager should theoretically see that and deploy the routers?
grs
commented
4 years ago You need a configmap called skupper-site first (that is what triggers the setup of the routers with all other necessary setup), and then yes, the connection-token can just be applied and that will configure the routers to connect to the site the token originates from.
Kampe
commented
4 years ago So I have this somewhat working, the configmap does lay down the router, however when I apply my secret - nothing happens - none of my skuppered services become available or spawn specific router pods for the service's proxy.
grs
commented
4 years ago What do you see for skupper list-exposed? From 0.3.0 there are no longer separate pods launched per service, the bridging is done internally in the skupper-router pod.
Kampe
commented
4 years ago Unfortunately, I don't have 0.3.0 version of the cli tooling - is that around somewhere yet? - looks like 0.2.0 is the latest and I just get Error: unknown command "list-exposed" for "skupper"
Also, wouldn't I be able to still see the services available to my skupper'd cluster for the services in which are skuppered?
I do get this for skupper status
Skupper enabled for namespace "default" in edge mode. It is not connected to any other sites.
My token is applied currently as a secret as well in the default namespace.
I created a new connection token in cloud cluster in GKE (connection created via skupper init && skupper connection-token) and then on boot I apply the secret token to "edge" cluster and apply the skupper manifests
bridge server logs:
│ 2020-08-20T06:37:09.177Z bridge-server info config read: {}
│ 2020-08-20T06:37:09.183Z bridge-server info no httpConnectors defined
│ 2020-08-20T06:37:09.188Z bridge-server info checking for org.apache.qpid.dispatch.router.httpConnector updates...
│ 2020-08-20T06:37:09.188Z bridge-server info no httpListeners defined
│ 2020-08-20T06:37:09.188Z bridge-server info checking for org.apache.qpid.dispatch.router.httpListener updates...
│ 2020-08-20T06:37:09.188Z bridge-server info no http2Connectors defined
│ 2020-08-20T06:37:09.188Z bridge-server info checking for org.apache.qpid.dispatch.router.http2Connector updates...
│ 2020-08-20T06:37:09.188Z bridge-server info no http2Listeners defined
│ 2020-08-20T06:37:09.188Z bridge-server info checking for org.apache.qpid.dispatch.router.http2Listener updates...
│ 2020-08-20T06:37:09.188Z bridge-server info no tcpConnectors defined
│ 2020-08-20T06:37:09.188Z bridge-server info checking for org.apache.qpid.dispatch.router.tcpConnector updates...
│ 2020-08-20T06:37:09.188Z bridge-server info no tcpListeners defined
│ 2020-08-20T06:37:09.189Z bridge-server info checking for org.apache.qpid.dispatch.router.tcpListener updates...
│ [connection-1] disconnected Error: connect ECONNREFUSED 127.0.0.1:5672 service-controller logs
│ E0820 06:39:13.650199 1 site_query.go:87] Failed to create amqp connection for site query server: x509: certificate signed by unknown authority
│ 2020/08/20 06:39:14 Establishing connection to skupper-messaging service for service sync
│ E0820 06:39:14.560062 1 service_sync.go:209] Failed to create amqp connection x509: certificate signed by unknown authority router-controller
│ 2020-08-20 06:37:34.331489 +0000 SERVER (info) [C39] Accepted connection to 0.0.0.0:5671 from 10.42.0.42:40814
│ 2020-08-20 06:37:34.335604 +0000 SERVER (info) [C39] Connection from 10.42.0.42:40814 (to 0.0.0.0:5671) failed: amqp:connection:framing-error SSL Failure: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
│ 2020-08-20 06:37:35.320841 +0000 SERVER (info) [C40] Accepted connection to 0.0.0.0:5671 from 10.42.0.42:40824
│ 2020-08-20 06:37:35.324861 +0000 SERVER (info) [C40] Connection from 10.42.0.42:40824 (to 0.0.0.0:5671) failed: amqp:connection:framing-error SSL Failure: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificatesite-controller
2020/08/20 07:11:32 Skupper site controller watching all namespaces
│ 2020/08/20 07:11:32 Starting the Skupper site controller informers
│ 2020/08/20 07:11:32 Waiting for informer caches to sync
│ 2020/08/20 07:11:32 Starting workers
│ 2020/08/20 07:11:32 Started workers
│ 2020/08/20 07:11:32 Initialising skupper site ...
│ Secret already exists: skupper
│ 2020/08/20 07:11:35 Skupper site initialised
│ 2020/08/20 07:11:35 Checking tokens...
│ 2020/08/20 07:11:35 Checking token requests...
│ 2020/08/20 07:11:35 Done.Development of the cli has now moved to the skupper repo (not skupper-cli) so releases are there: https://github.com/skupperproject/skupper/releases.
However I think I see the issue you have. The token from the 0.2.0 client has the name 'skupper' and by having this applied before the site controller is created, it prevents the site controller writing the expected secret for internal communication (hence the ssl errors in the logs you are seeing).
So, for now you need to delete the 'skupper' secret and 'skupper-site' configmap, then recreate the configmap first. You then need to edit the token to change the name from 'skupper' to something else before you apply it. That should then hopefully work.
Kampe
commented
4 years ago Okay excellent, in reference to the secret that needs to be renamed and applied (from skupper -> skupper-connection-certs), should I expect to put a specific annotation on said secret (skupper-connection-certs) for skupper to know this is the renamed secret to utilize as its connection token?
Kampe
commented
4 years ago Aha, figured that out - can I have this secret exist before the site-controller stands up?
secret does need
labels:
skupper.io/type: connection-tokenedit: yes - yes you can
got it all working! this is great!
Thanks Ted!
grs
commented
4 years ago The token should already have the label on it when generated. Providing the name doesn't clash, then yes the secret can be applied before the site controller is deployed.
(The problem is that at present the site-controller assumes that the secret called 'skupper' is previously generated certs for the local site, but the default token name from 0.2.0 will also be 'skupper')
grs
commented
4 years ago Great, glad it is working! (Will try to ensure we handle the clash better in future).
Kampe
commented
4 years ago Well - may have spoke a little prematurely! I do have my services skuppered into my other cluster - however my connections all just fail when attempting to utilize the services.
output from skupper list-exposed
cloud:
Services exposed through Skupper:
global-api (tcp port 5443) with targets
=> app=global-api,application=skupper-router,skupper.io/component=router name=global-api
global-service (tcp port 8080) with targets
=> app=global-service,application=skupper-router,skupper.io/component=router name=global-serviceedge:
Services exposed through Skupper:
global-service (tcp port 8080)
global-api (tcp port 5443)****Where's a good place to look to diagnose this issue?
Here's my cloud router logs:
│ 2020-08-20 21:24:08.271312 +0000 SERVER (info) [C5] Accepted connection to 0.0.0.0:45671 from 127.0.0.1:60830
│ 2020-08-20 21:24:08.465163 +0000 ROUTER (info) [C5] Connection Opened: dir=in host=127.0.0.1:60830 vhost= encrypted=TLSv1.3 auth=EXTERNAL user=CN=skupper container_id=skupp3r-edge-site-skupper-router-796464b4d5-clg95 props={:product="qpid-dispatch-router", :version="1.13.0", :"qd.conn-id"=1}
│ 2020-08-20 21:24:08.523461 +0000 ROUTER_CORE (info) [C5][L19] Link attached: dir=in source={<none> expire:sess} target={<none> expire:sess}
│ 2020-08-20 21:24:08.523666 +0000 ROUTER_CORE (info) [C5][L20] Link attached: dir=out source={skupp3r-edge-site-skupper-router-796464b4d5-clg95 expire:link caps::"qd.router-edge-downlink"} target={<none> expire:sess caps::"qd.router-edge-downlink"}
│ 2020-08-20 21:24:08.523821 +0000 ROUTER_CORE (info) [C5][L21] Link attached: dir=out source={_$qd.edge_addr_tracking expire:link} target={<none> expire:sess}
│ 2020-08-20 21:24:08.523875 +0000 ROUTER_CORE (info) [C5][L22] Link attached: dir=in source={<none> expire:sess} target={$management expire:link}
│ 2020-08-20 21:24:08.523996 +0000 ROUTER_CORE (info) [C5][L23] Link attached: dir=out source={<dynamic> expire:link} target={<none> expire:sess}
│ 2020-08-20 21:24:08.524043 +0000 ROUTER_CORE (info) [C5][L24] Link attached: dir=in source={<none> expire:sess} target={_$qd.addr_lookup expire:link}
│ 2020-08-20 21:24:08.524129 +0000 ROUTER_CORE (info) [C5][L25] Link attached: dir=out source={<dynamic> expire:link} target={<none> expire:sess}
│ 2020-08-20 21:25:25.375441 +0000 ROUTER_CORE (info) [C5][L26] Link attached: dir=out source={skupp3r-edge-site-skupper-router-796464b4d5-clg95/bridge-server/$management expire:link} target={<none> expire:sess}
│ 2020-08-20 21:25:34.291827 +0000 ROUTER_CORE (info) [C5][L27] Link attached: dir=out source={8d1bcd80-66f2-4638-9c87-3256f739856b/skupper-site-query expire:link} target={<none> expire:sess}
│ 2020-08-20 21:25:34.291894 +0000 ROUTER_CORE (info) [C5][L28] Link attached: dir=out source={mc/$skupper-service-sync expire:link} target={<none> expire:sess}
│ 2020-08-20 21:25:34.351926 +0000 ROUTER_CORE (info) [C5][L29] Link attached: dir=in source={<none> expire:sess} target={mc/$skupper-service-sync expire:link}
│ 2020-08-20 21:29:51.910421 +0000 ROUTER_CORE (info) [C5][L30] Link attached: dir=in source={<none> expire:sess} target={global-api expire:link}
│ 2020-08-20 21:30:11.946477 +0000 ROUTER_CORE (info) [C5][L30] Link blocked with zero credit for 21 seconds
│ 2020-08-20 21:50:52.483896 +0000 ROUTER_CORE (info) [C2][L6] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-20 21:50:52.484023 +0000 ROUTER_CORE (info) [C5][L28] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-20 21:50:52.877490 +0000 SERVER (info) [C5] Connection from 127.0.0.1:60830 (to 0.0.0.0:45671) failed: amqp:resource-limit-exceeded local-idle-timeout expired
│ 2020-08-20 21:50:52.877789 +0000 ROUTER_CORE (info) [C5][L19] Link lost: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
│ 2020-08-20 21:50:52.877844 +0000 ROUTER_CORE (info) [C5][L20] Link lost: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
│ 2020-08-20 21:50:52.877958 +0000 ROUTER_CORE (info) [C5][L22] Link lost: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
│ 2020-08-20 21:50:52.877984 +0000 ROUTER_CORE (info) [C5][L23] Link lost: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
│ 2020-08-20 21:50:52.878055 +0000 ROUTER_CORE (info) [C5][L25] Link lost: del=40 presett=40 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
│ 2020-08-20 21:50:52.878082 +0000 ROUTER_CORE (info) [C5][L26] Link lost: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
│ 2020-08-20 21:50:52.878096 +0000 ROUTER_CORE (info) [C5][L27] Link lost: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
│ 2020-08-20 21:50:52.878174 +0000 ROUTER_CORE (info) [C5][L28] Link lost: del=302 presett=0 psdrop=0 acc=301 rej=0 rel=0 mod=1 delay1=0 delay10=1 blocked=no
│ 2020-08-20 21:50:52.878245 +0000 ROUTER_CORE (info) [C5][L29] Link lost: del=300 presett=0 psdrop=0 acc=300 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
│ 2020-08-20 21:50:52.878308 +0000 ROUTER_CORE (info) [C5][L30] Link lost: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=yes
│ 2020-08-20 21:50:52.878473 +0000 ROUTER_CORE (info) [C5][L21] Link closed due to connection loss: del=1 presett=1 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=1 blocked=no
│ 2020-08-20 21:50:52.878508 +0000 ROUTER_CORE (info) [C5][L24] Link closed due to connection loss: del=40 presett=40 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
│ 2020-08-20 21:50:52.878518 +0000 ROUTER_CORE (info) [C5] Connection Closed[C5] Connection from 127.0.0.1:60830 (to 0.0.0.0:45671) failed: amqp:resource-limit-exceeded local-idle-timeout expired
That suggests that the router connection from the edge site was lost, due to the edge site router not sending a heartbeat. That in turn would suggest either some connectivity issue or a scheduling/cpu issue with the edge router. What do the edge router logs look like?
Also if you run qdstat -n and qdstat -c and qdstat -l against the router pod in each site that will give some useful information. (We have an issue open to simplify the collection of all this, so apologies for the many steps at this stage!).
Kampe
commented
4 years ago Interesting! I'll poke around some more with qdstat, here's the edge router's logs:
2020-08-21 16:42:45.524035 +0000 ROUTER (info) Router started in Edge mode
2020-08-21 16:42:45.524060 +0000 ROUTER (info) Version: 1.13.0
2020-08-21 16:42:45.562336 +0000 ROUTER_CORE (info) Core module enabled: edge_router
2020-08-21 16:42:45.562389 +0000 ROUTER_CORE (info) Core module present but disabled: core_test_hooks
2020-08-21 16:42:45.562400 +0000 ROUTER_CORE (info) Core module present but disabled: edge_addr_tracking
2020-08-21 16:42:45.562406 +0000 ROUTER_CORE (info) Core module present but disabled: address_lookup_server
2020-08-21 16:42:45.562414 +0000 ROUTER_CORE (info) Core module enabled: address_lookup_client
2020-08-21 16:42:45.562458 +0000 ROUTER_CORE (info) Stuck delivery detection: Scan interval: 30 seconds, Delivery age threshold: 10 seconds
2020-08-21 16:42:45.562467 +0000 ROUTER_CORE (info) Core module enabled: stuck_delivery_detection
2020-08-21 16:42:45.562474 +0000 ROUTER_CORE (info) Core module present but disabled: mobile_sync
2020-08-21 16:42:45.562481 +0000 ROUTER_CORE (info) Streaming link scrubber: Scan interval: 30 seconds, max free pool: 128 links
2020-08-21 16:42:45.562487 +0000 ROUTER_CORE (info) Core module enabled: streaming_link_scruber
2020-08-21 16:42:45.562494 +0000 ROUTER_CORE (info) Router Core thread running. 0/skupp3r-edge-site-skupper-router-dbb7b67d9-6ml7j
2020-08-21 16:42:45.562503 +0000 ROUTER_CORE (info) In-process subscription M/$management
2020-08-21 16:42:45.562529 +0000 ROUTER_CORE (info) In-process subscription L/$management
2020-08-21 16:42:45.574334 +0000 AGENT (info) Activating management agent on $_management_internal
2020-08-21 16:42:45.575751 +0000 ROUTER_CORE (info) In-process subscription L/$_management_internal
2020-08-21 16:42:45.578551 +0000 POLICY (info) Policy configured maxConnections: 65535, policyDir: '',access rules enabled: 'false', use hostname patterns: 'false'
2020-08-21 16:42:45.579787 +0000 POLICY (info) Policy fallback defaultVhost is defined: '$default'
2020-08-21 16:42:45.586871 +0000 CONN_MGR (info) Created SSL Profile with name skupper-amqps
2020-08-21 16:42:45.589239 +0000 CONN_MGR (info) Created SSL Profile with name skupper-cloud-certificates-profile
2020-08-21 16:42:45.592093 +0000 CONN_MGR (info) Configured Listener: localhost:5672 proto=any, role=normal
2020-08-21 16:42:45.596553 +0000 CONN_MGR (info) Configured Listener: 0.0.0.0:5671 proto=any, role=normal, sslProfile=skupper-amqps
2020-08-21 16:42:45.606714 +0000 CONN_MGR (info) Configured Listener: 0.0.0.0:8080 proto=any, role=normal, http
2020-08-21 16:42:45.610316 +0000 SERVER (info) HTTP server thread running
2020-08-21 16:42:45.610622 +0000 SERVER (notice) Listening for HTTP on 0.0.0.0:8080
2020-08-21 16:42:45.622384 +0000 CONN_MGR (info) Configured Listener: 0.0.0.0:9090 proto=any, role=normal, http
2020-08-21 16:42:45.622784 +0000 SERVER (notice) Listening for HTTP on 0.0.0.0:9090
2020-08-21 16:42:45.625619 +0000 CONN_MGR (info) Configured Connector: xx.xx.xx.xx:45671 proto=any, role=edge, sslProfile=skupper-cloud-certificates-profile
2020-08-21 16:42:45.630699 +0000 SERVER (notice) Operational, 4 Threads Running (process ID 1)
2020-08-21 16:42:45.630937 +0000 SERVER (notice) Process VmSize 186.89 MiB (3.84 GiB available memory)
2020-08-21 16:42:45.631177 +0000 SERVER (notice) Listening on localhost:5672
2020-08-21 16:42:45.631344 +0000 SERVER (notice) Listening on 0.0.0.0:5671
2020-08-21 16:42:45.968952 +0000 ROUTER (info) [C1] Connection Opened: dir=out host=xx.xx.xx.xx:45671 vhost= encrypted=TLSv1.3 auth=EXTERNAL user=(null) container_id=skupp3r-cloud-skupper-router-69f98b59f7-5k876 props={:product="qpid-dispatch-router", :version="1.13.0", :"qd.conn-id"=25}
2020-08-21 16:42:45.969317 +0000 ROUTER_CORE (info) Edge connection (id=1) to interior established
2020-08-21 16:42:45.969564 +0000 ROUTER_CORE (info) [C1][L2] Link attached: dir=out source={<none> expire:link} target={<none> expire:link}
2020-08-21 16:42:45.969658 +0000 ROUTER_CORE (info) [C1][L3] Link attached: dir=in source={skupp3r-edge-site-skupper-router-dbb7b67d9-6ml7j expire:link caps::"qd.router-edge-downlink"} target={<none> expire:link caps::"qd.router-edge-downlink"}
2020-08-21 16:42:45.969695 +0000 ROUTER_CORE (info) [C1][L4] Link attached: dir=in source={_$qd.edge_addr_tracking expire:link} target={<none> expire:link}
2020-08-21 16:42:45.969743 +0000 ROUTER_CORE (info) [C1][L5] Link attached: dir=out source={} target={$management expire:link}
2020-08-21 16:42:45.969754 +0000 ROUTER_CORE (info) [C1][L6] Link attached: dir=in source={<dynamic> expire:link} target={}
2020-08-21 16:42:45.969769 +0000 ROUTER_CORE (info) [C1][L7] Link attached: dir=out source={} target={_$qd.addr_lookup expire:link}
2020-08-21 16:42:45.969873 +0000 ROUTER_CORE (info) [C1][L8] Link attached: dir=in source={<dynamic> expire:link} target={}
2020-08-21 16:44:34.604155 +0000 SERVER (info) [C2] Accepted connection to localhost:5672 from 127.0.0.1:36618
2020-08-21 16:44:34.648853 +0000 ROUTER (info) [C2] Connection Opened: dir=in host=127.0.0.1:36618 vhost= encrypted=no auth=ANONYMOUS user=anonymous container_id=bridge-server-skupper-router-dbb7b67d9-6ml7j props=
2020-08-21 16:44:34.679338 +0000 ROUTER_CORE (info) [C2][L9] Link attached: dir=out source={skupp3r-edge-site-skupper-router-dbb7b67d9-6ml7j/bridge-server/$management expire:sess} target={<none> expire:sess}
2020-08-21 16:44:34.743693 +0000 ROUTER_CORE (info) [C1][L10] Link attached: dir=in source={skupp3r-edge-site-skupper-router-dbb7b67d9-6ml7j/bridge-server/$management expire:link} target={<none> expire:link}
2020-08-21 16:44:42.438130 +0000 SERVER (info) [C3] Accepted connection to 0.0.0.0:5671 from 10.42.0.32:46118
2020-08-21 16:44:42.448491 +0000 SERVER (info) [C4] Accepted connection to 0.0.0.0:5671 from 10.42.0.32:46116
2020-08-21 16:44:42.462754 +0000 ROUTER (info) [C3] Connection Opened: dir=in host=10.42.0.32:46118 vhost= encrypted=TLSv1.3 auth=EXTERNAL user=CN=skupper-messaging container_id=JbBgJodC6yRbF_meVkQrWq-AI8ri4Oe8v8Qfsh4LdZ2Wtp49q7G2uw props=
2020-08-21 16:44:42.468054 +0000 ROUTER_CORE (info) [C3][L11] Link attached: dir=out source={mc/$skupper-service-sync expire:sess} target={<none> expire:sess}
2020-08-21 16:44:42.469607 +0000 ROUTER (info) [C4] Connection Opened: dir=in host=10.42.0.32:46116 vhost= encrypted=TLSv1.3 auth=EXTERNAL user=CN=skupper-messaging container_id=pDx97KDwZ-_U2sj_6t2vEGCFZUhROtJRgkrhYd82j2XxYg-wz4zlIw props=
2020-08-21 16:44:42.472305 +0000 ROUTER_CORE (info) [C4][L12] Link attached: dir=out source={2fd785b5-fca6-4e10-8634-1aefb2ad57ff/skupper-site-query expire:sess} target={<none> expire:sess}
2020-08-21 16:44:42.524422 +0000 ROUTER_CORE (info) [C1][L13] Link attached: dir=in source={mc/$skupper-service-sync expire:link} target={<none> expire:link}
2020-08-21 16:44:42.528724 +0000 ROUTER_CORE (info) [C1][L14] Link attached: dir=in source={2fd785b5-fca6-4e10-8634-1aefb2ad57ff/skupper-site-query expire:link} target={<none> expire:link}
2020-08-21 16:44:42.532000 +0000 ROUTER_CORE (info) [C3][L15] Link attached: dir=in source={<none> expire:sess} target={mc/$skupper-service-sync expire:sess}
2020-08-21 16:44:42.532097 +0000 ROUTER_CORE (info) [C4][L16] Link attached: dir=in source={<none> expire:sess} target={<none> expire:sess}
2020-08-21 16:44:42.586261 +0000 ROUTER_CORE (info) [C1][L17] Link attached: dir=out source={<none> expire:link} target={mc/$skupper-service-sync expire:link}
2020-08-21 16:45:44.131170 +0000 SERVER (info) [C5] Accepted connection to localhost:5672 from 127.0.0.1:38174
2020-08-21 16:45:44.137893 +0000 ROUTER (info) [C5] Connection Opened: dir=in host=127.0.0.1:38174 vhost= encrypted=no auth=ANONYMOUS user=anonymous container_id=skupper-router-dbb7b67d9-6ml7j_http_1025_to_amqp_global-api props=
2020-08-21 16:45:44.141923 +0000 ROUTER_CORE (info) [C5][L18] Link attached: dir=in source={<none> expire:sess} target={global-api expire:sess}
2020-08-21 16:45:44.144363 +0000 ROUTER_CORE (info) [C5][L19] Link attached: dir=out source={<dynamic> expire:sess} target={<none> expire:sess}
2020-08-21 16:45:44.202894 +0000 ROUTER_CORE (info) [C1][L20] Link attached: dir=out source={<none> expire:link} target={global-api expire:link}
2020-08-21 16:49:58.099344 +0000 SERVER (info) [C6] Accepted connection to localhost:5672 from 127.0.0.1:46378
2020-08-21 16:49:58.100987 +0000 ROUTER (info) [C6] Connection Opened: dir=in host=127.0.0.1:46378 vhost= encrypted=no auth=no user=anonymous container_id=3abc7a25-4f34-4f16-97f7-a6052c221d89 props=
2020-08-21 16:49:58.105822 +0000 ROUTER_CORE (info) [C6][L21] Link attached: dir=in source={<none> expire:sess} target={$management expire:sess}
2020-08-21 16:49:58.165970 +0000 ROUTER_CORE (info) [C6][L22] Link attached: dir=out source={<dynamic> expire:sess} target={<none> expire:sess}
2020-08-21 16:49:58.198153 +0000 SERVER (info) [C6] Connection from 127.0.0.1:46378 (to localhost:5672) failed: amqp:connection:framing-error connection aborted
2020-08-21 16:49:58.198543 +0000 ROUTER_CORE (info) [C6][L21] Link lost: del=1 presett=0 psdrop=0 acc=1 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-21 16:49:58.198763 +0000 ROUTER_CORE (info) [C6][L22] Link lost: del=1 presett=1 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-21 16:49:58.198839 +0000 ROUTER_CORE (info) [C6] Connection ClosedIn that edge log (which is for a different day) there does not appear to be any loss of connection to the other site. Are you still seeing failure to communicate?
Kampe
commented
4 years ago Yeah different day there, new cluster too - same results though unfortunately - eventually leading to
│ 2020-08-21 16:45:44.131170 +0000 SERVER (info) [C5] Accepted connection to localhost:5672 from 127.0.0.1:38174
│ 2020-08-21 16:45:44.137893 +0000 ROUTER (info) [C5] Connection Opened: dir=in host=127.0.0.1:38174 vhost= encrypted=no auth=ANONYMOUS user=anonymous container_id=skupper-router-dbb7b67d9-6ml7j_http_1025_to_amqp_global-api props=
│ 2020-08-21 16:45:44.141923 +0000 ROUTER_CORE (info) [C5][L18] Link attached: dir=in source={<none> expire:sess} target={global-api expire:sess}
│ 2020-08-21 16:45:44.144363 +0000 ROUTER_CORE (info) [C5][L19] Link attached: dir=out source={<dynamic> expire:sess} target={<none> expire:sess}
│ 2020-08-21 16:45:44.202894 +0000 ROUTER_CORE (info) [C1][L20] Link attached: dir=out source={<none> expire:link} target={global-api expire:link}
│ 2020-08-21 16:49:58.099344 +0000 SERVER (info) [C6] Accepted connection to localhost:5672 from 127.0.0.1:46378
│ 2020-08-21 16:49:58.100987 +0000 ROUTER (info) [C6] Connection Opened: dir=in host=127.0.0.1:46378 vhost= encrypted=no auth=no user=anonymous container_id=3abc7a25-4f34-4f16-97f7-a6052c221d89 props=
│ 2020-08-21 16:49:58.105822 +0000 ROUTER_CORE (info) [C6][L21] Link attached: dir=in source={<none> expire:sess} target={$management expire:sess}
│ 2020-08-21 16:49:58.165970 +0000 ROUTER_CORE (info) [C6][L22] Link attached: dir=out source={<dynamic> expire:sess} target={<none> expire:sess}
│ 2020-08-21 16:49:58.198153 +0000 SERVER (info) [C6] Connection from 127.0.0.1:46378 (to localhost:5672) failed: amqp:connection:framing-error connection aborted
│ 2020-08-21 16:49:58.198543 +0000 ROUTER_CORE (info) [C6][L21] Link lost: del=1 presett=0 psdrop=0 acc=1 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
│ 2020-08-21 16:49:58.198763 +0000 ROUTER_CORE (info) [C6][L22] Link lost: del=1 presett=1 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
│ 2020-08-21 16:49:58.198839 +0000 ROUTER_CORE (info) [C6] Connection Closed
│ 2020-08-21 16:49:58.578386 +0000 SERVER (info) [C7] Accepted connection to localhost:5672 from 127.0.0.1:46396
│ 2020-08-21 16:49:58.578468 +0000 ROUTER (info) [C7] Connection Opened: dir=in host=127.0.0.1:46396 vhost= encrypted=no auth=no user=anonymous container_id=ce882158-de88-4079-8d0d-fec87315dffb props=
│ 2020-08-21 16:49:58.580027 +0000 ROUTER_CORE (info) [C7][L23] Link attached: dir=in source={<none> expire:sess} target={_topo/0/skupp3r-cloud-skupper-router-69f98b59f7-5k876/$management expire:sess}
│ 2020-08-21 16:49:58.651485 +0000 ROUTER_CORE (info) [C7][L24] Link attached: dir=out source={<dynamic> expire:sess} target={<none> expire:sess}
│ 2020-08-21 16:49:58.752659 +0000 SERVER (info) [C7] Connection from 127.0.0.1:46396 (to localhost:5672) failed: amqp:connection:framing-error connection aborted
│ 2020-08-21 16:49:58.756237 +0000 ROUTER_CORE (info) [C7][L23] Link lost: del=1 presett=0 psdrop=0 acc=1 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
│ 2020-08-21 16:49:58.756367 +0000 ROUTER_CORE (info) [C7][L24] Link lost: del=1 presett=1 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
│ 2020-08-21 16:49:58.756404 +0000 ROUTER_CORE (info) [C7] Connection Closed
│ 2020-08-21 16:49:59.064811 +0000 SERVER (info) [C8] Accepted connection to localhost:5672 from 127.0.0.1:46422
│ 2020-08-21 16:49:59.066424 +0000 ROUTER (info) [C8] Connection Opened: dir=in host=127.0.0.1:46422 vhost= encrypted=no auth=no user=anonymous container_id=1a470db4-96f7-456d-895f-4cc7e464b026 props=
│ 2020-08-21 16:49:59.069828 +0000 ROUTER_CORE (info) [C8][L25] Link attached: dir=in source={<none> expire:sess} target={_topo/0/skupp3r-cloud-skupper-router-69f98b59f7-5k876/$management expire:sess}
│ 2020-08-21 16:49:59.128545 +0000 ROUTER_CORE (info) [C8][L26] Link attached: dir=out source={<dynamic> expire:sess} target={<none> expire:sess}
│ 2020-08-21 16:49:59.210903 +0000 SERVER (info) [C8] Connection from 127.0.0.1:46422 (to localhost:5672) failed: amqp:connection:framing-error connection aborted
│ 2020-08-21 16:49:59.211311 +0000 ROUTER_CORE (info) [C8][L25] Link lost: del=1 presett=0 psdrop=0 acc=1 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
│ 2020-08-21 16:49:59.211433 +0000 ROUTER_CORE (info) [C8][L26] Link lost: del=1 presett=1 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
│ 2020-08-21 16:49:59.211625 +0000 ROUTER_CORE (info) [C8] Connection Closed
│ 2020-08-21 16:51:33.188429 +0000 ROUTER_CORE (info) [C5][L18] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-21 16:51:33.188502 +0000 ROUTER_CORE (info) [C1][L20] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-21 16:52:35.273804 +0000 ROUTER_CORE (info) [C5][L18] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-21 16:52:35.274199 +0000 ROUTER_CORE (info) [C1][L20] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-21 16:53:37.347489 +0000 ROUTER_CORE (info) [C5][L18] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-21 16:53:37.347551 +0000 ROUTER_CORE (info) [C1][L20] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-21 16:54:39.412803 +0000 ROUTER_CORE (info) [C5][L18] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-21 16:54:39.413340 +0000 ROUTER_CORE (info) [C1][L20] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-21 16:55:41.459286 +0000 ROUTER_CORE (info) [C5][L18] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-21 16:55:41.461036 +0000 ROUTER_CORE (info) [C1][L20] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-21 16:56:43.510436 +0000 ROUTER_CORE (info) [C5][L18] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-21 16:56:43.512683 +0000 ROUTER_CORE (info) [C1][L20] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-21 16:59:49.715080 +0000 ROUTER_CORE (info) [C5][L18] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-21 16:59:49.715925 +0000 ROUTER_CORE (info) [C1][L20] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-21 17:00:51.807536 +0000 ROUTER_CORE (info) [C5][L18] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-21 17:00:51.807690 +0000 ROUTER_CORE (info) [C1][L20] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-21 19:15:20.397860 +0000 ROUTER_CORE (info) [C5][L18] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-21 19:15:20.398136 +0000 ROUTER_CORE (info) [C1][L20] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 secondshere's the outputs of qdstat
Cloud:
[root@skupper-router-845f7bdf87-r56dn bin]# qdstat -n
2020-08-21 19:26:41.361765 UTC
skupp3r-cloud-skupper-router-845f7bdf87-r56dn
Routers in the Network
router-id next-hop link
===============================================================
skupp3r-cloud-skupper-router-845f7bdf87-r56dn (self) -
[root@skupper-router-845f7bdf87-r56dn bin]# qdstat -c
2020-08-21 19:26:44.348823 UTC
skupp3r-cloud-skupper-router-845f7bdf87-r56dn
Connections
id host container role dir security authentication tenant last dlv uptime
============================================================================================================================================================================================
4 127.0.0.1:42632 bridge-server-skupper-router-845f7bdf87-r56dn normal in no-security anonymous-user - 000:00:08:58
8 127.0.0.1:50888 4zoY8m_38LQaXO_6JQg5zn0-XjJ7L2JR23H_UF-EFQSDvZL8UPywTA normal in TLSv1.3(TLS_AES_128_GCM_SHA256) CN=skupper-messaging(x.509) - 000:00:08:02
9 127.0.0.1:50890 awooH5uVaBx_qVubFFANT4OMhVx-S2ExHDO0jM8iQ9v6xXKSG88caQ normal in TLSv1.3(TLS_AES_128_GCM_SHA256) CN=skupper-messaging(x.509) 000:00:00:01 000:00:08:02
10 127.0.0.1:54296 skupp3r-edge-site-skupper-router-dbb7b67d9-6ml7j edge in TLSv1.3(TLS_AES_256_GCM_SHA384) CN=skupper(x.509) 000:00:00:01 000:00:08:01
12 127.0.0.1:48022 b851767f-47b3-4aeb-aba9-14e6bfff58ee normal in no-security no-auth 000:00:00:00 000:00:00:00
[root@skupper-router-845f7bdf87-r56dn bin]# qdstat -l
2020-08-21 19:26:47.924876 UTC
skupp3r-cloud-skupper-router-845f7bdf87-r56dn
Router Links
type dir conn id id peer class addr phs cap pri undel unsett deliv presett psdrop acc rej rel mod delay rate stuck cred blkd
=====================================================================================================================================================================================================================================
endpoint out 4 6 mobile skupp3r-cloud-skupper-router-845f7bdf87-r56dn/bridge-server/$management 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 1000 -
endpoint out 8 25 mobile e6014882-211a-4aa3-acd9-7049c75ec4d3/skupper-site-query 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 10 -
endpoint in 8 26 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 9 27 mobile mc/$skupper-service-sync 0 250 0 0 0 194 0 0 194 0 0 0 0 0 0 6 -
endpoint in 9 28 mobile mc/$skupper-service-sync 0 250 0 0 0 97 0 0 97 0 0 0 0 0 0 250 -
endpoint in 10 29 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
edge-downlink out 10 30 edge skupp3r-edge-site-skupper-router-dbb7b67d9-6ml7j 250 0 0 0 2 0 0 2 0 0 0 0 0 0 250 -
endpoint out 10 31 mobile _$qd.edge_addr_tracking 0 250 0 0 0 4 4 0 0 0 0 0 4 0 0 32 -
endpoint out 10 32 mobile skupp3r-edge-site-skupper-router-dbb7b67d9-6ml7j/bridge-server/$management 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 10 33 mobile mc/$skupper-service-sync 0 250 0 0 0 97 0 0 97 0 0 0 0 0 0 250 -
endpoint in 10 34 mobile mc/$skupper-service-sync 0 250 0 0 0 97 0 0 97 0 0 0 0 0 0 250 -
endpoint out 10 35 mobile 2fd785b5-fca6-4e10-8634-1aefb2ad57ff/skupper-site-query 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint in 10 37 mobile $management 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 10 38 local temp.nXvOriTZ3kP9Wr7 250 0 0 0 0 0 0 0 0 0 0 0 0 0 100 -
endpoint in 10 39 mobile _$qd.addr_lookup 0 250 0 0 0 4 4 0 0 0 0 0 0 0 0 32 -
endpoint out 10 40 local temp.nwHneexLuRwBAoV 250 0 0 0 4 4 0 0 0 0 0 0 0 0 250 -
endpoint in 10 41 mobile global-api 0 250 0 0 0 1 0 0 1 0 0 0 0 0 0 250 -
endpoint in 13 46 mobile $management 0 250 0 0 0 2 0 0 2 0 0 0 0 0 0 250 -
endpoint out 13 47 local temp.jIqI7iVIGWkJljx 250 0 0 0 1 1 0 0 0 0 0 0 0 0 1 -Edge:
[root@skupper-router-dbb7b67d9-6ml7j bin]# qdstat -n
2020-08-21 19:24:29.678192 UTC
skupp3r-edge-site-skupper-router-dbb7b67d9-6ml7j
Router is Standalone - No Router List
[root@skupper-router-dbb7b67d9-6ml7j bin]# qdstat -c
2020-08-21 19:24:37.942228 UTC
skupp3r-edge-site-skupper-router-dbb7b67d9-6ml7j
Connections
id host container role dir security authentication tenant last dlv uptime
========================================================================================================================================================================================================
2 127.0.0.1:36618 bridge-server-skupper-router-dbb7b67d9-6ml7j normal in no-security anonymous-user - 000:02:39:54
3 10.42.0.32:46118 JbBgJodC6yRbF_meVkQrWq-AI8ri4Oe8v8Qfsh4LdZ2Wtp49q7G2uw normal in TLSv1.3(TLS_AES_128_GCM_SHA256) CN=skupper-messaging(x.509) 000:00:00:01 000:02:39:46
4 10.42.0.32:46116 pDx97KDwZ-_U2sj_6t2vEGCFZUhROtJRgkrhYd82j2XxYg-wz4zlIw normal in TLSv1.3(TLS_AES_128_GCM_SHA256) CN=skupper-messaging(x.509) - 000:02:39:46
10 xx.xx.xx.xx:45671 skupp3r-cloud-skupper-router-845f7bdf87-r56dn edge out TLSv1.3(TLS_AES_256_GCM_SHA384) x.509 000:00:00:00 000:00:05:55
11 127.0.0.1:57544 skupper-router-dbb7b67d9-6ml7j_tcp_1025_to_amqp_global-api normal in no-security anonymous-user 000:00:05:46 000:00:05:46
13 127.0.0.1:36320 cca911a7-4b51-4774-8bff-5c3b74f45d1b normal in no-security no-auth 000:00:00:00 000:00:00:01
[root@skupper-router-dbb7b67d9-6ml7j bin]# qdstat -l
2020-08-21 19:24:44.567518 UTC
skupp3r-edge-site-skupper-router-dbb7b67d9-6ml7j
Router Links
type dir conn id id peer class addr phs cap pri undel unsett deliv presett psdrop acc rej rel mod delay rate stuck cred blkd
=================================================================================================================================================================================================================================
endpoint out 2 9 mobile skupp3r-edge-site-skupper-router-dbb7b67d9-6ml7j/bridge-server/$management 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 1000 -
endpoint out 3 11 mobile mc/$skupper-service-sync 0 250 0 0 0 3839 0 0 3839 0 0 0 0 0 0 6 -
endpoint out 4 12 mobile 2fd785b5-fca6-4e10-8634-1aefb2ad57ff/skupper-site-query 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 10 -
endpoint in 3 15 mobile mc/$skupper-service-sync 0 250 0 0 0 1920 0 0 1920 0 0 0 0 0 0 250 -
endpoint in 4 16 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 10 39 local _edge 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint in 10 40 250 0 0 0 2 0 0 2 0 0 0 0 0 0 250 -
endpoint in 10 41 250 0 0 0 4 4 0 0 0 0 0 0 0 0 32 -
endpoint in 10 42 mobile skupp3r-edge-site-skupper-router-dbb7b67d9-6ml7j/bridge-server/$management 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint in 10 43 mobile mc/$skupper-service-sync 0 250 0 0 0 72 0 0 72 0 0 0 0 0 0 250 -
endpoint out 10 44 mobile mc/$skupper-service-sync 0 250 0 0 0 72 0 0 72 0 0 0 0 0 0 250 -
endpoint in 10 45 mobile 2fd785b5-fca6-4e10-8634-1aefb2ad57ff/skupper-site-query 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 10 47 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint in 10 48 250 0 0 0 0 0 0 0 0 0 0 0 0 0 100 -
endpoint out 10 49 250 0 0 0 4 4 0 0 0 0 0 4 0 0 32 -
endpoint in 10 50 250 0 0 0 4 4 0 0 0 0 0 0 0 0 250 -
endpoint out 11 51 local temp.jQ9zu69x8cVZ79k 250 0 0 0 1 0 0 1 0 0 0 0 0 0 999 -
endpoint in 11 52 mobile global-api 0 250 0 0 0 1 0 0 1 0 0 0 0 0 0 250 -
endpoint out 10 53 250 0 0 0 1 0 0 1 0 0 0 0 0 0 250 -
endpoint in 14 58 mobile $management 0 250 0 0 0 2 0 0 2 0 0 0 0 0 0 250 -
endpoint out 14 59 local temp.0bSAv+BWA3mCZb2 250 0 0 0 1 1 0 0 0 0 0 0 0 0 1 -The routers are connected this time. Could you do one more query, qdstat --linkroute against each router?
Kampe
commented
4 years ago Aye aye can do
Cloud:
[root@skupper-router-845f7bdf87-r56dn bin]# qdstat --linkroute
2020-08-21 20:22:04.342959 UTC
skupp3r-cloud-skupper-router-845f7bdf87-r56dn
Link Routes
No Link Routes foundEdge:
[root@skupper-router-dbb7b67d9-6ml7j bin]# qdstat --linkroute
2020-08-21 20:21:12.501948 UTC
skupp3r-edge-site-skupper-router-dbb7b67d9-6ml7j
Link Routes
No Link Routes foundAh didn't notice I'm seeing these in edge bridge server - may be useful
│ 2020-08-21T19:17:47.879Z bridge-server error Cannot find correlation id for failed delivery
│ Could not deliver request
│ (node:1) UnhandledPromiseRejectionWarning: ReferenceError: site is not defined
│ at /opt/app-root/lib/bridges.js:181:120
│ at process._tickCallback (internal/process/next_tick.js:68:7)
│ (node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
│ (node:1) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
│ Could not deliver request
│ (node:1) UnhandledPromiseRejectionWarning: ReferenceError: site is not defined
│ at /opt/app-root/lib/bridges.js:181:120
│ at process._tickCallback (internal/process/next_tick.js:68:7)
│ (node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
│ 2020-08-21T19:17:54.651Z bridge-server info outgoing request global-api:5443 (global-api/***/***)
│ 2020-08-21T19:17:54.651Z bridge-server info client sending message: GET global-api/***/***
│ 2020-08-21T19:18:40.460Z bridge-server error Cannot find correlation id for failed delivery
│ Could not deliver request
│ (node:1) UnhandledPromiseRejectionWarning: ReferenceError: site is not defined
│ at /opt/app-root/lib/bridges.js:181:120
│ at process._tickCallback (internal/process/next_tick.js:68:7)
│ (node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 3)Thanks! What image version is the bridge-server container?
Kampe
commented
4 years ago Both clusters are using Image: quay.io/skupper/bridge-server:0.3
grs
commented
4 years ago @Kampe Sorry for the delay in helping you to resolve this and thanks for your patience! In the last edge log snippet from bridge-server you pasted, it looks like you are exposing the 'global-api' service as http, whereas previously it was as tcp. Was this an explicit change?
Also, going back through the comments here I noticed one issue with the previous tcp service set-up. namely the target labels look wrong. You had:
Services exposed through Skupper:
global-api (tcp port 5443) with targets
=> app=global-api,application=skupper-router,skupper.io/component=router name=global-api
global-service (tcp port 8080) with targets
=> app=global-service,application=skupper-router,skupper.io/component=router name=global-serviceIn each case I think the 'application=skupper-router,skupper.io/component=router' should not be there. That possibly came from having a service created by a previous install of skupper? How are you exposing the service? With the cli, with an annotation on the deployment or with an annotation on the service?
In the last log from the bridge-server, using http bridging, there are essentially three layers to the issue from what I can see. The first, the ReferenceError, is a trivial bug in error handling which I can post a fix for. That however is triggered by an earlier error, 'Cannot find correlation id for failed delivery'. I don't yet understand how that can actually come about. Logs from the bridge server on the cloud site would help. This too however is essentially part of handling a failure to route a request, which is ultimately the 'root cause' (though clearly fixing the other handling issue will also be important).
I do again apologise for taking so long to get to the bottom of this, and appreciate your willingness to gather various bits of information for me. If I could impose further, then I think what we need is the following, all at the same time of testing: (1) output from skupper status and skupper list-exposed in both sites (or the contents of skupper-services if a cli with the latter option is not available), (2) logs from bridge-server and router containers in the skupper-router pod in both sites, (3) qdstat -c and qdstat -l output as run in the router container of the skupper-router pod in each site.
(Again we do have a task open for automating the collection of this sort of data to make it easier in future. I'm sorry thats not in place yet!)
Kampe
commented
4 years ago Good find, yes I did change that global api service to utilize an http router instead of TCP for testing to see if that was part of the issue. No need to apologize, this is an incredible tool! I'm glad to be able to help!
I'm currently using an annotation on the service to define all the skupper'd connections - ie:
apiVersion: v1
kind: Service
metadata:
name: global-api
labels:
app: global-api
service: global-api
annotations:
skupper.io/proxy: http
spec:
selector:
app: global-api
ports:
- name: http
port: 5443Here's the logs from the cloud skupper status and skupper list-exposed:
Skupper is enabled for namespace '"default" in interior mode'. It is connected to 1 other site. It has 2 exposed services.Services exposed through Skupper:
global-api (http port 5443) with targets
=> app=global-api, name=global-api
edge-httptest (http port 8000) with targets
=> app=edge-httptest, name=edge-httptestHere's the logs from the edge skupper status and skupper list-exposed:
Skupper is enabled for namespace '"default" in edge mode'. It is connected to 1 other site. It has 2 exposed services.Services exposed through Skupper:
global-api (http port 5443)
edge-httptest (http port 8000)Here's the logs from the edge router:
│ 2020-08-27T19:40:58.482Z bridge-server info config read: {"httpListeners":[{"address":"edge-httptest","aggregation":"","eventChannel":false,"host":"0.0.0.0","hostOverride":"","http2 │
│ 2020-08-27T19:40:58.483Z bridge-server info no httpConnectors defined
│ 2020-08-27T19:40:58.484Z bridge-server info checking for org.apache.qpid.dispatch.router.httpConnector updates...
│ 2020-08-27T19:40:58.484Z bridge-server info httpListeners defined: [{"address":"edge-httptest","aggregation":"","eventChannel":false,"host":"0.0.0.0","hostOverride":"","http2":false
│ 2020-08-27T19:40:58.484Z bridge-server info checking for org.apache.qpid.dispatch.router.httpListener updates...
│ 2020-08-27T19:40:58.484Z bridge-server info adding org.apache.qpid.dispatch.router.httpListener named edge-httptest
│ 2020-08-27T19:40:58.484Z bridge-server info Created HTTP to AMQP bridge 1026 => edge-httptest
│ 2020-08-27T19:40:58.492Z bridge-server info listening for http on 1026
│ 2020-08-27T19:40:58.492Z bridge-server info adding org.apache.qpid.dispatch.router.httpListener named global-api
│ 2020-08-27T19:40:58.493Z bridge-server info Created HTTP to AMQP bridge 1027 => global-api
│ 2020-08-27T19:40:58.494Z bridge-server info listening for http on 1027
│ 2020-08-27T19:40:58.494Z bridge-server info no http2Connectors defined
│ 2020-08-27T19:40:58.494Z bridge-server info checking for org.apache.qpid.dispatch.router.http2Connector updates...
│ 2020-08-27T19:40:58.494Z bridge-server info no http2Listeners defined
│ 2020-08-27T19:40:58.494Z bridge-server info checking for org.apache.qpid.dispatch.router.http2Listener updates...
│ 2020-08-27T19:40:58.494Z bridge-server info no tcpConnectors defined
│ 2020-08-27T19:40:58.494Z bridge-server info checking for org.apache.qpid.dispatch.router.tcpConnector updates... Here's the logs from the cloud router:
│ 2020-08-27T19:48:23.166Z bridge-server info config read: {"httpConnectors":[{"address":"edge-httptest","aggregation":"","eventChannel":false,"host":"10.48.7.48","hostOverride":"","http2":false,"name":"edge-httptest@10.48.7.48","port":8000,"siteId":"e6014882-211a-4aa3-acd9-7049c75e
│ 2020-08-27T19:48:23.169Z bridge-server info httpConnectors defined: [{"address":"edge-httptest","aggregation":"","eventChannel":false,"host":"10.48.7.48","hostOverride":"","http2":false,"name":"edge-httptest@10.48.7.48","port":8000,"siteId":"e6014882-211a-4aa3-acd9-7049c75ec4d3"},
│ 2020-08-27T19:48:23.170Z bridge-server info checking for org.apache.qpid.dispatch.router.httpConnector updates...
│ 2020-08-27T19:48:23.170Z bridge-server info adding org.apache.qpid.dispatch.router.httpConnector named edge-httptest@10.48.7.48
│ 2020-08-27T19:48:23.171Z bridge-server info Created AMQP to HTTP bridge edge-httptest => 10.48.7.48:8000
│ 2020-08-27T19:48:23.198Z bridge-server info adding org.apache.qpid.dispatch.router.httpConnector named global-api@10.48.8.53
│ 2020-08-27T19:48:23.199Z bridge-server info Created AMQP to HTTP bridge global-api => 10.48.8.53:5443
│ 2020-08-27T19:48:23.202Z bridge-server info adding org.apache.qpid.dispatch.router.httpConnector named global-api@10.48.7.47
│ 2020-08-27T19:48:23.202Z bridge-server info Created AMQP to HTTP bridge global-api => 10.48.7.47:5443
│ 2020-08-27T19:48:23.203Z bridge-server info httpListeners defined: [{"address":"edge-httptest","aggregation":"","eventChannel":false,"host":"0.0.0.0","hostOverride":"","http2":false,"name":"edge-httptest","port":1027,"siteId":"e6014882-211a-4aa3-acd9-7049c75ec4d3"},{"address":"xxx
│ 2020-08-27T19:48:23.204Z bridge-server info checking for org.apache.qpid.dispatch.router.httpListener updates...
│ 2020-08-27T19:48:23.204Z bridge-server info adding org.apache.qpid.dispatch.router.httpListener named edge-httptest
│ 2020-08-27T19:48:23.204Z bridge-server info Created HTTP to AMQP bridge 1027 => edge-httptest
│ 2020-08-27T19:48:23.207Z bridge-server info listening for http on 1027
│ 2020-08-27T19:48:23.208Z bridge-server info adding org.apache.qpid.dispatch.router.httpListener named global-api
│ 2020-08-27T19:48:23.208Z bridge-server info Created HTTP to AMQP bridge 1028 => global-api
│ 2020-08-27T19:48:23.209Z bridge-server info listening for http on 1028
│ 2020-08-27T19:48:23.210Z bridge-server info no http2Connectors defined
│ 2020-08-27T19:48:23.210Z bridge-server info checking for org.apache.qpid.dispatch.router.http2Connector updates...
│ 2020-08-27T19:48:23.210Z bridge-server info no http2Listeners defined
│ 2020-08-27T19:48:23.210Z bridge-server info checking for org.apache.qpid.dispatch.router.http2Listener updates...
│ 2020-08-27T19:48:23.210Z bridge-server info no tcpConnectors defined
│ 2020-08-27T19:48:23.211Z bridge-server info checking for org.apache.qpid.dispatch.router.tcpConnector updates...
│ [connection-1] disconnected Error: connect ECONNREFUSED 127.0.0.1:5672
│ [connection-2] disconnected Error: connect ECONNREFUSED 127.0.0.1:5672
│ [connection-3] disconnected Error: connect ECONNREFUSED 127.0.0.1:5672
│ [connection-6] disconnected Error: connect ECONNREFUSED 127.0.0.1:5672
│ 2020-08-27T19:48:23.582Z bridge-server info ignoring watch event for bridge-server (does not match ..data)
│ 2020-08-27T19:48:23.583Z bridge-server info ignoring watch event for ..2020_08_27_19_48_09.721793184 (does not match ..data)
│ 2020-08-27T19:48:24.688Z bridge-server info ignoring watch event for bridge-server (does not match ..data)
│ 2020-08-27T19:48:24.688Z bridge-server info ignoring watch event for ..2020_08_27_19_48_09.721793184 (does not match ..data) cloud router:
Traceback (most recent call last):
File "/home/qdrouterd/bin/auto_mesh.py", line 279, in <module>
connectors = query()
File "/home/qdrouterd/bin/auto_mesh.py", line 254, in query
response = api_request(host, port, path, token)
File "/home/qdrouterd/bin/auto_mesh.py", line 46, in api_request
req = conn.request("GET", path , headers=headers)
File "/usr/lib64/python2.7/httplib.py", line 1082, in request
self._send_request(method, url, body, headers)
File "/usr/lib64/python2.7/httplib.py", line 1122, in _send_request
self.endheaders(body)
File "/usr/lib64/python2.7/httplib.py", line 1078, in endheaders
self._send_output(message_body)
File "/usr/lib64/python2.7/httplib.py", line 894, in _send_output
self.send(msg)
File "/usr/lib64/python2.7/httplib.py", line 856, in send
self.connect()
File "/usr/lib64/python2.7/httplib.py", line 1295, in connect
HTTPConnection.connect(self)
File "/usr/lib64/python2.7/httplib.py", line 833, in connect
self.timeout, self.source_address)
File "/usr/lib64/python2.7/socket.py", line 575, in create_connection
raise err
error: [Errno 111] Connection refused
Error configuring automesh: [Errno 111] Connection refused
---------------------------------------
router {
mode: interior
id: skupp3r-cloud-skupper-router-845f7bdf87-55qhg
metadata: e6014882-211a-4aa3-acd9-7049c75ec4d3
}
listener {
name: amqp
host: localhost
port: 5672
role: normal
}
listener {
name: amqps
host: 0.0.0.0
port: 5671
role: normal
authenticatePeer: true
saslMechanisms: EXTERNAL
sslProfile: skupper-amqps
}
listener {
name: console
host: 0.0.0.0
port: 8080
role: normal
http: true
authenticatePeer: true
}
listener {
host: 0.0.0.0
port: 9090
role: normal
http: true
httpRootDir: disabled
websockets: false
healthz: true
metrics: true
}
listener {
name: interior-listener
role: inter-router
host: 0.0.0.0
port: 55671
saslMechanisms: EXTERNAL
authenticatePeer: true
sslProfile: skupper-internal
}
listener {
name: edge-listener
role: edge
host: 0.0.0.0
port: 45671
saslMechanisms: EXTERNAL
authenticatePeer: true
sslProfile: skupper-internal
}
sslProfile {
name: skupper-amqps
certFile: /etc/qpid-dispatch-certs/skupper-amqps/tls.crt
privateKeyFile: /etc/qpid-dispatch-certs/skupper-amqps/tls.key
caCertFile: /etc/qpid-dispatch-certs/skupper-amqps/ca.crt
}
sslProfile {
name: skupper-internal
certFile: /etc/qpid-dispatch-certs/skupper-internal/tls.crt
privateKeyFile: /etc/qpid-dispatch-certs/skupper-internal/tls.key
caCertFile: /etc/qpid-dispatch-certs/skupper-internal/ca.crt
}
address {
prefix: mc
distribution: multicast
}
## Connectors: ##
---------------------------------------
cat /etc/qpid-dispatch/sasl-users//rubble | saslpasswd2 -c -p -u skupper-router rubble -f /tmp/qdrouterd.sasldb
2020-08-27 19:48:23.209807 +0000 SERVER (info) Container Name: skupp3r-cloud-skupper-router-845f7bdf87-55qhg
2020-08-27 19:48:23.210140 +0000 ROUTER (info) Router started in Interior mode, area=0 id=skupp3r-cloud-skupper-router-845f7bdf87-55qhg
2020-08-27 19:48:23.210195 +0000 ROUTER (info) Version: 1.13.0
2020-08-27 19:48:23.210944 +0000 ROUTER (info) Router Engine Instantiated: id=skupp3r-cloud-skupper-router-845f7bdf87-55qhg instance=1598557703 max_routers=128
2020-08-27 19:48:23.233309 +0000 ROUTER_CORE (info) Core module present but disabled: edge_router
2020-08-27 19:48:23.233433 +0000 ROUTER_CORE (info) Core module present but disabled: core_test_hooks
2020-08-27 19:48:23.233514 +0000 ROUTER_CORE (info) Core module enabled: edge_addr_tracking
2020-08-27 19:48:23.233546 +0000 ROUTER_CORE (info) Core module enabled: address_lookup_server
2020-08-27 19:48:23.233581 +0000 ROUTER_CORE (info) Core module enabled: address_lookup_client
2020-08-27 19:48:23.233654 +0000 ROUTER_CORE (info) Stuck delivery detection: Scan interval: 30 seconds, Delivery age threshold: 10 seconds
2020-08-27 19:48:23.233695 +0000 ROUTER_CORE (info) Core module enabled: stuck_delivery_detection
2020-08-27 19:48:23.233891 +0000 ROUTER_CORE (info) Core module enabled: mobile_sync
2020-08-27 19:48:23.234007 +0000 ROUTER_CORE (info) Streaming link scrubber: Scan interval: 30 seconds, max free pool: 128 links
2020-08-27 19:48:23.234046 +0000 ROUTER_CORE (info) Core module enabled: streaming_link_scruber
2020-08-27 19:48:23.234083 +0000 ROUTER_CORE (info) Router Core thread running. 0/skupp3r-cloud-skupper-router-845f7bdf87-55qhg
2020-08-27 19:48:23.234110 +0000 ROUTER_CORE (info) In-process subscription M/$management
2020-08-27 19:48:23.234190 +0000 ROUTER_CORE (info) In-process subscription L/$management
2020-08-27 19:48:23.234222 +0000 ROUTER_CORE (info) In-process subscription L/qdrouter
2020-08-27 19:48:23.234267 +0000 ROUTER_CORE (info) In-process subscription T/qdrouter
2020-08-27 19:48:23.234302 +0000 ROUTER_CORE (info) In-process subscription L/qdhello
2020-08-27 19:48:23.234404 +0000 ROUTER_CORE (info) In-process subscription L/qdrouter.ma
2020-08-27 19:48:23.234453 +0000 ROUTER_CORE (info) In-process subscription T/qdrouter.ma
2020-08-27 19:48:23.234582 +0000 AGENT (info) Activating management agent on $_management_internal
2020-08-27 19:48:23.234668 +0000 ROUTER_CORE (info) In-process subscription L/$_management_internal
2020-08-27 19:48:23.237681 +0000 POLICY (info) Policy configured maxConnections: 65535, policyDir: '',access rules enabled: 'false', use hostname patterns: 'false'
2020-08-27 19:48:23.239308 +0000 POLICY (info) Policy fallback defaultVhost is defined: '$default'
2020-08-27 19:48:23.240178 +0000 CONN_MGR (info) Created SSL Profile with name skupper-amqps
2020-08-27 19:48:23.241632 +0000 CONN_MGR (info) Created SSL Profile with name skupper-internal
2020-08-27 19:48:23.243658 +0000 CONN_MGR (info) Configured Listener: localhost:5672 proto=any, role=normal
2020-08-27 19:48:23.246374 +0000 CONN_MGR (info) Configured Listener: 0.0.0.0:5671 proto=any, role=normal, sslProfile=skupper-amqps
2020-08-27 19:48:23.248815 +0000 CONN_MGR (info) Configured Listener: 0.0.0.0:8080 proto=any, role=normal, http
2020-08-27 19:48:23.249150 +0000 SERVER (info) HTTP server thread running
2020-08-27 19:48:23.249408 +0000 SERVER (notice) Listening for HTTP on 0.0.0.0:8080
2020-08-27 19:48:23.251394 +0000 CONN_MGR (info) Configured Listener: 0.0.0.0:9090 proto=any, role=normal, http
2020-08-27 19:48:23.251558 +0000 SERVER (notice) Listening for HTTP on 0.0.0.0:9090
2020-08-27 19:48:23.253946 +0000 CONN_MGR (info) Configured Listener: 0.0.0.0:55671 proto=any, role=inter-router, sslProfile=skupper-internal
2020-08-27 19:48:23.256489 +0000 CONN_MGR (info) Configured Listener: 0.0.0.0:45671 proto=any, role=edge, sslProfile=skupper-internal
2020-08-27 19:48:23.259882 +0000 SERVER (notice) Operational, 4 Threads Running (process ID 1)
2020-08-27 19:48:23.260182 +0000 SERVER (notice) Process VmSize 186.91 MiB (7.30 GiB available memory)
2020-08-27 19:48:23.260499 +0000 SERVER (notice) Listening on 0.0.0.0:5671
2020-08-27 19:48:23.260573 +0000 SERVER (notice) Listening on 0.0.0.0:55671
2020-08-27 19:48:23.260605 +0000 SERVER (notice) Listening on 0.0.0.0:45671
2020-08-27 19:48:23.260755 +0000 SERVER (notice) Listening on localhost:5672
2020-08-27 19:48:23.344540 +0000 SERVER (info) [C1] Accepted connection to localhost:5672 from 127.0.0.1:50618
2020-08-27 19:48:23.349472 +0000 SERVER (info) [C2] Accepted connection to localhost:5672 from 127.0.0.1:50620
2020-08-27 19:48:23.349720 +0000 SERVER (info) [C3] Accepted connection to localhost:5672 from 127.0.0.1:50622
2020-08-27 19:48:23.349846 +0000 SERVER (info) [C4] Accepted connection to localhost:5672 from 127.0.0.1:50624
2020-08-27 19:48:23.349957 +0000 SERVER (info) [C5] Accepted connection to localhost:5672 from 127.0.0.1:50626
2020-08-27 19:48:23.350061 +0000 SERVER (info) [C6] Accepted connection to localhost:5672 from 127.0.0.1:50628
2020-08-27 19:48:23.360097 +0000 ROUTER (info) [C2] Connection Opened: dir=in host=127.0.0.1:50620 vhost= encrypted=no auth=ANONYMOUS user=anonymous container_id=skupper-router-845f7bdf87-55qhg_amqp_global-api_to_http_10.48.8.53_5443 props=
2020-08-27 19:48:23.362391 +0000 ROUTER (info) [C3] Connection Opened: dir=in host=127.0.0.1:50622 vhost= encrypted=no auth=ANONYMOUS user=anonymous container_id=skupper-router-845f7bdf87-55qhg_amqp_global-api_to_http_10.48.7.47_5443 props=
2020-08-27 19:48:23.363391 +0000 ROUTER (info) [C1] Connection Opened: dir=in host=127.0.0.1:50618 vhost= encrypted=no auth=ANONYMOUS user=anonymous container_id=skupper-router-845f7bdf87-55qhg_amqp_edge-httptest_to_http_10.48.7.48_8000 props=
2020-08-27 19:48:23.363655 +0000 ROUTER (info) [C5] Connection Opened: dir=in host=127.0.0.1:50626 vhost= encrypted=no auth=ANONYMOUS user=anonymous container_id=skupper-router-845f7bdf87-55qhg_http_1028_to_amqp_global-api props=
2020-08-27 19:48:23.365989 +0000 ROUTER_CORE (info) [C2][L2] Link attached: dir=out source={global-api expire:sess} target={<none> expire:sess}
2020-08-27 19:48:23.366220 +0000 ROUTER_CORE (info) [C5][L3] Link attached: dir=in source={<none> expire:sess} target={global-api expire:sess}
2020-08-27 19:48:23.366270 +0000 ROUTER_CORE (info) [C5][L4] Link attached: dir=out source={<dynamic> expire:sess} target={<none> expire:sess}
2020-08-27 19:48:23.366593 +0000 ROUTER (info) [C4] Connection Opened: dir=in host=127.0.0.1:50624 vhost= encrypted=no auth=ANONYMOUS user=anonymous container_id=skupper-router-845f7bdf87-55qhg_http_1027_to_amqp_edge-httptest props=
2020-08-27 19:48:23.366661 +0000 ROUTER_CORE (info) [C3][L5] Link attached: dir=out source={global-api expire:sess} target={<none> expire:sess}
2020-08-27 19:48:23.366765 +0000 ROUTER_CORE (info) [C4][L6] Link attached: dir=in source={<none> expire:sess} target={edge-httptest expire:sess}
2020-08-27 19:48:23.366835 +0000 ROUTER_CORE (info) [C4][L7] Link attached: dir=out source={<dynamic> expire:sess} target={<none> expire:sess}
2020-08-27 19:48:23.367662 +0000 ROUTER_CORE (info) [C1][L8] Link attached: dir=out source={edge-httptest expire:sess} target={<none> expire:sess}
2020-08-27 19:48:23.369018 +0000 ROUTER (info) [C6] Connection Opened: dir=in host=127.0.0.1:50628 vhost= encrypted=no auth=ANONYMOUS user=anonymous container_id=bridge-server-skupper-router-845f7bdf87-55qhg props=
2020-08-27 19:48:23.375327 +0000 ROUTER_CORE (info) [C6][L9] Link attached: dir=out source={skupp3r-cloud-skupper-router-845f7bdf87-55qhg/bridge-server/$management expire:sess} target={<none> expire:sess}
2020-08-27 19:48:26.356113 +0000 SERVER (info) [C8] Accepted connection to 0.0.0.0:5671 from 127.0.0.1:55666
2020-08-27 19:48:26.361676 +0000 SERVER (info) [C7] Accepted connection to 0.0.0.0:5671 from 127.0.0.1:55664
2020-08-27 19:48:26.369930 +0000 ROUTER (info) [C8] Connection Opened: dir=in host=127.0.0.1:55666 vhost= encrypted=TLSv1.3 auth=EXTERNAL user=CN=skupper-messaging container_id=R5x2t1VhAE_eVhG0X9MvVSWukz7cTt8BUpIvVJ2W48AA2Yn_l2fMNA props=
2020-08-27 19:48:26.370322 +0000 ROUTER (info) [C7] Connection Opened: dir=in host=127.0.0.1:55664 vhost= encrypted=TLSv1.3 auth=EXTERNAL user=CN=skupper-messaging container_id=TLaBQOB8K-NpqDssiaskEOn7JvhyNsEm4pjpfrqkjhSVDDOAHGVVWg props=
2020-08-27 19:48:26.372312 +0000 ROUTER_CORE (info) [C8][L10] Link attached: dir=out source={mc/$skupper-service-sync expire:sess} target={<none> expire:sess}
2020-08-27 19:48:26.372355 +0000 ROUTER_CORE (info) [C7][L11] Link attached: dir=out source={e6014882-211a-4aa3-acd9-7049c75ec4d3/skupper-site-query expire:sess} target={<none> expire:sess}
2020-08-27 19:48:26.375174 +0000 ROUTER_CORE (info) [C8][L12] Link attached: dir=in source={<none> expire:sess} target={mc/$skupper-service-sync expire:sess}
2020-08-27 19:48:26.375216 +0000 ROUTER_CORE (info) [C7][L13] Link attached: dir=in source={<none> expire:sess} target={<none> expire:sess}
2020-08-27 19:48:31.689585 +0000 SERVER (info) [C9] Accepted connection to 0.0.0.0:45671 from 127.0.0.1:59074
2020-08-27 19:48:31.867448 +0000 ROUTER (info) [C9] Connection Opened: dir=in host=127.0.0.1:59074 vhost= encrypted=TLSv1.3 auth=EXTERNAL user=CN=skupper container_id=skupp3r-edge-site-skupper-router-57578fd697-7zrwk props={:product="qpid-dispatch-router", :version="1.13.0", :"qd.conn-id"=11}
2020-08-27 19:48:31.926644 +0000 ROUTER_CORE (info) [C9][L14] Link attached: dir=in source={<none> expire:sess} target={<none> expire:sess}
2020-08-27 19:48:31.926805 +0000 ROUTER_CORE (info) [C9][L15] Link attached: dir=out source={skupp3r-edge-site-skupper-router-57578fd697-7zrwk expire:link caps::"qd.router-edge-downlink"} target={<none> expire:sess caps::"qd.router-edge-downlink"}
2020-08-27 19:48:31.926945 +0000 ROUTER_CORE (info) [C9][L16] Link attached: dir=out source={_$qd.edge_addr_tracking expire:link} target={<none> expire:sess}
2020-08-27 19:48:31.926988 +0000 ROUTER_CORE (info) [C9][L17] Link attached: dir=in source={<none> expire:sess} target={edge-httptest expire:link}
2020-08-27 19:48:31.927010 +0000 ROUTER_CORE (info) [C9][L18] Link attached: dir=in source={<none> expire:sess} target={global-api expire:link}
2020-08-27 19:48:31.927037 +0000 ROUTER_CORE (info) [C9][L19] Link attached: dir=out source={skupp3r-edge-site-skupper-router-57578fd697-7zrwk/bridge-server/$management expire:link} target={<none> expire:sess}
2020-08-27 19:48:31.927057 +0000 ROUTER_CORE (info) [C9][L20] Link attached: dir=out source={338fb13d-cb9b-40b7-b622-f5694d0a9232/skupper-site-query expire:link} target={<none> expire:sess}
2020-08-27 19:48:31.927075 +0000 ROUTER_CORE (info) [C9][L21] Link attached: dir=out source={mc/$skupper-service-sync expire:link} target={<none> expire:sess}
2020-08-27 19:48:31.927093 +0000 ROUTER_CORE (info) [C9][L22] Link attached: dir=in source={<none> expire:sess} target={mc/$skupper-service-sync expire:link}
2020-08-27 19:48:31.927109 +0000 ROUTER_CORE (info) [C9][L23] Link attached: dir=in source={<none> expire:sess} target={$management expire:link}
2020-08-27 19:48:31.927155 +0000 ROUTER_CORE (info) [C9][L24] Link attached: dir=out source={<dynamic> expire:link} target={<none> expire:sess}
2020-08-27 19:48:31.927210 +0000 ROUTER_CORE (info) [C9][L25] Link attached: dir=in source={<none> expire:sess} target={_$qd.addr_lookup expire:link}
2020-08-27 19:48:31.927225 +0000 ROUTER_CORE (info) [C9][L26] Link attached: dir=out source={<dynamic> expire:link} target={<none> expire:sess}Edge router:
cat /etc/qpid-dispatch/sasl-users//rubble | saslpasswd2 -c -p -u qdrouterd rubble -f /tmp/qdrouterd.sasldb
2020-08-27 19:40:58.126816 +0000 SERVER (info) Container Name: skupp3r-edge-site-skupper-router-57578fd697-7zrwk
2020-08-27 19:40:58.127002 +0000 ROUTER (info) Router started in Edge mode
2020-08-27 19:40:58.127017 +0000 ROUTER (info) Version: 1.13.0
2020-08-27 19:40:58.136828 +0000 AGENT (info) Activating management agent on $_management_internal
2020-08-27 19:40:58.137269 +0000 ROUTER_CORE (info) Core module enabled: edge_router
2020-08-27 19:40:58.137288 +0000 ROUTER_CORE (info) Core module present but disabled: core_test_hooks
2020-08-27 19:40:58.137293 +0000 ROUTER_CORE (info) Core module present but disabled: edge_addr_tracking
2020-08-27 19:40:58.137296 +0000 ROUTER_CORE (info) Core module present but disabled: address_lookup_server
2020-08-27 19:40:58.137300 +0000 ROUTER_CORE (info) Core module enabled: address_lookup_client
2020-08-27 19:40:58.137339 +0000 ROUTER_CORE (info) Stuck delivery detection: Scan interval: 30 seconds, Delivery age threshold: 10 seconds
2020-08-27 19:40:58.137344 +0000 ROUTER_CORE (info) Core module enabled: stuck_delivery_detection
2020-08-27 19:40:58.137348 +0000 ROUTER_CORE (info) Core module present but disabled: mobile_sync
2020-08-27 19:40:58.137352 +0000 ROUTER_CORE (info) Streaming link scrubber: Scan interval: 30 seconds, max free pool: 128 links
2020-08-27 19:40:58.137356 +0000 ROUTER_CORE (info) Core module enabled: streaming_link_scruber
2020-08-27 19:40:58.137360 +0000 ROUTER_CORE (info) Router Core thread running. 0/skupp3r-edge-site-skupper-router-57578fd697-7zrwk
2020-08-27 19:40:58.137380 +0000 ROUTER_CORE (info) In-process subscription M/$management
2020-08-27 19:40:58.137394 +0000 ROUTER_CORE (info) In-process subscription L/$management
2020-08-27 19:40:58.137400 +0000 ROUTER_CORE (info) In-process subscription L/$_management_internal
2020-08-27 19:40:58.139084 +0000 POLICY (info) Policy configured maxConnections: 65535, policyDir: '',access rules enabled: 'false', use hostname patterns: 'false'
2020-08-27 19:40:58.140300 +0000 POLICY (info) Policy fallback defaultVhost is defined: '$default'
2020-08-27 19:40:58.140789 +0000 CONN_MGR (info) Created SSL Profile with name skupper-amqps
2020-08-27 19:40:58.141909 +0000 CONN_MGR (info) Created SSL Profile with name skupper-cloud-certificates-profile
2020-08-27 19:40:58.142960 +0000 CONN_MGR (info) Configured Listener: localhost:5672 proto=any, role=normal
2020-08-27 19:40:58.146457 +0000 CONN_MGR (info) Configured Listener: 0.0.0.0:5671 proto=any, role=normal, sslProfile=skupper-amqps
2020-08-27 19:40:58.148699 +0000 CONN_MGR (info) Configured Listener: 0.0.0.0:8080 proto=any, role=normal, http
2020-08-27 19:40:58.149260 +0000 SERVER (info) HTTP server thread running
2020-08-27 19:40:58.149342 +0000 SERVER (notice) Listening for HTTP on 0.0.0.0:8080
2020-08-27 19:40:58.150805 +0000 CONN_MGR (info) Configured Listener: 0.0.0.0:9090 proto=any, role=normal, http
2020-08-27 19:40:58.151112 +0000 SERVER (notice) Listening for HTTP on 0.0.0.0:9090
2020-08-27 19:40:58.152915 +0000 CONN_MGR (info) Configured Connector: xx.xxx.xxx.xxx:45671 proto=any, role=edge, sslProfile=skupper-cloud-certificates-profile
2020-08-27 19:40:58.155334 +0000 SERVER (notice) Operational, 4 Threads Running (process ID 1)
2020-08-27 19:40:58.155903 +0000 SERVER (notice) Process VmSize 186.95 MiB (3.84 GiB available memory)
2020-08-27 19:40:58.156403 +0000 SERVER (notice) Listening on localhost:5672
2020-08-27 19:40:58.156452 +0000 SERVER (notice) Listening on 0.0.0.0:5671
2020-08-27 19:40:58.465431 +0000 ROUTER (info) [C1] Connection Opened: dir=out host=xx.xxx.xxx.xxx:45671 vhost= encrypted=TLSv1.3 auth=EXTERNAL user=(null) container_id=skupp3r-cloud-skupper-router-845f7bdf87-5pwsf props={:product="qpid-dispatch-router", :version="1.13.0", :"qd.conn-id"=19}
2020-08-27 19:40:58.465494 +0000 ROUTER_CORE (info) Edge connection (id=1) to interior established
2020-08-27 19:40:58.465655 +0000 ROUTER_CORE (info) [C1][L2] Link attached: dir=out source={<none> expire:link} target={<none> expire:link}
2020-08-27 19:40:58.465711 +0000 ROUTER_CORE (info) [C1][L3] Link attached: dir=in source={skupp3r-edge-site-skupper-router-57578fd697-7zrwk expire:link caps::"qd.router-edge-downlink"} target={<none> expire:link caps::"qd.router-edge-downlink"}
2020-08-27 19:40:58.465736 +0000 ROUTER_CORE (info) [C1][L4] Link attached: dir=in source={_$qd.edge_addr_tracking expire:link} target={<none> expire:link}
2020-08-27 19:40:58.465769 +0000 ROUTER_CORE (info) [C1][L5] Link attached: dir=out source={} target={$management expire:link}
2020-08-27 19:40:58.465776 +0000 ROUTER_CORE (info) [C1][L6] Link attached: dir=in source={<dynamic> expire:link} target={}
2020-08-27 19:40:58.465786 +0000 ROUTER_CORE (info) [C1][L7] Link attached: dir=out source={} target={_$qd.addr_lookup expire:link}
2020-08-27 19:40:58.465793 +0000 ROUTER_CORE (info) [C1][L8] Link attached: dir=in source={<dynamic> expire:link} target={}
2020-08-27 19:40:58.502821 +0000 SERVER (info) [C2] Accepted connection to localhost:5672 from 127.0.0.1:51716
2020-08-27 19:40:58.502869 +0000 SERVER (info) [C3] Accepted connection to localhost:5672 from 127.0.0.1:51718
2020-08-27 19:40:58.502953 +0000 SERVER (info) [C4] Accepted connection to localhost:5672 from 127.0.0.1:51720
2020-08-27 19:40:58.509018 +0000 ROUTER (info) [C2] Connection Opened: dir=in host=127.0.0.1:51716 vhost= encrypted=no auth=ANONYMOUS user=anonymous container_id=skupper-router-57578fd697-7zrwk_http_1026_to_amqp_edge-httptest props=
2020-08-27 19:40:58.510298 +0000 ROUTER_CORE (info) [C1][L11] Link attached: dir=out source={<none> expire:link} target={edge-httptest expire:link}
2020-08-27 19:40:58.510326 +0000 ROUTER_CORE (info) [C2][L9] Link attached: dir=in source={<none> expire:sess} target={edge-httptest expire:sess}
2020-08-27 19:40:58.510339 +0000 ROUTER_CORE (info) [C2][L10] Link attached: dir=out source={<dynamic> expire:sess} target={<none> expire:sess}
2020-08-27 19:40:58.514250 +0000 ROUTER (info) [C3] Connection Opened: dir=in host=127.0.0.1:51718 vhost= encrypted=no auth=ANONYMOUS user=anonymous container_id=skupper-router-57578fd697-7zrwk_http_1027_to_amqp_global-api props=
2020-08-27 19:40:58.514535 +0000 ROUTER_CORE (info) [C1][L14] Link attached: dir=out source={<none> expire:link} target={global-api expire:link}
2020-08-27 19:40:58.514558 +0000 ROUTER_CORE (info) [C3][L12] Link attached: dir=in source={<none> expire:sess} target={global-api expire:sess}
2020-08-27 19:40:58.514574 +0000 ROUTER_CORE (info) [C3][L13] Link attached: dir=out source={<dynamic> expire:sess} target={<none> expire:sess}
2020-08-27 19:40:58.514750 +0000 ROUTER (info) [C4] Connection Opened: dir=in host=127.0.0.1:51720 vhost= encrypted=no auth=ANONYMOUS user=anonymous container_id=bridge-server-skupper-router-57578fd697-7zrwk props=
2020-08-27 19:40:58.517981 +0000 ROUTER_CORE (info) [C1][L16] Link attached: dir=in source={skupp3r-edge-site-skupper-router-57578fd697-7zrwk/bridge-server/$management expire:link} target={<none> expire:link}
2020-08-27 19:40:58.518009 +0000 ROUTER_CORE (info) [C4][L15] Link attached: dir=out source={skupp3r-edge-site-skupper-router-57578fd697-7zrwk/bridge-server/$management expire:sess} target={<none> expire:sess}
2020-08-27 19:40:59.273261 +0000 SERVER (info) [C5] Accepted connection to 0.0.0.0:5671 from 10.42.0.38:45848
2020-08-27 19:40:59.289965 +0000 ROUTER (info) [C5] Connection Opened: dir=in host=10.42.0.38:45848 vhost= encrypted=TLSv1.3 auth=EXTERNAL user=CN=skupper-messaging container_id=qguw3g8FdylqahPnZtAxmhsBZCiRszso5f2VArE611QsabjAmrcrTQ props=
2020-08-27 19:40:59.293357 +0000 ROUTER_CORE (info) [C5][L17] Link attached: dir=out source={338fb13d-cb9b-40b7-b622-f5694d0a9232/skupper-site-query expire:sess} target={<none> expire:sess}
2020-08-27 19:40:59.348181 +0000 ROUTER_CORE (info) [C1][L18] Link attached: dir=in source={338fb13d-cb9b-40b7-b622-f5694d0a9232/skupper-site-query expire:link} target={<none> expire:link}
2020-08-27 19:40:59.349794 +0000 ROUTER_CORE (info) [C5][L19] Link attached: dir=in source={<none> expire:sess} target={<none> expire:sess}
2020-08-27 19:41:02.461459 +0000 SERVER (info) [C6] Accepted connection to 0.0.0.0:5671 from 10.42.0.38:45762
2020-08-27 19:41:02.468319 +0000 ROUTER (info) [C6] Connection Opened: dir=in host=10.42.0.38:45762 vhost= encrypted=TLSv1.3 auth=EXTERNAL user=CN=skupper-messaging container_id=IUxpPCIoJP7a4efPAyS1bzlPASgtxslVvOVMzwudQgeOXFajLDEBKA props=
2020-08-27 19:41:02.468905 +0000 ROUTER_CORE (info) [C6][L20] Link attached: dir=out source={mc/$skupper-service-sync expire:sess} target={<none> expire:sess}
2020-08-27 19:41:02.523113 +0000 ROUTER_CORE (info) [C1][L21] Link attached: dir=in source={mc/$skupper-service-sync expire:link} target={<none> expire:link}
2020-08-27 19:41:02.523898 +0000 ROUTER_CORE (info) [C6][L22] Link attached: dir=in source={<none> expire:sess} target={mc/$skupper-service-sync expire:sess}
2020-08-27 19:41:02.578895 +0000 ROUTER_CORE (info) [C1][L23] Link attached: dir=out source={<none> expire:link} target={mc/$skupper-service-sync expire:link}
2020-08-27 19:48:09.313617 +0000 SERVER (info) [C1] Connection to xx.xxx.xxx.xxx:45671 failed: amqp:connection:framing-error SSL Failure: Unknown error
2020-08-27 19:48:09.314023 +0000 ROUTER_CORE (info) [C1][L2] Link lost: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:48:09.314093 +0000 ROUTER_CORE (info) [C1][L3] Link lost: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:48:09.314108 +0000 ROUTER_CORE (info) [C1][L11] Link lost: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:48:09.314116 +0000 ROUTER_CORE (info) [C1][L14] Link lost: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:48:09.314148 +0000 ROUTER_CORE (info) [C1][L16] Link lost: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:48:09.314156 +0000 ROUTER_CORE (info) [C1][L18] Link lost: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:48:09.314161 +0000 ROUTER_CORE (info) [C1][L21] Link lost: del=85 presett=0 psdrop=0 acc=85 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:48:09.314168 +0000 ROUTER_CORE (info) [C1][L23] Link lost: del=85 presett=0 psdrop=0 acc=85 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:48:09.314175 +0000 ROUTER_CORE (info) [C1][L4] Link closed due to connection loss: del=3 presett=3 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:48:09.314182 +0000 ROUTER_CORE (info) [C1][L5] Link closed due to connection loss: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:48:09.314204 +0000 ROUTER_CORE (info) [C1][L6] Link closed due to connection loss: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:48:09.314250 +0000 ROUTER_CORE (info) [C1][L7] Link closed due to connection loss: del=3 presett=3 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=2 delay10=0 blocked=no
2020-08-27 19:48:09.314261 +0000 ROUTER_CORE (info) [C1][L8] Link closed due to connection loss: del=3 presett=3 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:48:09.314272 +0000 ROUTER_CORE (info) Edge connection (id=1) to interior lost, no alternate connection available
2020-08-27 19:48:09.314277 +0000 ROUTER_CORE (info) [C1] Connection Closed
2020-08-27 19:48:11.379192 +0000 SERVER (info) [C7] Connection to xx.xxx.xxx.xxx:45671 failed: proton:io Connection refused - disconnected xx.xxx.xxx.xxx:45671
2020-08-27 19:48:16.443889 +0000 SERVER (info) [C8] Connection to xx.xxx.xxx.xxx:45671 failed: proton:io Connection refused - disconnected xx.xxx.xxx.xxx:45671
2020-08-27 19:48:21.511419 +0000 SERVER (info) [C9] Connection to xx.xxx.xxx.xxx:45671 failed: proton:io Connection refused - disconnected xx.xxx.xxx.xxx:45671
2020-08-27 19:48:26.574652 +0000 SERVER (info) [C10] Connection to xx.xxx.xxx.xxx:45671 failed: proton:io Connection refused - disconnected xx.xxx.xxx.xxx:45671
2020-08-27 19:48:31.893137 +0000 ROUTER (info) [C11] Connection Opened: dir=out host=xx.xxx.xxx.xxx:45671 vhost= encrypted=TLSv1.3 auth=EXTERNAL user=(null) container_id=skupp3r-cloud-skupper-router-845f7bdf87-55qhg props={:product="qpid-dispatch-router", :version="1.13.0", :"qd.conn-id"=9}
2020-08-27 19:48:31.893363 +0000 ROUTER_CORE (info) Edge connection (id=11) to interior established
2020-08-27 19:48:31.893398 +0000 ROUTER_CORE (info) [C11][L24] Link attached: dir=out source={<none> expire:link} target={<none> expire:link}
2020-08-27 19:48:31.893614 +0000 ROUTER_CORE (info) [C11][L25] Link attached: dir=in source={skupp3r-edge-site-skupper-router-57578fd697-7zrwk expire:link caps::"qd.router-edge-downlink"} target={<none> expire:link caps::"qd.router-edge-downlink"}
2020-08-27 19:48:31.893638 +0000 ROUTER_CORE (info) [C11][L26] Link attached: dir=in source={_$qd.edge_addr_tracking expire:link} target={<none> expire:link}
2020-08-27 19:48:31.893649 +0000 ROUTER_CORE (info) [C11][L27] Link attached: dir=out source={<none> expire:link} target={edge-httptest expire:link}
2020-08-27 19:48:31.893659 +0000 ROUTER_CORE (info) [C11][L28] Link attached: dir=out source={<none> expire:link} target={global-api expire:link}
2020-08-27 19:48:31.893666 +0000 ROUTER_CORE (info) [C11][L29] Link attached: dir=in source={skupp3r-edge-site-skupper-router-57578fd697-7zrwk/bridge-server/$management expire:link} target={<none> expire:link}
2020-08-27 19:48:31.893675 +0000 ROUTER_CORE (info) [C11][L30] Link attached: dir=in source={338fb13d-cb9b-40b7-b622-f5694d0a9232/skupper-site-query expire:link} target={<none> expire:link}
2020-08-27 19:48:31.893684 +0000 ROUTER_CORE (info) [C11][L31] Link attached: dir=in source={mc/$skupper-service-sync expire:link} target={<none> expire:link}
2020-08-27 19:48:31.893692 +0000 ROUTER_CORE (info) [C11][L32] Link attached: dir=out source={<none> expire:link} target={mc/$skupper-service-sync expire:link}
2020-08-27 19:48:31.893701 +0000 ROUTER_CORE (info) [C11][L33] Link attached: dir=out source={} target={$management expire:link}
2020-08-27 19:48:31.893708 +0000 ROUTER_CORE (info) [C11][L34] Link attached: dir=in source={<dynamic> expire:link} target={}
2020-08-27 19:48:31.893903 +0000 ROUTER_CORE (info) [C11][L35] Link attached: dir=out source={} target={_$qd.addr_lookup expire:link}
2020-08-27 19:48:31.893925 +0000 ROUTER_CORE (info) [C11][L36] Link attached: dir=in source={<dynamic> expire:link} target={}
2020-08-27 19:56:42.073163 +0000 SERVER (info) [C11] Connection to xx.xxx.xxx.xxx:45671 failed: amqp:connection:framing-error SSL Failure: Unknown error
2020-08-27 19:56:42.073298 +0000 ROUTER_CORE (info) [C11][L24] Link lost: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:56:42.073438 +0000 ROUTER_CORE (info) [C11][L25] Link lost: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:56:42.073480 +0000 ROUTER_CORE (info) [C11][L27] Link lost: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:56:42.073561 +0000 ROUTER_CORE (info) [C11][L28] Link lost: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:56:42.073645 +0000 ROUTER_CORE (info) [C11][L29] Link lost: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:56:42.073692 +0000 ROUTER_CORE (info) [C11][L30] Link lost: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:56:42.073718 +0000 ROUTER_CORE (info) [C11][L31] Link lost: del=98 presett=0 psdrop=0 acc=98 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:56:42.073739 +0000 ROUTER_CORE (info) [C11][L32] Link lost: del=98 presett=0 psdrop=0 acc=98 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:56:42.073797 +0000 ROUTER_CORE (info) [C11][L26] Link closed due to connection loss: del=3 presett=3 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:56:42.073892 +0000 ROUTER_CORE (info) [C11][L33] Link closed due to connection loss: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:56:42.073940 +0000 ROUTER_CORE (info) [C11][L34] Link closed due to connection loss: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:56:42.073981 +0000 ROUTER_CORE (info) [C11][L35] Link closed due to connection loss: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:56:42.074242 +0000 ROUTER_CORE (info) [C11][L36] Link closed due to connection loss: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
2020-08-27 19:56:42.074416 +0000 ROUTER_CORE (info) Edge connection (id=11) to interior lost, no alternate connection available
2020-08-27 19:56:42.074552 +0000 ROUTER_CORE (info) [C11] Connection Closed
2020-08-27 19:56:44.383076 +0000 ROUTER (info) [C12] Connection Opened: dir=out host=xx.xxx.xxx.xxx:45671 vhost= encrypted=TLSv1.3 auth=EXTERNAL user=(null) container_id=skupp3r-cloud-skupper-router-845f7bdf87-55qhg props={:product="qpid-dispatch-router", :version="1.13.0", :"qd.conn-id"=12}
2020-08-27 19:56:44.383145 +0000 ROUTER_CORE (info) Edge connection (id=12) to interior established
2020-08-27 19:56:44.383178 +0000 ROUTER_CORE (info) [C12][L37] Link attached: dir=out source={<none> expire:link} target={<none> expire:link}
2020-08-27 19:56:44.383198 +0000 ROUTER_CORE (info) [C12][L38] Link attached: dir=in source={skupp3r-edge-site-skupper-router-57578fd697-7zrwk expire:link caps::"qd.router-edge-downlink"} target={<none> expire:link caps::"qd.router-edge-downlink"}
2020-08-27 19:56:44.383208 +0000 ROUTER_CORE (info) [C12][L39] Link attached: dir=in source={_$qd.edge_addr_tracking expire:link} target={<none> expire:link}
2020-08-27 19:56:44.383218 +0000 ROUTER_CORE (info) [C12][L40] Link attached: dir=out source={<none> expire:link} target={edge-httptest expire:link}
2020-08-27 19:56:44.383233 +0000 ROUTER_CORE (info) [C12][L41] Link attached: dir=out source={<none> expire:link} target={global-api expire:link}
2020-08-27 19:56:44.383242 +0000 ROUTER_CORE (info) [C12][L42] Link attached: dir=in source={skupp3r-edge-site-skupper-router-57578fd697-7zrwk/bridge-server/$management expire:link} target={<none> expire:link}
2020-08-27 19:56:44.383251 +0000 ROUTER_CORE (info) [C12][L43] Link attached: dir=in source={338fb13d-cb9b-40b7-b622-f5694d0a9232/skupper-site-query expire:link} target={<none> expire:link}
2020-08-27 19:56:44.383260 +0000 ROUTER_CORE (info) [C12][L44] Link attached: dir=in source={mc/$skupper-service-sync expire:link} target={<none> expire:link}
2020-08-27 19:56:44.383269 +0000 ROUTER_CORE (info) [C12][L45] Link attached: dir=out source={<none> expire:link} target={mc/$skupper-service-sync expire:link}
2020-08-27 19:56:44.383293 +0000 ROUTER_CORE (info) [C12][L46] Link attached: dir=out source={} target={$management expire:link}
2020-08-27 19:56:44.383302 +0000 ROUTER_CORE (info) [C12][L47] Link attached: dir=in source={<dynamic> expire:link} target={}
2020-08-27 19:56:44.383311 +0000 ROUTER_CORE (info) [C12][L48] Link attached: dir=out source={} target={_$qd.addr_lookup expire:link}
2020-08-27 19:56:44.383317 +0000 ROUTER_CORE (info) [C12][L49] Link attached: dir=in source={<dynamic> expire:link} target={}edge qdstat
[root@skupper-router-57578fd697-7zrwk bin]# qdstat -c && qdstat -l
2020-08-27 20:00:11.746025 UTC
skupp3r-edge-site-skupper-router-57578fd697-7zrwk
Connections
id host container role dir security authentication tenant last dlv uptime
========================================================================================================================================================================================================================
2 127.0.0.1:51716 skupper-router-57578fd697-7zrwk_http_1026_to_amqp_edge-httptest normal in no-security anonymous-user - 000:00:19:12
3 127.0.0.1:51718 skupper-router-57578fd697-7zrwk_http_1027_to_amqp_global-api normal in no-security anonymous-user - 000:00:19:12
4 127.0.0.1:51720 bridge-server-skupper-router-57578fd697-7zrwk normal in no-security anonymous-user - 000:00:19:12
5 10.42.0.38:45848 qguw3g8FdylqahPnZtAxmhsBZCiRszso5f2VArE611QsabjAmrcrTQ normal in TLSv1.3(TLS_AES_128_GCM_SHA256) CN=skupper-messaging(x.509) - 000:00:19:11
6 10.42.0.38:45762 IUxpPCIoJP7a4efPAyS1bzlPASgtxslVvOVMzwudQgeOXFajLDEBKA normal in TLSv1.3(TLS_AES_128_GCM_SHA256) CN=skupper-messaging(x.509) 000:00:00:03 000:00:19:08
12 xx.xxx.xxx.xxx:45671 skupp3r-cloud-skupper-router-845f7bdf87-55qhg edge out TLSv1.3(TLS_AES_256_GCM_SHA384) x.509 000:00:00:00 000:00:03:27
13 127.0.0.1:46084 5751d92b-4f70-43b8-bb51-19ad81a6750c normal in no-security no-auth 000:00:00:00 000:00:00:00
2020-08-27 20:00:11.952788 UTC
skupp3r-edge-site-skupper-router-57578fd697-7zrwk
Router Links
type dir conn id id peer class addr phs cap pri undel unsett deliv presett psdrop acc rej rel mod delay rate stuck cred blkd
=================================================================================================================================================================================================================================
endpoint in 2 9 mobile edge-httptest 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 2 10 local temp.XSvPpBLu8Tmxi3Z 250 0 0 0 0 0 0 0 0 0 0 0 0 0 1000 -
endpoint in 3 12 mobile global-api 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 3 13 local temp.ibNUkWOkOG+drrg 250 0 0 0 0 0 0 0 0 0 0 0 0 0 1000 -
endpoint out 4 15 mobile skupp3r-edge-site-skupper-router-57578fd697-7zrwk/bridge-server/$management 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 1000 -
endpoint out 5 17 mobile 338fb13d-cb9b-40b7-b622-f5694d0a9232/skupper-site-query 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 10 -
endpoint in 5 19 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 6 20 mobile mc/$skupper-service-sync 0 250 0 0 0 453 0 0 453 0 0 0 0 0 0 7 -
endpoint in 6 22 mobile mc/$skupper-service-sync 0 250 0 0 0 229 0 0 229 0 0 0 0 0 0 250 -
endpoint out 12 37 local _edge 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint in 12 38 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint in 12 39 250 0 0 0 3 3 0 0 0 0 0 0 0 0 32 -
endpoint out 12 40 mobile edge-httptest 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 12 41 mobile global-api 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint in 12 42 mobile skupp3r-edge-site-skupper-router-57578fd697-7zrwk/bridge-server/$management 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint in 12 43 mobile 338fb13d-cb9b-40b7-b622-f5694d0a9232/skupper-site-query 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint in 12 44 mobile mc/$skupper-service-sync 0 250 0 0 0 41 0 0 41 0 0 0 0 0 0 250 -
endpoint out 12 45 mobile mc/$skupper-service-sync 0 250 0 0 0 41 0 0 41 0 0 0 0 0 0 250 -
endpoint out 12 46 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint in 12 47 250 0 0 0 0 0 0 0 0 0 0 0 0 0 100 -
endpoint out 12 48 250 0 0 0 2 2 0 0 0 0 0 2 0 0 32 -
endpoint in 12 49 250 0 0 0 2 2 0 0 0 0 0 0 0 0 250 -
endpoint in 14 52 mobile $management 0 250 0 0 0 2 0 0 2 0 0 0 0 0 0 250 -
endpoint out 14 53 local temp.vBNucN78G6jvn4h 250 0 0 0 1 1 0 0 0 0 0 0 0 0 1 -cloud qdstat
[root@skupper-router-845f7bdf87-55qhg bin]# qdstat -c && qdstat -l
2020-08-27 20:06:10.083653 UTC
skupp3r-cloud-skupper-router-845f7bdf87-55qhg
Connections
id host container role dir security authentication tenant last dlv uptime
==============================================================================================================================================================================================================================
2 127.0.0.1:50620 skupper-router-845f7bdf87-55qhg_amqp_global-api _to_http_10.48.8.53_5443 normal in no-security anonymous-user - 000:00:17:46
3 127.0.0.1:50622 skupper-router-845f7bdf87-55qhg_amqp_global-api _to_http_10.48.7.47_5443 normal in no-security anonymous-user - 000:00:17:46
1 127.0.0.1:50618 skupper-router-845f7bdf87-55qhg_amqp_edge-httptest _to_http_10.48.7.48_8000 normal in no-security anonymous-user - 000:00:17:46
5 127.0.0.1:50626 skupper-router-845f7bdf87-55qhg_http_1028_to_amqp_global-api normal in no-security anonymous-user - 000:00:17:46
4 127.0.0.1:50624 skupper-router-845f7bdf87-55qhg_http_1027_to_amqp_edge-httptest normal in no-security anonymous-user - 000:00:17:46
6 127.0.0.1:50628 bridge-server-skupper-router-845f7bdf87-55qhg normal in no-security anonymous-user - 000:00:17:46
11 127.0.0.1:57844 5_7fSByefrkl-8aLza6q_NAOoms1J2v4_dfSqkazXJ2G3WW2UDQrVQ normal in TLSv1.3(TLS_AES_128_GCM_SHA256) CN=skupper-messaging(x.509) - 000:00:09:27
10 127.0.0.1:57842 TbIVfl583-ZWHwiQ3YbdPxntezT7wjuHXpcdxcxzaunRi5FKwus4Ng normal in TLSv1.3(TLS_AES_128_GCM_SHA256) CN=skupper-messaging(x.509) 000:00:00:02 000:00:09:27
12 127.0.0.1:33006 skupp3r-edge-site-skupper-router-57578fd697-7zrwk edge in TLSv1.3(TLS_AES_256_GCM_SHA384) CN=skupper(x.509) 000:00:00:02 000:00:09:26
13 127.0.0.1:55298 3751e02f-9691-437b-becd-cc13595f47ad normal in no-security no-auth 000:00:00:00 000:00:00:00
2020-08-27 20:06:10.224702 UTC
skupp3r-cloud-skupper-router-845f7bdf87-55qhg
Router Links
type dir conn id id peer class addr phs cap pri undel unsett deliv presett psdrop acc rej rel mod delay rate stuck cred blkd
======================================================================================================================================================================================================================================
endpoint out 2 2 mobile global-api 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 10 -
endpoint in 5 3 mobile global-api 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 5 4 local temp.vPWzs4zAms+_FX8 250 0 0 0 0 0 0 0 0 0 0 0 0 0 1000 -
endpoint out 3 5 mobile global-api 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 10 -
endpoint in 4 6 mobile edge-httptest 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 4 7 local temp.Cj_aVni70FIPgUt 250 0 0 0 0 0 0 0 0 0 0 0 0 0 1000 -
endpoint out 1 8 mobile edge-httptest 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 10 -
endpoint out 6 9 mobile skupp3r-cloud-skupper-router-845f7bdf87-55qhg/bridge-server/$management 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 1000 -
endpoint out 11 27 mobile e6014882-211a-4aa3-acd9-7049c75ec4d3/skupper-site-query 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 10 -
endpoint out 10 28 mobile mc/$skupper-service-sync 0 250 0 0 0 226 0 0 226 0 0 0 0 0 0 9 -
endpoint in 11 29 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint in 10 30 mobile mc/$skupper-service-sync 0 250 0 0 0 113 0 0 113 0 0 0 0 0 0 250 -
endpoint in 12 31 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
edge-downlink out 12 32 edge skupp3r-edge-site-skupper-router-57578fd697-7zrwk 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 12 33 mobile _$qd.edge_addr_tracking 0 250 0 0 0 3 3 0 0 0 0 0 3 0 0 32 -
endpoint in 12 34 mobile edge-httptest 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint in 12 35 mobile global-api 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 12 36 mobile skupp3r-edge-site-skupper-router-57578fd697-7zrwk/bridge-server/$management 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 12 37 mobile 338fb13d-cb9b-40b7-b622-f5694d0a9232/skupper-site-query 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 12 38 mobile mc/$skupper-service-sync 0 250 0 0 0 113 0 0 113 0 0 0 0 0 0 250 -
endpoint in 12 39 mobile mc/$skupper-service-sync 0 250 0 0 0 113 0 0 113 0 0 0 0 0 0 250 -
endpoint in 12 40 mobile $management 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 12 41 local temp.4Q8AUpV2+OfU7SP 250 0 0 0 0 0 0 0 0 0 0 0 0 0 100 -
endpoint in 12 42 mobile _$qd.addr_lookup 0 250 0 0 0 2 2 0 0 0 0 0 0 0 0 32 -
endpoint out 12 43 local temp.atnjfB2iS9ecfvq 250 0 0 0 2 2 0 0 0 0 0 0 0 0 250 -
endpoint in 14 46 mobile $management 0 250 0 0 0 2 0 0 2 0 0 0 0 0 0 250 -
endpoint out 14 47 local temp.YXsQswx5tBoRo5I 250 0 0 0 1 1 0 0 0 0 0 0 0 0 1 -Can you get the logs for the cloud bridge-server again. I suspect they should have been updated (the configmap changes take some time to be propagated to the pod - this is something that will be improved in the next release). We should see some connectors being created there for the targets defined for each service.
grs
commented
4 years ago The error in the router log is odd, though it won't affect the scenario. It looks like that pod fails to query the kubernetes API server for some reason.
grs
commented
4 years ago The edge router log again shows that it seems to be getting disconnected continually from the cloud router. It is not an issue with the certs or anything as the connection does get established, it just seems to then be immediately disconnected. My guess would be that this may be something related to the loadbalancer service on the public cloud used here? Seems to be about every 8 minutes. May I ask the details of the cloud kubernetes cluster? I.e. version, vendor?
Kampe
commented
4 years ago The edge router log again shows that it seems to be getting disconnected continually from the cloud router. It is not an issue with the certs or anything as the connection does get established, it just seems to then be immediately disconnected. My guess would be that this may be something related to the loadbalancer service on the public cloud used here? Seems to be about every 8 minutes. May I ask the details of the cloud kubernetes cluster? I.e. version, vendor?
cloud clusters: GKE version 1.16.11-gke.5
edge cluster: k3s v1.18.6+k3s1
grs
commented
4 years ago Ok, I'll look into that separately. For now lets try and debug the services. Maybe the first test if possible is to verify whether they work within the cloud cluster itself?
Kampe
commented
4 years ago Ok, I'll look into that separately. For now lets try and debug the services. Maybe the first test if possible is to verify whether they work within the cloud cluster itself?
just hitting services on each of their healthcheck endpoints returns 200 internally, however contemplating complete tear down of cloud infra and will report back, been swapping global-api between tcp and http proxies (if there's any inconsistancies in the logs it's due to that)
grs
commented
4 years ago Are you able to run a client against the actual service in that cloud cluster? Just to verify that the kubernetes service as skupperized and the bridges etc are all ok there.
Kampe
commented
4 years ago yep, that's what I was testing, from another service in the same namespace I'm getting 200's
grs
commented
4 years ago Ok, but still not working when trying from edge? Is it intermittent? Or it never works? (Could we get the logs and qdstats again after trying that?)
Kampe
commented
4 years ago From edge I never get any requests through to the cloud services
tail of edge bridge logs:
│ 2020-08-27T20:15:50.082Z bridge-server info no tcpListeners defined
│ 2020-08-27T20:15:50.082Z bridge-server info checking for org.apache.qpid.dispatch.router.tcpListener updates...
│ 2020-08-27T21:02:19.970Z bridge-server info outgoing request global-api:5443 (global-api/xxxxxxxxxxxx)
│ 2020-08-27T21:02:19.971Z bridge-server info client sending message: GET global-api/xxxxxxxxxxxx
│ 2020-08-27T21:03:20.032Z bridge-server error Cannot find correlation id for failed delivery Cloud bridge server fills up with these:
2020-08-27T21:05:20.158Z bridge-server error request to http://global-api/xxxxxxxxxxxxxxxx timed out
2020-08-27T21:05:20.167Z bridge-server error Cannot find correlation id for failed delivery Cloud router tail:
│ 2020-08-27 20:15:50.073638 +0000 SERVER (info) [C10] Accepted connection to 0.0.0.0:45671 from 127.0.0.1:38376
│ 2020-08-27 20:15:50.254670 +0000 ROUTER (info) [C10] Connection Opened: dir=in host=127.0.0.1:38376 vhost= encrypted=TLSv1.3 auth=EXTERNAL user=CN=skupper container_id=skupp3r-edge-site-skupper-router-57578fd697-gzrhp props={:product="qpid-dispatch-router", :version="1.13.0", :"qd.conn-id"=1}
│ 2020-08-27 20:15:50.311969 +0000 ROUTER_CORE (info) [C10][L27] Link attached: dir=in source={<none> expire:sess} target={<none> expire:sess}
│ 2020-08-27 20:15:50.312182 +0000 ROUTER_CORE (info) [C10][L28] Link attached: dir=out source={skupp3r-edge-site-skupper-router-57578fd697-gzrhp expire:link caps::"qd.router-edge-downlink"} target={<none> expire:sess caps::"qd.router-edge-downlink"}
│ 2020-08-27 20:15:50.312234 +0000 ROUTER_CORE (info) [C10][L29] Link attached: dir=out source={_$qd.edge_addr_tracking expire:link} target={<none> expire:sess}
│ 2020-08-27 20:15:50.312266 +0000 ROUTER_CORE (info) [C10][L30] Link attached: dir=in source={<none> expire:sess} target={edge-httptest expire:link}
│ 2020-08-27 20:15:50.312291 +0000 ROUTER_CORE (info) [C10][L31] Link attached: dir=in source={<none> expire:sess} target={global-api expire:link}
│ 2020-08-27 20:15:50.312322 +0000 ROUTER_CORE (info) [C10][L32] Link attached: dir=out source={skupp3r-edge-site-skupper-router-57578fd697-gzrhp/bridge-server/$management expire:link} target={<none> expire:sess}
│ 2020-08-27 20:15:50.312484 +0000 ROUTER_CORE (info) [C10][L33] Link attached: dir=in source={<none> expire:sess} target={$management expire:link}
│ 2020-08-27 20:15:50.312530 +0000 ROUTER_CORE (info) [C10][L34] Link attached: dir=out source={<dynamic> expire:link} target={<none> expire:sess}
│ 2020-08-27 20:15:50.312557 +0000 ROUTER_CORE (info) [C10][L35] Link attached: dir=in source={<none> expire:sess} target={_$qd.addr_lookup expire:link}
│ 2020-08-27 20:15:50.312597 +0000 ROUTER_CORE (info) [C10][L36] Link attached: dir=out source={<dynamic> expire:link} target={<none> expire:sess}
│ 2020-08-27 20:15:51.563430 +0000 ROUTER_CORE (info) [C10][L37] Link attached: dir=out source={338fb13d-cb9b-40b7-b622-f5694d0a9232/skupper-site-query expire:link} target={<none> expire:sess}
│ 2020-08-27 20:15:51.626646 +0000 ROUTER_CORE (info) [C10][L38] Link attached: dir=out source={mc/$skupper-service-sync expire:link} target={<none> expire:sess}
│ 2020-08-27 20:15:51.683094 +0000 ROUTER_CORE (info) [C10][L39] Link attached: dir=in source={<none> expire:sess} target={mc/$skupper-service-sync expire:link}
│ 2020-08-27 20:19:42.213690 +0000 ROUTER_CORE (info) [C3][L3] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 20:19:42.213764 +0000 ROUTER_CORE (info) [C4][L4] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 20:20:13.222182 +0000 ROUTER_CORE (info) [C3][L3] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 20:20:13.222257 +0000 ROUTER_CORE (info) [C4][L4] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 20:21:15.242895 +0000 ROUTER_CORE (info) [C3][L3] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 20:21:15.242978 +0000 ROUTER_CORE (info) [C4][L4] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 20:22:17.267005 +0000 ROUTER_CORE (info) [C3][L3] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 20:22:17.267088 +0000 ROUTER_CORE (info) [C4][L4] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 20:23:19.290301 +0000 ROUTER_CORE (info) [C3][L3] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 20:23:19.290626 +0000 ROUTER_CORE (info) [C4][L4] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 20:24:21.319450 +0000 ROUTER_CORE (info) [C3][L3] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 20:24:21.319533 +0000 ROUTER_CORE (info) [C4][L4] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 20:25:11.497882 +0000 ROUTER_CORE (info) [C3][L40] Link attached: dir=in source={<none> expire:sess} target={<none> expire:sess}
│ 2020-08-27 20:25:23.343189 +0000 ROUTER_CORE (info) [C3][L3] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 20:25:23.343288 +0000 ROUTER_CORE (info) [C4][L4] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 20:26:25.364076 +0000 ROUTER_CORE (info) [C3][L3] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 secondsedge router
2020-08-27 20:15:51.651499 +0000 ROUTER_CORE (info) [C1][L23] Link attached: dir=out source={<none> expire:link} target={mc/$skupper-service-sync expire:link}
│ 2020-08-27 21:02:22.060416 +0000 ROUTER_CORE (info) [C2][L4] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 21:02:22.060493 +0000 ROUTER_CORE (info) [C1][L11] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 21:03:24.091958 +0000 ROUTER_CORE (info) [C2][L4] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 21:03:24.091998 +0000 ROUTER_CORE (info) [C1][L11] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 21:04:26.121157 +0000 ROUTER_CORE (info) [C2][L4] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 21:04:26.121256 +0000 ROUTER_CORE (info) [C1][L11] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 21:05:28.161132 +0000 ROUTER_CORE (info) [C2][L4] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 21:05:28.161182 +0000 ROUTER_CORE (info) [C1][L11] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 21:06:30.210465 +0000 ROUTER_CORE (info) [C2][L4] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 21:06:30.210541 +0000 ROUTER_CORE (info) [C1][L11] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 21:07:32.232326 +0000 ROUTER_CORE (info) [C2][L4] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds
│ 2020-08-27 21:07:32.232460 +0000 ROUTER_CORE (info) [C1][L11] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 secondsqdstats:
edge:
[root@skupper-router-57578fd697-gzrhp bin]# qdstat -c && qdstat -l
2020-08-27 21:11:17.616176 UTC
skupp3r-edge-site-skupper-router-57578fd697-gzrhp
Connections
id host container role dir security authentication tenant last dlv uptime
========================================================================================================================================================================================================================
3 127.0.0.1:36152 skupper-router-57578fd697-gzrhp_http_1026_to_amqp_edge-httptest normal in no-security anonymous-user - 000:00:55:25
2 127.0.0.1:36150 skupper-router-57578fd697-gzrhp_http_1027_to_amqp_global-api normal in no-security anonymous-user 000:00:00:55 000:00:55:25
4 127.0.0.1:36154 bridge-server-skupper-router-57578fd697-gzrhp normal in no-security anonymous-user - 000:00:55:25
1 xx.xxx.xxx.xxx:45671 skupp3r-cloud-skupper-router-845f7bdf87-s2zxn edge out TLSv1.3(TLS_AES_256_GCM_SHA384) x.509 000:00:00:00 000:00:55:25
5 10.42.0.38:58512 ocfH9kM5ZKuVHkeIQhlWhmIgCxvhCpoHvf5sRkuReprTuN0Q1QpWAQ normal in TLSv1.3(TLS_AES_128_GCM_SHA256) CN=skupper-messaging(x.509) - 000:00:55:24
6 10.42.0.38:58514 IqoVo6hbh1N4N1y6ruG2YInbZ41FSm-UPiGENrP54O1ovIzwR4x2aQ normal in TLSv1.3(TLS_AES_128_GCM_SHA256) CN=skupper-messaging(x.509) 000:00:00:01 000:00:55:24
7 127.0.0.1:44808 9ac0fde4-26c3-42d0-b267-fc9a39c01470 normal in no-security no-auth 000:00:00:00 000:00:00:00
2020-08-27 21:11:17.798038 UTC
skupp3r-edge-site-skupper-router-57578fd697-gzrhp
Router Links
type dir conn id id peer class addr phs cap pri undel unsett deliv presett psdrop acc rej rel mod delay rate stuck cred blkd
==================================================================================================================================================================================================================================
endpoint in 3 2 mobile edge-httptest 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 3 3 local temp.tuF97jT+Xp_BatQ 250 0 0 0 0 0 0 0 0 0 0 0 0 0 1000 -
endpoint in 2 4 mobile global-api 0 250 0 0 0 6 0 0 0 0 0 6 6 0 0 250 -
endpoint out 2 5 local temp.F05opszFXqbqMXr 250 0 0 0 4 0 0 4 0 0 0 0 0 0 996 -
endpoint out 4 6 mobile skupp3r-edge-site-skupper-router-57578fd697-gzrhp/bridge-server/$management 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 1000 -
endpoint out 1 7 local _edge 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint in 1 8 250 0 0 0 4 0 0 4 0 0 0 0 0 0 250 -
endpoint in 1 9 250 0 0 0 3 3 0 0 0 0 0 0 0 0 32 -
endpoint out 1 10 mobile edge-httptest 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 1 11 mobile global-api 0 250 0 0 0 6 0 0 0 0 0 6 6 0 0 250 -
endpoint in 1 12 mobile skupp3r-edge-site-skupper-router-57578fd697-gzrhp/bridge-server/$management 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 1 13 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint in 1 14 250 0 0 0 0 0 0 0 0 0 0 0 0 0 100 -
endpoint out 1 15 250 0 0 0 5 5 0 0 0 0 0 2 0 0 32 -
endpoint in 1 16 250 0 0 0 5 5 0 0 0 0 0 0 0 0 250 -
endpoint out 5 17 mobile 338fb13d-cb9b-40b7-b622-f5694d0a9232/skupper-site-query 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 10 -
endpoint in 1 18 mobile 338fb13d-cb9b-40b7-b622-f5694d0a9232/skupper-site-query 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint in 5 19 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 6 20 mobile mc/$skupper-service-sync 0 250 0 0 0 1330 0 0 1330 0 0 0 0 0 0 10 -
endpoint in 1 21 mobile mc/$skupper-service-sync 0 250 0 0 0 665 0 0 665 0 0 0 0 0 0 250 -
endpoint in 6 22 mobile mc/$skupper-service-sync 0 250 0 0 0 665 0 0 665 0 0 0 0 0 0 250 -
endpoint out 1 23 mobile mc/$skupper-service-sync 0 250 0 0 0 665 0 0 665 0 0 0 0 0 0 250 -
endpoint in 8 26 mobile $management 0 250 0 0 0 2 0 0 2 0 0 0 0 0 0 250 -
endpoint out 8 27 local temp.jfZoHn2UG4C1Ce5 250 0 0 0 1 1 0 0 0 0 0 0 0 0 1 -cloud:
[root@skupper-router-845f7bdf87-s2zxn bin]# qdstat -c && qdstat -l
2020-08-27 21:13:22.532331 UTC
skupp3r-cloud-skupper-router-845f7bdf87-s2zxn
Connections
id host container role dir security authentication tenant last dlv uptime
==============================================================================================================================================================================================================================
2 127.0.0.1:50592 skupper-router-845f7bdf87-s2zxn_amqp_global-api _to_http_10.48.8.53_5443 normal in no-security anonymous-user 000:00:00:01 000:00:58:18
3 127.0.0.1:50594 skupper-router-845f7bdf87-s2zxn_amqp_edge-httptest _to_http_10.48.7.48_8000 normal in no-security anonymous-user 000:00:00:08 000:00:58:18
4 127.0.0.1:50596 skupper-router-845f7bdf87-s2zxn_http_1027_to_amqp_edge-httptest normal in no-security anonymous-user 000:00:00:08 000:00:58:18
5 127.0.0.1:50598 skupper-router-845f7bdf87-s2zxn_http_1028_to_amqp_global-api normal in no-security anonymous-user 000:00:00:01 000:00:58:18
1 127.0.0.1:50590 skupper-router-845f7bdf87-s2zxn_amqp_global-api _to_http_10.48.7.47_5443 normal in no-security anonymous-user 000:00:00:01 000:00:58:18
6 127.0.0.1:50600 bridge-server-skupper-router-845f7bdf87-s2zxn normal in no-security anonymous-user - 000:00:58:18
7 127.0.0.1:60584 5kam1W2sBkkeo_01SBBlKXLQfLkQwRRHwtY8JJ2p8TLvdbVSyXd43g normal in TLSv1.3(TLS_AES_128_GCM_SHA256) CN=skupper-messaging(x.509) - 000:00:58:13
8 127.0.0.1:60586 5oiHyZ2PCDRPZ3HzpF5tX8NqxKAu_AeRhgv2unb_hkHZwNmILaamoA normal in TLSv1.3(TLS_AES_128_GCM_SHA256) CN=skupper-messaging(x.509) 000:00:00:01 000:00:58:13
10 127.0.0.1:38376 skupp3r-edge-site-skupper-router-57578fd697-gzrhp edge in TLSv1.3(TLS_AES_256_GCM_SHA384) CN=skupper(x.509) 000:00:00:01 000:00:57:31
11 127.0.0.1:38768 e157034b-19ab-46ed-bfc2-3268678fa0e2 normal in no-security no-auth 000:00:00:00 000:00:00:01
2020-08-27 21:13:22.781440 UTC
skupp3r-cloud-skupper-router-845f7bdf87-s2zxn
Router Links
type dir conn id id peer class addr phs cap pri undel unsett deliv presett psdrop acc rej rel mod delay rate stuck cred blkd
==========================================================================================================================================================================================================================================
endpoint out 2 2 mobile global-api 0 250 0 0 125 1506 0 0 0 0 0 1381 1381 0 0 10 -
endpoint out 3 3 mobile edge-httptest 0 250 0 0 250 13750 0 0 0 0 0 13500 13500 0 0 9 -
endpoint in 4 4 mobile edge-httptest 0 250 0 0 250 13750 0 0 0 0 0 13500 13500 0 0 13 -
endpoint out 4 5 local temp.AYLtiUhyyIbb0ib 250 0 0 0 12250 0 0 12250 0 0 0 0 0 0 1000 -
endpoint in 5 6 mobile global-api 0 250 0 0 250 3000 0 0 0 0 0 2750 2750 0 0 5 -
endpoint out 5 7 local temp.aGuYuuigpnlwGfd 250 0 0 0 1500 0 0 1500 0 0 0 0 50 0 1000 -
endpoint out 1 8 mobile global-api 0 250 0 0 125 1500 0 0 0 0 0 1375 1375 0 0 10 -
endpoint out 6 9 mobile skupp3r-cloud-skupper-router-845f7bdf87-s2zxn/bridge-server/$management 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 1000 -
endpoint out 7 10 mobile e6014882-211a-4aa3-acd9-7049c75ec4d3/skupper-site-query 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 10 -
endpoint in 7 11 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 8 12 mobile mc/$skupper-service-sync 0 250 0 0 0 1395 0 0 1395 0 0 0 0 0 0 10 -
endpoint in 8 13 mobile mc/$skupper-service-sync 0 250 0 0 0 698 0 0 698 0 0 0 0 0 0 250 -
endpoint in 10 27 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
edge-downlink out 10 28 edge skupp3r-edge-site-skupper-router-57578fd697-gzrhp 250 0 0 0 6 0 0 6 0 0 0 0 0 0 250 -
endpoint out 10 29 mobile _$qd.edge_addr_tracking 0 250 0 0 0 3 3 0 0 0 0 0 3 0 0 32 -
endpoint in 10 30 mobile edge-httptest 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint in 10 31 mobile global-api 0 250 0 0 0 6 0 0 0 0 0 6 6 0 0 250 -
endpoint out 10 32 mobile skupp3r-edge-site-skupper-router-57578fd697-gzrhp/bridge-server/$management 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint in 10 33 mobile $management 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 10 34 local temp.2aL31EbPIhSQOlZ 250 0 0 0 0 0 0 0 0 0 0 0 0 0 100 -
endpoint in 10 35 mobile _$qd.addr_lookup 0 250 0 0 0 5 5 0 0 0 0 0 0 0 0 32 -
endpoint out 10 36 local temp.HyP+cLjXpP4yIKS 250 0 0 0 5 5 0 0 0 0 0 0 0 0 250 -
endpoint out 10 37 mobile 338fb13d-cb9b-40b7-b622-f5694d0a9232/skupper-site-query 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 10 38 mobile mc/$skupper-service-sync 0 250 0 0 0 690 0 0 690 0 0 0 0 0 0 250 -
endpoint in 10 39 mobile mc/$skupper-service-sync 0 250 0 0 0 690 0 0 690 0 0 0 0 0 0 250 -
endpoint in 3 40 250 0 0 0 12250 0 0 12250 0 0 0 0 0 0 250 -
endpoint in 2 41 250 0 0 0 756 0 0 756 0 0 0 0 25 0 250 -
endpoint in 1 42 250 0 0 0 750 0 0 750 0 0 0 0 25 0 250 -
endpoint in 12 45 mobile $management 0 250 0 0 0 2 0 0 2 0 0 0 0 0 0 250 -
endpoint out 12 46 local temp.H5Ahpax3_e0lhpv 250 0 0 0 1 1 0 0 0 0 0 0 0 0 1 -What does the global-api kubernetes service yaml look like on cloud cluster? Also the configmaps skupper-internal and skupper-services? And what port is the server targetted by global-api listening on?
Kampe
commented
4 years ago global-api service yaml - service is listening on 5443
apiVersion: v1
kind: Service
metadata:
name: global-api
labels:
app: global-api
service: global-api
annotations:
skupper.io/proxy: http
spec:
selector:
app: global-api
ports:
- name: http
port: 5443skupper-internal configmap
Name: skupper-internal
Namespace: default
Labels: <none>
Annotations: <none>
Data
====
bridges.json:
----
[["httpConnector",{"address":"global-api","aggregation":"","eventChannel":false,"host":"10.48.7.47","hostOverride":"","http2":false,"name":"global-api@10.48.7.47","port":5443,"siteId":"e6014882-211a-4aa3-acd9-7049c75ec4d3"}],["httpConnector",{"address":"global-api","aggregation":"","eventChannel":false,"host":"10.48.8.53","hostOverride":"","http2":false,"name":"global-api@10.48.8.53","port":5443,"siteId":"e6014882-211a-4aa3-acd9-7049c75ec4d3"}],["httpConnector",{"address":"edge-httptest","aggregation":"","eventChannel":false,"host":"10.48.7.48","hostOverride":"","http2":false,"name":"edge-httptest@10.48.7.48","port":8000,"siteId":"e6014882-211a-4aa3-acd9-7049c75ec4d3"}],["httpListener",{"address":"edge-httptest","aggregation":"","eventChannel":false,"host":"0.0.0.0","hostOverride":"","http2":false,"name":"edge-httptest","port":1027,"siteId":"e6014882-211a-4aa3-acd9-7049c75ec4d3"}],["httpListener",{"address":"global-api","aggregation":"","eventChannel":false,"host":"0.0.0.0","hostOverride":"","http2":false,"name":"global-api","port":1028,"siteId":"e6014882-211a-4aa3-acd9-7049c75ec4d3"}]]
Events: <none>skupper services configmap
Name: skupper-services
Namespace: default
Labels: <none>
Annotations: <none>
Data
====
global-api:
----
{"address":"global-api","protocol":"http","port":5443,"targets":[{"name":"global-api","selector":"app=global-api","targetPort":5443}],"origin":"annotation"}
edge-httptest:
----
{"address":"edge-httptest","protocol":"http","port":8000,"targets":[{"name":"edge-httptest","selector":"app=edge-httptest","targetPort":8000}],"origin":"annotation"}
Events: <none>To isolate and test the services individually I'm able to port-forward myself on to those services and hit them directly and get a 200 response from both http://global-api:5443 and http://edge-httptest:8000 - for your sanity :D
grs
commented
4 years ago Looking through the stats, at the time they were collected there had been 6 requests from the edge site for global-api. It looks like all of these failed. However there were also 3000 requests from the cloud site itself on global-api. Does that match your expectations? These also seem to have all failed.
Is the service yaml above what is actually active? Or was it the yaml as originally applied? (With the annotation, skupper should update the selector so if the latter could you check what the active yaml looks like? If this is the active yaml, then that would mean requests to that service are not going through skupper - which would explain the 200s you saw which don't seem to be represented in the stats).
The failure seems to be from skupper to the two global-api server pods (10.48/8.53 and 10.48.7.47). From the log snippet it sounds like it is timing out (there is a fixed timeout of 60 seconds) rather than getting an error (that also matches the stats and router logs).
If the 3000 requests aren't something you would expect then my suspicion would be some sort of configuration loop perhaps, though I can't see anything incorrect.
Kampe
commented
4 years ago The service yaml there is what is originally applied, there should be nothing calling that 3000 times in the cloud as this is my development cluster so that seems a little odd. I do have istio (permissive mode) enabled within some pods in this namespace, however there's no envoy proxies on these pods in particular nor are they part of any mesh at the moment.
Name: global-api
Namespace: default
Labels: app=global-api
service=global-api
Annotations: skupper.io/proxy: http
Selector: application=skupper-router,skupper.io/component=router
Type: ClusterIP
IP: 10.115.10.142
Port: http 5443/TCP
TargetPort: 1028/TCP
Endpoints: 10.48.7.50:1028
Session Affinity: None
Events: <none>Can I suggest deleting the global-api service and then annotating the deployment that implements that service with the skupper.io/proxy annotation, or using the skupper cli's 'expose' command?
Kampe
commented
4 years ago I did manage to get this working with no extra configuration on my end, the state of that service was the issue :)
grs
commented
4 years ago @Kampe glad to hear you got that working! Thanks for the update!
Kampe
commented
4 years ago Thank you for your work on this, incredible tool!
Is it possible to allow for installation of skupper as an operator? For my use case, it's a pain including the binary in edge bootstrapping, however I'm already managing a handful of operator resources on init, would be ideal if I could just pull down a manifest for the operator and just apply the secret.yaml to skupper up