Kampe
commented
3 years ago I noticed the services didn't have a cluster IP when listing the services out, so I deleted and recreated them - now they do. I also disabled the TCP service for the time being and still have stuck deliveries just using http proxy and sending the requests locally via a port-forward over the skuppered service to no avail:
2020-12-10 07:37:14.639832 +0000 HTTP_ADAPTOR (info) Accepting HTTP/1.x connection on 0.0.0.0:1026
2020-12-10 07:37:14.640171 +0000 ROUTER_CORE (info) [C28] Connection Opened: dir=in host=127.0.0.1:38250 vhost= encrypted=no auth=no user= container_id=HTTP/1.x Adaptor props=
2020-12-10 07:37:14.640413 +0000 ROUTER_CORE (info) [C28][L66] Link attached: dir=out source={(dyn)<none> expire:link} target={<none> expire:link}
2020-12-10 07:37:14.640468 +0000 ROUTER_CORE (info) [C28][L67] Link attached: dir=in source={<none> expire:link} target={cloud-api expire:link}
2020-12-10 07:37:50.844384 +0000 ROUTER_CORE (info) [C28][L67] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 secondsHowever the router still hasn't crashed yet. Here's what it's links look like currently:
Router Links
type dir conn id id peer class addr phs cap pri undel unsett deliv presett psdrop acc rej rel mod delay rate stuck cred blkd
==================================================================================================================================================================================================================
endpoint in 1 2 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 1 3 mobile cloud-api 0 250 0 0 7 7 0 0 0 0 0 0 0 0 7 250 -
endpoint in 2 4 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 2 5 mobile cloud-api 0 250 0 0 7 7 0 0 0 0 0 0 0 0 7 250 -
endpoint out 3 6 mobile mc/$skupper-service-sync 0 250 0 0 0 272 0 0 272 0 0 0 0 0 0 8 -
endpoint out 4 7 mobile 92f5bd9b-f921-4408-aa22-4ccd3f5f2c6b/skupper-site-query 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 10 -
endpoint in 3 8 mobile mc/$skupper-service-sync 0 250 0 0 0 136 0 0 136 0 0 0 0 0 0 250 -
endpoint in 4 9 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 5 10 local temp.3zy7TuBZKHic_Hb 250 0 0 0 5 5 0 0 0 0 0 0 0 0 10 -
endpoint in 5 11 mobile $management 0 250 0 0 0 5 0 0 5 0 0 0 0 0 0 250 -
endpoint in 5 12 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint in 6 13 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
edge-downlink out 6 14 edge test-edge-skupper-router-7f45bdfb7c-92pww 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 6 15 mobile _$qd.edge_addr_tracking 0 250 0 0 0 2 2 0 0 0 0 0 2 0 0 32 -
endpoint out 6 16 mobile d5f5e229-5b7e-4553-97d3-24591e1f9555/skupper-site-query 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 6 17 mobile mc/$skupper-service-sync 0 250 0 0 0 136 0 0 136 0 0 0 0 0 0 250 -
endpoint in 6 18 mobile mc/$skupper-service-sync 0 250 0 0 0 136 0 0 136 0 0 0 0 0 0 250 -
endpoint in 6 19 mobile cloud-api 0 250 0 0 3 3 0 0 0 0 0 0 0 0 3 250 -
endpoint in 6 20 mobile $management 0 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 6 21 local temp.aBG0HwycWi9015q 250 0 0 0 0 0 0 0 0 0 0 0 0 0 100 -
endpoint in 6 22 mobile _$qd.addr_lookup 0 250 0 0 0 1 1 0 0 0 0 0 0 0 0 32 -
endpoint out 6 23 local temp.D2t5AWqAc9Ww9lK 250 0 0 0 1 1 0 0 0 0 0 0 0 0 250 -
endpoint out 7 24 local temp.eudw80wKeS5p_cd 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint out 8 25 local temp.Iofgp72C4VN_Sqk 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint in 8 26 mobile cloud-api 0 250 0 0 1 1 0 0 0 0 0 0 0 0 1 250 -
endpoint in 7 27 mobile cloud-api 0 250 0 0 1 1 0 0 0 0 0 0 0 0 1 250 -
endpoint out 28 66 local temp.hdDlLM2ylwQW4Ut 250 0 0 0 0 0 0 0 0 0 0 0 0 0 250 -
endpoint in 28 67 mobile cloud-api 0 250 0 0 1 1 0 0 0 0 0 0 0 0 1 250 -
endpoint in 29 68 mobile $management 0 250 0 0 0 2 0 0 2 0 0 0 0 0 0 250 -
endpoint out 29 69 local temp.JuL682dfZ_gHiQU 250 0 0 0 1 1 0 0 0 0 0 0 0 0 1 -
grs
kungfuchicken
ted-ross
Hello!
Just updated my clusters to utilize 0.4.0 of the site-controller as well as the new service-controller:0.4.0
Ran into some very interesting issues attempting to utilize my services, currently testing the HTTP endpoint manually, while I also have a test service running testing the TCP proxy. Here's the logs from the router before it crashed.
Here's how we currently configure skupper:
cloud hubedgewe have two services exposed:
Before the (cloud hub) router crashed I hopped on the pod and ran
qdstat -land noticed there were many links piling up for the http transfer. Here's an example of them.Be sure to let me know if there's any other information you'd be interested to see.