Need help with webpa docker-compose setup

[karthika-ab]

Hi , We are setting up a minimal webpa cluster with tr1d1um,talaria,petasos,scytale and prometheus using docker-compose following below documentation https://github.com/xmidt-org/xmidt/tree/master/deploy/docker-compose.The simulator tests given in the above link are all pass.

But we are not able to perform get and set operations from the parodus client (RDKB rpi) which is outside the docker environment as it is giving a HTTP 403 response.

root@RaspberryPi-Gateway:~# curl -i -H "Authorization: Basic dXNlcjpwYXNz" "http://192.168.1.11:6100/api/v2/device/mac:b827ebbed712/config?names=Device.DeviceInfo.Manufacturer" -H "X-Xmidt-Partner-ID: comcast,nbc,sky" HTTP/1.1 403 Forbidden X-Scytale-Build: unkown X-Scytale-Flavor: mint X-Scytale-Region: east X-Scytale-Server: 29a2b5ddfbdc X-Scytale-Start-Time: 01 Jul 21 10:46 UTC X-Talaria-Build: unkown X-Talaria-Flavor: mint X-Talaria-Region: east X-Talaria-Server: 779c0c610c6e X-Talaria-Start-Time: 01 Jul 21 10:46 UTC X-Tr1d1um-Build: unkown X-Tr1d1um-Flavor: mint X-Tr1d1um-Region: east X-Tr1d1um-Server: 996fb098e8f9 X-Tr1d1um-Start-Time: 01 Jul 21 10:46 UTC X-Webpa-Transaction-Id: nR0fW8v7WdJUV5LgWrsC1g X-Xmidt-Span: "http://779c0c610c6e:6200/api/v2/device/send","2021-07-01T11:00:50Z","2.172427ms" Date: Thu, 01 Jul 2021 11:00:50 GMT Content-Length: 0

The talaria logs have the following errors seen:

{"caller":"manager.go:215","id":"mac:b827ebbed712","level":"error","msg":"**missing security information**","ts":"2021-07-01T10:52:08.904603746Z"} {"caller":"manager.go:219","convey":{"boot-time":1625475738,"fw-name":"rdkb-generic-broadband-image_rdk-next_20210531050507","hw-manufacturer":"Raspberry","hw-model":"RPI","hw-serial-number":"000000002dbed712","webpa-interface-used":"erouter0","webpa-last-reconnect-reason":"webpa_process_starts","webpa-protocol":"PARODUS-2.0-1.1.4-22-gbdc2733"},"id":"mac:b827ebbed712","level":"info","ts":"2021-07-01T10:52:08.904651036Z"}

tr1d1um logs:

{"caller":"utils.go:83","duration":"3.174742ms","level":"info","msg":"record","request":{"address":"192.168.1.6:49682","path":"/api/v2/device/mac:b827ebbed712/config","query":"names=Device.DeviceInfo.Manufacturer","method":"GET"},"response":{"code":403,"headers":{"X-Scytale-Build":["unkown"],"X-Scytale-Flavor":["mint"],"X-Scytale-Region":["east"],"X-Scytale-Server":["29a2b5ddfbdc"],"X-Scytale-Start-Time":["01 Jul 21 10:46 UTC"],"X-Talaria-Build":["unkown"],"X-Talaria-Flavor":["mint"],"X-Talaria-Region":["east"],"X-Talaria-Server":["779c0c610c6e"],"X-Talaria-Start-Time":["01 Jul 21 10:46 UTC"],"X-Tr1d1um-Build":["unkown"],"X-Tr1d1um-Flavor":["mint"],"X-Tr1d1um-Region":["east"],"X-Tr1d1um-Server":["996fb098e8f9"],"X-Tr1d1um-Start-Time":["01 Jul 21 10:46 UTC"],"X-Webpa-Transaction-Id":["2P5vZmNk-1lcVS-9PHpuFw"],"X-Xmidt-Span":["\"http://779c0c610c6e:6200/api/v2/device/send\",\"2021-07-01T10:54:18Z\",\"1.357226ms\""]}},"satClientID":"user","tid":"2P5vZmNk-1lcVS-9PHpuFw","ts":"2021-07-01T10:54:18.605668225Z"}

scytale logs

Entrypoint script for scytale Server started. {"configurationFile":"/etc/scytale/scytale.yaml","level":"info","msg":"initialized Viper environment","ts":"2021-07-01T11:20:37.818348785Z"} {"could not create CPU profile: ":"open cpuprofile: permission denied","ts":"2021-07-01T11:20:37.818814738Z"} {"configurationFile":"/etc/scytale/scytale.yaml","level":"info","ts":"2021-07-01T11:20:38.022351022Z"} {"enabled":false,"level":"info","msg":"tracing status","ts":"2021-07-01T11:20:38.022516749Z"} {"level":"info","msg":"using consul for service discovery","ts":"2021-07-01T11:20:38.023459546Z"} {"datacenter":"dc0","instances":1,"level":"info","passingOnly":true,"service":"talaria","tags":"[stage=dev flavor=mint]","ts":"2021-07-01T11:20:38.202480577Z"} {"caller":"primaryHandler.go:238","level":"error","msg":"creating primary handler","ts":"2021-07-01T11:20:38.202852938Z"} {"level":"info","msg":"using service discovery for fanout","ts":"2021-07-01T11:20:38.202875616Z"} {"datacenter":"dc0","eventCount":0,"level":"info","msg":"subscription monitor starting","passingOnly":true,"service":"talaria","tags":["stage=dev","flavor=mint"],"ts":"2021-07-01T11:20:38.204161659Z"} {"datacenter":"dc0","eventCount":1,"instances":["http://779c0c610c6e:6200"],"level":"error","msg":"service discovery update","passingOnly":true,"service":"talaria","tags":["stage=dev","flavor=mint"],"ts":"2021-07-01T11:20:38.204208127Z"} {"bindAddress":":6300","level":"error","msg":"starting server","serverName":"scytale","ts":"2021-07-01T11:20:38.204524009Z"} {"bindAddress":":6301","level":"error","msg":"starting server","serverName":"scytale.health","ts":"2021-07-01T11:20:38.204574339Z"} {"bindAddress":":6302","level":"error","msg":"starting server","serverName":"scytale.pprof","ts":"2021-07-01T11:20:38.204652786Z"} {"bindAddress":":6303","level":"error","msg":"starting server","serverName":"scytale.metrics","ts":"2021-07-01T11:20:38.204719483Z"} {"X-Webpa-Device-Name":"mac:b827ebbed712/config","level":"error","msg":"all fanout requests failed","remoteAddr":"172.29.0.9:41396","requestMethod":"POST","requestURI":"/api/v2/device","statusCode":403,"ts":"2021-07-01T11:21:29.729039113Z","url":"/api/v2/device"}

It would be of great help if someone can point out what I am missing in the configuration.

Regards Karthika

[Sachin4403]

Hi @karthika-ab, Can you check whether the RPI is connected to talaria or not. You can check the same in Prometheus

metric name xmidt_talaria_active_connections

[karthika-ab]

Hi @Sachin4403 , I have checked the talaria connection to the device by using the below api using curl:

root@RaspberryPi-Gateway:~# curl -i -H "Authorization: Basic dXNlcjpwYXNz" "http://192.168.1.11:6200/api/v2/devices"
HTTP/1.1 200 OK
Content-Type: application/json
X-Talaria-Build: unkown
X-Talaria-Flavor: mint
X-Talaria-Region: east
X-Talaria-Server: e702e1c718c1
X-Talaria-Start-Time: 01 Jul 21 06:47 UTC
Date: Thu, 01 Jul 2021 06:57:11 GMT
Content-Length: 483

{"devices":[{"id": "mac:b827ebbed712", "pending": 0, "statistics": {"bytesSent": 0, "messagesSent": 0, "bytesReceived": 586, "messagesReceived": 1, "duplications": 0, "connectedAt": "2021-07-01T06:48:16.132110625Z", "upTime": "8m54.936107512s"}},{"id": "mac:112233445566", "pending": 0, "statistics": {"bytesSent": 606, "messagesSent": 3, "bytesReceived": 1992, "messagesReceived": 3, "duplications": 0, "connectedAt": "2021-07-01T06:48:02.295492698Z", "upTime": "9m8.772739965s"}}]}root@RaspberryPi-Gateway:~#
root@RaspberryPi-Gateway:~#

Attaching the screenshot of the query of the metric xmidt_talaria_activeconnections from prometheus. 

[Sachin4403]

Hi @karthika-ab, is /stat call is working from Tr1d1um, scytale, and talaria?

[karthika-ab]

Hi @Sachin,

Please see the below results ,got HTTP 200 for all 3 services: Talaria

root@RaspberryPi-Gateway:~#  curl -i -H "Authorization: Basic dXNlcjpwYXNz" "http://192.168.1.11:6200/api/v2/device/mac:b827ebbed712/stat"
HTTP/1.1 200 OK
Content-Type: application/json
X-Talaria-Build: unkown
X-Talaria-Flavor: mint
X-Talaria-Region: east
X-Talaria-Server: 0bf575ec1952
X-Talaria-Start-Time: 01 Jul 21 11:29 UTC
Date: Tue, 06 Jul 2021 08:19:38 GMT
Content-Length: 237

{"id": "mac:b827ebbed712", "pending": 0, "statistics": {"bytesSent": 0, "messagesSent": 0, "bytesReceived": 2935, "messagesReceived": 5, "duplications": 0, "connectedAt": "2021-07-01T11:34:29.463115884Z", "upTime": "116h45m9.29305974s"}}root@RaspberryPi-Gateway:~#

Tr1d1um

root@RaspberryPi-Gateway:~#
root@RaspberryPi-Gateway:~#  curl -i -H "Authorization: Basic dXNlcjpwYXNz" "http://192.168.1.11:6100/api/v2/device/mac:b827ebbed712/stat"
HTTP/1.1 200 OK
Content-Type: application/json
X-Scytale-Build: unkown
X-Scytale-Flavor: mint
X-Scytale-Region: east
X-Scytale-Server: b21caf3a1482
X-Scytale-Start-Time: 01 Jul 21 11:30 UTC
X-Talaria-Build: unkown
X-Talaria-Flavor: mint
X-Talaria-Region: east
X-Talaria-Server: 0bf575ec1952
X-Talaria-Start-Time: 01 Jul 21 11:29 UTC
X-Tr1d1um-Build: unkown
X-Tr1d1um-Flavor: mint
X-Tr1d1um-Region: east
X-Tr1d1um-Server: adcff9c890ae
X-Tr1d1um-Start-Time: 01 Jul 21 11:30 UTC
X-Webpa-Transaction-Id: 4d0Ly45gxeq7QKm-Dju0Bg
X-Xmidt-Span: "http://0bf575ec1952:6200/api/v2/device/mac:b827ebbed712/stat","2021-07-06T08:19:46Z","1.207261ms"
Date: Tue, 06 Jul 2021 08:19:46 GMT
Content-Length: 239

{"id": "mac:b827ebbed712", "pending": 0, "statistics": {"bytesSent": 0, "messagesSent": 0, "bytesReceived": 2935, "messagesReceived": 5, "duplications": 0, "connectedAt": "2021-07-01T11:34:29.463115884Z", "upTime": "116h45m16.740228263s"}}root@RaspberryPi-Gateway:~#

Scytale

root@RaspberryPi-Gateway:~#
root@RaspberryPi-Gateway:~#  curl -i -H "Authorization: Basic dXNlcjpwYXNz" "http://192.168.1.11:6300/api/v2/device/mac:b827ebbed712/stat"
HTTP/1.1 200 OK
Content-Length: 239
Content-Type: application/json
X-Scytale-Build: unkown
X-Scytale-Flavor: mint
X-Scytale-Region: east
X-Scytale-Server: b21caf3a1482
X-Scytale-Start-Time: 01 Jul 21 11:30 UTC
X-Talaria-Build: unkown
X-Talaria-Flavor: mint
X-Talaria-Region: east
X-Talaria-Server: 0bf575ec1952
X-Talaria-Start-Time: 01 Jul 21 11:29 UTC
X-Xmidt-Span: "http://0bf575ec1952:6200/api/v2/device/mac:b827ebbed712/stat","2021-07-06T08:19:58Z","702.204µs"
Date: Tue, 06 Jul 2021 08:19:58 GMT

{"id": "mac:b827ebbed712", "pending": 0, "statistics": {"bytesSent": 0, "messagesSent": 0, "bytesReceived": 2935, "messagesReceived": 5, "duplications": 0, "connectedAt": "2021-07-01T11:34:29.463115884Z", "upTime": "116h45m28.714852232s"}}root@RaspberryPi-Gateway:~#
root@RaspberryPi-Gateway:~#

[Sachin4403]

@karthika-ab Can you share your talaria configuration

[karthika-ab]

yes Sachin ,Please find the yaml files for talaria ,Tr1d1um and scytale.Also attaching the modified docker-compose,yml file for your reference. docker-compose.zip

[Sachin4403]

Hi @karthika-ab, There are 2 problems in the configuration of scytale and talaria due to this you are getting 403.

1. scytale.fanout.authorization has to set the same as talaria.inbound.authKey
2. talaria.inbound.authKey is wrong it should be base64 of base64 (username:password) currently it is authHeader which will be rejected in this check

You can keep anything any base64 encoded string in these but that must have a colon in between i.e dXNlcjpwYXNz is encoded string of user:pass

[karthika-ab]

@Sachin4403 ,Thanks for pointing out the above misconfiguration.I have rectified the same and added a dummy authKey which is base64 encoded .But i am still getting HTTP 403 for the webpa (the parodus client is outside the docker-compose setup). The talaria logs are below:

{"caller":"manager.go:215","id":"mac:b827ebbed712","level":"error","**msg":"missing security information"**,"ts":"2021-07-06T20:02:14.751076357Z"}
{"caller":"manager.go:219","convey":{"boot-time":1625640701,"fw-name":"rdkb-generic-broadband-image_rdk-next_20210531050507","hw-manufacturer":"Raspberry","hw-model":"RPI","hw-serial-number":"000000002dbed712","webpa-interface-used":"erouter0","webpa-last-reconnect-reason":"SSL_Socket_Close","webpa-protocol":"PARODUS-2.0-1.1.4-22-gbdc2733"},"id":"mac:b827ebbed712","level":"info","ts":"2021-07-06T20:02:14.751169406Z"}
{"datacenter":"","eventCount":2,"instances":["http://4e7b64e25c91:6200"],"level":"error","msg":"service discovery update","passingOnly":true,"service":"talaria","tags":["stage=dev","flavor=mint"],"ts":"2021-07-06T20:02:33.754668331Z"}
{"datacenter":"","eventCount":2,"level":"info","msg":"rehash starting","passingOnly":true,"service":"talaria","tags":["stage=dev","flavor=mint"],"ts":"2021-07-06T20:02:33.756108236Z"}
{"datacenter":"","eventCount":2,"id":"mac:b827ebbed712","instance":"http://4e7b64e25c91:6200","level":"info","msg":"disconnecting device: rehashed to another instance","passingOnly":true,"service":"talaria","tags":["stage=dev","flavor=mint"],"ts":"2021-07-06T20:02:33.756168925Z"}
{"datacenter":"","disconnectCount":1,"duration":"115.038µs","eventCount":2,"level":"info","msg":"rehash complete","passingOnly":true,"service":"talaria","tags":["stage=dev","flavor=mint"],"ts":"2021-07-06T20:02:33.75627732Z"}
{"caller":"manager.go:373","error":"read tcp 192.168.144.5:6200->192.168.1.6:58328: use of closed network connection","id":"mac:b827ebbed712","level":"error","msg":"read error","ts":"2021-07-06T20:02:33.757059714Z"}
{"caller":"manager.go:294","closeError":"close tcp 192.168.144.5:6200->192.168.1.6:58328: use of closed network connection","finalStatistics":"{\"bytesSent\": 0, \"messagesSent\": 0, \"bytesReceived\": 587, \"messagesReceived\": 1, \"duplications\": 0, \"connectedAt\": \"2021-07-06T20:02:14.751031841Z\", \"upTime\": \"19.006180828s\"}","id":"mac:b827ebbed712","level":"error","msg":"Closed device connection","reason":"readerror","reasonError":null,"ts":"2021-07-06T20:02:33.75724131Z"}
{"caller":"manager.go:215","id":"mac:b827ebbed712","level":"error","msg":"missing security information","ts":"2021-07-06T20:02:35.265344921Z"}
{"caller":"manager.go:219","convey":{"boot-time":1625640701,"fw-name":"rdkb-generic-broadband-image_rdk-next_20210531050507","hw-manufacturer":"Raspberry","hw-model":"RPI","hw-serial-number":"000000002dbed712","webpa-interface-used":"erouter0","webpa-last-reconnect-reason":"SSL_Socket_Close","webpa-protocol":"PARODUS-2.0-1.1.4-22-gbdc2733"},"id":"mac:b827ebbed712","level":"info","ts":"2021-07-06T20:02:35.265399625Z"}

Attaching the modified yaml files talaria.zip

[Sachin4403]

Hi @karthika-ab,

I don't see consul configuration in your yaml but logs says you are using consul. Can you share the right configuration or right configuration files.

[karthika-ab]

@Sachin4403 ,I have shared the correct yaml and docker-compose,yml file i am using .From what i see the consul environment is defined in the docker-compose.yml file.

[Sachin4403]

@karthika-ab, But in your talaria configuration files, I can't see the consul configuration. can you share the files from Docker containers?

file path will be /etc/appname/appname.yaml

appname will be talaria or scytale.

[joe94]

@Sachin4403 thanks a lot for jumping in and helping with this! I had a quick debugging session with @karthika-ab earlier today to gather clues as to what might be happening. Here's what I think is happening:

@karthika-ab's device is connecting to XMiDT without a Themis token because MTLS wasn't successful. In this case, Parodus attempts to get the token a few times, and then it just connects without one. For this reason, Talaria does not have a secure source from which to fetch the partnerID metadata for the device (the underlying default value is unknown).

Additionally, the deviceAccessCheck feature was enabled in strict mode by default in our docker configs. This made the request fail with a 403 because the partnerID device metadata was not contained in the list of partnerIDs in the WRP message.

Proposed solution

I made this fix https://github.com/xmidt-org/talaria/commit/dace2f9ead7eb23e4cddbb4d0d6a8a977b5f1fa2 that allows modifying the enforcement type and made a new release for talaria v0.5.12

Use this new version and switch the check to not be enforced but only monitored. Add the env var to the docker-compose under each talaria: Example:

  talaria-1:
    <<: *talaria
    environment:
      - CONSUL_HOST=consul0:8500
      - TRACING_PROVIDER_NAME=zipkin
      - TRACING_PROVIDER_ENDPOINT=http://zipkin:9411/api/v2/spans
      - DEVICE_ACCESS_CHECK_TYPE=monitor

[joe94]

Finally, I should also mention that in addition to deviceAccessCheck done by Talaria (more info on it here -> https://github.com/xmidt-org/talaria/blob/main/talaria.yaml#L340), parodus does its own check to ensure that parodus' partnerID is contained in the list of partners of the WRP message.

The way to distinguish them is that for the parodus check, a body will be returned of the form:

{"statusCode":403,"message":"Invalid partner_id"}

[joe94]

@karthika-ab, FYI you might not want to expose sensitive data like your device mac addresses in here.

[karthika-ab]

@karthika-ab, FYI you might not want to expose sensitive data like your device mac addresses in here. -->i exposed the ip ,as it is not a real device ,but a reference device(rpi) used for checking which is only available in local networks.

[karthika-ab]

@joe94 ,Many many thanks again changing DEVICE_ACCESS_CHECK_TYPE to monitor from enforce solves the issue .

[karthika-ab]

@Sachin4403 ,thanks for all the help and support in this ticket.I truly appreciate it .