search

chirpstack / chirpstack-gateway-os

OpenWrt based gateway images including ChirpStack components.
https://www.chirpstack.io
132 stars 57 forks source link

Choosing a gateway leeds to empty page #39

Closed joau-mms closed 3 years ago

joau-mms commented 4 years ago

Version: 13.02.2020 chirpstack-gateway-os-full-raspberrypi3-20200213123140.rootfs.wic.gz

Problem: After rebooting choosing an existing gateway results in empty page and error message: "context deadline exceeded (code: 2)"

image click on "testgw" and I got: image It takes a while for the error message to pop up.

Edit: Gateways -> Maps The Gateway is still present on the map, status is "a few seconds ago"

Edit2 Problem disappears after a long time or after madly clicking arround.

/var/log/messages


Feb 24 13:17:04 raspberrypi3 user.info chirpstack-application-server[338]: time="2020-02-24T13:17:04Z" level=info msg="finished unary call with code OK" ctx_id=5a15d627-51f5-4683-940f-1a9196d92a97 grpc.code=OK grpc.method=List grpc.service=api.GatewayService grpc.start_time
Feb 24 13:17:04 raspberrypi3 user.info chirpstack-application-server[338]: time="2020-02-24T13:17:04Z" level=info msg="finished unary call with code OK" ctx_id=6c37d010-a691-47e0-b278-94168564da80 grpc.code=OK grpc.method=List grpc.service=api.GatewayService grpc.start_time
Feb 24 13:17:04 raspberrypi3 user.info chirpstack-application-server[338]: time="2020-02-24T13:17:04Z" level=info msg="finished unary call with code OK" ctx_id=3860b6a3-a4ae-4ba0-93e6-99a19e09c33e grpc.code=OK grpc.method=GetStats grpc.service=api.GatewayService grpc.start_
Feb 24 13:17:06 raspberrypi3 user.warn chirpstack-application-server[338]: time="2020-02-24T13:17:06Z" level=warning msg="creating insecure network-server client" server="localhost:8000"
Feb 24 13:17:10 raspberrypi3 auth.err getty[1499]: tcgetattr: Input/output error^M
Feb 24 13:17:11 raspberrypi3 user.err chirpstack-application-server[338]: time="2020-02-24T13:17:11Z" level=error msg="finished unary call with code Unknown" ctx_id=1e039f37-d8f1-4716-93cc-8cb1cf22fa99 error="rpc error: code = Unknown desc = context deadline exceeded" grpc
Feb 24 13:17:16 raspberrypi3 user.info chirpstack-concentratord-sx1301[517]: Publishing stats event, stats_id: a2ae067a-0a52-427a-a6a5-d34f5fef838d, rx_received: 0, rx_received_ok: 0, tx_received: 0, tx_emitted: 0
Feb 24 13:17:16 raspberrypi3 user.info chirpstack-gateway-bridge[473]: time="2020-02-24T13:17:16Z" level=info msg="backend/concentratord: stats event received" stats_id=a2ae067a-0a52-427a-a6a5-d34f5fef838d
Feb 24 13:17:16 raspberrypi3 user.info chirpstack-gateway-bridge[473]: time="2020-02-24T13:17:16Z" level=info msg="integration/mqtt: publishing event" event=stats qos=0 stats_id=a2ae067a-0a52-427a-a6a5-d34f5fef838d topic=gateway/4223422342234223/event/stats
Feb 24 13:17:16 raspberrypi3 user.info chirpstack-network-server[362]: time="2020-02-24T13:17:16Z" level=info msg="gateway/mqtt: gateway stats packet received" gateway_id=4223422342234223 stats_id=a2ae067a-0a52-427a-a6a5-d34f5fef838d
Feb 24 13:17:16 raspberrypi3 user.info chirpstack-network-server[362]: time="2020-02-24T13:17:16Z" level=info msg="gateway updated" ctx_id=a2ae067a-0a52-427a-a6a5-d34f5fef838d gateway_id=4223422342234223
Feb 24 13:17:16 raspberrypi3 user.info chirpstack-application-server[338]: time="2020-02-24T13:17:16Z" level=info msg="gateway updated" ctx_id=dd12a7e8-866d-4bdb-a39c-1ed5a865d016 id=4223422342234223 name=testgw
Feb 24 13:17:16 raspberrypi3 user.info chirpstack-application-server[338]: time="2020-02-24T13:17:16Z" level=info msg="metrics saved" aggregation="[MINUTE HOUR DAY MONTH]" ctx_id=dd12a7e8-866d-4bdb-a39c-1ed5a865d016 name="gw:4223422342234223"
Feb 24 13:17:16 raspberrypi3 user.info chirpstack-application-server[338]: time="2020-02-24T13:17:16Z" level=info msg="finished unary call with code OK" ctx_id=dd12a7e8-866d-4bdb-a39c-1ed5a865d016 grpc.code=OK grpc.method=HandleGatewayStats grpc.service=as.ApplicationServer
Feb 24 13:17:16 raspberrypi3 user.info chirpstack-network-server[362]: time="2020-02-24T13:17:16Z" level=info msg="finished client unary call" ctx_id=a2ae067a-0a52-427a-a6a5-d34f5fef838d grpc.code=OK grpc.ctx_id=dd12a7e8-866d-4bdb-a39c-1ed5a865d016 grpc.duration=16.41915ms```
joau-mms commented 4 years ago

Same Story but with Gateway-Profiles:

Click on existing GW-profile, get empty page and get "context deadline exceeded (code: 2)"

/var/log/messages

Feb 24 13:22:27 raspberrypi3 daemon.info connmand[312]: ntp: adjust (slew): -0.001566 sec
Feb 24 13:22:31 raspberrypi3 user.info chirpstack-application-server[338]: time="2020-02-24T13:22:31Z" level=info msg="finished unary call with code OK" ctx_id=2733619b-ea4d-4720-8fe6-7ee9149e77da grpc.code=OK grpc.method=List grpc.service=api.GatewayProfileService grpc.sta
Feb 24 13:22:33 raspberrypi3 user.warn chirpstack-application-server[338]: time="2020-02-24T13:22:33Z" level=warning msg="creating insecure network-server client" server="localhost:8000"
Feb 24 13:22:38 raspberrypi3 user.err chirpstack-application-server[338]: time="2020-02-24T13:22:38Z" level=error msg="finished unary call with code Unknown" ctx_id=dbacbc86-092c-4d94-8be0-1f00594cf6aa error="rpc error: code = Unknown desc = context deadline exceeded" grpc
joau-mms commented 4 years ago

Rebooting, and requesting the Gateway-profile "fixes" the problem somehow.

/var/log/messages


Feb 24 13:36:00 raspberrypi3 user.info chirpstack-application-server[337]: time="2020-02-24T13:36:00Z" level=info msg="finished unary call with code OK" ctx_id=730e917d-8fad-41ec-896a-e334d57b2e68 grpc.code=OK grpc.method=Branding grpc.service=api.InternalService grpc.start
Feb 24 13:36:00 raspberrypi3 user.info chirpstack-application-server[337]: time="2020-02-24T13:36:00Z" level=info msg="finished unary call with code OK" ctx_id=cab4e344-a38e-4768-a502-eb3d8749e62a grpc.code=OK grpc.method=Profile grpc.service=api.InternalService grpc.start_
Feb 24 13:36:01 raspberrypi3 user.info chirpstack-application-server[337]: time="2020-02-24T13:36:01Z" level=info msg="finished unary call with code OK" ctx_id=d73b86eb-24b6-4101-a8fa-1f05eb6f67c1 grpc.code=OK grpc.method=List grpc.service=api.OrganizationService grpc.start
Feb 24 13:36:01 raspberrypi3 user.info chirpstack-application-server[337]: time="2020-02-24T13:36:01Z" level=info msg="finished unary call with code OK" ctx_id=e059b87b-f44f-4069-9edd-6a87594e2b0e grpc.code=OK grpc.method=Get grpc.service=api.OrganizationService grpc.start_
Feb 24 13:36:01 raspberrypi3 user.info chirpstack-application-server[337]: time="2020-02-24T13:36:01Z" level=info msg="finished unary call with code OK" ctx_id=176be0f6-9a1f-42ac-8d2e-86e36f7c721e grpc.code=OK grpc.method=Get grpc.service=api.OrganizationService grpc.start_
Feb 24 13:36:01 raspberrypi3 user.info chirpstack-application-server[337]: time="2020-02-24T13:36:01Z" level=info msg="finished unary call with code OK" ctx_id=31532c2f-3b2e-4d17-8214-ceaa0800cf07 grpc.code=OK grpc.method=List grpc.service=api.OrganizationService grpc.start
Feb 24 13:36:01 raspberrypi3 user.info chirpstack-application-server[337]: time="2020-02-24T13:36:01Z" level=info msg="finished unary call with code OK" ctx_id=630f6b1e-6817-4d0f-ad1b-d7964fb5941b grpc.code=OK grpc.method=Get grpc.service=api.OrganizationService grpc.start_
Feb 24 13:36:01 raspberrypi3 user.info chirpstack-application-server[337]: time="2020-02-24T13:36:01Z" level=info msg="finished unary call with code OK" ctx_id=47146ecf-0c5e-4874-a583-66de27ec0eea grpc.code=OK grpc.method=List grpc.service=api.OrganizationService grpc.start
Feb 24 13:36:01 raspberrypi3 user.info chirpstack-application-server[337]: time="2020-02-24T13:36:01Z" level=info msg="finished unary call with code OK" ctx_id=e936c372-83fa-444e-a098-73cfc9703e61 grpc.code=OK grpc.method=Get grpc.service=api.OrganizationService grpc.start_
Feb 24 13:36:02 raspberrypi3 auth.err getty[581]: tcgetattr: Input/output error^M
Feb 24 13:36:02 raspberrypi3 user.info chirpstack-application-server[337]: time="2020-02-24T13:36:02Z" level=info msg="finished unary call with code OK" ctx_id=417f3748-0c09-4123-9970-60753ba55450 grpc.code=OK grpc.method=List grpc.service=api.ApplicationService grpc.start_
Feb 24 13:36:02 raspberrypi3 user.info chirpstack-application-server[337]: time="2020-02-24T13:36:02Z" level=info msg="finished unary call with code OK" ctx_id=09165418-11f8-4a56-9db3-492c7e4fafd0 grpc.code=OK grpc.method=List grpc.service=api.ApplicationService grpc.start_
Feb 24 13:36:02 raspberrypi3 user.info chirpstack-application-server[337]: time="2020-02-24T13:36:02Z" level=info msg="finished unary call with code OK" ctx_id=9aa7e47f-4eb4-4722-bc1c-bca6d2e92581 grpc.code=OK grpc.method=List grpc.service=api.GatewayService grpc.start_time
Feb 24 13:36:02 raspberrypi3 user.info chirpstack-application-server[337]: time="2020-02-24T13:36:02Z" level=info msg="finished unary call with code OK" ctx_id=fb2126b6-e8e6-4fdb-b011-7401153a4ba7 grpc.code=OK grpc.method=List grpc.service=api.GatewayService grpc.start_time
Feb 24 13:36:03 raspberrypi3 user.info chirpstack-application-server[337]: time="2020-02-24T13:36:03Z" level=info msg="finished unary call with code OK" ctx_id=53e5d369-1926-4c5e-b39b-b1cae70d0a4f grpc.code=OK grpc.method=GetStats grpc.service=api.GatewayService grpc.start_
Feb 24 13:36:04 raspberrypi3 user.warn chirpstack-application-server[337]: time="2020-02-24T13:36:04Z" level=warning msg="creating insecure network-server client" server="localhost:8000"
Feb 24 13:36:07 raspberrypi3 user.info chirpstack-application-server[337]: time="2020-02-24T13:36:07Z" level=info msg="finished unary call with code OK" ctx_id=79887536-0485-46b0-bdf4-abd7d9d9c330 grpc.code=OK grpc.method=List grpc.service=api.GatewayService grpc.start_time
Feb 24 13:36:07 raspberrypi3 user.info chirpstack-application-server[337]: time="2020-02-24T13:36:07Z" level=info msg="finished unary call with code OK" ctx_id=20d8da60-1461-4481-910a-d40df67c9628 grpc.code=OK grpc.method=List grpc.service=api.GatewayService grpc.start_time
Feb 24 13:36:07 raspberrypi3 user.info chirpstack-application-server[337]: time="2020-02-24T13:36:07Z" level=info msg="finished unary call with code OK" ctx_id=128ce868-1426-4bde-978e-efe2ff7d51c4 grpc.code=OK grpc.method=GetStats grpc.service=api.GatewayService grpc.start_
Feb 24 13:36:09 raspberrypi3 user.warn chirpstack-application-server[337]: time="2020-02-24T13:36:09Z" level=warning msg="creating insecure network-server client" server="localhost:8000"
Feb 24 13:36:09 raspberrypi3 user.err chirpstack-application-server[337]: time="2020-02-24T13:36:09Z" level=error msg="finished unary call with code Unknown" ctx_id=b290e62c-f58f-4e96-b8f9-f26ba36feb91 error="rpc error: code = Unknown desc = context deadline exceeded" grpc
Feb 24 13:36:09 raspberrypi3 user.info chirpstack-application-server[337]: time="2020-02-24T13:36:09Z" level=info msg="finished unary call with code OK" ctx_id=ef5b0979-7dcc-4f77-8f29-59d859a4e77b grpc.code=OK grpc.method=List grpc.service=api.GatewayProfileService grpc.sta
Feb 24 13:36:12 raspberrypi3 auth.err getty[600]: tcgetattr: Input/output error^M
Feb 24 13:36:14 raspberrypi3 user.err chirpstack-application-server[337]: time="2020-02-24T13:36:14Z" level=error msg="finished unary call with code Unknown" ctx_id=8871c2cd-1e0a-4cdf-8949-b3e1db91dd20 error="rpc error: code = Unknown desc = context deadline exceeded" grpc
Feb 24 13:36:14 raspberrypi3 user.warn chirpstack-application-server[337]: time="2020-02-24T13:36:14Z" level=warning msg="creating insecure network-server client" server="localhost:8000"
Feb 24 13:36:14 raspberrypi3 user.info chirpstack-network-server[362]: time="2020-02-24T13:36:14Z" level=info msg="finished unary call with code OK" ctx_id=739b9291-2796-429d-83ba-00bfb7844578 grpc.code=OK grpc.method=GetGatewayProfile grpc.service=ns.NetworkServerService g
Feb 24 13:36:14 raspberrypi3 user.info chirpstack-application-server[337]: time="2020-02-24T13:36:14Z" level=info msg="finished client unary call" ctx_id=02684ef9-c936-4233-9b4b-13fd7904123e grpc.code=OK grpc.ctx_id=739b9291-2796-429d-83ba-00bfb7844578 grpc.duration=27.5785
Feb 24 13:36:14 raspberrypi3 user.info chirpstack-application-server[337]: time="2020-02-24T13:36:14Z" level=info msg="finished unary call with code OK" ctx_id=02684ef9-c936-4233-9b4b-13fd7904123e grpc.code=OK grpc.method=Get grpc.service=api.GatewayProfileService grpc.star
Feb 24 13:36:15 raspberrypi3 user.info chirpstack-concentratord-sx1301[524]: Publishing stats event, stats_id: 17bf9dbe-43a4-4eaa-a7ea-f71487f95838, rx_received: 1, rx_received_ok: 1, tx_received: 1, tx_emitted: 1
Feb 24 13:36:15 raspberrypi3 user.info chirpstack-gateway-bridge[478]: time="2020-02-24T13:36:15Z" level=info msg="backend/concentratord: stats event received" stats_id=17bf9dbe-43a4-4eaa-a7ea-f71487f95838
Feb 24 13:36:15 raspberrypi3 user.info chirpstack-gateway-bridge[478]: time="2020-02-24T13:36:15Z" level=info msg="integration/mqtt: publishing event" event=stats qos=0 stats_id=17bf9dbe-43a4-4eaa-a7ea-f71487f95838 topic=gateway/4223422342234223/event/stats
Feb 24 13:36:15 raspberrypi3 user.info chirpstack-network-server[362]: time="2020-02-24T13:36:15Z" level=info msg="gateway/mqtt: gateway stats packet received" gateway_id=4223422342234223 stats_id=17bf9dbe-43a4-4eaa-a7ea-f71487f95838
Feb 24 13:36:15 raspberrypi3 user.info chirpstack-network-server[362]: time="2020-02-24T13:36:15Z" level=info msg="gateway updated" ctx_id=17bf9dbe-43a4-4eaa-a7ea-f71487f95838 gateway_id=4223422342234223
Feb 24 13:36:15 raspberrypi3 user.warn chirpstack-network-server[362]: time="2020-02-24T13:36:15Z" level=warning msg="creating insecure application-server client" server="localhost:8001"```
joau-mms commented 4 years ago

Same Problem can occur with

Recovery of functionality is only possible after a random number of reboots.

joau-mms commented 4 years ago

Random error message poped up after some reboots

error parsing service config: error from DNS TXT record lookup: lookup _grpc_config.localhost on 127.0.0.1:53: read udp 127.0.0.1:51928->127.0.0.1:53: i/o timeout (code: 14)

Cloolalang commented 4 years ago

Similar issue here: I set up Chirpstack on R-pi 3B+ Arm with Ubuntu standard install. When I remove my SD card and copy the image to my PC as a back-up, (lost too much work to failed SD cards in the past) and then refit the SD card and boot I get the same "context deadline exceeded (code: 2)" when I try to navigate the application server web UI.

Yes, I do very carefully shut down the PI and power off before I remove the SD card.

https://godoc.org/context#DeadlineExceeded Something to do with GO

https://golang.org/pkg/context/

Cloolalang commented 4 years ago

I tried: 1) full re-format SD card. 2) Run a scandisk and fixed errors (windows) 2) Run fake SD card tester 3) Reloaded Pi Image using win32diskimager and did a verify. Problem persists (SD card is a 8gb class 4)

Cloolalang commented 4 years ago

The Log shows: Oct 07 10:31:04 ubuntu chirpstack-application-server[1911]: time="2020-10-07T10:31:04Z" level=error msg="finished unary call with code Unknown" ctx_id=4394b56c-654b-4008-9ee0-50abf9d8f476 error="rpc error: code = Unknown desc = context deadline exceeded" grpc.code=Unknown grpc.method=Get grpc.service=api.GatewayService grpc.start_time="2020-10-07T10:31:03Z" grpc.time_ms=512.965 peer.address="127.0.0.1:48182" span.kind=server system=grpc

Its taking too long to do something,

Cloolalang commented 4 years ago

I the end I solved this by re-flashing my SD card and doing a verify using win32diskimager, freeing up space on my sd card and removing dead swap files and it all came back running OK.