search

open-horizon / exchange-api

Horizon Exchange REST API Server
Apache License 2.0
11 stars 30 forks source link

Tables not getting cleaned up #515

Closed dlarson04 closed 3 years ago

dlarson04 commented 3 years ago

Exchange has actors that are supposed to periodically cleanup resourcechanges table and expired messages in both agbotmsgs and nodemsgs. Those actors don't seem to be running as tables are not being pruned.

Exchange image: hyc-edge-team-nightly-docker-virtual.artifactory.swg-devops.com/amd64_exchange-api:2.77.0-425.202106171447.bf0a157

I see these log messages when the exchanges start so the time interval is correct

Jun 17 12:02:23 ibm-edge-exchange-59875464d9-z98nv ibm-edge-exchange INFO [INFO] [06/17/2021 16:02:23.930] [main] [ExchApi(akka://actors)] Resource changes cleanup Interval: 3600
Jun 17 12:02:23 ibm-edge-exchange-59875464d9-z98nv ibm-edge-exchange INFO [INFO] [06/17/2021 16:02:23.931] [main] [ExchApi(akka://actors)] Remove expired msgs cleanup Interval: 1800
Jun 17 12:02:25 ibm-edge-exchange-59875464d9-sf7nt ibm-edge-exchange INFO [INFO] [06/17/2021 16:02:25.418] [main] [ExchApi(akka://actors)] Resource changes cleanup Interval: 3600
Jun 17 12:02:25 ibm-edge-exchange-59875464d9-sf7nt ibm-edge-exchange INFO [INFO] [06/17/2021 16:02:25.419] [main] [ExchApi(akka://actors)] Remove expired msgs cleanup Interval: 1800
Jun 17 12:03:03 ibm-edge-exchange-59875464d9-vrbwj ibm-edge-exchange INFO [INFO] [06/17/2021 16:03:03.162] [main] [ExchApi(akka://actors)] Resource changes cleanup Interval: 3600
Jun 17 12:03:03 ibm-edge-exchange-59875464d9-vrbwj ibm-edge-exchange INFO [INFO] [06/17/2021 16:03:03.163] [main] [ExchApi(akka://actors)] Remove expired msgs cleanup Interval: 1800

But today on Jun18 at 7AM EST, I see these things still in resourcechanges table and agbotmsgs table

ibmclouddb=> select * from resourcechanges order by changeid asc limit 2;
 changeid | orgid |  id   | category | public | resource | operation |        lastupdated
----------+-------+-------+----------+--------+----------+-----------+----------------------------
 94986542 | IBM   | agbot | agbot    | false  | agbot    | modified  | 2021-06-15 13:50:22.570635
 94986543 | IBM   | agbot | agbot    | false  | agbot    | modified  | 2021-06-15 13:50:49.372317
ibmclouddb=> select msgid,nodeid,timesent,timeexpires from agbotmsgs order by msgid limit 5;
  msgid   |                        nodeid                        |             timesent             |           timeexpires
----------+------------------------------------------------------+----------------------------------+----------------------------------
 15552310 | dominion/edge-nodes-svl1-24.fyre.ibm.com_z_horizon10 | 2021-06-17T18:18:47.521312Z[UTC] | 2021-06-17T18:48:47.521322Z[UTC]
 15552315 | dominion/edge-nodes-svl1-24.fyre.ibm.com_z_horizon11 | 2021-06-17T18:18:47.748954Z[UTC] | 2021-06-17T18:48:47.748963Z[UTC]
 15552320 | dominion/edge-nodes-svl1-24.fyre.ibm.com_z_horizon12 | 2021-06-17T18:18:48.312915Z[UTC] | 2021-06-17T18:48:48.312922Z[UTC]
 15552324 | dominion/edge-nodes-svl1-24.fyre.ibm.com_z_horizon7  | 2021-06-17T18:18:48.903538Z[UTC] | 2021-06-17T18:48:48.903547Z[UTC]
 15552336 | dominion/edge-nodes-svl1-36.fyre.ibm.com_z_horizon11 | 2021-06-17T18:18:50.136032Z[UTC] | 2021-06-17T18:48:50.136041Z[UTC]
(5 rows)

Not sure if this is related to akka upgrade but pretty certain this worked in IEAM 4.2.1

naphelps commented 3 years ago

@sf2ne The scheduler.schedule is now legacy functionality in Akka 2.6/10.4. Also needed to replum this with the tls work since we standup two independent server binds now.

dlarson04 commented 3 years ago

verified in 2.78.0 version