LAMP message queue,nats, worker, and server are stopping and starting frequently

[lukeoftheshire]

As mentioned in the title message queue, nats worker, and server services appear to be stopping and starting fairly frequently in portainer. @avaidyam and I are worried that there is a possibility of data loss due to these frequent stop-starts, so we are making this issue to keep track of it and log any information we find. The first thought we have is wondering if this could be related to automation with LAMP_worker in some way?

Please see the following image for an example of the frequently restarting services.

Relevant logs: Redis:

# all terminated redis containers have these exact same logs:

1:C 02 Nov 2021 18:28:35.924 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 02 Nov 2021 18:28:35.924 # Redis version=6.0.8, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 02 Nov 2021 18:28:35.924 # Configuration loaded
1:M 02 Nov 2021 18:28:35.925 * Running mode=standalone, port=6379.
1:M 02 Nov 2021 18:28:35.925 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 02 Nov 2021 18:28:35.925 # Server initialized
1:M 02 Nov 2021 18:28:35.925 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 02 Nov 2021 18:28:35.925 * Ready to accept connections
1:signal-handler (1635927627) Received SIGTERM scheduling shutdown...
1:M 03 Nov 2021 08:20:27.947 # User requested shutdown...
1:M 03 Nov 2021 08:20:27.947 # Redis is now ready to exit, bye bye...

Server

# server

2021-11-08T05:04:56.503471420Z GET /participant/U3497957254/sensor_event?origin=lamp.analytics&limit=1000 200 - 4.104 ms
2021-11-08T05:04:56.511630282Z GET /participant/U3142310106/sensor_event?origin=lamp.analytics&limit=1000 200 - 3.933 ms
2021-11-08T05:04:56.521692686Z GET /participant/U2049596189/sensor_event?origin=lamp.analytics&limit=1000 200 - 4.034 ms
2021-11-08T05:04:56.531045551Z GET /participant/U1660388766/sensor_event?origin=lamp.analytics&limit=1000 200 - 5.079 ms
2021-11-08T05:04:56.539532002Z GET /participant/U4112643738/sensor_event?origin=lamp.analytics&limit=1000 200 - 4.259 ms
2021-11-08T05:04:56.552536822Z GET /participant/U9232944603/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.793 ms
2021-11-08T05:04:56.562586592Z GET /participant/U8800226069/sensor_event?origin=lamp.analytics&limit=1000 200 - 4.053 ms
2021-11-08T05:04:56.570892444Z GET /participant/U3251451662/sensor_event?origin=lamp.analytics&limit=1000 200 - 4.037 ms
2021-11-08T05:04:56.579535933Z GET /participant/U5997877700/sensor_event?origin=lamp.analytics&limit=1000 200 - 4.350 ms
2021-11-08T05:04:56.586261352Z Fetching acTivity--
2021-11-08T05:04:56.593837144Z GET /study/z5fp592a0mt23j99mmyw/activity?ignore_binary=true 200 - 9.865 ms
2021-11-08T05:04:56.601750537Z Fetching acTivity--
2021-11-08T05:04:56.608943771Z GET /study/w7najqrzq4v0cy6yc3gq/activity?ignore_binary=true 200 - 9.643 ms
2021-11-08T05:04:56.619398203Z GET /researcher/mshs5a9p7bxkg4hhqfp8/study 200 - 6.072 ms
2021-11-08T05:04:56.625889364Z Fetching acTivity--
2021-11-08T05:04:56.649856552Z GET /study/h5r0e27ctd5sakv2xhxs/activity?ignore_binary=true 200 - 26.202 ms
2021-11-08T05:04:56.684045274Z GET /study/h5r0e27ctd5sakv2xhxs/participant 200 - 27.313 ms
2021-11-08T05:04:56.698394602Z GET /participant/U74398612/sensor_event?origin=lamp.analytics&limit=1000 200 - 9.563 ms
2021-11-08T05:04:56.714334556Z GET /participant/U3364247567/sensor_event?origin=lamp.analytics&limit=1000 200 - 11.559 ms
2021-11-08T05:04:56.740480661Z GET /participant/U650585221/sensor_event?origin=lamp.analytics&limit=1000 200 - 21.805 ms
2021-11-08T05:04:56.761366199Z GET /participant/U3301171773/sensor_event?origin=lamp.analytics&limit=1000 200 - 13.991 ms
2021-11-08T05:04:56.780415380Z GET /participant/U2873307322/sensor_event?origin=lamp.analytics&limit=1000 200 - 12.975 ms
2021-11-08T05:04:56.804275666Z GET /participant/U4009815427/sensor_event?origin=lamp.analytics&limit=1000 200 - 19.455 ms
2021-11-08T05:04:56.832072825Z GET /participant/U2216383209/sensor_event?origin=lamp.analytics&limit=1000 200 - 23.443 ms
2021-11-08T05:04:56.848695305Z GET /participant/U3185823493/sensor_event?origin=lamp.analytics&limit=1000 200 - 12.137 ms
2021-11-08T05:04:56.859142484Z GET /participant/U2220392219/sensor_event?origin=lamp.analytics&limit=1000 200 - 6.030 ms
2021-11-08T05:04:56.872265528Z GET /participant/U3388367831/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.954 ms
2021-11-08T05:04:56.888325736Z GET /participant/U545281964/sensor_event?origin=lamp.analytics&limit=1000 200 - 11.722 ms
2021-11-08T05:04:56.902187673Z GET /participant/U105240996/sensor_event?origin=lamp.analytics&limit=1000 200 - 9.551 ms
2021-11-08T05:04:56.912872742Z GET /participant/U3420262132/sensor_event?origin=lamp.analytics&limit=1000 200 - 6.541 ms
2021-11-08T05:04:56.927340951Z GET /participant/U4233010501/sensor_event?origin=lamp.analytics&limit=1000 200 - 9.467 ms
2021-11-08T05:04:56.943646410Z GET /participant/U1306386901/sensor_event?origin=lamp.analytics&limit=1000 200 - 12.179 ms
2021-11-08T05:04:56.956955610Z GET /participant/U15911961/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.787 ms
2021-11-08T05:04:56.970090892Z GET /participant/U1678656022/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.827 ms
2021-11-08T05:04:56.983647328Z GET /participant/U1750909798/sensor_event?origin=lamp.analytics&limit=1000 200 - 9.266 ms
2021-11-08T05:04:56.993945323Z GET /participant/U129474160/sensor_event?origin=lamp.analytics&limit=1000 200 - 5.847 ms
2021-11-08T05:04:57.005242295Z GET /participant/U331993895/sensor_event?origin=lamp.analytics&limit=1000 200 - 7.075 ms
2021-11-08T05:04:57.018090294Z GET /participant/U3979547651/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.658 ms
2021-11-08T05:04:57.030432608Z GET /participant/U4253955812/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.159 ms
2021-11-08T05:04:57.031434899Z POST /participant/U0816569523/sensor_event 200 - 3.659 ms
2021-11-08T05:04:57.032059577Z Nats server is disconnected
2021-11-08T05:04:57.032489483Z Nats server is disconnected
2021-11-08T05:04:57.040027103Z POST /participant/U0816569523/sensor_event 200 - 3.552 ms
2021-11-08T05:04:57.040840095Z Nats server is disconnected
2021-11-08T05:04:57.041941873Z Nats server is disconnected
2021-11-08T05:04:57.048342778Z GET /participant/U822393088/sensor_event?origin=lamp.analytics&limit=1000 200 - 11.411 ms
2021-11-08T05:04:57.065324378Z GET /participant/U2641237368/sensor_event?origin=lamp.analytics&limit=1000 200 - 11.119 ms
2021-11-08T05:04:57.081478357Z GET /participant/U1451232375/sensor_event?origin=lamp.analytics&limit=1000 200 - 11.782 ms
2021-11-08T05:04:57.091702918Z GET /participant/U3397872497/sensor_event?origin=lamp.analytics&limit=1000 200 - 5.906 ms
2021-11-08T05:04:57.105982723Z GET /participant/U1901851494/sensor_event?origin=lamp.analytics&limit=1000 200 - 10.000 ms
2021-11-08T05:04:57.118770209Z GET /participant/U504317083/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.181 ms
2021-11-08T05:04:57.134988419Z GET /participant/U1950095227/sensor_event?origin=lamp.analytics&limit=1000 200 - 11.675 ms
2021-11-08T05:04:57.149236698Z GET /participant/U1648287542/sensor_event?origin=lamp.analytics&limit=1000 200 - 9.407 ms
2021-11-08T05:04:57.167703236Z GET /participant/U2492145138/sensor_event?origin=lamp.analytics&limit=1000 200 - 13.332 ms
2021-11-08T05:04:57.184509076Z GET /participant/U2740606872/sensor_event?origin=lamp.analytics&limit=1000 200 - 12.080 ms
2021-11-08T05:04:57.198413964Z GET /participant/U4205883735/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.483 ms
2021-11-08T05:04:57.216266286Z GET /participant/U1335698569/sensor_event?origin=lamp.analytics&limit=1000 200 - 13.332 ms
2021-11-08T05:04:57.234041770Z GET /participant/U3206254569/sensor_event?origin=lamp.analytics&limit=1000 200 - 11.131 ms
2021-11-08T05:04:57.247126318Z GET /participant/U3070209781/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.634 ms
2021-11-08T05:04:57.267087119Z GET /participant/U3529444568/sensor_event?origin=lamp.analytics&limit=1000 200 - 15.669 ms
2021-11-08T05:04:57.281185670Z GET /participant/U3875207727/sensor_event?origin=lamp.analytics&limit=1000 200 - 9.314 ms
2021-11-08T05:04:57.293549405Z GET /participant/U2964907141/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.181 ms
2021-11-08T05:04:57.305740738Z GET /participant/U2582523240/sensor_event?origin=lamp.analytics&limit=1000 200 - 7.952 ms
2021-11-08T05:04:57.317446521Z GET /participant/U3333326031/sensor_event?origin=lamp.analytics&limit=1000 200 - 7.429 ms
2021-11-08T05:04:57.331604100Z GET /participant/U517481429/sensor_event?origin=lamp.analytics&limit=1000 200 - 9.691 ms
2021-11-08T05:04:57.344573384Z GET /participant/U2610873497/sensor_event?origin=lamp.analytics&limit=1000 200 - 7.924 ms
2021-11-08T05:04:57.357531011Z GET /participant/U2334901698/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.557 ms
2021-11-08T05:04:57.372274845Z GET /participant/U354752578/sensor_event?origin=lamp.analytics&limit=1000 200 - 10.274 ms
2021-11-08T05:04:57.384943777Z GET /participant/U3998756602/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.177 ms
2021-11-08T05:04:57.398471562Z GET /participant/U2258900661/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.960 ms
2021-11-08T05:04:57.412996843Z GET /participant/U2990317473/sensor_event?origin=lamp.analytics&limit=1000 200 - 9.994 ms
2021-11-08T05:04:57.427048408Z GET /participant/U2761310183/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.449 ms
2021-11-08T05:04:57.438229359Z GET /participant/U2853535444/sensor_event?origin=lamp.analytics&limit=1000 200 - 6.703 ms
2021-11-08T05:04:57.448964878Z GET /participant/U3984145087/sensor_event?origin=lamp.analytics&limit=1000 200 - 6.532 ms
2021-11-08T05:04:57.461148572Z GET /participant/U4079570273/sensor_event?origin=lamp.analytics&limit=1000 200 - 7.937 ms
2021-11-08T05:04:57.469604345Z GET /participant/U3189272257/sensor_event?origin=lamp.analytics&limit=1000 200 - 4.189 ms
2021-11-08T05:04:57.481927719Z GET /participant/U1827406012/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.127 ms
2021-11-08T05:04:57.494371266Z GET /participant/U3748700695/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.175 ms
2021-11-08T05:04:57.508723637Z GET /participant/U1602953801/sensor_event?origin=lamp.analytics&limit=1000 200 - 10.181 ms
2021-11-08T05:04:57.520349312Z GET /participant/U2405560385/sensor_event?origin=lamp.analytics&limit=1000 200 - 6.981 ms
2021-11-08T05:04:57.532523924Z GET /participant/U464628524/sensor_event?origin=lamp.analytics&limit=1000 200 - 7.907 ms
2021-11-08T05:04:57.544880739Z GET /participant/U3239194597/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.104 ms
2021-11-08T05:04:57.558916174Z GET /participant/U1416167928/sensor_event?origin=lamp.analytics&limit=1000 200 - 9.840 ms
2021-11-08T05:04:57.571106459Z GET /participant/U77672147/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.152 ms
2021-11-08T05:04:58.645427041Z Nats server is disconnected
2021-11-08T05:04:58.645862061Z Nats server is disconnected
2021-11-08T05:04:58.646205554Z Nats server is disconnected
2021-11-08T05:04:58.713443618Z Nats server is disconnected
2021-11-08T05:04:58.713959748Z Nats server is disconnected
2021-11-08T05:04:58.714399694Z Nats server is disconnected
2021-11-08T05:05:03.181964221Z Nats server is disconnected
2021-11-08T05:05:03.182416824Z Nats server is disconnected
2021-11-08T05:05:03.209232576Z Nats server is disconnected
2021-11-08T05:05:03.209732929Z Nats server is disconnected
2021-11-08T05:05:03.210126941Z Nats server is disconnected
2021-11-08T05:05:06.467009915Z Nats server is disconnected
2021-11-08T05:05:06.467361186Z Nats server is disconnected
2021-11-08T05:05:06.489747157Z Nats server is disconnected
2021-11-08T05:05:06.490135232Z Nats server is disconnected
2021-11-08T05:05:06.490488506Z Nats server is disconnected

--- LAMP_server.1.l1ahx6xzr8pi289jdacyi08q1 TERMINATED ---

2021-11-08T08:18:57.368722789Z POST /participant/U4593963689/sensor_event 200 - 10.442 ms
2021-11-08T08:18:57.435567486Z POST /participant/U4593963689/sensor_event 200 - 6.228 ms
2021-11-08T08:18:57.540186261Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632609174293&limit=10000 200 - 343.159 ms
2021-11-08T08:18:57.540870468Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632843639441&limit=10000 200 - 352.157 ms
2021-11-08T08:18:57.542105095Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630221550164&limit=10000 200 - 380.624 ms
2021-11-08T08:18:57.913685741Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632607168648&limit=10000 200 - 371.970 ms
2021-11-08T08:18:57.952030276Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632841636401&limit=10000 200 - 383.155 ms
2021-11-08T08:18:57.986839879Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630219561160&limit=10000 200 - 413.656 ms
2021-11-08T08:18:58.357722005Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632605162949&limit=10000 200 - 366.920 ms
2021-11-08T08:18:58.357758015Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632839632896&limit=10000 200 - 360.930 ms
2021-11-08T08:18:58.392863631Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630217572233&limit=10000 200 - 382.833 ms
2021-11-08T08:18:58.773092867Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632837629908&limit=10000 200 - 344.720 ms
2021-11-08T08:18:58.773598426Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632603157770&limit=10000 200 - 372.952 ms
2021-11-08T08:18:58.832878854Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630215583217&limit=10000 200 - 405.918 ms
2021-11-08T08:18:59.165635792Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632835626968&limit=10000 200 - 349.689 ms
2021-11-08T08:18:59.166584328Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632601152586&limit=10000 200 - 366.906 ms
2021-11-08T08:18:59.254252613Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630213594340&limit=10000 200 - 395.020 ms
2021-11-08T08:18:59.426364996Z GET /participant/3m0k1a5fvh6kfvahchvh/sensor_event?origin=lamp.analytics 304 - 12.652 ms
2021-11-08T08:18:59.433115377Z GET /type/3m0k1a5fvh6kfvahchvh/attachment/lamp.messaging 404 - 17.342 ms
2021-11-08T08:18:59.538311595Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632833623780&limit=10000 200 - 336.293 ms
2021-11-08T08:18:59.543639997Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632599146483&limit=10000 200 - 346.791 ms
2021-11-08T08:18:59.626156165Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630211605426&limit=10000 200 - 335.940 ms
2021-11-08T08:18:59.975100475Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632831620076&limit=10000 200 - 373.037 ms
2021-11-08T08:18:59.975419032Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632597140325&limit=10000 200 - 404.447 ms
2021-11-08T08:19:00.022508668Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630209616539&limit=10000 200 - 369.469 ms
2021-11-08T08:19:00.038763906Z Fetching acTivity--
2021-11-08T08:19:00.045635802Z GET /activity/nkc57v55fcs8weam6zgk?ignore_binary=true 200 - 21.244 ms
2021-11-08T08:19:00.081474975Z GET /type/nkc57v55fcs8weam6zgk/parent 200 - 25.529 ms
2021-11-08T08:19:00.095705014Z GET /study/9n5tbtf3caf74gd4dwh4/participant 200 - 9.190 ms
2021-11-08T08:19:00.282646131Z GET /participant/U2201132710/sensor_event?origin=lamp.analytics&limit=1000 200 - 180.635 ms
2021-11-08T08:19:00.479258970Z GET /participant/U0176919012/sensor_event?origin=lamp.analytics&limit=1000 200 - 183.762 ms
2021-11-08T08:19:00.513618175Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630207627433&limit=10000 200 - 396.697 ms
2021-11-08T08:19:00.513716344Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632829616523&limit=10000 200 - 477.274 ms
2021-11-08T08:19:00.515102757Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632595135208&limit=10000 200 - 489.735 ms
2021-11-08T08:19:00.575905885Z GET /participant/U8640322252/sensor_event?origin=lamp.analytics&limit=1000 200 - 63.091 ms
2021-11-08T08:19:00.955080597Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630205638432&limit=10000 200 - 387.885 ms
2021-11-08T08:19:00.989403030Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632593129745&limit=10000 200 - 414.626 ms
2021-11-08T08:19:00.989582494Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632827612198&limit=10000 200 - 449.089 ms
2021-11-08T08:19:01.365610283Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632825609342&limit=10000 200 - 343.364 ms
2021-11-08T08:19:01.432458974Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632591120196&limit=10000 200 - 382.789 ms
2021-11-08T08:19:01.434845136Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630203649527&limit=10000 200 - 442.526 ms
2021-11-08T08:19:01.625052595Z POST /participant/U0176919012/sensor_event 200 - 7.711 ms
2021-11-08T08:19:01.852395777Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632823606449&limit=10000 200 - 409.731 ms
2021-11-08T08:19:01.918113156Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632589112497&limit=10000 200 - 428.030 ms
2021-11-08T08:19:01.918146748Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630201660649&limit=10000 200 - 457.728 ms
2021-11-08T08:19:02.285123959Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632821603332&limit=10000 200 - 361.442 ms
2021-11-08T08:19:02.373726520Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630199671859&limit=10000 200 - 421.111 ms
2021-11-08T08:19:02.672070846Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632819600102&limit=10000 200 - 353.731 ms
2021-11-08T08:19:02.781829121Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630197682949&limit=10000 200 - 381.295 ms
2021-11-08T08:19:03.075647476Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632817596706&limit=10000 200 - 371.653 ms
2021-11-08T08:19:03.194585568Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630195694015&limit=10000 200 - 379.839 ms
2021-11-08T08:19:03.409473506Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632587105235&limit=10000 200 - 371.023 ms
2021-11-08T08:19:03.466046164Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632815593777&limit=10000 200 - 360.029 ms
2021-11-08T08:19:03.550642299Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630193705055&limit=10000 200 - 331.119 ms
2021-11-08T08:19:04.492870166Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630191716116&limit=10000 200 - 351.905 ms
2021-11-08T08:19:04.494797985Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632585098142&limit=10000 200 - 368.662 ms
2021-11-08T08:20:04.746403899Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632813590920&limit=10000 200 - 309.668 ms
2021-11-08T08:20:04.829542118Z GET /participant/yhe0wtfn6n6sbvsan0js/sensor_event?origin=lamp.analytics 304 - 7.373 ms
2021-11-08T08:20:04.830206121Z GET /participant/z5ys6yv5w1rj31zq3w4r/sensor_event?origin=lamp.analytics 304 - 8.681 ms
2021-11-08T08:20:04.831931154Z GET /type/z5ys6yv5w1rj31zq3w4r/attachment/lamp.messaging 404 - 19.810 ms
2021-11-08T08:20:04.832418472Z GET /type/yhe0wtfn6n6sbvsan0js/attachment/lamp.messaging 404 - 19.389 ms
2021-11-08T08:20:05.018253825Z POST /participant/U4233494401/sensor_event 200 - 12.749 ms
2021-11-08T08:20:05.030053832Z POST /participant/U4233494401/sensor_event 200 - 13.366 ms
2021-11-08T08:20:05.068959944Z POST /participant/U6368562055/sensor_event 200 - 25.038 ms
2021-11-08T08:20:05.080558442Z POST /participant/U6368562055/sensor_event 200 - 26.957 ms
2021-11-08T08:20:05.184530592Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630189727213&limit=10000 200 - 402.685 ms
2021-11-08T08:20:05.186184426Z POST /participant/U1912527657/sensor_event 200 - 41.802 ms
2021-11-08T08:20:05.216720351Z POST /participant/U1912527657/sensor_event 200 - 34.113 ms
2021-11-08T08:20:05.225903323Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632583089722&limit=10000 200 - 446.972 ms
2021-11-08T08:20:05.227514798Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632811587941&limit=10000 200 - 428.699 ms
2021-11-08T08:20:05.447958800Z POST /participant/U6299504587/sensor_event 200 - 7.120 ms
2021-11-08T08:20:05.449065029Z POST /participant/U6299504587/sensor_event 200 - 5.849 ms
2021-11-08T08:20:05.510704159Z POST /participant/U7645962928/sensor_event 200 - 5.783 ms
2021-11-08T08:20:05.512100452Z POST /participant/U7645962928/sensor_event 200 - 4.248 ms
2021-11-08T08:20:05.580512321Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632809585153&limit=10000 200 - 317.288 ms
2021-11-08T08:20:05.615058735Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632581082434&limit=10000 200 - 346.063 ms
2021-11-08T08:20:09.561384497Z [ioredis] Unhandled error event: Error: connect ECONNREFUSED 10.0.6.7:6379
2021-11-08T08:20:09.561412633Z     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1146:16)
2021-11-08T08:20:09.662477824Z [ioredis] Unhandled error event: Error: connect ECONNREFUSED 10.0.6.7:6379
2021-11-08T08:20:09.662501396Z     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1146:16)
2021-11-08T08:20:09.813758255Z [ioredis] Unhandled error event: Error: connect ECONNREFUSED 10.0.6.7:6379
2021-11-08T08:20:09.813770963Z     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1146:16)
2021-11-08T08:20:10.030331082Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-08T08:20:10.030382935Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-08T08:20:10.281118022Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-08T08:20:10.281135429Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-08T08:20:10.582255312Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-08T08:20:10.582270980Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-08T08:20:10.933942375Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-08T08:20:10.933963077Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-08T08:20:11.334202438Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-08T08:20:11.334219963Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-08T08:20:11.785596098Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-08T08:20:11.785624180Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-08T08:20:12.287151670Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-08T08:20:12.287176901Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-08T08:20:12.838690477Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-08T08:20:12.838716338Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-08T08:20:13.440402315Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-08T08:20:13.440417426Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)

--- LAMP_server.1.gpqb989olgtbfbesr4o9tjcpv TERMINATED ---

2021-11-08T11:20:17.554507102Z Nats server is disconnected
2021-11-08T11:20:17.554995623Z Nats server is disconnected
2021-11-08T11:20:17.555364098Z Nats server is disconnected
2021-11-08T11:20:19.204530182Z GET /participant/U8479747510/sensor_event?origin=lamp.analytics 304 - 4.665 ms
2021-11-08T11:20:19.225886863Z GET /type/U8479747510/attachment/lamp.messaging 404 - 19.356 ms
2021-11-08T11:20:22.791111753Z POST /participant/U2201583859/sensor_event 200 - 4.072 ms
2021-11-08T11:20:22.791615676Z Nats server is disconnected
2021-11-08T11:20:22.791990496Z Nats server is disconnected
2021-11-08T11:20:22.792319805Z Nats server is disconnected
2021-11-08T11:20:22.793136469Z POST /participant/U2201583859/sensor_event 200 - 3.407 ms
2021-11-08T11:20:22.793477166Z Nats server is disconnected
2021-11-08T11:20:22.793929254Z Nats server is disconnected
2021-11-08T11:20:22.794380651Z Nats server is disconnected
2021-11-08T11:20:22.794818020Z Nats server is disconnected
2021-11-08T11:20:22.795288014Z Nats server is disconnected
2021-11-08T11:20:23.275389670Z GET /participant/0vayybdfqxfpv74acj5b/sensor_event?origin=lamp.analytics 304 - 5.526 ms
2021-11-08T11:20:23.282991501Z GET /type/0vayybdfqxfpv74acj5b/attachment/lamp.messaging 404 - 10.300 ms
2021-11-08T11:20:34.277361545Z GET /participant/0vayybdfqxfpv74acj5b/sensor_event?origin=lamp.analytics 304 - 5.974 ms
2021-11-08T11:20:34.283982184Z GET /type/0vayybdfqxfpv74acj5b/attachment/lamp.messaging 404 - 11.392 ms
2021-11-08T11:20:37.701980860Z POST /participant/U6529862821/sensor_event 200 - 8.087 ms
2021-11-08T11:20:37.702908014Z POST /participant/U6529862821/sensor_event 200 - 3.414 ms
2021-11-08T11:20:37.703654365Z Nats server is disconnected
2021-11-08T11:20:37.704210119Z Nats server is disconnected
2021-11-08T11:20:37.704721303Z Nats server is disconnected
2021-11-08T11:20:37.705212131Z Nats server is disconnected
2021-11-08T11:20:37.705741400Z Nats server is disconnected
2021-11-08T11:20:37.706229034Z Nats server is disconnected
2021-11-08T11:20:37.706705460Z Nats server is disconnected
2021-11-08T11:20:37.707191412Z Nats server is disconnected
2021-11-08T11:20:42.800129073Z POST /participant/U5688551159/sensor_event 200 - 3.502 ms
2021-11-08T11:20:42.801478553Z Nats server is disconnected
2021-11-08T11:20:42.801905113Z Nats server is disconnected
2021-11-08T11:20:42.802251197Z Nats server is disconnected
2021-11-08T11:20:42.802583209Z Nats server is disconnected
2021-11-08T11:20:42.895156572Z POST /participant/U5688551159/sensor_event 200 - 3.545 ms
2021-11-08T11:20:42.895832615Z Nats server is disconnected
2021-11-08T11:20:42.896297549Z Nats server is disconnected
2021-11-08T11:20:42.896737186Z Nats server is disconnected
2021-11-08T11:20:42.897170176Z Nats server is disconnected
2021-11-08T11:20:42.954618856Z Store_Size 32335
2021-11-08T11:20:44.034413696Z write started timestamp 290-1636370444034
2021-11-08T11:20:47.497123323Z Initializing LAMP API server...
2021-11-08T11:20:47.529026691Z   Connected to Nats Pub Server
2021-11-08T11:20:47.529205582Z   Connected to Nats Sub Server
2021-11-08T11:20:47.613631976Z   Initializing database connection...
2021-11-08T11:20:47.614146247Z     MONGODB adapter in use
2021-11-08T11:20:47.640568877Z       ActivitySpec database online.
2021-11-08T11:20:47.640584283Z       SensorSpec database online.
2021-11-08T11:20:47.640587581Z       Researcher database online.
2021-11-08T11:20:47.640591166Z       Study database online.
2021-11-08T11:20:47.640594416Z       Participant database online.
2021-11-08T11:20:47.640624164Z       Activity database online.
2021-11-08T11:20:47.640640885Z       Sensor database online.
2021-11-08T11:20:47.640643293Z       ActivityEvent database online.
2021-11-08T11:20:47.640650675Z       SensorEvent database online.
2021-11-08T11:20:47.640652728Z       Tag database online.
2021-11-08T11:20:47.640673104Z       Credential database online.
2021-11-08T11:20:47.640753818Z   Database verification complete.
2021-11-08T11:20:47.640803472Z   Server routing initialized.
2021-11-08T11:20:47.640808114Z Initialization complete.
2021-11-08T11:20:47.643394788Z undefined
2021-11-08T11:20:51.756229078Z GET /participant/m6rrb59rpsncgxqkz5ft/sensor_event?origin=lamp.analytics 304 - 16.532 ms
2021-11-08T11:20:51.771231971Z GET /type/m6rrb59rpsncgxqkz5ft/attachment/lamp.messaging 404 - 23.310 ms
2021-11-08T11:20:55.634176540Z GET /participant/U9610582616/sensor_event?origin=lamp.analytics 304 - 8.002 ms
2021-11-08T11:20:55.655236061Z GET /participant/yhe0wtfn6n6sbvsan0js/sensor_event?origin=lamp.analytics 304 - 10.025 ms
2021-11-08T11:20:55.658382885Z GET /participant/z5ys6yv5w1rj31zq3w4r/sensor_event?origin=lamp.analytics 304 - 12.143 ms
2021-11-08T11:20:55.660516030Z GET /type/yhe0wtfn6n6sbvsan0js/attachment/lamp.messaging 404 - 24.880 ms
2021-11-08T11:20:55.669945235Z GET /type/z5ys6yv5w1rj31zq3w4r/attachment/lamp.messaging 404 - 33.359 ms
2021-11-08T11:20:55.670863392Z GET /type/U9610582616/attachment/lamp.messaging 304 - 46.294 ms
2021-11-08T11:20:56.391277993Z Fetching acTivity--
2021-11-08T11:20:56.406941754Z GET /participant/U7931361682/sensor 304 - 20.122 ms
2021-11-08T11:20:56.423732427Z GET /participant/U7931361682/activity?ignore_binary=true 304 - 38.762 ms
2021-11-08T11:20:57.706956407Z POST /participant/U1954110644/sensor_event 200 - 5.195 ms
2021-11-08T11:20:58.846583791Z POST /participant/U1954110644/sensor_event 200 - 3.744 ms
2021-11-08T11:20:58.915556172Z POST /participant/U1954110644/sensor_event 200 - 5.541 ms
2021-11-08T11:21:40.021408592Z Store_Size 30918
2021-11-08T11:21:41.164443179Z write started timestamp 291-1636370501164
2021-11-08T11:21:44.708716021Z write finished timestamp 291-1636370504708
2021-11-08T11:22:06.282137626Z Store_Size 31144
2021-11-08T11:22:09.732845844Z Store_Size 31107
2021-11-08T11:22:10.806241866Z write started timestamp 293-1636370530806
2021-11-08T11:22:13.679592291Z write started timestamp 294-1636370533679
2021-11-08T11:22:14.164672020Z write finished timestamp 293-1636370534164
2021-11-08T11:22:17.105497485Z write finished timestamp 294-1636370537105
2021-11-08T11:22:19.174452759Z write started timestamp 296-1636370539174
2021-11-08T11:22:20.588959790Z Store_Size 31203
2021-11-08T11:22:23.399460725Z write finished timestamp 296-1636370543399
2021-11-08T11:22:23.642292978Z Store_Size 31527
2021-11-08T11:22:26.116024095Z Store_Size 31009
2021-11-08T11:22:27.184332947Z write started timestamp 300-1636370547184
2021-11-08T11:22:31.705768959Z Store_Size 31199
2021-11-08T11:22:31.772244089Z write finished timestamp 300-1636370551772
2021-11-08T11:22:56.908945181Z Store_Size 30474
2021-11-08T11:23:44.518238483Z write started timestamp 304-1636370624518
2021-11-08T11:23:47.937927382Z write finished timestamp 304-1636370627937
2021-11-08T11:24:00.346086324Z Store_Size 30102
2021-11-08T11:24:16.097066186Z Store_Size 33626
2021-11-08T11:24:17.194932720Z write started timestamp 306-1636370657194
2021-11-08T11:24:20.501904161Z Store_Size 30672
2021-11-08T11:24:20.663096236Z write finished timestamp 306-1636370660662

--- LAMP_server.1.kx99bjpgmh422l1hpxoxrd4yb TERMINATED ---

2021-11-10T09:18:29.969134093Z GET /participant/U1699389494/sensor_event?origin=lamp.accelerometer&from=1635912000000&to=1636167002817&limit=10000 200 - 3698.791 ms
2021-11-10T09:18:30.225862112Z GET /participant/U8282161095/sensor_event?origin=lamp.accelerometer&from=0&to=1629977289423&limit=10000 200 - 380.475 ms
2021-11-10T09:19:58.096461455Z POST /participant/U9158417286/sensor_event 200 - 19.108 ms
2021-11-10T09:19:58.098314082Z POST /participant/U9158417286/sensor_event 200 - 17.039 ms
2021-11-10T09:19:58.102399596Z POST /participant/U3446920050/sensor_event 200 - 13.688 ms
2021-11-10T09:19:58.103211453Z GET /participant/0vayybdfqxfpv74acj5b/sensor_event?origin=lamp.analytics 304 - 35.695 ms
2021-11-10T09:19:58.106103906Z POST /participant/U3446920050/sensor_event 200 - 15.033 ms
2021-11-10T09:19:58.106798981Z POST /participant/U4097925880/sensor_event 200 - 23.298 ms
2021-11-10T09:19:58.107191619Z GET /participant/0vayybdfqxfpv74acj5b/sensor_event?origin=lamp.analytics 304 - 36.826 ms
2021-11-10T09:19:58.108165607Z POST /participant/U4097925880/sensor_event 200 - 22.216 ms
2021-11-10T09:19:58.124651849Z GET /participant/U2762632331/sensor_event?origin=lamp.analytics 304 - 23.639 ms
2021-11-10T09:19:58.136961717Z GET /participant/yhe0wtfn6n6sbvsan0js/sensor_event?origin=lamp.analytics 304 - 36.209 ms
2021-11-10T09:19:58.149952303Z POST /participant/U8939345176/sensor_event 200 - 14.861 ms
2021-11-10T09:19:58.151526492Z GET /type/0vayybdfqxfpv74acj5b/attachment/lamp.messaging 404 - 87.913 ms
2021-11-10T09:19:58.168018946Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 104.858 ms
2021-11-10T09:19:58.169384236Z GET /type/0vayybdfqxfpv74acj5b/attachment/lamp.messaging 404 - 102.388 ms
2021-11-10T09:19:58.169719141Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 107.449 ms
2021-11-10T09:19:58.171873263Z GET /type/yhe0wtfn6n6sbvsan0js/attachment/lamp.messaging 404 - 71.408 ms
2021-11-10T09:19:58.244656193Z GET /type/U2762632331/attachment/lamp.messaging 304 - 144.679 ms
2021-11-10T09:19:58.321700150Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 106.394 ms
2021-11-10T09:19:58.322306359Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 107.670 ms
2021-11-10T09:19:58.373734513Z POST /participant/U6299504587/sensor_event 200 - 56.645 ms
2021-11-10T09:19:58.379544752Z GET /participant/U4179451929/sensor_event?origin=lamp.accelerometer&from=1635912000000&to=1635933001750&limit=10000 200 - 240.489 ms
2021-11-10T09:19:58.387041951Z GET /participant/U1699389494/sensor_event?origin=lamp.accelerometer&from=1635912000000&to=1636165000707&limit=10000 200 - 300.201 ms
2021-11-10T09:19:58.420610175Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 39.910 ms
2021-11-10T09:19:58.421530595Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 40.325 ms
2021-11-10T09:19:58.425922957Z POST /participant/U6299504587/sensor_event 200 - 7.383 ms
2021-11-10T09:19:58.453318235Z GET /participant/0vayybdfqxfpv74acj5b/sensor_event?origin=lamp.analytics 304 - 14.514 ms
2021-11-10T09:19:58.486397093Z GET /type/0vayybdfqxfpv74acj5b/attachment/lamp.messaging 404 - 47.088 ms
2021-11-10T09:19:58.526095632Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 49.447 ms
2021-11-10T09:19:58.531582727Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 54.478 ms
2021-11-10T09:19:58.595859654Z GET /participant/U4179451929/sensor_event?origin=lamp.accelerometer&from=1635912000000&to=1635931999599&limit=10000 200 - 184.535 ms
2021-11-10T09:19:58.613451730Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 21.226 ms
2021-11-10T09:19:58.614573382Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 21.795 ms
2021-11-10T09:19:58.658528349Z POST /participant/U9444966227/sensor_event 200 - 15.028 ms
2021-11-10T09:19:58.733321124Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 76.626 ms
2021-11-10T09:19:58.746419984Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 89.010 ms
2021-11-10T09:19:58.801379111Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 29.475 ms
2021-11-10T09:19:58.820364235Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 39.848 ms
2021-11-10T09:19:58.876896539Z GET /participant/U4179451929/sensor_event?origin=lamp.accelerometer&from=1635912000000&to=1635930997508&limit=10000 200 - 241.493 ms
2021-11-10T09:19:58.884437868Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 19.790 ms
2021-11-10T09:19:58.885028084Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 19.823 ms
2021-11-10T09:19:58.945178470Z POST /participant/U8939345176/sensor_event 200 - 7.271 ms
2021-11-10T09:19:58.974748594Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 52.047 ms
2021-11-10T09:19:58.981869799Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 59.674 ms
2021-11-10T09:19:59.054790136Z POST /participant/U0128440310/sensor_event 200 - 8.778 ms
2021-11-10T09:19:59.102661039Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 84.785 ms
2021-11-10T09:19:59.103011875Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 82.650 ms
2021-11-10T09:19:59.105322630Z GET /participant/U4179451929/sensor_event?origin=lamp.accelerometer&from=1635912000000&to=1635929995252&limit=10000 200 - 199.557 ms
2021-11-10T09:19:59.109062594Z POST /participant/U8939345176/sensor_event 200 - 5.016 ms
2021-11-10T09:19:59.150401949Z Store_Size 36695
2021-11-10T09:19:59.178101163Z POST /participant/U8939345176/sensor_event 200 - 18.150 ms
2021-11-10T09:19:59.263376312Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 123.213 ms
2021-11-10T09:19:59.276299502Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 135.022 ms
2021-11-10T09:19:59.367048181Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 68.092 ms
2021-11-10T09:19:59.372659984Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 54.279 ms
2021-11-10T09:19:59.375022906Z GET /participant/U4179451929/sensor_event?origin=lamp.accelerometer&from=1635912000000&to=1635928993106&limit=10000 200 - 235.433 ms
2021-11-10T09:19:59.431983661Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 22.675 ms
2021-11-10T09:19:59.432313296Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 29.204 ms
2021-11-10T09:19:59.507146772Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 33.023 ms
2021-11-10T09:19:59.515583873Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 42.014 ms
2021-11-10T09:19:59.618379803Z GET /participant/U4179451929/sensor_event?origin=lamp.accelerometer&from=1635912000000&to=1635927991020&limit=10000 200 - 213.691 ms
2021-11-10T09:19:59.619385475Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 57.423 ms
2021-11-10T09:19:59.620343829Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 48.965 ms
2021-11-10T09:19:59.722556863Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 43.972 ms
2021-11-10T09:19:59.736993795Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 50.481 ms
2021-11-10T09:19:59.822275115Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 31.944 ms
2021-11-10T09:19:59.826834482Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 37.606 ms
2021-11-10T09:19:59.885633658Z GET /participant/U4179451929/sensor_event?origin=lamp.accelerometer&from=1635912000000&to=1635926687962&limit=10000 200 - 235.021 ms
2021-11-10T09:19:59.897532080Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 18.661 ms
2021-11-10T09:19:59.898116192Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 19.764 ms
2021-11-10T09:19:59.977095588Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 41.910 ms
2021-11-10T09:19:59.978859499Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 43.204 ms
2021-11-10T09:20:00.098234862Z GET /participant/U4179451929/sensor_event?origin=lamp.accelerometer&from=1635912000000&to=1635925685883&limit=10000 200 - 188.619 ms
2021-11-10T09:20:00.190224521Z POST /participant/U9444966227/sensor_event 200 - 4.349 ms
2021-11-10T09:20:00.280890491Z write started timestamp 5306-1636536000280
2021-11-10T09:20:00.786762059Z POST /participant/U0128440310/sensor_event 200 - 4.967 ms
2021-11-10T09:20:01.052626009Z Store_Size 30071
2021-11-10T09:20:01.842924388Z GET /participant/U1699389494/sensor_event?origin=lamp.accelerometer&from=1635912000000&to=1636162998474&limit=10000 200 - 3423.226 ms
2021-11-10T09:20:02.119467082Z write started timestamp 5307-1636536002119
2021-11-10T09:20:05.372850603Z write finished timestamp 5306-1636536005372
2021-11-10T09:20:06.055643516Z [ioredis] Unhandled error event: Error: connect ECONNREFUSED 10.0.6.7:6379
2021-11-10T09:20:06.055669893Z     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1146:16)
2021-11-10T09:20:06.157284597Z [ioredis] Unhandled error event: Error: connect ECONNREFUSED 10.0.6.7:6379
2021-11-10T09:20:06.157308629Z     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1146:16)
2021-11-10T09:20:06.308355508Z [ioredis] Unhandled error event: Error: connect ECONNREFUSED 10.0.6.7:6379
2021-11-10T09:20:06.308378351Z     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1146:16)
2021-11-10T09:20:06.509463304Z [ioredis] Unhandled error event: Error: connect ECONNREFUSED 10.0.6.7:6379
2021-11-10T09:20:06.509484996Z     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1146:16)
2021-11-10T09:20:06.761649449Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-10T09:20:06.761671807Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-10T09:20:07.062383926Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-10T09:20:07.062408431Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-10T09:20:07.271692403Z write finished timestamp 5307-1636536007271
2021-11-10T09:20:07.413800003Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-10T09:20:07.413821711Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-10T09:20:07.815357412Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-10T09:20:07.815376732Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-10T09:20:08.266897525Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-10T09:20:08.266914169Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)

--- LAMP_server.1.drlwcrmt19r0adf6swzd0bhha TERMINATED ---

Worker

2021-11-06T07:24:36.202196895Z initialize queue
2021-11-06T07:24:36.215008861Z Initialized the queues
2021-11-06T07:24:36.216768438Z Clean all queues...
2021-11-06T07:24:36.217041440Z CLEANING ALL QUEUE
2021-11-06T07:24:36.296416766Z DONE--CLEANING ALL QUEUE
2021-11-06T07:24:36.296438878Z Initializing schedulers...
2021-11-06T07:24:36.311950283Z Running with automation disabled.
2021-11-06T07:24:36.313312103Z server listening in 3000
2021-11-06T07:24:36.313420394Z Initiating nats server
2021-11-06T07:24:36.321638606Z error---while nats connect Error: getaddrinfo ENOTFOUND nats
2021-11-06T07:24:36.321668691Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26) {
2021-11-06T07:24:36.321671901Z   errno: -3008,
2021-11-06T07:24:36.321674320Z   code: 'ENOTFOUND',
2021-11-06T07:24:36.321676540Z   syscall: 'getaddrinfo',
2021-11-06T07:24:36.321678537Z   hostname: 'nats'
2021-11-06T07:24:36.321680439Z }
2021-11-06T07:24:37.330102918Z error while scheduling initially--- FetchError: request to https://api.lamp.digital/researcher failed, reason: connect ECONNREFUSED 10.0.0.2:443
2021-11-06T07:24:37.330128300Z     at ClientRequest.<anonymous> (/usr/src/app/node_modules/node-fetch/lib/index.js:1461:11)
2021-11-06T07:24:37.330131635Z     at ClientRequest.emit (node:events:394:28)
2021-11-06T07:24:37.330133585Z     at TLSSocket.socketErrorListener (node:_http_client:447:9)
2021-11-06T07:24:37.330135586Z     at TLSSocket.emit (node:events:394:28)
2021-11-06T07:24:37.330137485Z     at emitErrorNT (node:internal/streams/destroy:157:8)
2021-11-06T07:24:37.330139471Z     at emitErrorCloseNT (node:internal/streams/destroy:122:3)
2021-11-06T07:24:37.330141280Z     at processTicksAndRejections (node:internal/process/task_queues:83:21) {
2021-11-06T07:24:37.330143323Z   type: 'system',
2021-11-06T07:24:37.330145756Z   errno: 'ECONNREFUSED',
2021-11-06T07:24:37.330149066Z   code: 'ECONNREFUSED'
2021-11-06T07:24:37.330151713Z }

--- LAMP_worker.1.euh0cxgo7k7pkux7f8fcd1nhy TERMINATED ---

2021-11-08T08:19:00.005522787Z Preparing to fetch activities
2021-11-08T08:19:00.046734113Z activity_id given nkc57v55fcs8weam6zgk
2021-11-08T08:19:00.046754928Z Saving to redis
2021-11-08T08:19:00.046757459Z Processing 1 activities for push notifications.
2021-11-08T08:19:00.046759478Z locked job on activity_scheduler
2021-11-08T08:19:00.582880094Z start_date======== 2021-02-26T12:19:00.000Z
2021-11-08T08:19:00.584576875Z Saving to Redis completed....
2021-11-08T08:19:00.584598475Z release lock  on success  activity_scheduler
2021-11-08T08:19:00.584602000Z Rescheduled job after notificcation process-  nkc57v55fcs8weam6zgk
2021-11-08T08:19:00.584605722Z release lock  on success  nkc57v55fcs8weam6zgk
2021-11-08T08:19:01.629986160Z participant_id listened U0176919012
2021-11-08T08:19:01.630017903Z sensordata listened {
2021-11-08T08:19:01.630022393Z   action: 'notification',
2021-11-08T08:19:01.630025274Z   user_action: 'Open App',
2021-11-08T08:19:01.630028439Z   payload: '{expiry=21600000, actions=[{"name":"Open App","page":"\\/participant\\/U0176919012\\/activity\\/nkc57v55fcs8weam6zgk"}], page=/participant/U0176919012/activity/nkc57v55fcs8weam6zgk, title=Baloon, notificationId=519628, message=You have a mindLAMP activity waiting for you: Baloon.}'
2021-11-08T08:19:01.630033493Z }
2021-11-08T08:20:00.002835202Z today_date while notify 2021-11-08T08:20:00.000Z
2021-11-08T08:20:00.002861522Z start_date while notify 2021-05-19T15:20:00.000Z
2021-11-08T08:20:00.002864169Z {
2021-11-08T08:20:00.002866362Z   device_token: 'REDACTED',
2021-11-08T08:20:00.002868674Z   device_type: 'android',
2021-11-08T08:20:00.002870581Z   payload: {
2021-11-08T08:20:00.002872572Z     participant_id: 'U0176919012',
2021-11-08T08:20:00.002874477Z     activity_id: '39m9gp4da1r20fsdvqjz',
2021-11-08T08:20:00.002876942Z     message: 'You have a mindLAMP activity waiting for you: b1.',
2021-11-08T08:20:00.002880263Z     title: 'b1',
2021-11-08T08:20:00.002883252Z     url: '/participant/U0176919012/activity/39m9gp4da1r20fsdvqjz',
2021-11-08T08:20:00.002886399Z     notificationId: 906379
2021-11-08T08:20:00.002888943Z   }
2021-11-08T08:20:00.002891712Z }
2021-11-08T08:20:00.002894293Z /participant/U0176919012/activity/39m9gp4da1r20fsdvqjz
2021-11-08T08:20:00.004613263Z Completed  job state on 39m9gp4da1r20fsdvqjz
2021-11-08T08:20:00.004657287Z locked job on 39m9gp4da1r20fsdvqjz
2021-11-08T08:20:00.004671092Z jobs in queue
2021-11-08T08:20:00.004674260Z Preparing to fetch activities
2021-11-08T08:20:00.009562860Z release lock  on exception2  39m9gp4da1r20fsdvqjz
2021-11-08T08:20:05.048041434Z participant_id listened U4233494401
2021-11-08T08:20:05.048068154Z sensordata listened {
2021-11-08T08:20:05.048070584Z   latitude: REDACTED,
2021-11-08T08:20:05.048072642Z   longitude: REDACTED,
2021-11-08T08:20:05.048074580Z   accuracy: 4.775681811813515,
2021-11-08T08:20:05.048076489Z   altitude: 312.71183001063764
2021-11-08T08:20:05.048078438Z }
2021-11-08T08:20:05.214713334Z participant_id listened U4233494401
2021-11-08T08:20:05.214736454Z sensordata listened {
2021-11-08T08:20:05.214738868Z   latitude: REDACTED,
2021-11-08T08:20:05.214741266Z   longitude: REDACTED,
2021-11-08T08:20:05.214743187Z   accuracy: 4.786619216730543,
2021-11-08T08:20:05.214745111Z   altitude: 312.71183001063764
2021-11-08T08:20:05.214747091Z }
2021-11-08T08:20:05.222477535Z participant_id listened U6368562055
2021-11-08T08:20:05.222497302Z sensordata listened {
2021-11-08T08:20:05.222499336Z   latitude: REDACTED,
2021-11-08T08:20:05.222501096Z   longitude: REDACTED,
2021-11-08T08:20:05.222503108Z   accuracy: 14.262297327788108,
2021-11-08T08:20:05.222505046Z   altitude: 230.2040985794738
2021-11-08T08:20:05.222506922Z }
2021-11-08T08:20:05.228750004Z participant_id listened U6368562055
2021-11-08T08:20:05.228761361Z sensordata listened {
2021-11-08T08:20:05.228763364Z   latitude: REDACTED,
2021-11-08T08:20:05.228765676Z   longitude: REDACTED,
2021-11-08T08:20:05.228767531Z   accuracy: 14.262618541848493,
2021-11-08T08:20:05.228769346Z   altitude: 230.2040985794738
2021-11-08T08:20:05.228771275Z }
2021-11-08T08:20:05.231691378Z participant_id listened U1912527657
2021-11-08T08:20:05.231709450Z sensordata listened {
2021-11-08T08:20:05.231712015Z   latitude: REDACTED,
2021-11-08T08:20:05.231714235Z   longitude: REDACTED,
2021-11-08T08:20:05.231716164Z   accuracy: 14.246829593421227,
2021-11-08T08:20:05.231718100Z   altitude: 7.289181161671877
2021-11-08T08:20:05.231720233Z }
2021-11-08T08:20:05.234595637Z participant_id listened U1912527657
2021-11-08T08:20:05.234608192Z sensordata listened {
2021-11-08T08:20:05.234611238Z   latitude: REDACTED,
2021-11-08T08:20:05.234613357Z   longitude: REDACTED,
2021-11-08T08:20:05.234615358Z   accuracy: 14.247632504082615,
2021-11-08T08:20:05.234617276Z   altitude: 7.289181161671877
2021-11-08T08:20:05.234619080Z }
2021-11-08T08:20:05.450339201Z participant_id listened U6299504587
2021-11-08T08:20:05.450363146Z sensordata listened { x: 0.06829833984375, y: -0.0142364501953125, z: -1.0013427734375 }
2021-11-08T08:20:05.453017363Z participant_id listened U6299504587
2021-11-08T08:20:05.453033604Z sensordata listened {
2021-11-08T08:20:05.453036186Z   x: 0.0678558349609375,
2021-11-08T08:20:05.453038134Z   y: -0.0136260986328125,
2021-11-08T08:20:05.453039975Z   z: -1.0009307861328125
2021-11-08T08:20:05.453041820Z }
2021-11-08T08:20:05.517296003Z participant_id listened U7645962928
2021-11-08T08:20:05.517318942Z sensordata listened {
2021-11-08T08:20:05.517322966Z   latitude: REDACTED,
2021-11-08T08:20:05.517326298Z   longitude: REDACTED,
2021-11-08T08:20:05.517329676Z   accuracy: 10,
2021-11-08T08:20:05.517332783Z   altitude: 19.375179290771484
2021-11-08T08:20:05.517336345Z }
2021-11-08T08:20:05.524529373Z participant_id listened U7645962928
2021-11-08T08:20:05.524552526Z sensordata listened {
2021-11-08T08:20:05.524554807Z   latitude: REDACTED,
2021-11-08T08:20:05.524556893Z   longitude: REDACTED,
2021-11-08T08:20:05.524558845Z   accuracy: 10,
2021-11-08T08:20:05.524560778Z   altitude: 19.36281967163086
2021-11-08T08:20:05.524562749Z }
---  LAMP_worker.1.ltv0p7vghnk4ltzicgexbpskc TERMINATED ---
2021-11-08T11:59:18.509269556Z     y: 12.539999961853027,
2021-11-08T11:59:18.509272429Z     z: -14.15999984741211
2021-11-08T11:59:18.509275238Z   }
2021-11-08T11:59:18.509277995Z }
2021-11-08T11:59:18.877902420Z participant_id listened U0719341792
2021-11-08T11:59:18.877977567Z sensordata listened {
2021-11-08T11:59:18.877981005Z   gravity: {
2021-11-08T11:59:18.877984176Z     x: 0.001065264455974102,
2021-11-08T11:59:18.877986250Z     y: -0.002130528911948204,
2021-11-08T11:59:18.877988268Z     z: -0.001065264455974102
2021-11-08T11:59:18.877990133Z   }
2021-11-08T11:59:18.877991883Z }
2021-11-08T11:59:19.321976076Z participant_id listened U0719341792
2021-11-08T11:59:19.322048102Z sensordata listened {
2021-11-08T11:59:19.322057331Z   motion: {
2021-11-08T11:59:19.322059597Z     x: -0.08140286058187485,
2021-11-08T11:59:19.322061553Z     y: -0.026336219161748886,
2021-11-08T11:59:19.322063597Z     z: 9.876082420349121
2021-11-08T11:59:19.322065491Z   }
2021-11-08T11:59:19.322067437Z }
2021-11-08T11:59:19.776248425Z participant_id listened U0719341792
2021-11-08T11:59:19.776270112Z sensordata listened {
2021-11-08T11:59:19.776274228Z   motion: {
2021-11-08T11:59:19.776277055Z     x: -0.090979665517807,
2021-11-08T11:59:19.776279944Z     y: -0.023942016065120697,
2021-11-08T11:59:19.776282544Z     z: 9.876082420349121
2021-11-08T11:59:19.776285366Z   }
2021-11-08T11:59:19.776288047Z }
2021-11-08T11:59:20.191797345Z participant_id listened U0719341792
2021-11-08T11:59:20.191838301Z sensordata listened {
2021-11-08T11:59:20.191842910Z   gravity: {
2021-11-08T11:59:20.191846229Z     x: -0.001065264455974102,
2021-11-08T11:59:20.191848806Z     y: -0.000532632227987051,
2021-11-08T11:59:20.191851511Z     z: 0.000532632227987051
2021-11-08T11:59:20.191854145Z   }
2021-11-08T11:59:20.191856849Z }
2021-11-08T11:59:20.590169289Z participant_id listened U0719341792
2021-11-08T11:59:20.590225744Z sensordata listened {
2021-11-08T11:59:20.590232063Z   motion: {
2021-11-08T11:59:20.590235061Z     x: -0.0933738648891449,
2021-11-08T11:59:20.590237989Z     y: -0.019153613597154617,
2021-11-08T11:59:20.590239923Z     z: 9.86650562286377
2021-11-08T11:59:20.590241955Z   }
2021-11-08T11:59:20.590243769Z }
2021-11-08T11:59:20.998172899Z participant_id listened U0719341792
2021-11-08T11:59:20.998188203Z sensordata listened {
2021-11-08T11:59:20.998191355Z   rotation: {
2021-11-08T11:59:20.998194478Z     x: -0.003573794150725007,
2021-11-08T11:59:20.998197671Z     y: 0.0024797802325338125,
2021-11-08T11:59:20.998200398Z     z: 0.4774126410484314
2021-11-08T11:59:20.998203011Z   }
2021-11-08T11:59:20.998205804Z }
2021-11-08T11:59:21.447888928Z participant_id listened U0719341792
2021-11-08T11:59:21.447923031Z sensordata listened {
2021-11-08T11:59:21.447927683Z   magnetic: {
2021-11-08T11:59:21.447930491Z     x: 19.260000228881836,
2021-11-08T11:59:21.447933453Z     y: 12.119999885559082,
2021-11-08T11:59:21.447945851Z     z: -13.559999465942383
2021-11-08T11:59:21.447948942Z   }
2021-11-08T11:59:21.447951579Z }
2021-11-08T11:59:21.878267509Z participant_id listened U0719341792
2021-11-08T11:59:21.878312756Z sensordata listened {
2021-11-08T11:59:21.878316671Z   motion: {
2021-11-08T11:59:21.878320103Z     x: -0.08858546614646912,
2021-11-08T11:59:21.878323735Z     y: -0.033518824726343155,
2021-11-08T11:59:21.878326744Z     z: 9.873687744140625
2021-11-08T11:59:21.878330054Z   }
2021-11-08T11:59:21.878333029Z }
2021-11-08T11:59:22.239922802Z participant_id listened U0719341792
2021-11-08T11:59:22.239947879Z sensordata listened {
2021-11-08T11:59:22.239951281Z   magnetic: {
2021-11-08T11:59:22.239953965Z     x: 19.020000457763672,
2021-11-08T11:59:22.239956960Z     y: 12.239999771118164,
2021-11-08T11:59:22.239960459Z     z: -13.4399995803833
2021-11-08T11:59:22.239962609Z   }
2021-11-08T11:59:22.239965308Z }
2021-11-08T11:59:22.627761860Z participant_id listened U0719341792
2021-11-08T11:59:22.627826932Z sensordata listened {
2021-11-08T11:59:22.627837246Z   magnetic: {
2021-11-08T11:59:22.627840435Z     x: 19.31999969482422,
2021-11-08T11:59:22.627843210Z     y: 12.359999656677246,
2021-11-08T11:59:22.627846020Z     z: -14.34000015258789
2021-11-08T11:59:22.627848931Z   }
2021-11-08T11:59:22.627851859Z }
2021-11-08T11:59:23.003636028Z participant_id listened U0719341792
2021-11-08T11:59:23.003697660Z sensordata listened {
2021-11-08T11:59:23.003707923Z   gravity: {
2021-11-08T11:59:23.003710475Z     x: -0.000532632227987051,
2021-11-08T11:59:23.003712540Z     y: -0.000532632227987051,
2021-11-08T11:59:23.003714605Z     z: -0.000532632227987051
2021-11-08T11:59:23.003716550Z   }
2021-11-08T11:59:23.003718536Z }
2021-11-08T11:59:23.353067933Z participant_id listened U0719341792
2021-11-08T11:59:23.353084195Z sensordata listened {
2021-11-08T11:59:23.353100123Z   rotation: {
2021-11-08T11:59:23.353103096Z     x: -0.003732413984835148,
2021-11-08T11:59:23.353106293Z     y: 0.0027173981070518494,
2021-11-08T11:59:23.353109459Z     z: 0.4790908396244049
2021-11-08T11:59:23.353112380Z   }
2021-11-08T11:59:23.353115349Z }

--- LAMP_worker.1.qxvo8mrms2wj1uzlfuqa77aze TERMINATED ---

2021-11-10T09:19:58.946995803Z sensordata listened {
2021-11-10T09:19:58.947001326Z   magnetic: {
2021-11-10T09:19:58.947004375Z     x: -24.318750381469727,
2021-11-10T09:19:58.947007047Z     y: -11.700000762939453,
2021-11-10T09:19:58.947009672Z     z: -39.01875305175781
2021-11-10T09:19:58.947012450Z   }
2021-11-10T09:19:58.947015167Z }
2021-11-10T09:19:59.056761481Z participant_id listened U0128440310
2021-11-10T09:19:59.056789957Z sensordata listened { magnetic: { x: 293.3125, y: -750.7999877929688, z: -764.4375 } }
2021-11-10T09:19:59.111401990Z participant_id listened U8939345176
2021-11-10T09:19:59.111421069Z sensordata listened {
2021-11-10T09:19:59.111424916Z   motion: {
2021-11-10T09:19:59.111427703Z     x: 0.5814558863639832,
2021-11-10T09:19:59.111430688Z     y: 0.8201399445533752,
2021-11-10T09:19:59.111433432Z     z: 9.896116256713867
2021-11-10T09:19:59.111436440Z   }
2021-11-10T09:19:59.111439430Z }
2021-11-10T09:19:59.270585336Z participant_id listened U8939345176
2021-11-10T09:19:59.270616530Z sensordata listened {
2021-11-10T09:19:59.270621113Z   rotation: {
2021-11-10T09:19:59.270624342Z     x: 0.00011796114267781377,
2021-11-10T09:19:59.270627960Z     y: -0.050691016018390656,
2021-11-10T09:19:59.270631288Z     z: -0.8104769587516785
2021-11-10T09:19:59.270634650Z   }
2021-11-10T09:19:59.270638188Z }
2021-11-10T09:20:00.003270797Z today_date while notify 2021-11-10T09:20:00.000Z
2021-11-10T09:20:00.003299851Z start_date while notify 2021-11-05T09:20:00.000Z
2021-11-10T09:20:00.003303371Z {
2021-11-10T09:20:00.003306086Z   device_token: 'REDACTED',
2021-11-10T09:20:00.003309343Z   device_type: 'ios',
2021-11-10T09:20:00.003312104Z   payload: {
2021-11-10T09:20:00.003314621Z     participant_id: 'U3494874569',
2021-11-10T09:20:00.003317306Z     activity_id: 'mgwry4j8za55x34yzs21',
2021-11-10T09:20:00.003320030Z     message: 'You have a mindLAMP activity waiting for you: Scr 1.',
2021-11-10T09:20:00.003323051Z     title: 'Scr 1',
2021-11-10T09:20:00.003335919Z     url: '/participant/U3494874569/activity/mgwry4j8za55x34yzs21',
2021-11-10T09:20:00.003338655Z     notificationId: 947940
2021-11-10T09:20:00.003341561Z   }
2021-11-10T09:20:00.003344084Z }
2021-11-10T09:20:00.003348918Z /participant/U3494874569/activity/mgwry4j8za55x34yzs21
2021-11-10T09:20:00.004385360Z Completed  job state on mgwry4j8za55x34yzs21
2021-11-10T09:20:00.005568729Z today_date while notify 2021-11-10T09:20:00.000Z
2021-11-10T09:20:00.005624513Z start_date while notify 2021-05-19T15:20:00.000Z
2021-11-10T09:20:00.005630533Z {
2021-11-10T09:20:00.005633475Z   device_token: 'REDACTED',
2021-11-10T09:20:00.005636957Z   device_type: 'android',
2021-11-10T09:20:00.005639972Z   payload: {
2021-11-10T09:20:00.005642709Z     participant_id: 'U0176919012',
2021-11-10T09:20:00.005645534Z     activity_id: '39m9gp4da1r20fsdvqjz',
2021-11-10T09:20:00.005648396Z     message: 'You have a mindLAMP activity waiting for you: b1.',
2021-11-10T09:20:00.005651227Z     title: 'b1',
2021-11-10T09:20:00.005654131Z     url: '/participant/U0176919012/activity/39m9gp4da1r20fsdvqjz',
2021-11-10T09:20:00.005657025Z     notificationId: 906379
2021-11-10T09:20:00.005659753Z   }
2021-11-10T09:20:00.005662337Z }
2021-11-10T09:20:00.005666792Z /participant/U0176919012/activity/39m9gp4da1r20fsdvqjz
2021-11-10T09:20:00.006522392Z Completed  job state on 39m9gp4da1r20fsdvqjz
2021-11-10T09:20:00.020225973Z Error encountered sending GCM push notification.
2021-11-10T09:20:00.020923870Z response Response {
2021-11-10T09:20:00.020930398Z   size: 0,
2021-11-10T09:20:00.020932640Z   timeout: 0,
2021-11-10T09:20:00.020935247Z   [Symbol(Body internals)]: {
2021-11-10T09:20:00.020937221Z     body: PassThrough {
2021-11-10T09:20:00.020939139Z       _readableState: [ReadableState],
2021-11-10T09:20:00.020941286Z       _events: [Object: null prototype],
2021-11-10T09:20:00.020943234Z       _eventsCount: 2,
2021-11-10T09:20:00.020945038Z       _maxListeners: undefined,
2021-11-10T09:20:00.020947136Z       _writableState: [WritableState],
2021-11-10T09:20:00.020948938Z       allowHalfOpen: true,
2021-11-10T09:20:00.020950833Z       [Symbol(kCapture)]: false,
2021-11-10T09:20:00.020952718Z       [Symbol(kCallback)]: null
2021-11-10T09:20:00.020954501Z     },
2021-11-10T09:20:00.020966065Z     disturbed: false,
2021-11-10T09:20:00.020968001Z     error: null
2021-11-10T09:20:00.020980261Z   },
2021-11-10T09:20:00.020982205Z   [Symbol(Response internals)]: {
2021-11-10T09:20:00.020984000Z     url: 'https://app-gateway.lamp.digital/push',
2021-11-10T09:20:00.020985912Z     status: 404,
2021-11-10T09:20:00.020988003Z     statusText: 'Not Found',
2021-11-10T09:20:00.020989830Z     headers: Headers { [Symbol(map)]: [Object: null prototype] },
2021-11-10T09:20:00.020991858Z     counter: 0
2021-11-10T09:20:00.020993730Z   }
2021-11-10T09:20:00.020995579Z }
2021-11-10T09:20:00.021035963Z "Error encountered sending APN push notification."--Error: HTTP error!
2021-11-10T09:20:00.192021360Z participant_id listened U9444966227
2021-11-10T09:20:00.192043114Z sensordata listened {
2021-11-10T09:20:00.192046054Z   motion: {
2021-11-10T09:20:00.192048093Z     x: 0.17237511277198792,
2021-11-10T09:20:00.192050350Z     y: -0.03112328238785267,
2021-11-10T09:20:00.192052527Z     z: 9.806227684020996
2021-11-10T09:20:00.192054420Z   }
2021-11-10T09:20:00.192056198Z }
2021-11-10T09:20:00.788429637Z participant_id listened U0128440310
2021-11-10T09:20:00.788451798Z sensordata listened {
2021-11-10T09:20:00.788455459Z   gravity: {
2021-11-10T09:20:00.788458327Z     x: 0.00026631588116288185,
2021-11-10T09:20:00.788461046Z     y: -0.00013315794058144093,
2021-11-10T09:20:00.788463647Z     z: -0.00013315794058144093
2021-11-10T09:20:00.788466359Z   }
2021-11-10T09:20:00.788469099Z }=

--- LAMP_worker.1.qcfz0h4k4u7v4748egvle96rk TERMINATED ---

[lukeoftheshire]

Hi @Linoy339

That was a good catch! For some reason our yaml file and the the actual container were mismatched - the yaml file said 2021, but the container was using the latest image! Aditya and I have fixed this so we should now be using the 2021 image for production.

Also, I have tried the staging API and I am able to make some basic requests successfully.

[Linoy339]

OKay. Let me update the staging stack once again and check. will revert you here soon

[Linoy339]

@lukeoftheshire . It worked after I removed some labels from staging-worker stack :

       traefik.enable: 'true'
        traefik.docker.network: '****'
        traefik.http.routers.lamp_worker_staging.entryPoints: '*****'
        traefik.http.routers.lamp_worker_staging.tls.certresolver: '****'
        traefik.http.services.lamp_worker_staging.loadbalancer.server.port: ****

[Linoy339]

@lukeoftheshire . Can you please restart LAMP_server and LAMP_worker, as nats found to be disconnected.

[lukeoftheshire]

Hi @Linoy339 - I have restarted the server and worker services!

[Linoy339]

Thanks @lukeoftheshire

[Linoy339]

@lukeoftheshire . The lamp_worker staging instance is not updating with latest code changes. Can you please check on this?

[lukeoftheshire]

Hi @Linoy339 - after doing an image update for lamp_worker, I also did a little digging and it looks like the lamp_worker service is using an image from ~29 hours ago, which appears to match up timewise with the latest release found at https://github.com/BIDMCDigitalPsychiatry/LAMP-worker/pkgs/container/lamp-worker/versions.

Is there a later version that should be being used?

[Linoy339]

@lukeoftheshire . Apologies. Now, it seems like updated. I am not sure what happened, may be the update happened little late

Thank you

[Linoy339]

@lukeoftheshire Can you please restart LAMP_server and LAMP_worker for production environment ?

[lukeoftheshire]

@Linoy339 - Done!

[Linoy339]

Thank you

[avaidyam]

@lukeoftheshire . It worked after I removed some labels from staging-worker stack :

     traefik.enable: 'true'
      traefik.docker.network: '****'
      traefik.http.routers.lamp_worker_staging.entryPoints: '*****'
      traefik.http.routers.lamp_worker_staging.tls.certresolver: '****'
      traefik.http.services.lamp_worker_staging.loadbalancer.server.port: ****

This is a good catch, thanks for letting us know!

[Linoy339]

@lukeoftheshire Can you please restart LAMP_server and LAMP_worker for production environment ?

[lukeoftheshire]

@Linoy339 Done! Apologies for the delay.

[Linoy339]

@lukeoftheshire . We were troubleshooting this issue. Can you please confirm : 1) Can you please confirm hostname of LAMP_nats ? 2) Is there any permission issue for connecting Lamp_server and Lamp_nats at this moment?

It seems like Lamp_server and Lamp_worker got crashed and restarted. Its unable to connect nats there after.

[avaidyam]

@Linoy339 You should be able to see all these details in the console in the Stack page. There's no public hostname for NATS, so it's nats:4222.

As it's been a few weeks, I need to ask -- are we getting closer to resolving this issue at all?

[Linoy339]

@avaidyam . We tried in many ways. We were testing staging with fixes. We checked with memory and all those were fine. We observed that the production along with staging is getting restarted at some point. This is where we stuck with the question that why the both stacks getting restarted.

[michaelmenon]

Hi @avaidyam We will be updating some possible fix asap and will let you know. First we are trying to have the worker re-connect successfully to NATS even if its re-started A fix for that will be updated asap. Once that is done then we will have time to analyze why the whole stack is getting restarted and figure out the root cause. We will keep you updated on this.

[avaidyam]

@Linoy339

LAMP-server:
[ioredis] Unhandled error event: Error: connect ECONNREFUSED 10.0.6.7:6379

LAMP-redis: 
1:signal-handler (1638439813) Received SIGTERM scheduling shutdown...
1:M 02 Dec 2021 10:10:13.793 # User requested shutdown...
1:M 02 Dec 2021 10:10:13.793 # Redis is now ready to exit, bye bye...

LAMP-nats:
[1] 2021/12/02 08:10:24.346357 [ERR] 10.0.6.4:60046 - cid:137 - processPub Parse Error: 'lamp.dashboard.experimental.Anxiety Short Form_graph 176'
[1] 2021/12/02 08:10:24.487041 [ERR] 10.0.6.4:60050 - cid:138 - processPub Parse Error: 'lamp.dashboard.experimental.Mood Short Form_graph 170'
[1] 2021/12/02 08:10:29.687110 [ERR] 10.0.6.4:60052 - cid:139 - processPub Parse Error: 'lamp.dashboard.experimental.Anxiety Short Form_graph 176'

I'm not sure why Redis is being shutdown by Docker, but it's not being killed for out-of-memory. There may be a debug log mode to enable for it.

NATS might be crashing due to this error. It looks like LAMP-server is allowing the string lamp.dashboard.experimental.Anxiety Short Form_graph 176 to be used as a key, which is an illegal reverse DNS identifier. The proper regular expression for a reverse DNS identifier is:

^[A-Za-z]{2,6}((?!-)\.[A-Za-z0-9-]{1,63}(?<!-))+$

//cc @lukeoftheshire Can you try to reproduce a crash by poking a new value for one of these keys that has a space in it?

[michaelmenon]

The Redis server is being stopped by docker when the stack is getting restarted and the docker is sending SIGTERM signal to Redis process. so its not an issue of Redis. we need to figure out why the stack is getting restarted.

@Linoy please confirm in production all the NATS tokens are custom tokens so illegal token exception should not cause this issues. in staging we have automation where you can push on different tokens but that is still not pushed to production and we have this issue in production as well. Linoy has already made the check for illegal tokens in his local code and this will be update3d with the fix we are trying for NATS reconnect.

[Linoy339]

@lukeoftheshire .Also, Can you restart Lamp-server and Lamp-worker, as the nats is disconnected now?

@avaidyamlamp.dashboard.experimental.Anxiety Short Form_graph 176. Its used in tags. Right?

[michaelmenon]

Also we need to put the depends_on key in the docker compose of the worker service, so that NATS/Redis is restarted properly before the worker. We will test this in staging along with other fix.

depends_on:

• nats
• message_queue

[avaidyam]

@michaelmenon depends_on is not supported by Docker Swarm:

The depends_on option is ignored when deploying a stack in swarm mode with a version 3 Compose file.

[michaelmenon]

ok thanks. Then there must be some other flag to define the start order in swarm mode. Will check we need to use that.

[michaelmenon]

Also we should use this flag --log-opt max-size=10m --log-opt max-file=2 with docker run for each container to prune the log files produced by each docker container. I think the docker log files produced by each container is growing to a big size and take-up a lot of space.

[michaelmenon]

I think instead of depending on docker flags we already have the reconnection logic in the worker code just need to make sure that actually waits until the NATS is not up. But needs to have the flags to prune the docker logs of each container so that they don't grow too big and consume all the space causing a stack restart.

[michaelmenon]

Need to put this in each of the docker service in the compose file logging: driver: "json-file" options: max-size: "50m"

[avaidyam]

Also we should use this flag --log-opt max-size=10m --log-opt max-file=2 with docker

These were already set quite some time ago by @lukeoftheshire for LAMP-server. NATS and Redis don't take up much logging space, so @lukeoftheshire, could you copy/paste the same logging options for LAMP-dashboard and LAMP-worker? It'll require a restart.

I think instead of depending on docker flags we already have the reconnection logic in the worker code just need to make sure that actually waits until the NATS is not up.

Agreed - the waiting logic must be in both LAMP-server and LAMP-worker.

[lukeoftheshire]

@avaidyam I have updated the loggin options for dashboard and worker - they have restarted so I assume those restrctions now apply

On a related note, I set the 'lamp.dashboard.experimental.Anxiety Short Form_graph 176' tag on one of my example particpants and nothing significant seems to have happened on the server as of a few minutes after setting that tag.

[Linoy339]

@lukeoftheshire . Can you restart Lamp-server and Lamp-worker in production ? Also please pull latest image for LAMP-worker in production

[lukeoftheshire]

@Linoy339 both done!

[Linoy339]

Lamp-worker is started properly. But, it seems LAMP-server didn't Can you check @lukeoftheshire ?

[lukeoftheshire]

I restarted it again - it looks to be running now!

[Linoy339]

@avaidyam . @lukeoftheshire . We have done some fixes in staging environment. @lukeoftheshire . Can you please restart LAMP-server for production ?

[SuruPa00]

@Linoy339 was this intentionally closed? it is still in testing status and being worked on, no?

[Linoy339]

Apologies. It was by mistake.

[Linoy339]

@lukeoftheshire .Please make sure that LAMP-server for production is restarted

[lukeoftheshire]

Hi @Linoy339 Apologies for the delay! LAMP-server is restarted in prod.

[michaelmenon]

Hi @avaidyam @lukeoftheshire can you let me know if Docker was updated recently I mean 3-4 weeks back ?

When I checked the LAMP server and Worker exit code is 137 and usually 137 means OOM error out of memory error but you can check if its out of memory in the inspect container section and there it clearly says OOMKilled as false so its not out of memory. And while researching I found this thread in the Docker forum:

https://forums.docker.com/t/container-fails-with-error-137-but-no-oom-flag-set-and-theres-plenty-of-ram/69336

Other users have also reported similar kind of issues where the containers are being stopped after 2-3 days with exit 137 and no OOMKilled flag. Please let us know the Docker version running on the production we may try to update the Docker version and check.

[avaidyam]

node-01 was upgraded about 4 months ago due to a version of Amazon Linux 1 being deprecated. It's running Docker v20, and node-02 is still running Docker v18. I would also note that it's possible to have an error code 137 and OOMKilled = false if the application (i.e. Java or NodeJS) killed itself before Docker killed it for a memory violation. This could be the underlying issue, but I wonder if there's something happening to both LAMP-server and LAMP-worker that causes an OOM error for NodeJS? (The issue is we haven't seen the NodeJS OOM exception log messages either.)

[michaelmenon]

ok yes not seeing any OOM. exception for for Node js. Can you run "dmesg" command in the node-01 machine terminal. This will show the signals sent by the Kernal to the processes . Please see if there is a "Killed process" message.

OOMKilled is set to false if Docker itself kills the process but it will not be set if Kernel kills a process for out of memory. We can see the "Killed process" signal sent by kernel by running this command "dmesg" in the command terminal of the host machine itself. Can you please check on that.

[lukeoftheshire]

Hi @michaelmenon @avaidyam - running dmesg -T | grep -i Killed in node-01 reveals two instances when the redis-server process was killed - however both timestamps were in October (the 15th and the 23rd, for the record). There doesn't appear to be anything more recent than that.

Hope this helps - it seems to me like these specific instances are not relevant to this issue so I won't clog the page by including logs or anything, but let me know if I can give more info about either of these issues or if a screenshot/text log would be helpful.

[Linoy339]

@lukeoftheshire . Seems like the production version of LAMP-server got deployed using lamp-server:latest instead of lamp-server:2021. Please check.

Also, the the crash was reported today. You can see that production and staging stack got down at the same time. As we said, It looks strange, as these are different stacks.

Please share your thoughts. If you get any spike report from your server logs, please let us know

[michaelmenon]

Thanks @lukeoftheshire . It does not seems to be from low memory as I was watching the memory for all the process in the node -01 yesterday night and all of them were taking very low memory. And the stack went down yesterday night so I don't think its because of low RAM or something. Also not sure why staging and production will go down together. Is there any auto update set for server ?

WE checked and it seems Cloudwatch doesn't have memory metrics so would be good if we can get grafana access

[michaelmenon]

Hi @lukeoftheshire after you update the correct image(lamp-server:2021) to production we will update some fixes to production by tomorrow so that server and worker connects to NATS/redis correctly after restart.

Also can you check if some backup script or auto update is running on the server for any specific time ?

[lukeoftheshire]

Hi @Linoy339 @michaelmenon . This is very strange - the YAML in portainer definitely uses the 2021 tag but the image was still 'latest'. I updated the service and it appears to be back to the 2021 release. I will look into this and see if I can figure out why this has happened twice now.

There are several scripts that run on node-02 at specific times, but none that run on node-01 to my knowledge.

[avaidyam]

A docker image can have multiple tags. If a new production release was created, it will be tagged with both :2021.x.y and :latest. This doesn't seem to be the case here so comparing SHA might be helpful.

[Linoy339]

Thanks @lukeoftheshire . The production is now properly tagged. It seems the fixes in staging in working fine, as the redis and nats are connecting properly to lamp server as well as lamp worker. We shall deploy in production by tomorrow morning (IST) and check there.

[Linoy339]

@lukeoftheshire @avaidyam . We suspect of auto deployment process.

Today, We have deployed dashboard around 2 pm (IST) to production and its seen that all stack components (production and staging) got restarted.

It seems like dashboard deployment is causing all containers to restart and In certain or most of the times, the order of container start would be : 1)Lamp-server 2)Lamp-worker 3)Lamp-message_queue 4)Lamp-nats which leads to the scenario that server/worker would not connect to redis and nats for the first time which in turn force us to restart server/worker containers manually, so that it get connected to redis as well as nats properly. This issue is resolved now , as the hosts tries connecting untill nats/message_queue gets alive. Also, this is pushed to production.

But, it seems the auto restart of containers is because of dashboard deployment. Also, we have done LAMP-server deployment too in production around 3.01 pm (IST ) and this led to the restart of dashboard container too. Please see the timestamps for both containers(LAMP-server and LAMP-dashboard). Can you please go through the script or yaml or anything related with CI/CD workflow?

Also, FYI: We can see that the image, Lamp-worker:latest is getting pulled by both staging as well as production stack after a merge is completed. This should happen only in staging and the production should be updated if we forcefully pull the image