LAMP message queue,nats, worker, and server are stopping and starting frequently

[lukeoftheshire]

As mentioned in the title message queue, nats worker, and server services appear to be stopping and starting fairly frequently in portainer. @avaidyam and I are worried that there is a possibility of data loss due to these frequent stop-starts, so we are making this issue to keep track of it and log any information we find. The first thought we have is wondering if this could be related to automation with LAMP_worker in some way?

Please see the following image for an example of the frequently restarting services.

Relevant logs: Redis:

# all terminated redis containers have these exact same logs:

1:C 02 Nov 2021 18:28:35.924 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 02 Nov 2021 18:28:35.924 # Redis version=6.0.8, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 02 Nov 2021 18:28:35.924 # Configuration loaded
1:M 02 Nov 2021 18:28:35.925 * Running mode=standalone, port=6379.
1:M 02 Nov 2021 18:28:35.925 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 02 Nov 2021 18:28:35.925 # Server initialized
1:M 02 Nov 2021 18:28:35.925 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 02 Nov 2021 18:28:35.925 * Ready to accept connections
1:signal-handler (1635927627) Received SIGTERM scheduling shutdown...
1:M 03 Nov 2021 08:20:27.947 # User requested shutdown...
1:M 03 Nov 2021 08:20:27.947 # Redis is now ready to exit, bye bye...

Server

# server

2021-11-08T05:04:56.503471420Z GET /participant/U3497957254/sensor_event?origin=lamp.analytics&limit=1000 200 - 4.104 ms
2021-11-08T05:04:56.511630282Z GET /participant/U3142310106/sensor_event?origin=lamp.analytics&limit=1000 200 - 3.933 ms
2021-11-08T05:04:56.521692686Z GET /participant/U2049596189/sensor_event?origin=lamp.analytics&limit=1000 200 - 4.034 ms
2021-11-08T05:04:56.531045551Z GET /participant/U1660388766/sensor_event?origin=lamp.analytics&limit=1000 200 - 5.079 ms
2021-11-08T05:04:56.539532002Z GET /participant/U4112643738/sensor_event?origin=lamp.analytics&limit=1000 200 - 4.259 ms
2021-11-08T05:04:56.552536822Z GET /participant/U9232944603/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.793 ms
2021-11-08T05:04:56.562586592Z GET /participant/U8800226069/sensor_event?origin=lamp.analytics&limit=1000 200 - 4.053 ms
2021-11-08T05:04:56.570892444Z GET /participant/U3251451662/sensor_event?origin=lamp.analytics&limit=1000 200 - 4.037 ms
2021-11-08T05:04:56.579535933Z GET /participant/U5997877700/sensor_event?origin=lamp.analytics&limit=1000 200 - 4.350 ms
2021-11-08T05:04:56.586261352Z Fetching acTivity--
2021-11-08T05:04:56.593837144Z GET /study/z5fp592a0mt23j99mmyw/activity?ignore_binary=true 200 - 9.865 ms
2021-11-08T05:04:56.601750537Z Fetching acTivity--
2021-11-08T05:04:56.608943771Z GET /study/w7najqrzq4v0cy6yc3gq/activity?ignore_binary=true 200 - 9.643 ms
2021-11-08T05:04:56.619398203Z GET /researcher/mshs5a9p7bxkg4hhqfp8/study 200 - 6.072 ms
2021-11-08T05:04:56.625889364Z Fetching acTivity--
2021-11-08T05:04:56.649856552Z GET /study/h5r0e27ctd5sakv2xhxs/activity?ignore_binary=true 200 - 26.202 ms
2021-11-08T05:04:56.684045274Z GET /study/h5r0e27ctd5sakv2xhxs/participant 200 - 27.313 ms
2021-11-08T05:04:56.698394602Z GET /participant/U74398612/sensor_event?origin=lamp.analytics&limit=1000 200 - 9.563 ms
2021-11-08T05:04:56.714334556Z GET /participant/U3364247567/sensor_event?origin=lamp.analytics&limit=1000 200 - 11.559 ms
2021-11-08T05:04:56.740480661Z GET /participant/U650585221/sensor_event?origin=lamp.analytics&limit=1000 200 - 21.805 ms
2021-11-08T05:04:56.761366199Z GET /participant/U3301171773/sensor_event?origin=lamp.analytics&limit=1000 200 - 13.991 ms
2021-11-08T05:04:56.780415380Z GET /participant/U2873307322/sensor_event?origin=lamp.analytics&limit=1000 200 - 12.975 ms
2021-11-08T05:04:56.804275666Z GET /participant/U4009815427/sensor_event?origin=lamp.analytics&limit=1000 200 - 19.455 ms
2021-11-08T05:04:56.832072825Z GET /participant/U2216383209/sensor_event?origin=lamp.analytics&limit=1000 200 - 23.443 ms
2021-11-08T05:04:56.848695305Z GET /participant/U3185823493/sensor_event?origin=lamp.analytics&limit=1000 200 - 12.137 ms
2021-11-08T05:04:56.859142484Z GET /participant/U2220392219/sensor_event?origin=lamp.analytics&limit=1000 200 - 6.030 ms
2021-11-08T05:04:56.872265528Z GET /participant/U3388367831/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.954 ms
2021-11-08T05:04:56.888325736Z GET /participant/U545281964/sensor_event?origin=lamp.analytics&limit=1000 200 - 11.722 ms
2021-11-08T05:04:56.902187673Z GET /participant/U105240996/sensor_event?origin=lamp.analytics&limit=1000 200 - 9.551 ms
2021-11-08T05:04:56.912872742Z GET /participant/U3420262132/sensor_event?origin=lamp.analytics&limit=1000 200 - 6.541 ms
2021-11-08T05:04:56.927340951Z GET /participant/U4233010501/sensor_event?origin=lamp.analytics&limit=1000 200 - 9.467 ms
2021-11-08T05:04:56.943646410Z GET /participant/U1306386901/sensor_event?origin=lamp.analytics&limit=1000 200 - 12.179 ms
2021-11-08T05:04:56.956955610Z GET /participant/U15911961/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.787 ms
2021-11-08T05:04:56.970090892Z GET /participant/U1678656022/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.827 ms
2021-11-08T05:04:56.983647328Z GET /participant/U1750909798/sensor_event?origin=lamp.analytics&limit=1000 200 - 9.266 ms
2021-11-08T05:04:56.993945323Z GET /participant/U129474160/sensor_event?origin=lamp.analytics&limit=1000 200 - 5.847 ms
2021-11-08T05:04:57.005242295Z GET /participant/U331993895/sensor_event?origin=lamp.analytics&limit=1000 200 - 7.075 ms
2021-11-08T05:04:57.018090294Z GET /participant/U3979547651/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.658 ms
2021-11-08T05:04:57.030432608Z GET /participant/U4253955812/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.159 ms
2021-11-08T05:04:57.031434899Z POST /participant/U0816569523/sensor_event 200 - 3.659 ms
2021-11-08T05:04:57.032059577Z Nats server is disconnected
2021-11-08T05:04:57.032489483Z Nats server is disconnected
2021-11-08T05:04:57.040027103Z POST /participant/U0816569523/sensor_event 200 - 3.552 ms
2021-11-08T05:04:57.040840095Z Nats server is disconnected
2021-11-08T05:04:57.041941873Z Nats server is disconnected
2021-11-08T05:04:57.048342778Z GET /participant/U822393088/sensor_event?origin=lamp.analytics&limit=1000 200 - 11.411 ms
2021-11-08T05:04:57.065324378Z GET /participant/U2641237368/sensor_event?origin=lamp.analytics&limit=1000 200 - 11.119 ms
2021-11-08T05:04:57.081478357Z GET /participant/U1451232375/sensor_event?origin=lamp.analytics&limit=1000 200 - 11.782 ms
2021-11-08T05:04:57.091702918Z GET /participant/U3397872497/sensor_event?origin=lamp.analytics&limit=1000 200 - 5.906 ms
2021-11-08T05:04:57.105982723Z GET /participant/U1901851494/sensor_event?origin=lamp.analytics&limit=1000 200 - 10.000 ms
2021-11-08T05:04:57.118770209Z GET /participant/U504317083/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.181 ms
2021-11-08T05:04:57.134988419Z GET /participant/U1950095227/sensor_event?origin=lamp.analytics&limit=1000 200 - 11.675 ms
2021-11-08T05:04:57.149236698Z GET /participant/U1648287542/sensor_event?origin=lamp.analytics&limit=1000 200 - 9.407 ms
2021-11-08T05:04:57.167703236Z GET /participant/U2492145138/sensor_event?origin=lamp.analytics&limit=1000 200 - 13.332 ms
2021-11-08T05:04:57.184509076Z GET /participant/U2740606872/sensor_event?origin=lamp.analytics&limit=1000 200 - 12.080 ms
2021-11-08T05:04:57.198413964Z GET /participant/U4205883735/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.483 ms
2021-11-08T05:04:57.216266286Z GET /participant/U1335698569/sensor_event?origin=lamp.analytics&limit=1000 200 - 13.332 ms
2021-11-08T05:04:57.234041770Z GET /participant/U3206254569/sensor_event?origin=lamp.analytics&limit=1000 200 - 11.131 ms
2021-11-08T05:04:57.247126318Z GET /participant/U3070209781/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.634 ms
2021-11-08T05:04:57.267087119Z GET /participant/U3529444568/sensor_event?origin=lamp.analytics&limit=1000 200 - 15.669 ms
2021-11-08T05:04:57.281185670Z GET /participant/U3875207727/sensor_event?origin=lamp.analytics&limit=1000 200 - 9.314 ms
2021-11-08T05:04:57.293549405Z GET /participant/U2964907141/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.181 ms
2021-11-08T05:04:57.305740738Z GET /participant/U2582523240/sensor_event?origin=lamp.analytics&limit=1000 200 - 7.952 ms
2021-11-08T05:04:57.317446521Z GET /participant/U3333326031/sensor_event?origin=lamp.analytics&limit=1000 200 - 7.429 ms
2021-11-08T05:04:57.331604100Z GET /participant/U517481429/sensor_event?origin=lamp.analytics&limit=1000 200 - 9.691 ms
2021-11-08T05:04:57.344573384Z GET /participant/U2610873497/sensor_event?origin=lamp.analytics&limit=1000 200 - 7.924 ms
2021-11-08T05:04:57.357531011Z GET /participant/U2334901698/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.557 ms
2021-11-08T05:04:57.372274845Z GET /participant/U354752578/sensor_event?origin=lamp.analytics&limit=1000 200 - 10.274 ms
2021-11-08T05:04:57.384943777Z GET /participant/U3998756602/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.177 ms
2021-11-08T05:04:57.398471562Z GET /participant/U2258900661/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.960 ms
2021-11-08T05:04:57.412996843Z GET /participant/U2990317473/sensor_event?origin=lamp.analytics&limit=1000 200 - 9.994 ms
2021-11-08T05:04:57.427048408Z GET /participant/U2761310183/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.449 ms
2021-11-08T05:04:57.438229359Z GET /participant/U2853535444/sensor_event?origin=lamp.analytics&limit=1000 200 - 6.703 ms
2021-11-08T05:04:57.448964878Z GET /participant/U3984145087/sensor_event?origin=lamp.analytics&limit=1000 200 - 6.532 ms
2021-11-08T05:04:57.461148572Z GET /participant/U4079570273/sensor_event?origin=lamp.analytics&limit=1000 200 - 7.937 ms
2021-11-08T05:04:57.469604345Z GET /participant/U3189272257/sensor_event?origin=lamp.analytics&limit=1000 200 - 4.189 ms
2021-11-08T05:04:57.481927719Z GET /participant/U1827406012/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.127 ms
2021-11-08T05:04:57.494371266Z GET /participant/U3748700695/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.175 ms
2021-11-08T05:04:57.508723637Z GET /participant/U1602953801/sensor_event?origin=lamp.analytics&limit=1000 200 - 10.181 ms
2021-11-08T05:04:57.520349312Z GET /participant/U2405560385/sensor_event?origin=lamp.analytics&limit=1000 200 - 6.981 ms
2021-11-08T05:04:57.532523924Z GET /participant/U464628524/sensor_event?origin=lamp.analytics&limit=1000 200 - 7.907 ms
2021-11-08T05:04:57.544880739Z GET /participant/U3239194597/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.104 ms
2021-11-08T05:04:57.558916174Z GET /participant/U1416167928/sensor_event?origin=lamp.analytics&limit=1000 200 - 9.840 ms
2021-11-08T05:04:57.571106459Z GET /participant/U77672147/sensor_event?origin=lamp.analytics&limit=1000 200 - 8.152 ms
2021-11-08T05:04:58.645427041Z Nats server is disconnected
2021-11-08T05:04:58.645862061Z Nats server is disconnected
2021-11-08T05:04:58.646205554Z Nats server is disconnected
2021-11-08T05:04:58.713443618Z Nats server is disconnected
2021-11-08T05:04:58.713959748Z Nats server is disconnected
2021-11-08T05:04:58.714399694Z Nats server is disconnected
2021-11-08T05:05:03.181964221Z Nats server is disconnected
2021-11-08T05:05:03.182416824Z Nats server is disconnected
2021-11-08T05:05:03.209232576Z Nats server is disconnected
2021-11-08T05:05:03.209732929Z Nats server is disconnected
2021-11-08T05:05:03.210126941Z Nats server is disconnected
2021-11-08T05:05:06.467009915Z Nats server is disconnected
2021-11-08T05:05:06.467361186Z Nats server is disconnected
2021-11-08T05:05:06.489747157Z Nats server is disconnected
2021-11-08T05:05:06.490135232Z Nats server is disconnected
2021-11-08T05:05:06.490488506Z Nats server is disconnected

--- LAMP_server.1.l1ahx6xzr8pi289jdacyi08q1 TERMINATED ---

2021-11-08T08:18:57.368722789Z POST /participant/U4593963689/sensor_event 200 - 10.442 ms
2021-11-08T08:18:57.435567486Z POST /participant/U4593963689/sensor_event 200 - 6.228 ms
2021-11-08T08:18:57.540186261Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632609174293&limit=10000 200 - 343.159 ms
2021-11-08T08:18:57.540870468Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632843639441&limit=10000 200 - 352.157 ms
2021-11-08T08:18:57.542105095Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630221550164&limit=10000 200 - 380.624 ms
2021-11-08T08:18:57.913685741Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632607168648&limit=10000 200 - 371.970 ms
2021-11-08T08:18:57.952030276Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632841636401&limit=10000 200 - 383.155 ms
2021-11-08T08:18:57.986839879Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630219561160&limit=10000 200 - 413.656 ms
2021-11-08T08:18:58.357722005Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632605162949&limit=10000 200 - 366.920 ms
2021-11-08T08:18:58.357758015Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632839632896&limit=10000 200 - 360.930 ms
2021-11-08T08:18:58.392863631Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630217572233&limit=10000 200 - 382.833 ms
2021-11-08T08:18:58.773092867Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632837629908&limit=10000 200 - 344.720 ms
2021-11-08T08:18:58.773598426Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632603157770&limit=10000 200 - 372.952 ms
2021-11-08T08:18:58.832878854Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630215583217&limit=10000 200 - 405.918 ms
2021-11-08T08:18:59.165635792Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632835626968&limit=10000 200 - 349.689 ms
2021-11-08T08:18:59.166584328Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632601152586&limit=10000 200 - 366.906 ms
2021-11-08T08:18:59.254252613Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630213594340&limit=10000 200 - 395.020 ms
2021-11-08T08:18:59.426364996Z GET /participant/3m0k1a5fvh6kfvahchvh/sensor_event?origin=lamp.analytics 304 - 12.652 ms
2021-11-08T08:18:59.433115377Z GET /type/3m0k1a5fvh6kfvahchvh/attachment/lamp.messaging 404 - 17.342 ms
2021-11-08T08:18:59.538311595Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632833623780&limit=10000 200 - 336.293 ms
2021-11-08T08:18:59.543639997Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632599146483&limit=10000 200 - 346.791 ms
2021-11-08T08:18:59.626156165Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630211605426&limit=10000 200 - 335.940 ms
2021-11-08T08:18:59.975100475Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632831620076&limit=10000 200 - 373.037 ms
2021-11-08T08:18:59.975419032Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632597140325&limit=10000 200 - 404.447 ms
2021-11-08T08:19:00.022508668Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630209616539&limit=10000 200 - 369.469 ms
2021-11-08T08:19:00.038763906Z Fetching acTivity--
2021-11-08T08:19:00.045635802Z GET /activity/nkc57v55fcs8weam6zgk?ignore_binary=true 200 - 21.244 ms
2021-11-08T08:19:00.081474975Z GET /type/nkc57v55fcs8weam6zgk/parent 200 - 25.529 ms
2021-11-08T08:19:00.095705014Z GET /study/9n5tbtf3caf74gd4dwh4/participant 200 - 9.190 ms
2021-11-08T08:19:00.282646131Z GET /participant/U2201132710/sensor_event?origin=lamp.analytics&limit=1000 200 - 180.635 ms
2021-11-08T08:19:00.479258970Z GET /participant/U0176919012/sensor_event?origin=lamp.analytics&limit=1000 200 - 183.762 ms
2021-11-08T08:19:00.513618175Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630207627433&limit=10000 200 - 396.697 ms
2021-11-08T08:19:00.513716344Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632829616523&limit=10000 200 - 477.274 ms
2021-11-08T08:19:00.515102757Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632595135208&limit=10000 200 - 489.735 ms
2021-11-08T08:19:00.575905885Z GET /participant/U8640322252/sensor_event?origin=lamp.analytics&limit=1000 200 - 63.091 ms
2021-11-08T08:19:00.955080597Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630205638432&limit=10000 200 - 387.885 ms
2021-11-08T08:19:00.989403030Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632593129745&limit=10000 200 - 414.626 ms
2021-11-08T08:19:00.989582494Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632827612198&limit=10000 200 - 449.089 ms
2021-11-08T08:19:01.365610283Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632825609342&limit=10000 200 - 343.364 ms
2021-11-08T08:19:01.432458974Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632591120196&limit=10000 200 - 382.789 ms
2021-11-08T08:19:01.434845136Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630203649527&limit=10000 200 - 442.526 ms
2021-11-08T08:19:01.625052595Z POST /participant/U0176919012/sensor_event 200 - 7.711 ms
2021-11-08T08:19:01.852395777Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632823606449&limit=10000 200 - 409.731 ms
2021-11-08T08:19:01.918113156Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632589112497&limit=10000 200 - 428.030 ms
2021-11-08T08:19:01.918146748Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630201660649&limit=10000 200 - 457.728 ms
2021-11-08T08:19:02.285123959Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632821603332&limit=10000 200 - 361.442 ms
2021-11-08T08:19:02.373726520Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630199671859&limit=10000 200 - 421.111 ms
2021-11-08T08:19:02.672070846Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632819600102&limit=10000 200 - 353.731 ms
2021-11-08T08:19:02.781829121Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630197682949&limit=10000 200 - 381.295 ms
2021-11-08T08:19:03.075647476Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632817596706&limit=10000 200 - 371.653 ms
2021-11-08T08:19:03.194585568Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630195694015&limit=10000 200 - 379.839 ms
2021-11-08T08:19:03.409473506Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632587105235&limit=10000 200 - 371.023 ms
2021-11-08T08:19:03.466046164Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632815593777&limit=10000 200 - 360.029 ms
2021-11-08T08:19:03.550642299Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630193705055&limit=10000 200 - 331.119 ms
2021-11-08T08:19:04.492870166Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630191716116&limit=10000 200 - 351.905 ms
2021-11-08T08:19:04.494797985Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632585098142&limit=10000 200 - 368.662 ms
2021-11-08T08:20:04.746403899Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632813590920&limit=10000 200 - 309.668 ms
2021-11-08T08:20:04.829542118Z GET /participant/yhe0wtfn6n6sbvsan0js/sensor_event?origin=lamp.analytics 304 - 7.373 ms
2021-11-08T08:20:04.830206121Z GET /participant/z5ys6yv5w1rj31zq3w4r/sensor_event?origin=lamp.analytics 304 - 8.681 ms
2021-11-08T08:20:04.831931154Z GET /type/z5ys6yv5w1rj31zq3w4r/attachment/lamp.messaging 404 - 19.810 ms
2021-11-08T08:20:04.832418472Z GET /type/yhe0wtfn6n6sbvsan0js/attachment/lamp.messaging 404 - 19.389 ms
2021-11-08T08:20:05.018253825Z POST /participant/U4233494401/sensor_event 200 - 12.749 ms
2021-11-08T08:20:05.030053832Z POST /participant/U4233494401/sensor_event 200 - 13.366 ms
2021-11-08T08:20:05.068959944Z POST /participant/U6368562055/sensor_event 200 - 25.038 ms
2021-11-08T08:20:05.080558442Z POST /participant/U6368562055/sensor_event 200 - 26.957 ms
2021-11-08T08:20:05.184530592Z GET /participant/U7919569091/sensor_event?origin=lamp.accelerometer&from=0&to=1630189727213&limit=10000 200 - 402.685 ms
2021-11-08T08:20:05.186184426Z POST /participant/U1912527657/sensor_event 200 - 41.802 ms
2021-11-08T08:20:05.216720351Z POST /participant/U1912527657/sensor_event 200 - 34.113 ms
2021-11-08T08:20:05.225903323Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632583089722&limit=10000 200 - 446.972 ms
2021-11-08T08:20:05.227514798Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632811587941&limit=10000 200 - 428.699 ms
2021-11-08T08:20:05.447958800Z POST /participant/U6299504587/sensor_event 200 - 7.120 ms
2021-11-08T08:20:05.449065029Z POST /participant/U6299504587/sensor_event 200 - 5.849 ms
2021-11-08T08:20:05.510704159Z POST /participant/U7645962928/sensor_event 200 - 5.783 ms
2021-11-08T08:20:05.512100452Z POST /participant/U7645962928/sensor_event 200 - 4.248 ms
2021-11-08T08:20:05.580512321Z GET /participant/U2600416933/sensor_event?origin=lamp.accelerometer&from=0&to=1632809585153&limit=10000 200 - 317.288 ms
2021-11-08T08:20:05.615058735Z GET /participant/U1760561177/sensor_event?origin=lamp.accelerometer&from=0&to=1632581082434&limit=10000 200 - 346.063 ms
2021-11-08T08:20:09.561384497Z [ioredis] Unhandled error event: Error: connect ECONNREFUSED 10.0.6.7:6379
2021-11-08T08:20:09.561412633Z     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1146:16)
2021-11-08T08:20:09.662477824Z [ioredis] Unhandled error event: Error: connect ECONNREFUSED 10.0.6.7:6379
2021-11-08T08:20:09.662501396Z     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1146:16)
2021-11-08T08:20:09.813758255Z [ioredis] Unhandled error event: Error: connect ECONNREFUSED 10.0.6.7:6379
2021-11-08T08:20:09.813770963Z     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1146:16)
2021-11-08T08:20:10.030331082Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-08T08:20:10.030382935Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-08T08:20:10.281118022Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-08T08:20:10.281135429Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-08T08:20:10.582255312Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-08T08:20:10.582270980Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-08T08:20:10.933942375Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-08T08:20:10.933963077Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-08T08:20:11.334202438Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-08T08:20:11.334219963Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-08T08:20:11.785596098Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-08T08:20:11.785624180Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-08T08:20:12.287151670Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-08T08:20:12.287176901Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-08T08:20:12.838690477Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-08T08:20:12.838716338Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-08T08:20:13.440402315Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-08T08:20:13.440417426Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)

--- LAMP_server.1.gpqb989olgtbfbesr4o9tjcpv TERMINATED ---

2021-11-08T11:20:17.554507102Z Nats server is disconnected
2021-11-08T11:20:17.554995623Z Nats server is disconnected
2021-11-08T11:20:17.555364098Z Nats server is disconnected
2021-11-08T11:20:19.204530182Z GET /participant/U8479747510/sensor_event?origin=lamp.analytics 304 - 4.665 ms
2021-11-08T11:20:19.225886863Z GET /type/U8479747510/attachment/lamp.messaging 404 - 19.356 ms
2021-11-08T11:20:22.791111753Z POST /participant/U2201583859/sensor_event 200 - 4.072 ms
2021-11-08T11:20:22.791615676Z Nats server is disconnected
2021-11-08T11:20:22.791990496Z Nats server is disconnected
2021-11-08T11:20:22.792319805Z Nats server is disconnected
2021-11-08T11:20:22.793136469Z POST /participant/U2201583859/sensor_event 200 - 3.407 ms
2021-11-08T11:20:22.793477166Z Nats server is disconnected
2021-11-08T11:20:22.793929254Z Nats server is disconnected
2021-11-08T11:20:22.794380651Z Nats server is disconnected
2021-11-08T11:20:22.794818020Z Nats server is disconnected
2021-11-08T11:20:22.795288014Z Nats server is disconnected
2021-11-08T11:20:23.275389670Z GET /participant/0vayybdfqxfpv74acj5b/sensor_event?origin=lamp.analytics 304 - 5.526 ms
2021-11-08T11:20:23.282991501Z GET /type/0vayybdfqxfpv74acj5b/attachment/lamp.messaging 404 - 10.300 ms
2021-11-08T11:20:34.277361545Z GET /participant/0vayybdfqxfpv74acj5b/sensor_event?origin=lamp.analytics 304 - 5.974 ms
2021-11-08T11:20:34.283982184Z GET /type/0vayybdfqxfpv74acj5b/attachment/lamp.messaging 404 - 11.392 ms
2021-11-08T11:20:37.701980860Z POST /participant/U6529862821/sensor_event 200 - 8.087 ms
2021-11-08T11:20:37.702908014Z POST /participant/U6529862821/sensor_event 200 - 3.414 ms
2021-11-08T11:20:37.703654365Z Nats server is disconnected
2021-11-08T11:20:37.704210119Z Nats server is disconnected
2021-11-08T11:20:37.704721303Z Nats server is disconnected
2021-11-08T11:20:37.705212131Z Nats server is disconnected
2021-11-08T11:20:37.705741400Z Nats server is disconnected
2021-11-08T11:20:37.706229034Z Nats server is disconnected
2021-11-08T11:20:37.706705460Z Nats server is disconnected
2021-11-08T11:20:37.707191412Z Nats server is disconnected
2021-11-08T11:20:42.800129073Z POST /participant/U5688551159/sensor_event 200 - 3.502 ms
2021-11-08T11:20:42.801478553Z Nats server is disconnected
2021-11-08T11:20:42.801905113Z Nats server is disconnected
2021-11-08T11:20:42.802251197Z Nats server is disconnected
2021-11-08T11:20:42.802583209Z Nats server is disconnected
2021-11-08T11:20:42.895156572Z POST /participant/U5688551159/sensor_event 200 - 3.545 ms
2021-11-08T11:20:42.895832615Z Nats server is disconnected
2021-11-08T11:20:42.896297549Z Nats server is disconnected
2021-11-08T11:20:42.896737186Z Nats server is disconnected
2021-11-08T11:20:42.897170176Z Nats server is disconnected
2021-11-08T11:20:42.954618856Z Store_Size 32335
2021-11-08T11:20:44.034413696Z write started timestamp 290-1636370444034
2021-11-08T11:20:47.497123323Z Initializing LAMP API server...
2021-11-08T11:20:47.529026691Z   Connected to Nats Pub Server
2021-11-08T11:20:47.529205582Z   Connected to Nats Sub Server
2021-11-08T11:20:47.613631976Z   Initializing database connection...
2021-11-08T11:20:47.614146247Z     MONGODB adapter in use
2021-11-08T11:20:47.640568877Z       ActivitySpec database online.
2021-11-08T11:20:47.640584283Z       SensorSpec database online.
2021-11-08T11:20:47.640587581Z       Researcher database online.
2021-11-08T11:20:47.640591166Z       Study database online.
2021-11-08T11:20:47.640594416Z       Participant database online.
2021-11-08T11:20:47.640624164Z       Activity database online.
2021-11-08T11:20:47.640640885Z       Sensor database online.
2021-11-08T11:20:47.640643293Z       ActivityEvent database online.
2021-11-08T11:20:47.640650675Z       SensorEvent database online.
2021-11-08T11:20:47.640652728Z       Tag database online.
2021-11-08T11:20:47.640673104Z       Credential database online.
2021-11-08T11:20:47.640753818Z   Database verification complete.
2021-11-08T11:20:47.640803472Z   Server routing initialized.
2021-11-08T11:20:47.640808114Z Initialization complete.
2021-11-08T11:20:47.643394788Z undefined
2021-11-08T11:20:51.756229078Z GET /participant/m6rrb59rpsncgxqkz5ft/sensor_event?origin=lamp.analytics 304 - 16.532 ms
2021-11-08T11:20:51.771231971Z GET /type/m6rrb59rpsncgxqkz5ft/attachment/lamp.messaging 404 - 23.310 ms
2021-11-08T11:20:55.634176540Z GET /participant/U9610582616/sensor_event?origin=lamp.analytics 304 - 8.002 ms
2021-11-08T11:20:55.655236061Z GET /participant/yhe0wtfn6n6sbvsan0js/sensor_event?origin=lamp.analytics 304 - 10.025 ms
2021-11-08T11:20:55.658382885Z GET /participant/z5ys6yv5w1rj31zq3w4r/sensor_event?origin=lamp.analytics 304 - 12.143 ms
2021-11-08T11:20:55.660516030Z GET /type/yhe0wtfn6n6sbvsan0js/attachment/lamp.messaging 404 - 24.880 ms
2021-11-08T11:20:55.669945235Z GET /type/z5ys6yv5w1rj31zq3w4r/attachment/lamp.messaging 404 - 33.359 ms
2021-11-08T11:20:55.670863392Z GET /type/U9610582616/attachment/lamp.messaging 304 - 46.294 ms
2021-11-08T11:20:56.391277993Z Fetching acTivity--
2021-11-08T11:20:56.406941754Z GET /participant/U7931361682/sensor 304 - 20.122 ms
2021-11-08T11:20:56.423732427Z GET /participant/U7931361682/activity?ignore_binary=true 304 - 38.762 ms
2021-11-08T11:20:57.706956407Z POST /participant/U1954110644/sensor_event 200 - 5.195 ms
2021-11-08T11:20:58.846583791Z POST /participant/U1954110644/sensor_event 200 - 3.744 ms
2021-11-08T11:20:58.915556172Z POST /participant/U1954110644/sensor_event 200 - 5.541 ms
2021-11-08T11:21:40.021408592Z Store_Size 30918
2021-11-08T11:21:41.164443179Z write started timestamp 291-1636370501164
2021-11-08T11:21:44.708716021Z write finished timestamp 291-1636370504708
2021-11-08T11:22:06.282137626Z Store_Size 31144
2021-11-08T11:22:09.732845844Z Store_Size 31107
2021-11-08T11:22:10.806241866Z write started timestamp 293-1636370530806
2021-11-08T11:22:13.679592291Z write started timestamp 294-1636370533679
2021-11-08T11:22:14.164672020Z write finished timestamp 293-1636370534164
2021-11-08T11:22:17.105497485Z write finished timestamp 294-1636370537105
2021-11-08T11:22:19.174452759Z write started timestamp 296-1636370539174
2021-11-08T11:22:20.588959790Z Store_Size 31203
2021-11-08T11:22:23.399460725Z write finished timestamp 296-1636370543399
2021-11-08T11:22:23.642292978Z Store_Size 31527
2021-11-08T11:22:26.116024095Z Store_Size 31009
2021-11-08T11:22:27.184332947Z write started timestamp 300-1636370547184
2021-11-08T11:22:31.705768959Z Store_Size 31199
2021-11-08T11:22:31.772244089Z write finished timestamp 300-1636370551772
2021-11-08T11:22:56.908945181Z Store_Size 30474
2021-11-08T11:23:44.518238483Z write started timestamp 304-1636370624518
2021-11-08T11:23:47.937927382Z write finished timestamp 304-1636370627937
2021-11-08T11:24:00.346086324Z Store_Size 30102
2021-11-08T11:24:16.097066186Z Store_Size 33626
2021-11-08T11:24:17.194932720Z write started timestamp 306-1636370657194
2021-11-08T11:24:20.501904161Z Store_Size 30672
2021-11-08T11:24:20.663096236Z write finished timestamp 306-1636370660662

--- LAMP_server.1.kx99bjpgmh422l1hpxoxrd4yb TERMINATED ---

2021-11-10T09:18:29.969134093Z GET /participant/U1699389494/sensor_event?origin=lamp.accelerometer&from=1635912000000&to=1636167002817&limit=10000 200 - 3698.791 ms
2021-11-10T09:18:30.225862112Z GET /participant/U8282161095/sensor_event?origin=lamp.accelerometer&from=0&to=1629977289423&limit=10000 200 - 380.475 ms
2021-11-10T09:19:58.096461455Z POST /participant/U9158417286/sensor_event 200 - 19.108 ms
2021-11-10T09:19:58.098314082Z POST /participant/U9158417286/sensor_event 200 - 17.039 ms
2021-11-10T09:19:58.102399596Z POST /participant/U3446920050/sensor_event 200 - 13.688 ms
2021-11-10T09:19:58.103211453Z GET /participant/0vayybdfqxfpv74acj5b/sensor_event?origin=lamp.analytics 304 - 35.695 ms
2021-11-10T09:19:58.106103906Z POST /participant/U3446920050/sensor_event 200 - 15.033 ms
2021-11-10T09:19:58.106798981Z POST /participant/U4097925880/sensor_event 200 - 23.298 ms
2021-11-10T09:19:58.107191619Z GET /participant/0vayybdfqxfpv74acj5b/sensor_event?origin=lamp.analytics 304 - 36.826 ms
2021-11-10T09:19:58.108165607Z POST /participant/U4097925880/sensor_event 200 - 22.216 ms
2021-11-10T09:19:58.124651849Z GET /participant/U2762632331/sensor_event?origin=lamp.analytics 304 - 23.639 ms
2021-11-10T09:19:58.136961717Z GET /participant/yhe0wtfn6n6sbvsan0js/sensor_event?origin=lamp.analytics 304 - 36.209 ms
2021-11-10T09:19:58.149952303Z POST /participant/U8939345176/sensor_event 200 - 14.861 ms
2021-11-10T09:19:58.151526492Z GET /type/0vayybdfqxfpv74acj5b/attachment/lamp.messaging 404 - 87.913 ms
2021-11-10T09:19:58.168018946Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 104.858 ms
2021-11-10T09:19:58.169384236Z GET /type/0vayybdfqxfpv74acj5b/attachment/lamp.messaging 404 - 102.388 ms
2021-11-10T09:19:58.169719141Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 107.449 ms
2021-11-10T09:19:58.171873263Z GET /type/yhe0wtfn6n6sbvsan0js/attachment/lamp.messaging 404 - 71.408 ms
2021-11-10T09:19:58.244656193Z GET /type/U2762632331/attachment/lamp.messaging 304 - 144.679 ms
2021-11-10T09:19:58.321700150Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 106.394 ms
2021-11-10T09:19:58.322306359Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 107.670 ms
2021-11-10T09:19:58.373734513Z POST /participant/U6299504587/sensor_event 200 - 56.645 ms
2021-11-10T09:19:58.379544752Z GET /participant/U4179451929/sensor_event?origin=lamp.accelerometer&from=1635912000000&to=1635933001750&limit=10000 200 - 240.489 ms
2021-11-10T09:19:58.387041951Z GET /participant/U1699389494/sensor_event?origin=lamp.accelerometer&from=1635912000000&to=1636165000707&limit=10000 200 - 300.201 ms
2021-11-10T09:19:58.420610175Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 39.910 ms
2021-11-10T09:19:58.421530595Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 40.325 ms
2021-11-10T09:19:58.425922957Z POST /participant/U6299504587/sensor_event 200 - 7.383 ms
2021-11-10T09:19:58.453318235Z GET /participant/0vayybdfqxfpv74acj5b/sensor_event?origin=lamp.analytics 304 - 14.514 ms
2021-11-10T09:19:58.486397093Z GET /type/0vayybdfqxfpv74acj5b/attachment/lamp.messaging 404 - 47.088 ms
2021-11-10T09:19:58.526095632Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 49.447 ms
2021-11-10T09:19:58.531582727Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 54.478 ms
2021-11-10T09:19:58.595859654Z GET /participant/U4179451929/sensor_event?origin=lamp.accelerometer&from=1635912000000&to=1635931999599&limit=10000 200 - 184.535 ms
2021-11-10T09:19:58.613451730Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 21.226 ms
2021-11-10T09:19:58.614573382Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 21.795 ms
2021-11-10T09:19:58.658528349Z POST /participant/U9444966227/sensor_event 200 - 15.028 ms
2021-11-10T09:19:58.733321124Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 76.626 ms
2021-11-10T09:19:58.746419984Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 89.010 ms
2021-11-10T09:19:58.801379111Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 29.475 ms
2021-11-10T09:19:58.820364235Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 39.848 ms
2021-11-10T09:19:58.876896539Z GET /participant/U4179451929/sensor_event?origin=lamp.accelerometer&from=1635912000000&to=1635930997508&limit=10000 200 - 241.493 ms
2021-11-10T09:19:58.884437868Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 19.790 ms
2021-11-10T09:19:58.885028084Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 19.823 ms
2021-11-10T09:19:58.945178470Z POST /participant/U8939345176/sensor_event 200 - 7.271 ms
2021-11-10T09:19:58.974748594Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 52.047 ms
2021-11-10T09:19:58.981869799Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 59.674 ms
2021-11-10T09:19:59.054790136Z POST /participant/U0128440310/sensor_event 200 - 8.778 ms
2021-11-10T09:19:59.102661039Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 84.785 ms
2021-11-10T09:19:59.103011875Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 82.650 ms
2021-11-10T09:19:59.105322630Z GET /participant/U4179451929/sensor_event?origin=lamp.accelerometer&from=1635912000000&to=1635929995252&limit=10000 200 - 199.557 ms
2021-11-10T09:19:59.109062594Z POST /participant/U8939345176/sensor_event 200 - 5.016 ms
2021-11-10T09:19:59.150401949Z Store_Size 36695
2021-11-10T09:19:59.178101163Z POST /participant/U8939345176/sensor_event 200 - 18.150 ms
2021-11-10T09:19:59.263376312Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 123.213 ms
2021-11-10T09:19:59.276299502Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 135.022 ms
2021-11-10T09:19:59.367048181Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 68.092 ms
2021-11-10T09:19:59.372659984Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 54.279 ms
2021-11-10T09:19:59.375022906Z GET /participant/U4179451929/sensor_event?origin=lamp.accelerometer&from=1635912000000&to=1635928993106&limit=10000 200 - 235.433 ms
2021-11-10T09:19:59.431983661Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 22.675 ms
2021-11-10T09:19:59.432313296Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 29.204 ms
2021-11-10T09:19:59.507146772Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 33.023 ms
2021-11-10T09:19:59.515583873Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 42.014 ms
2021-11-10T09:19:59.618379803Z GET /participant/U4179451929/sensor_event?origin=lamp.accelerometer&from=1635912000000&to=1635927991020&limit=10000 200 - 213.691 ms
2021-11-10T09:19:59.619385475Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 57.423 ms
2021-11-10T09:19:59.620343829Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 48.965 ms
2021-11-10T09:19:59.722556863Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 43.972 ms
2021-11-10T09:19:59.736993795Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 50.481 ms
2021-11-10T09:19:59.822275115Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 31.944 ms
2021-11-10T09:19:59.826834482Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 37.606 ms
2021-11-10T09:19:59.885633658Z GET /participant/U4179451929/sensor_event?origin=lamp.accelerometer&from=1635912000000&to=1635926687962&limit=10000 200 - 235.021 ms
2021-11-10T09:19:59.897532080Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 18.661 ms
2021-11-10T09:19:59.898116192Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 19.764 ms
2021-11-10T09:19:59.977095588Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 41.910 ms
2021-11-10T09:19:59.978859499Z GET /participant/U0009146886/sensor_event?origin=lamp.accelerometer&from=1609362000000.0&to=1609362000000 200 - 43.204 ms
2021-11-10T09:20:00.098234862Z GET /participant/U4179451929/sensor_event?origin=lamp.accelerometer&from=1635912000000&to=1635925685883&limit=10000 200 - 188.619 ms
2021-11-10T09:20:00.190224521Z POST /participant/U9444966227/sensor_event 200 - 4.349 ms
2021-11-10T09:20:00.280890491Z write started timestamp 5306-1636536000280
2021-11-10T09:20:00.786762059Z POST /participant/U0128440310/sensor_event 200 - 4.967 ms
2021-11-10T09:20:01.052626009Z Store_Size 30071
2021-11-10T09:20:01.842924388Z GET /participant/U1699389494/sensor_event?origin=lamp.accelerometer&from=1635912000000&to=1636162998474&limit=10000 200 - 3423.226 ms
2021-11-10T09:20:02.119467082Z write started timestamp 5307-1636536002119
2021-11-10T09:20:05.372850603Z write finished timestamp 5306-1636536005372
2021-11-10T09:20:06.055643516Z [ioredis] Unhandled error event: Error: connect ECONNREFUSED 10.0.6.7:6379
2021-11-10T09:20:06.055669893Z     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1146:16)
2021-11-10T09:20:06.157284597Z [ioredis] Unhandled error event: Error: connect ECONNREFUSED 10.0.6.7:6379
2021-11-10T09:20:06.157308629Z     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1146:16)
2021-11-10T09:20:06.308355508Z [ioredis] Unhandled error event: Error: connect ECONNREFUSED 10.0.6.7:6379
2021-11-10T09:20:06.308378351Z     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1146:16)
2021-11-10T09:20:06.509463304Z [ioredis] Unhandled error event: Error: connect ECONNREFUSED 10.0.6.7:6379
2021-11-10T09:20:06.509484996Z     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1146:16)
2021-11-10T09:20:06.761649449Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-10T09:20:06.761671807Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-10T09:20:07.062383926Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-10T09:20:07.062408431Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-10T09:20:07.271692403Z write finished timestamp 5307-1636536007271
2021-11-10T09:20:07.413800003Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-10T09:20:07.413821711Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-10T09:20:07.815357412Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-10T09:20:07.815376732Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)
2021-11-10T09:20:08.266897525Z [ioredis] Unhandled error event: Error: getaddrinfo ENOTFOUND message_queue
2021-11-10T09:20:08.266914169Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26)

--- LAMP_server.1.drlwcrmt19r0adf6swzd0bhha TERMINATED ---

Worker

2021-11-06T07:24:36.202196895Z initialize queue
2021-11-06T07:24:36.215008861Z Initialized the queues
2021-11-06T07:24:36.216768438Z Clean all queues...
2021-11-06T07:24:36.217041440Z CLEANING ALL QUEUE
2021-11-06T07:24:36.296416766Z DONE--CLEANING ALL QUEUE
2021-11-06T07:24:36.296438878Z Initializing schedulers...
2021-11-06T07:24:36.311950283Z Running with automation disabled.
2021-11-06T07:24:36.313312103Z server listening in 3000
2021-11-06T07:24:36.313420394Z Initiating nats server
2021-11-06T07:24:36.321638606Z error---while nats connect Error: getaddrinfo ENOTFOUND nats
2021-11-06T07:24:36.321668691Z     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:71:26) {
2021-11-06T07:24:36.321671901Z   errno: -3008,
2021-11-06T07:24:36.321674320Z   code: 'ENOTFOUND',
2021-11-06T07:24:36.321676540Z   syscall: 'getaddrinfo',
2021-11-06T07:24:36.321678537Z   hostname: 'nats'
2021-11-06T07:24:36.321680439Z }
2021-11-06T07:24:37.330102918Z error while scheduling initially--- FetchError: request to https://api.lamp.digital/researcher failed, reason: connect ECONNREFUSED 10.0.0.2:443
2021-11-06T07:24:37.330128300Z     at ClientRequest.<anonymous> (/usr/src/app/node_modules/node-fetch/lib/index.js:1461:11)
2021-11-06T07:24:37.330131635Z     at ClientRequest.emit (node:events:394:28)
2021-11-06T07:24:37.330133585Z     at TLSSocket.socketErrorListener (node:_http_client:447:9)
2021-11-06T07:24:37.330135586Z     at TLSSocket.emit (node:events:394:28)
2021-11-06T07:24:37.330137485Z     at emitErrorNT (node:internal/streams/destroy:157:8)
2021-11-06T07:24:37.330139471Z     at emitErrorCloseNT (node:internal/streams/destroy:122:3)
2021-11-06T07:24:37.330141280Z     at processTicksAndRejections (node:internal/process/task_queues:83:21) {
2021-11-06T07:24:37.330143323Z   type: 'system',
2021-11-06T07:24:37.330145756Z   errno: 'ECONNREFUSED',
2021-11-06T07:24:37.330149066Z   code: 'ECONNREFUSED'
2021-11-06T07:24:37.330151713Z }

--- LAMP_worker.1.euh0cxgo7k7pkux7f8fcd1nhy TERMINATED ---

2021-11-08T08:19:00.005522787Z Preparing to fetch activities
2021-11-08T08:19:00.046734113Z activity_id given nkc57v55fcs8weam6zgk
2021-11-08T08:19:00.046754928Z Saving to redis
2021-11-08T08:19:00.046757459Z Processing 1 activities for push notifications.
2021-11-08T08:19:00.046759478Z locked job on activity_scheduler
2021-11-08T08:19:00.582880094Z start_date======== 2021-02-26T12:19:00.000Z
2021-11-08T08:19:00.584576875Z Saving to Redis completed....
2021-11-08T08:19:00.584598475Z release lock  on success  activity_scheduler
2021-11-08T08:19:00.584602000Z Rescheduled job after notificcation process-  nkc57v55fcs8weam6zgk
2021-11-08T08:19:00.584605722Z release lock  on success  nkc57v55fcs8weam6zgk
2021-11-08T08:19:01.629986160Z participant_id listened U0176919012
2021-11-08T08:19:01.630017903Z sensordata listened {
2021-11-08T08:19:01.630022393Z   action: 'notification',
2021-11-08T08:19:01.630025274Z   user_action: 'Open App',
2021-11-08T08:19:01.630028439Z   payload: '{expiry=21600000, actions=[{"name":"Open App","page":"\\/participant\\/U0176919012\\/activity\\/nkc57v55fcs8weam6zgk"}], page=/participant/U0176919012/activity/nkc57v55fcs8weam6zgk, title=Baloon, notificationId=519628, message=You have a mindLAMP activity waiting for you: Baloon.}'
2021-11-08T08:19:01.630033493Z }
2021-11-08T08:20:00.002835202Z today_date while notify 2021-11-08T08:20:00.000Z
2021-11-08T08:20:00.002861522Z start_date while notify 2021-05-19T15:20:00.000Z
2021-11-08T08:20:00.002864169Z {
2021-11-08T08:20:00.002866362Z   device_token: 'REDACTED',
2021-11-08T08:20:00.002868674Z   device_type: 'android',
2021-11-08T08:20:00.002870581Z   payload: {
2021-11-08T08:20:00.002872572Z     participant_id: 'U0176919012',
2021-11-08T08:20:00.002874477Z     activity_id: '39m9gp4da1r20fsdvqjz',
2021-11-08T08:20:00.002876942Z     message: 'You have a mindLAMP activity waiting for you: b1.',
2021-11-08T08:20:00.002880263Z     title: 'b1',
2021-11-08T08:20:00.002883252Z     url: '/participant/U0176919012/activity/39m9gp4da1r20fsdvqjz',
2021-11-08T08:20:00.002886399Z     notificationId: 906379
2021-11-08T08:20:00.002888943Z   }
2021-11-08T08:20:00.002891712Z }
2021-11-08T08:20:00.002894293Z /participant/U0176919012/activity/39m9gp4da1r20fsdvqjz
2021-11-08T08:20:00.004613263Z Completed  job state on 39m9gp4da1r20fsdvqjz
2021-11-08T08:20:00.004657287Z locked job on 39m9gp4da1r20fsdvqjz
2021-11-08T08:20:00.004671092Z jobs in queue
2021-11-08T08:20:00.004674260Z Preparing to fetch activities
2021-11-08T08:20:00.009562860Z release lock  on exception2  39m9gp4da1r20fsdvqjz
2021-11-08T08:20:05.048041434Z participant_id listened U4233494401
2021-11-08T08:20:05.048068154Z sensordata listened {
2021-11-08T08:20:05.048070584Z   latitude: REDACTED,
2021-11-08T08:20:05.048072642Z   longitude: REDACTED,
2021-11-08T08:20:05.048074580Z   accuracy: 4.775681811813515,
2021-11-08T08:20:05.048076489Z   altitude: 312.71183001063764
2021-11-08T08:20:05.048078438Z }
2021-11-08T08:20:05.214713334Z participant_id listened U4233494401
2021-11-08T08:20:05.214736454Z sensordata listened {
2021-11-08T08:20:05.214738868Z   latitude: REDACTED,
2021-11-08T08:20:05.214741266Z   longitude: REDACTED,
2021-11-08T08:20:05.214743187Z   accuracy: 4.786619216730543,
2021-11-08T08:20:05.214745111Z   altitude: 312.71183001063764
2021-11-08T08:20:05.214747091Z }
2021-11-08T08:20:05.222477535Z participant_id listened U6368562055
2021-11-08T08:20:05.222497302Z sensordata listened {
2021-11-08T08:20:05.222499336Z   latitude: REDACTED,
2021-11-08T08:20:05.222501096Z   longitude: REDACTED,
2021-11-08T08:20:05.222503108Z   accuracy: 14.262297327788108,
2021-11-08T08:20:05.222505046Z   altitude: 230.2040985794738
2021-11-08T08:20:05.222506922Z }
2021-11-08T08:20:05.228750004Z participant_id listened U6368562055
2021-11-08T08:20:05.228761361Z sensordata listened {
2021-11-08T08:20:05.228763364Z   latitude: REDACTED,
2021-11-08T08:20:05.228765676Z   longitude: REDACTED,
2021-11-08T08:20:05.228767531Z   accuracy: 14.262618541848493,
2021-11-08T08:20:05.228769346Z   altitude: 230.2040985794738
2021-11-08T08:20:05.228771275Z }
2021-11-08T08:20:05.231691378Z participant_id listened U1912527657
2021-11-08T08:20:05.231709450Z sensordata listened {
2021-11-08T08:20:05.231712015Z   latitude: REDACTED,
2021-11-08T08:20:05.231714235Z   longitude: REDACTED,
2021-11-08T08:20:05.231716164Z   accuracy: 14.246829593421227,
2021-11-08T08:20:05.231718100Z   altitude: 7.289181161671877
2021-11-08T08:20:05.231720233Z }
2021-11-08T08:20:05.234595637Z participant_id listened U1912527657
2021-11-08T08:20:05.234608192Z sensordata listened {
2021-11-08T08:20:05.234611238Z   latitude: REDACTED,
2021-11-08T08:20:05.234613357Z   longitude: REDACTED,
2021-11-08T08:20:05.234615358Z   accuracy: 14.247632504082615,
2021-11-08T08:20:05.234617276Z   altitude: 7.289181161671877
2021-11-08T08:20:05.234619080Z }
2021-11-08T08:20:05.450339201Z participant_id listened U6299504587
2021-11-08T08:20:05.450363146Z sensordata listened { x: 0.06829833984375, y: -0.0142364501953125, z: -1.0013427734375 }
2021-11-08T08:20:05.453017363Z participant_id listened U6299504587
2021-11-08T08:20:05.453033604Z sensordata listened {
2021-11-08T08:20:05.453036186Z   x: 0.0678558349609375,
2021-11-08T08:20:05.453038134Z   y: -0.0136260986328125,
2021-11-08T08:20:05.453039975Z   z: -1.0009307861328125
2021-11-08T08:20:05.453041820Z }
2021-11-08T08:20:05.517296003Z participant_id listened U7645962928
2021-11-08T08:20:05.517318942Z sensordata listened {
2021-11-08T08:20:05.517322966Z   latitude: REDACTED,
2021-11-08T08:20:05.517326298Z   longitude: REDACTED,
2021-11-08T08:20:05.517329676Z   accuracy: 10,
2021-11-08T08:20:05.517332783Z   altitude: 19.375179290771484
2021-11-08T08:20:05.517336345Z }
2021-11-08T08:20:05.524529373Z participant_id listened U7645962928
2021-11-08T08:20:05.524552526Z sensordata listened {
2021-11-08T08:20:05.524554807Z   latitude: REDACTED,
2021-11-08T08:20:05.524556893Z   longitude: REDACTED,
2021-11-08T08:20:05.524558845Z   accuracy: 10,
2021-11-08T08:20:05.524560778Z   altitude: 19.36281967163086
2021-11-08T08:20:05.524562749Z }
---  LAMP_worker.1.ltv0p7vghnk4ltzicgexbpskc TERMINATED ---
2021-11-08T11:59:18.509269556Z     y: 12.539999961853027,
2021-11-08T11:59:18.509272429Z     z: -14.15999984741211
2021-11-08T11:59:18.509275238Z   }
2021-11-08T11:59:18.509277995Z }
2021-11-08T11:59:18.877902420Z participant_id listened U0719341792
2021-11-08T11:59:18.877977567Z sensordata listened {
2021-11-08T11:59:18.877981005Z   gravity: {
2021-11-08T11:59:18.877984176Z     x: 0.001065264455974102,
2021-11-08T11:59:18.877986250Z     y: -0.002130528911948204,
2021-11-08T11:59:18.877988268Z     z: -0.001065264455974102
2021-11-08T11:59:18.877990133Z   }
2021-11-08T11:59:18.877991883Z }
2021-11-08T11:59:19.321976076Z participant_id listened U0719341792
2021-11-08T11:59:19.322048102Z sensordata listened {
2021-11-08T11:59:19.322057331Z   motion: {
2021-11-08T11:59:19.322059597Z     x: -0.08140286058187485,
2021-11-08T11:59:19.322061553Z     y: -0.026336219161748886,
2021-11-08T11:59:19.322063597Z     z: 9.876082420349121
2021-11-08T11:59:19.322065491Z   }
2021-11-08T11:59:19.322067437Z }
2021-11-08T11:59:19.776248425Z participant_id listened U0719341792
2021-11-08T11:59:19.776270112Z sensordata listened {
2021-11-08T11:59:19.776274228Z   motion: {
2021-11-08T11:59:19.776277055Z     x: -0.090979665517807,
2021-11-08T11:59:19.776279944Z     y: -0.023942016065120697,
2021-11-08T11:59:19.776282544Z     z: 9.876082420349121
2021-11-08T11:59:19.776285366Z   }
2021-11-08T11:59:19.776288047Z }
2021-11-08T11:59:20.191797345Z participant_id listened U0719341792
2021-11-08T11:59:20.191838301Z sensordata listened {
2021-11-08T11:59:20.191842910Z   gravity: {
2021-11-08T11:59:20.191846229Z     x: -0.001065264455974102,
2021-11-08T11:59:20.191848806Z     y: -0.000532632227987051,
2021-11-08T11:59:20.191851511Z     z: 0.000532632227987051
2021-11-08T11:59:20.191854145Z   }
2021-11-08T11:59:20.191856849Z }
2021-11-08T11:59:20.590169289Z participant_id listened U0719341792
2021-11-08T11:59:20.590225744Z sensordata listened {
2021-11-08T11:59:20.590232063Z   motion: {
2021-11-08T11:59:20.590235061Z     x: -0.0933738648891449,
2021-11-08T11:59:20.590237989Z     y: -0.019153613597154617,
2021-11-08T11:59:20.590239923Z     z: 9.86650562286377
2021-11-08T11:59:20.590241955Z   }
2021-11-08T11:59:20.590243769Z }
2021-11-08T11:59:20.998172899Z participant_id listened U0719341792
2021-11-08T11:59:20.998188203Z sensordata listened {
2021-11-08T11:59:20.998191355Z   rotation: {
2021-11-08T11:59:20.998194478Z     x: -0.003573794150725007,
2021-11-08T11:59:20.998197671Z     y: 0.0024797802325338125,
2021-11-08T11:59:20.998200398Z     z: 0.4774126410484314
2021-11-08T11:59:20.998203011Z   }
2021-11-08T11:59:20.998205804Z }
2021-11-08T11:59:21.447888928Z participant_id listened U0719341792
2021-11-08T11:59:21.447923031Z sensordata listened {
2021-11-08T11:59:21.447927683Z   magnetic: {
2021-11-08T11:59:21.447930491Z     x: 19.260000228881836,
2021-11-08T11:59:21.447933453Z     y: 12.119999885559082,
2021-11-08T11:59:21.447945851Z     z: -13.559999465942383
2021-11-08T11:59:21.447948942Z   }
2021-11-08T11:59:21.447951579Z }
2021-11-08T11:59:21.878267509Z participant_id listened U0719341792
2021-11-08T11:59:21.878312756Z sensordata listened {
2021-11-08T11:59:21.878316671Z   motion: {
2021-11-08T11:59:21.878320103Z     x: -0.08858546614646912,
2021-11-08T11:59:21.878323735Z     y: -0.033518824726343155,
2021-11-08T11:59:21.878326744Z     z: 9.873687744140625
2021-11-08T11:59:21.878330054Z   }
2021-11-08T11:59:21.878333029Z }
2021-11-08T11:59:22.239922802Z participant_id listened U0719341792
2021-11-08T11:59:22.239947879Z sensordata listened {
2021-11-08T11:59:22.239951281Z   magnetic: {
2021-11-08T11:59:22.239953965Z     x: 19.020000457763672,
2021-11-08T11:59:22.239956960Z     y: 12.239999771118164,
2021-11-08T11:59:22.239960459Z     z: -13.4399995803833
2021-11-08T11:59:22.239962609Z   }
2021-11-08T11:59:22.239965308Z }
2021-11-08T11:59:22.627761860Z participant_id listened U0719341792
2021-11-08T11:59:22.627826932Z sensordata listened {
2021-11-08T11:59:22.627837246Z   magnetic: {
2021-11-08T11:59:22.627840435Z     x: 19.31999969482422,
2021-11-08T11:59:22.627843210Z     y: 12.359999656677246,
2021-11-08T11:59:22.627846020Z     z: -14.34000015258789
2021-11-08T11:59:22.627848931Z   }
2021-11-08T11:59:22.627851859Z }
2021-11-08T11:59:23.003636028Z participant_id listened U0719341792
2021-11-08T11:59:23.003697660Z sensordata listened {
2021-11-08T11:59:23.003707923Z   gravity: {
2021-11-08T11:59:23.003710475Z     x: -0.000532632227987051,
2021-11-08T11:59:23.003712540Z     y: -0.000532632227987051,
2021-11-08T11:59:23.003714605Z     z: -0.000532632227987051
2021-11-08T11:59:23.003716550Z   }
2021-11-08T11:59:23.003718536Z }
2021-11-08T11:59:23.353067933Z participant_id listened U0719341792
2021-11-08T11:59:23.353084195Z sensordata listened {
2021-11-08T11:59:23.353100123Z   rotation: {
2021-11-08T11:59:23.353103096Z     x: -0.003732413984835148,
2021-11-08T11:59:23.353106293Z     y: 0.0027173981070518494,
2021-11-08T11:59:23.353109459Z     z: 0.4790908396244049
2021-11-08T11:59:23.353112380Z   }
2021-11-08T11:59:23.353115349Z }

--- LAMP_worker.1.qxvo8mrms2wj1uzlfuqa77aze TERMINATED ---

2021-11-10T09:19:58.946995803Z sensordata listened {
2021-11-10T09:19:58.947001326Z   magnetic: {
2021-11-10T09:19:58.947004375Z     x: -24.318750381469727,
2021-11-10T09:19:58.947007047Z     y: -11.700000762939453,
2021-11-10T09:19:58.947009672Z     z: -39.01875305175781
2021-11-10T09:19:58.947012450Z   }
2021-11-10T09:19:58.947015167Z }
2021-11-10T09:19:59.056761481Z participant_id listened U0128440310
2021-11-10T09:19:59.056789957Z sensordata listened { magnetic: { x: 293.3125, y: -750.7999877929688, z: -764.4375 } }
2021-11-10T09:19:59.111401990Z participant_id listened U8939345176
2021-11-10T09:19:59.111421069Z sensordata listened {
2021-11-10T09:19:59.111424916Z   motion: {
2021-11-10T09:19:59.111427703Z     x: 0.5814558863639832,
2021-11-10T09:19:59.111430688Z     y: 0.8201399445533752,
2021-11-10T09:19:59.111433432Z     z: 9.896116256713867
2021-11-10T09:19:59.111436440Z   }
2021-11-10T09:19:59.111439430Z }
2021-11-10T09:19:59.270585336Z participant_id listened U8939345176
2021-11-10T09:19:59.270616530Z sensordata listened {
2021-11-10T09:19:59.270621113Z   rotation: {
2021-11-10T09:19:59.270624342Z     x: 0.00011796114267781377,
2021-11-10T09:19:59.270627960Z     y: -0.050691016018390656,
2021-11-10T09:19:59.270631288Z     z: -0.8104769587516785
2021-11-10T09:19:59.270634650Z   }
2021-11-10T09:19:59.270638188Z }
2021-11-10T09:20:00.003270797Z today_date while notify 2021-11-10T09:20:00.000Z
2021-11-10T09:20:00.003299851Z start_date while notify 2021-11-05T09:20:00.000Z
2021-11-10T09:20:00.003303371Z {
2021-11-10T09:20:00.003306086Z   device_token: 'REDACTED',
2021-11-10T09:20:00.003309343Z   device_type: 'ios',
2021-11-10T09:20:00.003312104Z   payload: {
2021-11-10T09:20:00.003314621Z     participant_id: 'U3494874569',
2021-11-10T09:20:00.003317306Z     activity_id: 'mgwry4j8za55x34yzs21',
2021-11-10T09:20:00.003320030Z     message: 'You have a mindLAMP activity waiting for you: Scr 1.',
2021-11-10T09:20:00.003323051Z     title: 'Scr 1',
2021-11-10T09:20:00.003335919Z     url: '/participant/U3494874569/activity/mgwry4j8za55x34yzs21',
2021-11-10T09:20:00.003338655Z     notificationId: 947940
2021-11-10T09:20:00.003341561Z   }
2021-11-10T09:20:00.003344084Z }
2021-11-10T09:20:00.003348918Z /participant/U3494874569/activity/mgwry4j8za55x34yzs21
2021-11-10T09:20:00.004385360Z Completed  job state on mgwry4j8za55x34yzs21
2021-11-10T09:20:00.005568729Z today_date while notify 2021-11-10T09:20:00.000Z
2021-11-10T09:20:00.005624513Z start_date while notify 2021-05-19T15:20:00.000Z
2021-11-10T09:20:00.005630533Z {
2021-11-10T09:20:00.005633475Z   device_token: 'REDACTED',
2021-11-10T09:20:00.005636957Z   device_type: 'android',
2021-11-10T09:20:00.005639972Z   payload: {
2021-11-10T09:20:00.005642709Z     participant_id: 'U0176919012',
2021-11-10T09:20:00.005645534Z     activity_id: '39m9gp4da1r20fsdvqjz',
2021-11-10T09:20:00.005648396Z     message: 'You have a mindLAMP activity waiting for you: b1.',
2021-11-10T09:20:00.005651227Z     title: 'b1',
2021-11-10T09:20:00.005654131Z     url: '/participant/U0176919012/activity/39m9gp4da1r20fsdvqjz',
2021-11-10T09:20:00.005657025Z     notificationId: 906379
2021-11-10T09:20:00.005659753Z   }
2021-11-10T09:20:00.005662337Z }
2021-11-10T09:20:00.005666792Z /participant/U0176919012/activity/39m9gp4da1r20fsdvqjz
2021-11-10T09:20:00.006522392Z Completed  job state on 39m9gp4da1r20fsdvqjz
2021-11-10T09:20:00.020225973Z Error encountered sending GCM push notification.
2021-11-10T09:20:00.020923870Z response Response {
2021-11-10T09:20:00.020930398Z   size: 0,
2021-11-10T09:20:00.020932640Z   timeout: 0,
2021-11-10T09:20:00.020935247Z   [Symbol(Body internals)]: {
2021-11-10T09:20:00.020937221Z     body: PassThrough {
2021-11-10T09:20:00.020939139Z       _readableState: [ReadableState],
2021-11-10T09:20:00.020941286Z       _events: [Object: null prototype],
2021-11-10T09:20:00.020943234Z       _eventsCount: 2,
2021-11-10T09:20:00.020945038Z       _maxListeners: undefined,
2021-11-10T09:20:00.020947136Z       _writableState: [WritableState],
2021-11-10T09:20:00.020948938Z       allowHalfOpen: true,
2021-11-10T09:20:00.020950833Z       [Symbol(kCapture)]: false,
2021-11-10T09:20:00.020952718Z       [Symbol(kCallback)]: null
2021-11-10T09:20:00.020954501Z     },
2021-11-10T09:20:00.020966065Z     disturbed: false,
2021-11-10T09:20:00.020968001Z     error: null
2021-11-10T09:20:00.020980261Z   },
2021-11-10T09:20:00.020982205Z   [Symbol(Response internals)]: {
2021-11-10T09:20:00.020984000Z     url: 'https://app-gateway.lamp.digital/push',
2021-11-10T09:20:00.020985912Z     status: 404,
2021-11-10T09:20:00.020988003Z     statusText: 'Not Found',
2021-11-10T09:20:00.020989830Z     headers: Headers { [Symbol(map)]: [Object: null prototype] },
2021-11-10T09:20:00.020991858Z     counter: 0
2021-11-10T09:20:00.020993730Z   }
2021-11-10T09:20:00.020995579Z }
2021-11-10T09:20:00.021035963Z "Error encountered sending APN push notification."--Error: HTTP error!
2021-11-10T09:20:00.192021360Z participant_id listened U9444966227
2021-11-10T09:20:00.192043114Z sensordata listened {
2021-11-10T09:20:00.192046054Z   motion: {
2021-11-10T09:20:00.192048093Z     x: 0.17237511277198792,
2021-11-10T09:20:00.192050350Z     y: -0.03112328238785267,
2021-11-10T09:20:00.192052527Z     z: 9.806227684020996
2021-11-10T09:20:00.192054420Z   }
2021-11-10T09:20:00.192056198Z }
2021-11-10T09:20:00.788429637Z participant_id listened U0128440310
2021-11-10T09:20:00.788451798Z sensordata listened {
2021-11-10T09:20:00.788455459Z   gravity: {
2021-11-10T09:20:00.788458327Z     x: 0.00026631588116288185,
2021-11-10T09:20:00.788461046Z     y: -0.00013315794058144093,
2021-11-10T09:20:00.788463647Z     z: -0.00013315794058144093
2021-11-10T09:20:00.788466359Z   }
2021-11-10T09:20:00.788469099Z }=

--- LAMP_worker.1.qcfz0h4k4u7v4748egvle96rk TERMINATED ---

[SuruPa00]

This may also be linked to the recent logout issues we have been seeing as automations is a recently developing addition and we as a group have collectively been still trying to pinpoint a source of these issues. There may be a problem where the app or dashboard code fails to reach the server. This may make it so it thinks the credentials are not correct and deletes them. Let's look into this to see if this issue may be related to the series of automatic logouts.

[Linoy339]

@lukeoftheshire . We used to restart the stack when we see the message - Nats server is disconnected Or similar redis disconnection error. Can you please keep an eye on memory when any of the container gets restarted and also, how frequent its getting restarted? @SuruPa00. This doesn't seems to be the reason for automatic logout issues.

From the logs, nats-server and redis-server is getting disconnected for some reason We shall implement more logging in codebase and check.

[avaidyam]

@Linoy339 Going forward, please do not manually restart anything - no changes to containers or. services should ever be made (only the stack file should be updated by myself once a change is made to components, which there should not be anymore).

Please resolve the actual issue. I still suspect this is linked to the automatic logouts until we have further evidence to the contrary.

[michaelmenon]

thanks Aditya, I will check with the team and update.

[Linoy339]

@lukeoftheshire , @avaidyam , @SuruPa00 Here, it seems that the nats connection is getting disconnected and closed after the default retry attempts (which is 10 now) from the client. As a solution, we have changed the retry value as infinite, so that nats client will look for nats-server until it is available. This is done in staging and we can move to production after monitoring it for 2 or 3 days.

[avaidyam]

@Linoy339 Can we diagnose why NATS is being disconnected? Both Redis and NATS should never be going down as a result of our code. We aren't fixing the actual issue with the patch you described.

[Linoy339]

@avaidyam . Yes, we are diagnosing it by putting more logs. We already tackled the issue related with max payload of nats earlier. Also, we made sure that redis is reconnected if its disconnected for some reason. Also, we need to introduce the above solution, so that we can make sure that nats will be alive soon after being disconnected

[Linoy339]

@lukeoftheshire . We have seen that both LAMP_server and worker in staging as well as production got restarted. Can you please restart the stack in order to make it work correctly.

As a step of diagnosing, can you please provide the whole logs (LAMP_server,LAMP_worker,LAMP_staging_server,LAMP_staging_worker,LAMP_message_queue,LAMP_staging_message_queue) for today?

[Linoy339]

@lukeoftheshire . Can you please restart the Lamp-server and Lamp-worker(staging and production), as Lamp-server and Lamp-worker is not connected to message_queue, which blocks notification process in LAMP-worker.

[avaidyam]

@Linoy339 You should be able to access the logs from the Portainer console still. Just please don't update any services, containers, or stacks. If you need to update any stacks, @lukeoftheshire and I can help do that.

[Linoy339]

@avaidyam . If you can give full logs as txt file, it would be helpful. Also, we won't update any service anymore. Instead, we will inform @lukeoftheshire for that

[avaidyam]

@Linoy339 We don't have access to anything more than you would through the Portainer console. The logs attached above were copied out of Portainer and manually redacted to hide IDs and sensitive info.

[Linoy339]

Okay. I will try that. Also, please restart the Lamp_staging as well as Lamp stack, as still seen like nats is disconnected

[Linoy339]

@avaidyam @lukeoftheshire .

Can you please check with Lamp stack (lamp-worker and lamp-server), as it seems like not restarted properly Please restart the stack(lamp-worker and lamp-server only)

[avaidyam]

@Linoy339 we are working on it! apologies for the delay.

[lukeoftheshire]

Hi @Linoy339 thank you for your patience - sorry about the delay. I have just restarted both lamp-worker and lamp-server. Please let me know if this fixes the issues and if you need anything else done.

[Linoy339]

@avaidyam @lukeoftheshire : No issues. It can be seen that the bull queue is not getting initialized after redis being reconnected, which should be the primary as well as the root issue out of all and this need to be resolved. Any ways, this couldn't be reproduced in our portainer environment.

[Linoy339]

Now, it got restarted properly

[Linoy339]

@avaidyam . Inorder to troubleshoot these problems, we might need to restart containers of our own. So, can you please grant us to handle the staging stack of our own?

[avaidyam]

The staging stack should be fine - NOT the production stack, though.

[Linoy339]

Sure. Thank you

[Linoy339]

@avaidyam @lukeoftheshire It seems like lamp server api is not reachable apart from LAMP-dashboard. Have you changed anything in the inbound rules or any particular network settings? Can you please check on this ?

[avaidyam]

This sounds like it could be on your end? AWS sends us a system status notification if api.lamp.digital is unreachable from the internet.

[Linoy339]

Apologies. Its api-staging.lamp.digital and not api.lamp.digital

It says error as Error: connect ECONNREFUSED 3.130.237.50:443

[avaidyam]

Works on my end?

[Linoy339]

Now, it worked from the postman.

But, its not getting resolved in lamp_staging_worker. It says : error while fetching activities--- FetchError: request to https://api-staging.lamp.digital/study/sdp35cpgddmcbp3f1qak/activity?ignore_binary=true failed, reason: connect ECONNREFUSED 3.130.237.50:443

[Linoy339]

I think connection is getting refused somehow and I got the same error in postman too. Also, lamp_staging worker is unable to fetch the api calls. The following error FetchError: request to https://api-staging.lamp.digital/researcher failed, reason: connect ECONNREFUSED 3.130.237.50:443

@lukeoftheshire . Can you check

[lukeoftheshire]

Seems to work for me as well - I can connect to the staging API.

[Linoy339]

Can you please try for some time, as we get some error in fetching. I have tried in another machine as well.

Please see the screenshot:

[avaidyam]

@Linoy339 You should be able to see the logs of what the server is receiving and denying here. If no request appears for your IP address, then it's likely on your end or a spam protection feature we're not aware of on AWS's end.

[Linoy339]

Okay. We can see that Lamp-staging-server is refusing Lamp-staging-worker.

Please find the logs here : https://console.lamp.digital/#!/4/docker/containers/5fdd1c28bcfc97cc1b07c759debced4e09dc30d6e8cb714a9532f0e4b57f0ad8/logs?nodeName=node-02.lamp.digital

Can you please try identify the issue if any exist?

[Linoy339]

@avaidyam. @lukeoftheshire. Please find the error message which I am getting in lamp_staging_worker container logs: "FetchError: request to https://api-staging.lamp.digital/participant/U2271608715/sensor_event?origin=lamp.analytics&limit=1000 failed, reason: connect ECONNREFUSED 3.130.237.50:443"

Also, when I tried staging instance, it says cannot restart container....no space left on device Is there any issue related with space in the host, node-02?

[Linoy339]

It seems to be a space issue. Can you please check on volumes and related data?

[lukeoftheshire]

Hi @Linoy339 Thank you for bringing this to our attention. This should be fixed for now although we will look into the root causes.

[Linoy339]

@lukeoftheshire . We still face issue in communication between lamp_staging_worker and lamp_staging_server, which blocks us for testing. It is suspected that space issue has not been resolved .

This is the error in lamp_staging_worker logs : request to https://api-staging.lamp.digital/type/6jf470r7z7peb6grgfr7/attachment/lamp.automation failed, reason: connect ECONNREFUSED 3.130.237.50:443

[Linoy339]

@lukeoftheshire Please find the screenshot stating the space issue:

[Linoy339]

@lukeoftheshire . Also, We can see that both staging and production dashboards are in the host, node-02 .

Can you confirm on this, as we suspect one of them to be in node-01?

[avaidyam]

@Linoy339 Only the production server components are in node-01. We are in the process of moving both the production and staging dashboards off of the infrastructure and onto GitHub Pages, which will allow for more rapid development and less latency when accessing pages.

[Linoy339]

Thank you @avaidyam This means both dashboards were in node-02.

[lukeoftheshire]

Hi @Linoy339 - thanks for bringing this to our attention. I have cleared out a small amount of space on the node - hopefully that is enough for now - over the course of the day I will clear out more space to stop this from becoming a recurring issue. I'll update here once that's done.

I can also confirm Aditya's statement - both dashboards are in node 2.

[Linoy339]

Thank you @avaidyam and @lukeoftheshire

And do you know by any chance, how node-02 became space out ?

We are in the process of moving both the production and staging dashboards off of the infrastructure and onto GitHub Pages, which will allow for more rapid development and less latency when accessing pages.

So, this will make dashboard to be called as static from (https://pages.github.com/)

[avaidyam]

@Linoy339 Yes that's correct. Node-02 is also used for internal development and data science so there's a lot of cache files generated. However, every time a new staging build is deployed, the used volumes and images pile up adding 6+ GB per build. We have been manually removing them thus far.

[Linoy339]

Okay. Now, we understand Thank you for the clarification

[lukeoftheshire]

@Linoy339 @avaidyam I have removed a good deal of data from the caching directory. Disk usage is back below 50%. I will continue to monitor this, of course, but this problem should no longer occur.

[Linoy339]

Thanks @lukeoftheshire

[Linoy339]

@lukeoftheshire . We can see that the space has been restored. But, Its seems that its still an issue and can be seen in the staging-worker logs: request to https://api-staging.lamp.digital/participant/U8315566314/sensor_event?origin=lamp.analytics&limit=1000 failed, reason: connect ECONNREFUSED 3.130.237.50:443

Do we need to restart anything after clearing the space?

[lukeoftheshire]

Hi @Linoy339 I have just restarted the lamp staging server and worker services.

I will try the staging api on my end and update this post with the results.

[Linoy339]

Okay.

Also, LAMP-server is provided with staging image - ghcr.io/bidmcdigitalpsychiatry/lamp-server:latest instead of ghcr.io/bidmcdigitalpsychiatry/lamp-server:2021.

We had a release today. we couldn't notice it earlier. Can you check this too ?

[avaidyam]

@Linoy339 :latest is always overwritten by master branch releases (staging). This is the correct tag to use.

[Linoy339]

Okay. From my understanding, the tag for the production will be the lamp-server:2021 and staging uses lamp-server:latest

Here is the screenshot:

It is seen that the production server uses lamp-server:latest. This is something new to us

If it is okay, this would be fine