Investigate why stack updates sometimes does not auto pull new configs.

[robinkwilson]

Likely a problem in ita. Could be related to permissions:

Dec 08 23:50:41 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:41.404Z ita:flush Flushing service postgrest...                           [108/1968]
Dec 08 23:50:41 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:41.706Z configtool:ps Requesting parameters for /keymaster/domain-rkw-1 (1)...
Dec 08 23:50:42 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:42.040Z configtool:ps Requesting parameters for /ita/domain-rkw-1/postgrest (1).
..
Dec 08 23:50:42 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:42.070Z ita:flush Computing delta for postgrest...
Dec 08 23:50:42 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:42.070Z configtool:hab Requesting Habitat config for postgrest.default.
Dec 08 23:50:42 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:42.073Z ita:flush All postgrest configs already up-to-date.
Dec 08 23:50:42 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:42.079Z ita:flush Flushing service reticulum...
Dec 08 23:50:42 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:42.377Z configtool:ps Requesting parameters for /keymaster/domain-rkw-1 (1)...
Dec 08 23:50:42 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:42.711Z configtool:ps Requesting parameters for /ita/domain-rkw-1/reticulum (1).
..
Dec 08 23:50:42 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:42.745Z ita:flush Computing delta for reticulum...
Dec 08 23:50:42 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:42.745Z configtool:hab Requesting Habitat config for reticulum.default.
Dec 08 23:50:42 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:42.752Z ita:flush Updating Habitat configs:
Dec 08 23:50:42 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:42.754Z configtool:hab Invoking hab: bio config apply -r 127.0.0.1:9632 -u polycosm-
config-user reticulum.default@domain-rkw-1 1607471442
Dec 08 23:50:42 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:42.764Z ita:app Auto-flush of reticulum failed.
Dec 08 23:50:42 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:42.765Z ita:app Error: bio exited with code 1:
Dec 08 23:50:42 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): ✗✗✗
Dec 08 23:50:42 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): ✗✗✗ Permission denied (os error 13)
Dec 08 23:50:42 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): ✗✗✗
Dec 08 23:50:42 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O):
Dec 08 23:50:42 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O):     at ChildProcess.<anonymous> (/hab/pkgs/mozillareality/ita/0.0.1/20200526203229/node_modules/hubs-
configtool/src/habitat.js:23:16)
Dec 08 23:50:42 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O):     at ChildProcess.emit (events.js:209:13)
Dec 08 23:50:42 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O):     at ChildProcess.EventEmitter.emit (domain.js:476:20)
Dec 08 23:50:42 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O):     at maybeClose (internal/child_process.js:1021:16)
Dec 08 23:50:42 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O):     at Process.ChildProcess._handle.onexit (internal/child_process.js:283:5)
Dec 08 23:50:42 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:42.768Z ita:flush Flushing service spoke...
Dec 08 23:50:43 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:43.045Z configtool:ps Requesting parameters for /keymaster/domain-rkw-1 (1)...
Dec 08 23:50:43 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:43.379Z configtool:ps Requesting parameters for /ita/domain-rkw-1/spoke (1)...
Dec 08 23:50:43 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:43.414Z ita:flush Computing delta for spoke...
Dec 08 23:50:43 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:43.414Z configtool:hab Requesting Habitat config for spoke.default.
Dec 08 23:50:43 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:43.418Z ita:flush All spoke configs already up-to-date.
Dec 08 23:50:43 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:43.422Z ita:flush Flushing service youtube-dl-api-server...
Dec 08 23:50:43 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:43.713Z configtool:ps Requesting parameters for /keymaster/domain-rkw-1 (1)...
Dec 08 23:50:44 an-aws-box.domain.net bash[2214]: ita.default@domain-rkw-1(O): 2020-12-08T23:50:44.048Z configtool:ps Requesting parameters for /ita/domain-rkw-1/youtube-dl-api
-server (1)...

┆Issue is synchronized with this Jira Task

[justincolangelo]

I am seeing this error on our servers. There isn't a polycosm-config-user on the servers, though, so I was wondering if it was there at one time and disappeared, or if we were supposed to create it. Did this get resolved by any chance?

[rawnsley]

I've also got this problem, but to be fair I have hacked the AWS CloudFormation template and added my own reticulum image, so I was expecting some trouble. In my script I tear down the mozillareality/reticulum habitat (or service or whatever it's called) and then install my own custom version in it's place. Perhaps I'm tearing down reticulum before it's executed some important initialization scripts?