getodk / central-backend

Node.js based backend for ODK Central
https://docs.getodk.org/central-intro/
Apache License 2.0
50 stars 76 forks source link

Server should not come up if migrations fail #461

Closed yanokwa closed 2 years ago

yanokwa commented 2 years ago

I upgraded to Central v1.4. I was able to login, but the the server audit logs page did not load. Looking at the logs, I saw this...

It would be ideal if the server does not load if migrations fail in some way (or in this case, run out of memory).

root@example:~/central# docker logs central_service_1
wait-for-it.sh: waiting 15 seconds for example.foo.bar:5432
wait-for-it.sh: example.foo.bar:5432 is available after 0 seconds
generating local service configuration..
running migrations..

<--- Last few GCs --->

[30:0x4a26ef0]    65185 ms: Mark-sweep (reduce) 968.3 (972.4) -> 968.2 (973.4) MB, 299.2 / 0.0 ms  (average mu = 0.552, current mu = 0.000) last resort GC in old space requested
[30:0x4a26ef0]    65457 ms: Mark-sweep (reduce) 968.2 (972.4) -> 968.2 (973.4) MB, 272.2 / 0.0 ms  (average mu = 0.365, current mu = 0.000) last resort GC in old space requested

<--- JS stacktrace --->

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: 0xa25510 node::Abort() [node]
 2: 0x9664d3 node::FatalError(char const*, char const*) [node]
 3: 0xb9a8be v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb9ac37 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd56ca5  [node]
 6: 0xd692c1 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 7: 0xd2e9fd v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node]
 8: 0xd28884 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [node]
 9: 0xd2abd1 v8::internal::FactoryBase<v8::internal::Factory>::NewRawTwoByteString(int, v8::internal::AllocationType) [node]
10: 0xd30b6c v8::internal::Factory::NewStringFromUtf8(v8::internal::Vector<char const> const&, v8::internal::AllocationType) [node]
11: 0xbb1912 v8::String::NewFromUtf8(v8::Isolate*, char const*, v8::NewStringType, int) [node]
12: 0xafe839 node::StringBytes::Encode(v8::Isolate*, char const*, unsigned long, node::encoding, v8::Local<v8::Value>*) [node]
13: 0xa01871  [node]
14: 0xc066ab  [node]
15: 0xc07c56  [node]
16: 0xc082d6 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [node]
17: 0x14269f9  [node]
/usr/odk/start-odk.sh: line 6:    30 Aborted                 (core dumped) node -e 'const { withDatabase, migrate } = require("./lib/model/migrate"); withDatabase(require("config").get("default.database"))(migrate);'
starting cron..
using 4 worker(s) based on available memory (1986316)..
starting server.
2022-03-06T17:19:22: PM2 log: Launching in no daemon mode
2022-03-06T17:19:22: PM2 log: App [service:0] starting in -fork mode-
2022-03-06T17:19:22: PM2 log: App [service:0] online
matthew-white commented 2 years ago

Duplicate of getodk/central#258 I think. I definitely think this would be a helpful change once we get the chance!