search

pelias / pip-service

Pelias point-in-polygon-service
https://pelias.io
MIT License
16 stars 18 forks source link

pip-service is not working with sqlite #120

Closed itz-sudo closed 4 years ago

itz-sudo commented 4 years ago

I've imported full planet recently. After starting pip-service, trying to send request to pip and I'm not getting any response. Imported wof with sqlite=true with only --admin-only and having sqlite files in the path /mnt/pelias/whosonfirst/sqlite/. My configuration in pelias.json for whosonfirst and pip service is

"services": {
        "pip": {
          "url": "http://127.0.0.1:3103"
        },
......
......
    "imports": {
......
......
    "whosonfirst": {
      "datapath": "/mnt/pelias/whosonfirst",
      "sqlite" : "true"
    }
......

Since i'm using pm2 to run this pip-service taken logs came at this time

0|pip      | pip-service is now running on port 3103
0|pip      | 2019-12-30T07:35:15.465Z - info: [wof-pip-service:master] ocean worker loaded 7 features in 5.147 seconds
0|pip      | 2019-12-30T07:35:15.645Z - info: [wof-pip-service:master] marinearea worker loaded 305 features in 5.321 seconds
0|pip      | 2019-12-30T07:35:15.711Z - info: [wof-pip-service:master] empire worker loaded 11 features in 5.419 seconds
0|pip      | 2019-12-30T07:35:15.774Z - info: [wof-pip-service:master] dependency worker loaded 40 features in 5.478 seconds
0|pip      | 2019-12-30T07:35:15.884Z - info: [wof-pip-service:master] continent worker loaded 8 features in 5.581 seconds
0|pip      | 2019-12-30T07:35:16.005Z - info: [wof-pip-service:master] borough worker loaded 273 features in 5.707 seconds
0|pip      | 2019-12-30T07:35:16.809Z - info: [wof-pip-service:master] macrocounty worker loaded 477 features in 6.488 seconds
0|pip      | 2019-12-30T07:35:17.080Z - info: [wof-pip-service:master] macroregion worker loaded 94 features in 6.789 seconds
0|pip      | 2019-12-30T07:35:21.036Z - info: [wof-pip-service:master] country worker loaded 210 features in 10.728 seconds
0|pip      | 2019-12-30T07:35:32.709Z - info: [wof-pip-service:master] neighbourhood worker loaded 29316 features in 21.903 seconds
0|pip      | 2019-12-30T07:35:47.685Z - info: [wof-pip-service:master] region worker loaded 4175 features in 37.304 seconds

0|pip  | 2019-12-30T07:36:23.598Z - info: [wof-pip-service:master] county worker loaded 40797 features in 72.245 seconds
0|pip  | 2019-12-30T07:36:33.769Z - info: [wof-pip-service:master] localadmin worker loaded 112503 features in 80.498 seconds
0|pip  | <--- Last few GCs --->
0|pip  | [3445:0x3972420]   109792 ms: Scavenge 1213.6 (1431.1) -> 1204.9 (1433.1) MB, 7.1 / 0.0 ms  (average mu = 0.343, current mu = 0.294) allocation failure 
0|pip  | [3445:0x3972420]   109860 ms: Scavenge 1218.9 (1434.1) -> 1209.2 (1438.6) MB, 21.0 / 0.0 ms  (average mu = 0.343, current mu = 0.294) allocation failure 
0|pip  | [3445:0x3972420]   110034 ms: Scavenge 1223.2 (1439.1) -> 1214.9 (1441.6) MB, 18.7 / 0.0 ms  (average mu = 0.343, current mu = 0.294) allocation failure 
0|pip  | <--- JS stacktrace --->
0|pip  | ==== JS stack trace =========================================
0|pip  |     0: ExitFrame [pc: 0x117132e063c1]
0|pip  | Security context: 0x0d935731e6e9 <JSObject>
0|pip  |     1: read [0x1a64f9949681] [_stream_readable.js:~389] [pc=0x11713328de22](this=0x26115a320e79 <Readable map = 0x22335df3c489>,n=0x1d1e081026f1 <undefined>)
0|pip  |     2: arguments adaptor frame: 0->1
0|pip  |     3: flow(aka flow) [0x3e7b2f244aa1] [_stream_readable.js:~969] [pc=0x117132e79ba0](this=0x1d1e081026f1 <undefined>,stream=0x26115a320e79 <Readable map = 0x22335df...
0|pip  | FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
0|pip  |  1: 0x8fa050 node::Abort() [locality]
0|pip  |  2: 0x8fa09c  [locality]
0|pip  |  3: 0xb0020e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [locality]
0|pip  |  4: 0xb00444 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [locality]
0|pip  |  5: 0xef4952  [locality]
0|pip  |  6: 0xef4a58 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [locality]
0|pip  |  7: 0xf00b32 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [locality]
0|pip  |  8: 0xf01464 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [locality]
0|pip  |  9: 0xf040d1 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [locality]
0|pip  | 10: 0xecc3b5  [locality]
0|pip  | 11: 0xed3c2a v8::internal::Factory::NewRawOneByteString(int, v8::internal::PretenureFlag) [locality]
0|pip  | 12: 0xed3f3b v8::internal::Factory::NewStringFromOneByte(v8::internal::Vector<unsigned char const>, v8::internal::PretenureFlag) [locality]
0|pip  | 13: 0xed498d v8::internal::Factory::NewStringFromUtf8(v8::internal::Vector<char const>, v8::internal::PretenureFlag) [locality]
0|pip  | 14: 0xb0e169 v8::String::NewFromUtf8(v8::Isolate*, char const*, v8::NewStringType, int) [locality]
0|pip  | 15: 0x7f090b4b25a9 Data::GetValueJS(v8::Isolate*, sqlite3_stmt*, int, bool) [/home/pelias/pip-service/node_modules/better-sqlite3/build/better_sqlite3.node]
0|pip  | 16: 0x7f090b4b27af Data::GetFlatRowJS(v8::Isolate*, v8::Local<v8::Context>, sqlite3_stmt*, bool) [/home/pelias/pip-service/node_modules/better-sqlite3/build/better_sqlite3.node]
0|pip  | 17: 0x7f090b4b2ad0 Data::GetRowJS(v8::Isolate*, v8::Local<v8::Context>, sqlite3_stmt*, bool, char) [/home/pelias/pip-service/node_modules/better-sqlite3/build/better_sqlite3.node]
0|pip  | 18: 0x7f090b4b5df3 StatementIterator::Next(v8::FunctionCallbackInfo<v8::Value> const&) [/home/pelias/pip-service/node_modules/better-sqlite3/build/better_sqlite3.node]
0|pip  | 19: 0x117132e063c1

It says FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory. I've increased --max_old_space_size from 8000 to 10000. But still it gives same error.

missinglink commented 4 years ago

This is mostly likely due to insufficient RAM

itz-sudo commented 4 years ago

I need to pass max_old_space_size value while starting pip-service with pm2. By default while reaching heap size 300MB pm2 restarting this pip-service. This works with command PORT=3103 pm2 start /home/pelias/pip-service/index.js --name "pip" --node-args="--max_old_space_size=8000"

orangejulius commented 4 years ago

Hi @itz-sudo, We don't recommend starting the service by calling index.js directly. We provide a command, ./bin/start, which will properly set any Node.js parameters, including max_old_space_size.

With pm2, you can use pm2 start ./bin/start.