Closed geopolitis closed 5 years ago
geopolitis
commented
5 years ago In mid-time, I tested to add swap file to the VM with the following commands:
sudo fallocate -l 4G /swapfile
sudo dd if=/dev/zero of=/swapfile bs=1024 count=4194304
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
seems working now.
rjurney
commented
5 years ago @geopolitis RAM requirements for the VM have always been a problem. 16GB is ideal. Very cool that you got it working.
rjurney
commented
5 years ago @geopolitis Thanks for this solution, I am just now adding 8192MB of swap in the Vagrant boot script.
I think I've got this right:
# Setup a swap partition
sudo fallocate -l 8G /swapfile
sudo dd if=/dev/zero of=/swapfile bs=1M count=8192
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
The following errors No such file or directory and mongoDB out of memory. What is the suggested memory for the VM? I tried with 12GB of ram and I have the same results.
====================== Inspecting the Data We can use gunzip -c to inspect the JSON records. In [9]:
%%bash gunzip -c ../data/on_time_performance.jsonl.gz/part-00000.gz | head -3 gzip: ../data/on_time_performance.jsonl.gz/part-00000.gz: No such file or directory
=======================
Publishing Flight Records to MongoDB
MongoDB log: 2019-08-13T20:46:31.309+0000 I COMMAND [conn5] command agile_data_science.on_time_performance command: insert { insert: "on_time_performance", ordered: true, $db: "agile_data_science" } ninserted:1000 keysInserted:1000 numYields:0 reslen:44 locks:{ Global: { acquireCount: { r: 16, w: 16 } }, Database: { acquireCount: { w: 16 } }, Collection: { acquireCount: { w: 16 } } } protocol:op_query 336ms 2019-08-13T20:46:31.352+0000 I COMMAND [conn7] command agile_data_science.on_time_performance command: insert { insert: "on_time_performance", ordered: true, $db: "agile_data_science" } ninserted:1000 keysInserted:1000 numYields:0 reslen:44 locks:{ Global: { acquireCount: { r: 16, w: 16 } }, Database: { acquireCount: { w: 16 } }, Collection: { acquireCount: { w: 16 } } } protocol:op_query 341ms 2019-08-13T20:46:31.826+0000 I COMMAND [conn5] command agile_data_science.on_time_performance command: insert { insert: "on_time_performance", ordered: true, $db: "agile_data_science" } ninserted:1000 keysInserted:1000 numYields:0 reslen:44 locks:{ Global: { acquireCount: { r: 16, w: 16 } }, Database: { acquireCount: { w: 16 } }, Collection: { acquireCount: { w: 16 } } } protocol:op_query 154ms 2019-08-13T20:46:31.826+0000 I COMMAND [conn7] command agile_data_science.on_time_performance command: insert { insert: "on_time_performance", ordered: true, $db: "agile_data_science" } ninserted:1000 keysInserted:1000 numYields:0 reslen:44 locks:{ Global: { acquireCount: { r: 16, w: 16 } }, Database: { acquireCount: { w: 16 } }, Collection: { acquireCount: { w: 16 } } } protocol:op_query 154ms 2019-08-13T20:46:34.204+0000 I COMMAND [ftdc] serverStatus was very slow: { after basic: 127, after asserts: 253, after backgroundFlushing: 253, after connections: 253, after dur: 253, after extra_info: 253, after globalLock: 253, after locks: 253, after logicalSessionRecordCache: 253, after network: 333, after opLatencies: 333, after opcounters: 333, after opcountersRepl: 401, after repl: 401, after security: 401, after storageEngine: 401, after tcmalloc: 401, after transactions: 401, after wiredTiger: 523, at end: 1089 } 2019-08-13T20:46:34.870+0000 F - [ftdc] out of memory.
0x5561b8d134ca 0x5561b8d12ba9 0x5561b8d074e1 0x5561b84d7d59 0x5561b84d860f 0x5561b84e0e68 0x5561b84dda58 0x5561b84d9727 0x7f5d53a4c66f 0x7f5d531c16db 0x7f5d52eea88f ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"5561B6CD8000","o":"203B4CA","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5561B6CD8000","o":"203ABA9","s":"_ZN5mongo29reportOutOfMemoryErrorAndExitEv"},{"b":"5561B6CD8000","o":"202F4E1","s":"_ZN5mongo12mongoReallocEPvm"},{"b":"5561B6CD8000","o":"17FFD59","s":"_ZN5mongo14FTDCCompressor20getCompressedSamplesEv"},{"b":"5561B6CD8000","o":"180060F","s":"_ZN5mongo14FTDCCompressor9addSampleERKNS_7BSONObjENS_6Date_tE"},{"b":"5561B6CD8000","o":"1808E68","s":"_ZN5mongo14FTDCFileWriter11writeSampleERKNS_7BSONObjENS_6Date_tE"},{"b":"5561B6CD8000","o":"1805A58","s":"_ZN5mongo15FTDCFileManager28writeSampleAndRotateIfNeededEPNS_6ClientERKNS_7BSONObjENS_6Date_tE"},{"b":"5561B6CD8000","o":"1801727","s":"_ZN5mongo14FTDCController6doLoopEv"},{"b":"7F5D5398F000","o":"BD66F"},{"b":"7F5D531BA000","o":"76DB"},{"b":"7F5D52DC9000","o":"12188F","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.3", "gitVersion" : "9586e557d54ef70f9ca4b43c26892cd55257e1a5", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.15.0-55-generic", "version" : "#60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "5561B6CD8000", "elfType" : 3, "buildId" : "40A22A63C3F04AF7F9D3983994C20023104C5804" }, { "b" : "7FFE7296E000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "A8F867803A5FF97DA0979B82ADE7AA1A4B2C9BB1" }, { "b" : "7F5D55C4F000", "path" : "/usr/lib/x86_64-linux-gnu/libstemmer.so.0d", "elfType" : 3, "buildId" : "278CA72E21C11FF2E15A86B0B2C13A8922951702" }, { "b" : "7F5D55A32000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "EF3E006DFE3132A41D4D4DC0E407D6EA658E11C4" }, { "b" : "7F5D5582A000", "path" : "/usr/lib/x86_64-linux-gnu/libsnappy.so.1", "elfType" : 3, "buildId" : "55765D88D03CC928130D788F1C7E4BF8415AC7E3" }, { "b" : "7F5D555B0000", "path" : "/usr/lib/x86_64-linux-gnu/libyaml-cpp.so.0.5", "elfType" : 3, "buildId" : "BF65D47C8CD968E616F7D179F84A80CA71DB8249" }, { "b" : "7F5D553A7000", "path" : "/usr/lib/x86_64-linux-gnu/libpcrecpp.so.0", "elfType" : 3, "buildId" : "089B8438CC1394E978E56C556C9CAE768BD2F18C" }, { "b" : "7F5D55126000", "path" : "/usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.65.1", "elfType" : 3, "buildId" : "9F69F11220BB1FAAB0B73A2B6F4B0E81D9B901CE" }, { "b" : "7F5D54F0C000", "path" : "/usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.65.1", "elfType" : 3, "buildId" : "32B8421A0643426D9FB008005F5A86688065008B" }, { "b" : "7F5D54D07000", "path" : "/usr/lib/x86_64-linux-gnu/libboost_system.so.1.65.1", "elfType" : 3, "buildId" : "4BA851D242F2DB710CB1817DE860CF97AE2F9714" }, { "b" : "7F5D54A97000", "path" : "/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4", "elfType" : 3, "buildId" : "572D5C17FBDA6B678DF653411F676819DE18CA6B" }, { "b" : "7F5D5487C000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "390E9CC4C215314B6D8ADE6D6E28F8518418039C" }, { "b" : "7F5D545EF000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "439A262CC0127BA401707DEC7A28884D617550E0" }, { "b" : "7F5D54124000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1", "elfType" : 3, "buildId" : "CB6876717C83B0CC01C3C919B9B6E86D8554F546" }, { "b" : "7F5D53F1C000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "9826FBDF57ED7D6965131074CB3C08B1009C1CD8" }, { "b" : "7F5D53D18000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "25AD56E902E23B490A9CCDB08A9744D89CB95BCC" }, { "b" : "7F5D5398F000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "570BF32E8698FCE3BFACC4A8B010827F842D1DD6" }, { "b" : "7F5D535F1000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "A33761AB8FB485311B3C85BF4253099D7CABE653" }, { "b" : "7F5D533D9000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "41BDC55C07D5E5B1D8AB38E2C19B1F535855E084" }, { "b" : "7F5D531BA000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "28C6AADE70B2D40D1F0F3D0A1A0CAD1AB816448F" }, { "b" : "7F5D52DC9000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B417C0BA7CC5CF06D1D1BED6652CEDB9253C60D0" }, { "b" : "7F5D52B57000", "path" : "/lib/x86_64-linux-gnu/libpcre.so.3", "elfType" : 3, "buildId" : "5B3416BB188EAF3FA4B7530AAE6C1890B38B0372" }, { "b" : "7F5D5293C000", "path" : "/usr/lib/x86_64-linux-gnu/libunwind.so.8", "elfType" : 3, "buildId" : "7995F03B59E1D6EB7968EEA5B8534910D4E8E8D6" }, { "b" : "7F5D55EA0000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "64DF1B961228382FE18684249ED800AB1DCEAAD4" }, { "b" : "7F5D52716000", "path" : "/lib/x86_64-linux-gnu/liblzma.so.5", "elfType" : 3, "buildId" : "8FBCCA354D964860B9E6EB3736E9B7BC6177B417" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x3A) [0x5561b8d134ca] mongod(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x79) [0x5561b8d12ba9] mongod(_ZN5mongo12mongoReallocEPvm+0x21) [0x5561b8d074e1] mongod(_ZN5mongo14FTDCCompressor20getCompressedSamplesEv+0x11C9) [0x5561b84d7d59] mongod(_ZN5mongo14FTDCCompressor9addSampleERKNS_7BSONObjENS_6Date_tE+0x20F) [0x5561b84d860f] mongod(_ZN5mongo14FTDCFileWriter11writeSampleERKNS_7BSONObjENS_6Date_tE+0x38) [0x5561b84e0e68] mongod(_ZN5mongo15FTDCFileManager28writeSampleAndRotateIfNeededEPNS_6ClientERKNS_7BSONObjENS_6Date_tE+0x38) [0x5561b84dda58] mongod(_ZN5mongo14FTDCController6doLoopEv+0x257) [0x5561b84d9727] libstdc++.so.6(+0xBD66F) [0x7f5d53a4c66f] libpthread.so.0(+0x76DB) [0x7f5d531c16db] libc.so.6(clone+0x3F) [0x7f5d52eea88f] ----- END BACKTRACE -----