datagovua / edrpou-api

API читання ЄДР
http://edr.data-gov-ua.org/
Other
16 stars 1 forks source link

Faster (batch) insert #1

Open Vanuan opened 8 years ago

Vanuan commented 8 years ago

Investigate why does it take so long to insert data.

Vanuan commented 8 years ago

Use redis for socket session: 7f37f815b8b90201fc64f229d4471c8202a75b1c Prevent crashes: 9218f50ebf32c9cb45c9cc9c8253ba0684533c8a Use redis for http session: 64bf8187b76b3ae92bc5660f796ab649ac02a64d

Vanuan commented 8 years ago

Additionally, add these lines to /etc/rc.local

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
Vanuan commented 8 years ago
** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
**        We suggest setting it to 'never'
** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
**        We suggest setting it to 'never'
Vanuan commented 8 years ago
edr_1           | FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
edr_1           | Aborted (core dumped)
edr_edr_1 exited with code 134
Vanuan commented 8 years ago
  File "<string>", line 3, in <module>
  File "compose/cli/main.py", line 57, in main
  File "compose/cli/main.py", line 108, in perform_command
  File "compose/cli/main.py", line 741, in up
  File "compose/cli/log_printer.py", line 86, in run
  File "compose/cli/log_printer.py", line 222, in consume_queue
MemoryError
docker-compose returned -1
Vanuan commented 8 years ago

https://www.npmjs.com/package/ofe

Vanuan commented 8 years ago
# WARNING overcommit_memory is set to 0!
Background save may fail under low memory condition.
To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf
and then reboot or run the command 'sysctl vm.overcommit_memory=1'
for this to take effect.