search

mysociety / alaveteli

Provide a Freedom of Information request system for your jurisdiction
https://alaveteli.org
Other
389 stars 196 forks source link

NoMemoryError: failed to allocate memory #4389

Open garethrees opened 6 years ago

garethrees commented 6 years ago

Seen a few of these NoMemoryError: failed to allocate memory since we moved server.

/usr/bin/env XAPIAN_MAX_CHANGESETS=10 run-with-lockfile -n /…/change-xapian-database.lock "/…/alaveteli/script/update-xapian-index verbose=true" >> /…/alaveteli/log/update-xapian-index.log || echo "stalled?"

rake aborted!
NoMemoryError: failed to allocate memory
/…/alaveteli-2017-11-21T12-40-42/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/core_ext/marshal.rb:6:in `getbyte'
/…/alaveteli-2017-11-21T12-40-42/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/core_ext/marshal.rb:6:in `load'
/…/alaveteli-2017-11-21T12-40-42/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/core_ext/marshal.rb:6:in `load_with_autoloading'
/…/alaveteli-2017-11-21T12-40-42/vendor/bundle/ruby/2.3.0/gems/open4-1.3.4/lib/open4.rb:125:in `propagate_exception'
/…/alaveteli-2017-11-21T12-40-42/vendor/bundle/ruby/2.3.0/gems/open4-1.3.4/lib/open4.rb:101:in `do_popen'
/…/alaveteli-2017-11-21T12-40-42/vendor/bundle/ruby/2.3.0/gems/open4-1.3.4/lib/open4.rb:34:in `popen4'
/…/alaveteli-2017-11-21T12-40-42/commonlib/rblib/external_command.rb:101:in `run'
/…/alaveteli-2017-11-21T12-40-42/lib/alaveteli_external_command.rb:33:in `run'
/…/alaveteli-2017-11-21T12-40-42/lib/mail_handler/mail_handler.rb:86:in `get_attachment_text_one_file'
/…/alaveteli-2017-11-21T12-40-42/app/models/incoming_message.rb:701:in `block in _extract_text'
/…/alaveteli-2017-11-21T12-40-42/app/models/incoming_message.rb:700:in `each'
/…/alaveteli-2017-11-21T12-40-42/app/models/incoming_message.rb:700:in `reduce'
/…/alaveteli-2017-11-21T12-40-42/app/models/incoming_message.rb:700:in `_extract_text'
/…/alaveteli-2017-11-21T12-40-42/app/models/incoming_message.rb:708:in `_get_attachment_text_internal'
/…/alaveteli-2017-11-21T12-40-42/app/models/incoming_message.rb:671:in `get_attachment_text_full'
/…/alaveteli-2017-11-21T12-40-42/app/models/incoming_message.rb:713:in `get_text_for_indexing_full'
/…/alaveteli-2017-11-21T12-40-42/app/models/info_request_event.rb:242:in `search_text_main'
/…/alaveteli-2017-11-21T12-40-42/lib/acts_as_xapian/acts_as_xapian.rb:842:in `single_xapian_value'
/…/alaveteli-2017-11-21T12-40-42/lib/acts_as_xapian/acts_as_xapian.rb:835:in `xapian_value'
/…/alaveteli-2017-11-21T12-40-42/lib/acts_as_xapian/acts_as_xapian.rb:956:in `block in xapian_index'
/…/alaveteli-2017-11-21T12-40-42/lib/acts_as_xapian/acts_as_xapian.rb:953:in `each'
/…/alaveteli-2017-11-21T12-40-42/lib/acts_as_xapian/acts_as_xapian.rb:953:in `xapian_index'
/…/alaveteli-2017-11-21T12-40-42/lib/acts_as_xapian/acts_as_xapian.rb:670:in `run_job'
/…/alaveteli-2017-11-21T12-40-42/lib/acts_as_xapian/acts_as_xapian.rb:650:in `block (2 levels) in update_index'
/…/alaveteli-2017-11-21T12-40-42/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
/…/alaveteli-2017-11-21T12-40-42/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
/…/alaveteli-2017-11-21T12-40-42/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
/…/alaveteli-2017-11-21T12-40-42/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.10/lib/active_record/transactions.rb:220:in `transaction'
/…/alaveteli-2017-11-21T12-40-42/lib/acts_as_xapian/acts_as_xapian.rb:639:in `block in update_index'
/…/alaveteli-2017-11-21T12-40-42/lib/acts_as_xapian/acts_as_xapian.rb:636:in `each'
/…/alaveteli-2017-11-21T12-40-42/lib/acts_as_xapian/acts_as_xapian.rb:636:in `update_index'
/…/alaveteli-2017-11-21T12-40-42/lib/acts_as_xapian/tasks/xapian.rake:12:in `block (2 levels) in <top (required)>'
/…/alaveteli-2017-11-21T12-40-42/vendor/bundle/ruby/2.3.0/gems/rake-12.1.0/exe/rake:27:in `<top (required)>'
/opt/rbenv/versions/2.3.5/bin/bundle:22:in `load'
/opt/rbenv/versions/2.3.5/bin/bundle:22:in `<main>'
Tasks: TOP => xapian:update_index
(See full trace by running task with --trace)
sagepe commented 6 years ago

I've had a quick look this afternoon following a chat with @crowbot. The various limits set for the foi user either haven't changed, or have increased. There are no OOM killer messages associated with the errors, or kernel messages suggesting limits have been reached, so it doesn't appear to be down to config from the old host that we'd missed transferring to the new one for any of the more obvious things, and everything seems to be captured in git anyway. I'll have a bit more of a think and another look in a day or two.

garethrees commented 6 years ago

Just looking at when these started and how many we've had:

sagepe commented 5 years ago

Looking through this again, it seems to me that the exception is being triggered while parsing an attachment of some kind. I wonder if this commit might be the culprit, this gets passed into the call to open4 via the library in commonlib, I think? (Bit shaky here, could do with a more skilled Ruby developer to have a look too!)

garethrees commented 5 years ago

Yeah, that probably has something to do with it (in that its limiting memory allocation), but that commit was made in 2013. I'm not sure why we started seeing so frequently (~100 instances) since late November 2017.