SOLR Read only (AVR couldn't write)

[davidschober]

Description

Molly reported that AVR couldn't handle edits. This did not seem to affect other systems.

Cause

The solr logs filled up causing the Docker containers to flip to read only.

Logs/ Symptom

Error: 'Server error writing document id c751d4a1-6dbe-405b-8044-2f2ee9bd23d0 to the index',

URI: http://solr.stack.vpc.rdc.library.northwestern.edu/solr/avr/update?wt=ruby&softCommit=true
Request Headers: {"Content-Type"=>"text/xml"}
Request Data: "<?xml version=\"1.0\" encoding=\"UTF-8\"?><add><doc><field name=\"system_create_dtsi\">2018-11-27T20:29:47Z</field><field name=\"system_modified_dtsi\">2018-11-27T20:29:47Z</field><field name=\"has_model_ssim\">Hydra::AccessControl</field><field name=\"id\">c751d4a1-6dbe-405b-8044-2f2ee9bd23d0</field></doc></add>"

Backtrace: /home/app/current/vendor/gems/ruby/2.4.0/gems/rsolr-1.1.2/lib/rsolr/client.rb:288:in `adapt_response'
/home/app/current/vendor/gems/ruby/2.4.0/gems/rsolr-1.1.2/lib/rsolr/client.rb:189:in `execute'
/home/app/current/vendor/gems/ruby/2.4.0/gems/rsolr-1.1.2/lib/rsolr/client.rb:175:in `send_and_receive'
(eval):2:in `post'
/home/app/current/vendor/gems/ruby/2.4.0/gems/rsolr-1.1.2/lib/rsolr/client.rb:84:in `update'
/home/app/current/vendor/gems/ruby/2.4.0/gems/rsolr-1.1.2/lib/rsolr/client.rb:104:in `add'
/home/app/current/vendor/gems/ruby/2.4.0/gems/active-fedora-11.5.2/lib/active_fedora/solr_service.rb:87:in `add'
/home/app/current/vendor/gems/ruby/2.4.0/gems/active-fedora-11.5.2/lib/active_fedora/indexing.rb:43:in `update_index'
/home/app/current/vendor/gems/ruby/2.4.0/gems/active-fedora-11.5.2/lib/active_fedora/indexing.rb:65:in `_create_record'
/home/app/current/vendor/gems/ruby/2.4.0/gems/active-fedora-11.5.2/lib/active_fedora/callbacks.rb:240:in `block in _create_record'
/home/app/current/vendor/gems/ruby/2.4.0/gems/activesupport-4.2.10/lib/active_support/callbacks.rb:117:in `call'

Resolution

1) ssh to bastion, then from there to each of the three solr nodes.

2) On each solr node, truncate /var/lib/docker/containers/*/*-json.log (I might have json and log backwards)

3) docker restart $(docker ps -q | head -1)

Slack Chat

https://nulib.slack.com/archives/G8YA2V3GR/p1543350723030600

[davidschober]

cc: @d-venckus @mbklein @Toputnal