Deposit of file through UI fails

[seanluyk]

The object in question: https://era-av.library.ualberta.ca/media_objects/fq977v77s

1.7 GB item uploaded through the UI fails: ""Incomplete response received from application"

File name: DSC-wiki-speaker-series-accountability-cut.mp4. Attempted the morning of October 1st, around 10:30-11

[jefferya]

@seanluyk investigating. My working hypothesis is this is related to other UI upload problem.

[seanluyk]

Thanks @jefferya!

[jefferya]

@seanluyk: could you add the name of the file used when the "incomplete response received from application" error was triggered? Looking through the ERA A+V log, I'm unable to map the media object id, fq977v77s to a UI upload attempt (failed or incomplete).

Also, approx time, if possible.

[jefferya]

Hypothesis: related to #676 - Apache logs record no space left on device errors (details below)

Checked the following:

• rollbar: no problem reported
• Tested within a dev env: no error; upload successful
• cursory inspection of the /var/www/avalon/log/production.log: no obvious error but unable to map the media object id to a master file upload and uncertain of the time the error occured
• /var/log/httpd/avalon6_error.log: no obvious related error in the past 5 days

• `/var/log/httpd/error_log:

  • App 24374 output: [ 2020-10-01 11:10:09.3938 24374/0x00000000040ed9c8(Worker 1) utils.rb ]: *** Exception Errno::ENOSPC in Rack application object (No space left on device @ io_write - /tmp/PassengerTeeInput-21stu8) (process 24374, thread 0x00000000040ed9c8(Worker 1)):
  • [ W 2020-10-01 11:10:09.5566 24121/Ta age/Cor/Con/InternalUtils.cpp:96 ]: [Client 2-4203] Sending 502 response: application did not send a complete response App 24374 output: [ 2020-10-01 11:39:24.3596 24374/0x00000000040ed9c8(Worker 1) utils.rb ]: *** Exception Errno::ENOSPC in Rack application object (No space left on device @ io_write - /tmp/PassengerTeeInput-c81obi) (process 24374, thread 0x00000000040ed9c8(Worker 1))
  • [ W 2020-10-01 11:39:24.5211 24121/Td age/Cor/Con/InternalUtils.cpp:96 ]: [Client 3-4233] Sending 502 response: application did not send a complete response App 24374 output: [ 2020-10-01 12:15:22.3766 24374/0x00000000040ed9c8(Worker 1) utils.rb ]: *** Exception Errno::ENOSPC in Rack application object (No space left on device @ io_write - /tmp/RackMultipart20201001-24374-63n7pq.mp4) (process 24374, thread 0x00000000040ed9c8(Worker 1)): App 24374 output: from /usr/lib64/ruby/2.5.0/delegate.rb:349:in write' App 24374 output: from /usr/lib64/ruby/2.5.0/delegate.rb:349:in<<' App 24374 output: from /usr/lib64/ruby/2.5.0/delegate.rb:349:in block in delegating_block' App 24374 output: from /var/www/avalon6/vendor/bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/multipart/parser.rb:147:inon_mime_body' App 24374 output: from /var/www/avalon6/vendor/bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/multipart/parser.rb:273:in handle_mime_body' App 24374 output: from /var/www/avalon6/vendor/bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/multipart/parser.rb:213:inblock in run_parser' App 24374 output: from /var/www/avalon6/vendor/bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/multipart/parser.rb:204:in loop' App 24374 output: from /var/www/avalon6/vendor/bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/multipart/parser.rb:204:inrun_parser' App 24374 output: from /var/www/avalon6/vendor/bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/multipart/parser.rb:188:in on_read' App 24374 output: from /var/www/avalon6/vendor/bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/multipart/parser.rb:77:inblock in parse' App 24374 output: from /var/www/avalon6/vendor/bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/multipart/parser.rb:75:in loop' App 24374 output: from /var/www/avalon6/vendor/bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/multipart/parser.rb:75:inparse' App 24374 output: from /var/www/avalon6/vendor/bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/multipart.rb:54:in extract_multipart' App 24374 output: from /var/www/avalon6/vendor/bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/request.rb:594:inparse_multipart' App 24374 output: from /var/www/avalon6/vendor/bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/request.rb:446:in POST' App 24374 output: from /var/www/avalon6/vendor/bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/method_override.rb:45:inmethod_override_param' App 24374 output: from /var/www/avalon6/vendor/bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/method_override.rb:29:in method_override' App 24374 output: from /var/www/avalon6/vendor/bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/method_override.rb:17:incall' App 24374 output: from /var/www/avalon6/vendor/bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/runtime.rb:22:in call' App 24374 output: from /var/www/avalon6/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/cache/strategy/local_cache_middleware.rb:29:incall' App 24374 output: from /var/www/avalon6/vendor/bundle/ruby/2.5.0/gems/actionpack-5.2.3/lib/action_dispatch/middleware/executor.rb:14:in call' App 24374 output: from /var/www/avalon6/vendor/bundle/ruby/2.5.0/gems/rack-2.2.3/lib/rack/sendfile.rb:110:incall' App 24374 output: from /var/www/avalon6/vendor/bundle/ruby/2.5.0/gems/rack-cors-1.0.6/lib/rack/cors.rb:98:in call' App 24374 output: from /var/www/avalon6/vendor/bundle/ruby/2.5.0/gems/railties-5.2.3/lib/rails/engine.rb:524:incall' App 24374 output: from /usr/share/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in process_request' App 24374 output: from /usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:inaccept_and_process_next_request' App 24374 output: from /usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in main_loop' App 24374 output: from /usr/share/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:inblock (3 levels) in start_threads' App 24374 output: from /usr/share/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'

Reference: sudo cat /var/log/httpd/error_log | grep -v 'after_action' | grep -v 'If you wish' | grep -v 'DEPRECATION' | grep -v 'each_line' | grep -v 'client denied by server'

[seanluyk]

@jefferya I've added some more details to this issue as requested

[jefferya]

workaround: use drop box instead of UI; side-effect, easier to preserve original media