search

datadryad / dryad-product-roadmap

Repository of issues for Dryad project boards
https://github.com/orgs/datadryad/projects
8 stars 0 forks source link

Timeouts when zenodo goes unresponsive #1290

Closed sfisher closed 2 years ago

sfisher commented 3 years ago

I see messages like this one in the delayed job workers.

execution expired (Delayed::Worker.max_run_time is only 21600 seconds)
/apps/dryad/apps/ui/releases/20210528004433/stash/stash_engine/lib/stash/zenodo_software/streamer.rb:65:in `write'
/apps/dryad/apps/ui/releases/20210528004433/stash/stash_engine/lib/stash/zenodo_software/streamer.rb:65:in `block in stream'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/http-4.4.1/lib/http/response/body.rb:37:in `each'
/apps/dryad/apps/ui/releases/20210528004433/stash/stash_engine/lib/stash/zenodo_software/streamer.rb:62:in `stream'
/apps/dryad/apps/ui/releases/20210528004433/stash/stash_engine/lib/stash/zenodo_software/file_collection.rb:44:in `block in upload_files'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activerecord-5.2.6/lib/active_record/relation/delegation.rb:71:in `each'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activerecord-5.2.6/lib/active_record/relation/delegation.rb:71:in `each'
/apps/dryad/apps/ui/releases/20210528004433/stash/stash_engine/lib/stash/zenodo_software/file_collection.rb:34:in `upload_files'
/apps/dryad/apps/ui/releases/20210528004433/stash/stash_engine/lib/stash/zenodo_software/file_collection.rb:23:in `synchronize_to_zenodo'
/apps/dryad/apps/ui/releases/20210528004433/stash/stash_engine/lib/stash/zenodo_replicate/copier.rb:71:in `add_to_zenodo'
/apps/dryad/apps/ui/releases/20210528004433/stash/stash_engine/app/jobs/stash_engine/zenodo_copy_job.rb:27:in `perform'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activejob-5.2.6/lib/active_job/execution.rb:39:in `block in perform_now'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activesupport-5.2.6/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/i18n-1.8.10/lib/i18n.rb:314:in `with_locale'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activejob-5.2.6/lib/active_job/translation.rb:9:in `block (2 levels) in <module:Translation>'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activesupport-5.2.6/lib/active_support/callbacks.rb:118:in `instance_exec'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activesupport-5.2.6/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activejob-5.2.6/lib/active_job/logging.rb:26:in `block (4 levels) in <module:Logging>'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activesupport-5.2.6/lib/active_support/notifications.rb:168:in `block in instrument'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activesupport-5.2.6/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activesupport-5.2.6/lib/active_support/notifications.rb:168:in `instrument'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activejob-5.2.6/lib/active_job/logging.rb:25:in `block (3 levels) in <module:Logging>'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activejob-5.2.6/lib/active_job/logging.rb:46:in `block in tag_logger'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activesupport-5.2.6/lib/active_support/tagged_logging.rb:71:in `block in tagged'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activesupport-5.2.6/lib/active_support/tagged_logging.rb:28:in `tagged'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activesupport-5.2.6/lib/active_support/tagged_logging.rb:71:in `tagged'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activejob-5.2.6/lib/active_job/logging.rb:46:in `tag_logger'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activejob-5.2.6/lib/active_job/logging.rb:22:in `block (2 levels) in <module:Logging>'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activesupport-5.2.6/lib/active_support/callbacks.rb:118:in `instance_exec'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activesupport-5.2.6/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activesupport-5.2.6/lib/active_support/callbacks.rb:136:in `run_callbacks'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activejob-5.2.6/lib/active_job/execution.rb:38:in `perform_now'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activejob-5.2.6/lib/active_job/execution.rb:24:in `block in execute'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activesupport-5.2.6/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activejob-5.2.6/lib/active_job/railtie.rb:28:in `block (4 levels) in <class:Railtie>'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activesupport-5.2.6/lib/active_support/execution_wrapper.rb:87:in `wrap'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activesupport-5.2.6/lib/active_support/reloader.rb:73:in `block in wrap'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activesupport-5.2.6/lib/active_support/execution_wrapper.rb:87:in `wrap'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activesupport-5.2.6/lib/active_support/reloader.rb:72:in `wrap'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activejob-5.2.6/lib/active_job/railtie.rb:27:in `block (3 levels) in <class:Railtie>'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activesupport-5.2.6/lib/active_support/callbacks.rb:118:in `instance_exec'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activesupport-5.2.6/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activesupport-5.2.6/lib/active_support/callbacks.rb:136:in `run_callbacks'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activejob-5.2.6/lib/active_job/execution.rb:22:in `execute'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/activejob-5.2.6/lib/active_job/queue_adapters/delayed_job_adapter.rb:42:in `perform'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/backend/base.rb:81:in `block in invoke_job'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:61:in `block in initialize'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:66:in `execute'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/backend/base.rb:78:in `invoke_job'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:230:in `block (2 levels) in run'
/apps/dryad/local/lib/ruby/2.6.0/timeout.rb:103:in `timeout'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:230:in `block in run'
/apps/dryad/local/lib/ruby/2.6.0/benchmark.rb:308:in `realtime'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:229:in `run'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:312:in `block in reserve_and_run_one_job'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:61:in `block in initialize'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:66:in `execute'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:312:in `reserve_and_run_one_job'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:213:in `block in work_off'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:212:in `times'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:212:in `work_off'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:175:in `block (4 levels) in start'
/apps/dryad/local/lib/ruby/2.6.0/benchmark.rb:308:in `realtime'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:174:in `block (3 levels) in start'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:61:in `block in initialize'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:66:in `execute'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:173:in `block (2 levels) in start'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:172:in `loop'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:172:in `block in start'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/plugins/clear_locks.rb:7:in `block (2 levels) in <class:ClearLocks>'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:79:in `block (2 levels) in add'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:61:in `block in initialize'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:79:in `block in add'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:66:in `execute'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/worker.rb:171:in `start'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/command.rb:137:in `run'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/command.rb:125:in `block in run_process'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/daemons-1.4.0/lib/daemons/application.rb:277:in `block in start_proc'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/daemons-1.4.0/lib/daemons/daemonize.rb:84:in `call_as_daemon'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/daemons-1.4.0/lib/daemons/application.rb:281:in `start_proc'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/daemons-1.4.0/lib/daemons/application.rb:307:in `start'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/daemons-1.4.0/lib/daemons/controller.rb:56:in `run'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/daemons-1.4.0/lib/daemons.rb:199:in `block in run_proc'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/daemons-1.4.0/lib/daemons/cmdline.rb:124:in `catch_exceptions'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/daemons-1.4.0/lib/daemons.rb:198:in `run_proc'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/command.rb:123:in `run_process'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/command.rb:104:in `block in daemonize'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/command.rb:102:in `times'
/apps/dryad/apps/ui/shared/bundle/ruby/2.6.0/gems/delayed_job-4.1.9/lib/delayed/command.rb:102:in `daemonize'
bin/delayed_job:5:in `<top (required)>'
/apps/dryad/local/lib/ruby/site_ruby/2.6.0/bundler/cli/exec.rb:63:in `load'
/apps/dryad/local/lib/ruby/site_ruby/2.6.0/bundler/cli/exec.rb:63:in `kernel_load'
/apps/dryad/local/lib/ruby/site_ruby/2.6.0/bundler/cli/exec.rb:28:in `run'
/apps/dryad/local/lib/ruby/site_ruby/2.6.0/bundler/cli.rb:476:in `exec'
/apps/dryad/local/lib/ruby/site_ruby/2.6.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/apps/dryad/local/lib/ruby/site_ruby/2.6.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/apps/dryad/local/lib/ruby/site_ruby/2.6.0/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/apps/dryad/local/lib/ruby/site_ruby/2.6.0/bundler/cli.rb:30:in `dispatch'
/apps/dryad/local/lib/ruby/site_ruby/2.6.0/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/apps/dryad/local/lib/ruby/site_ruby/2.6.0/bundler/cli.rb:24:in `start'
/apps/dryad/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/exe/bundle:46:in `block in <top (required)>'
/apps/dryad/local/lib/ruby/site_ruby/2.6.0/bundler/friendly_errors.rb:123:in `with_friendly_errors'
/apps/dryad/local/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/exe/bundle:34:in `<top (required)>'
/dryad/local/bin/bundle:23:in `load'
/dryad/local/bin/bundle:23:in `<main>'
sfisher commented 2 years ago

In ticket #1791 because of timeouts of delayed job and still timing out for large items at 1 day.