search

sul-dlss-deprecated / rialto-etl

ETL tools for RIALTO, Stanford Libraries' research intelligence project
https://library.stanford.edu/projects/rialto
Apache License 2.0
3 stars 0 forks source link

Sparql loader concurrency error #142

Closed justinlittman closed 6 years ago

justinlittman commented 6 years ago

Steps to reproduce:

  1. Instantiate a Blazegraph instance and add appropriate setting to settings.local.yml.
  2. Transform researchers to SPARQL statements (I use a small sample): exe/transform call StanfordPeople -i researchers_sample.ndj > researchers_sample.sparql
  3. Load the SPARQL statements to Blazegraph: exe/load call Sparql -i researchers_sample.sparql

Expected result: SPARQL statements are executed against Blazegraph instance.

Actual result:

Traceback (most recent call last):
    20: from exe/load:7:in `<main>'
    19: from /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/base.rb:466:in `start'
    18: from /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor.rb:387:in `dispatch'
    17: from /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
    16: from /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
    15: from /Users/jlittman/data/rialto/rialto-etl/lib/rialto/etl/cli/load.rb:19:in `call'
    14: from /Users/jlittman/data/rialto/rialto-etl/lib/rialto/etl/loaders/sparql.rb:22:in `load'
    13: from /Users/jlittman/data/rialto/rialto-etl/lib/rialto/etl/loaders/sparql.rb:22:in `open'
    12: from /Users/jlittman/data/rialto/rialto-etl/lib/rialto/etl/loaders/sparql.rb:23:in `block in load'
    11: from /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/traject-2.3.4/lib/traject/indexer.rb:453:in `process'
    10: from /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/traject-2.3.4/lib/traject/thread_pool.rb:154:in `shutdown_and_wait'
     9: from /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_executor_service.rb:49:in `wait_for_termination'
     8: from /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/atomic/event.rb:84:in `wait'
     7: from /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
     6: from /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
     5: from /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize'
     4: from /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/atomic/event.rb:87:in `block in wait'
     3: from /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/abstract_lockable_object.rb:43:in `ns_wait_until'
     2: from /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:43:in `ns_wait'
     1: from /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:43:in `wait'
/Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:43:in `sleep': No live threads left. Deadlock? (fatal)
2 threads, 2 sleeps current:0x00007fa0a8b0de30 main thread:0x00007fa0a52003f0
* #<Thread:0x00007fa0a687dc98 sleep_forever>
   rb_thread_t:0x00007fa0a52003f0 native:0x00007fffac7ee380 int:0
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:43:in `sleep'
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:43:in `wait'
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:43:in `ns_wait'
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/abstract_lockable_object.rb:43:in `ns_wait_until'
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/atomic/event.rb:87:in `block in wait'
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize'
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/atomic/event.rb:84:in `wait'
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_executor_service.rb:49:in `wait_for_termination'
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/traject-2.3.4/lib/traject/thread_pool.rb:154:in `shutdown_and_wait'
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/traject-2.3.4/lib/traject/indexer.rb:453:in `process'
   /Users/jlittman/data/rialto/rialto-etl/lib/rialto/etl/loaders/sparql.rb:23:in `block in load'
   /Users/jlittman/data/rialto/rialto-etl/lib/rialto/etl/loaders/sparql.rb:22:in `open'
   /Users/jlittman/data/rialto/rialto-etl/lib/rialto/etl/loaders/sparql.rb:22:in `load'
   /Users/jlittman/data/rialto/rialto-etl/lib/rialto/etl/cli/load.rb:19:in `call'
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor.rb:387:in `dispatch'
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/base.rb:466:in `start'
   exe/load:7:in `<main>'
* #<Thread:0x00007fa0a493ab60@/Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:317 sleep_forever>
   rb_thread_t:0x00007fa0a8b0de30 native:0x000070000ae08000 int:0
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize'
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:106:in `worker_task_completed'
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:349:in `run_task'
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in `block (3 levels) in create_worker'
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `loop'
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (2 levels) in create_worker'
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `catch'
   /Users/jlittman/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `block in create_worker'

Notes:

aaron-collier commented 6 years ago

This may need to be renamed/clarified, as if this happens when pointing to a local blazegraph, the sparql loader isn't being used. At least what we're calling sparql-loader

jcoyne commented 6 years ago

I'm sure Justin is referring to Rialto::Etl::Loaders::Sparql