search

superphy / prairiedog

next-gen pangenome graphs for predictive genomics
Other
0 stars 0 forks source link

"Only leader can commit or abort" #107

Closed kevinkle closed 5 years ago

kevinkle commented 5 years ago 
2019-07-04 12:06:37 panther prairiedog[27905] DEBUG 900000/5043852, 17%
2019-07-04 12:06:40 panther prairiedog[28039] DEBUG 800000/5008903, 15%
[Thu Jul  4 12:07:00 2019]
2019-07-04 12:07:00 panther snakemake.logging[27694] INFO [Thu Jul  4 12:07:00 2019]
Error in rule pangenome:
2019-07-04 12:07:00 panther snakemake.logging[27694] ERROR Error in rule pangenome:
    jobid: 0
2019-07-04 12:07:00 panther snakemake.logging[27694] ERROR     jobid: 0
    output: outputs/pangenome_SRR3933051.g
2019-07-04 12:07:00 panther snakemake.logging[27694] ERROR     output: outputs/pangenome_SRR3933051.g

2019-07-04 12:07:00 panther snakemake.logging[27694] ERROR
[Thu Jul  4 12:07:00 2019]
2019-07-04 12:07:00 panther snakemake.logging[27816] INFO [Thu Jul  4 12:07:00 2019]
Error in rule pangenome:
2019-07-04 12:07:00 panther snakemake.logging[27816] ERROR Error in rule pangenome:
    jobid: 0
2019-07-04 12:07:00 panther snakemake.logging[27816] ERROR     jobid: 0
    output: outputs/pangenome_SRR3295607.g
2019-07-04 12:07:00 panther snakemake.logging[27816] ERROR     output: outputs/pangenome_SRR3295607.g

2019-07-04 12:07:00 panther snakemake.logging[27816] ERROR
RuleException:
_Rendezvous in line 90 of /home/kevin/prairiedog/Snakefile:
<_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Only leader can decide to commit or abort"
        debug_error_string = "{"created":"@1562263620.570892871","description":"Error received from peer ipv6:[::1]:9080","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Only leader can decide to commit or abort","grpc_status":2}"
>
  File "/home/kevin/prairiedog/Snakefile", line 90, in __rule_pangenome
  File "/home/kevin/prairiedog/prairiedog/subgraph_ref.py", line 75, in update_graph
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 64, in save
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 211, in commit
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 232, in _common_except_commit
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 199, in commit
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/client_stub.py", line 62, in commit_or_abort
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/grpc/_channel.py", line 565, in __call__
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/concurrent/futures/thread.py", line 57, in run
2019-07-04 12:07:00 panther snakemake.logging[27694] ERROR RuleException:
_Rendezvous in line 90 of /home/kevin/prairiedog/Snakefile:
<_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Only leader can decide to commit or abort"
        debug_error_string = "{"created":"@1562263620.570892871","description":"Error received from peer ipv6:[::1]:9080","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Only leader can decide to commit or abort","grpc_status":2}"
>
  File "/home/kevin/prairiedog/Snakefile", line 90, in __rule_pangenome
  File "/home/kevin/prairiedog/prairiedog/subgraph_ref.py", line 75, in update_graph
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 64, in save
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 211, in commit
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 232, in _common_except_commit
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 199, in commit
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/client_stub.py", line 62, in commit_or_abort
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/grpc/_channel.py", line 565, in __call__
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/concurrent/futures/thread.py", line 57, in run
RuleException:
_Rendezvous in line 90 of /home/kevin/prairiedog/Snakefile:
<_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Only leader can decide to commit or abort"
        debug_error_string = "{"created":"@1562263620.545207049","description":"Error received from peer ipv6:[::1]:9080","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Only leader can decide to commit or abort","grpc_status":2}"
>
  File "/home/kevin/prairiedog/Snakefile", line 90, in __rule_pangenome
  File "/home/kevin/prairiedog/prairiedog/subgraph_ref.py", line 75, in update_graph
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 64, in save
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 211, in commit
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 232, in _common_except_commit
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 199, in commit
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/client_stub.py", line 62, in commit_or_abort
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/grpc/_channel.py", line 565, in __call__
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/concurrent/futures/thread.py", line 57, in run
2019-07-04 12:07:00 panther snakemake.logging[27816] ERROR RuleException:
_Rendezvous in line 90 of /home/kevin/prairiedog/Snakefile:
<_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Only leader can decide to commit or abort"
        debug_error_string = "{"created":"@1562263620.545207049","description":"Error received from peer ipv6:[::1]:9080","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Only leader can decide to commit or abort","grpc_status":2}"
>
  File "/home/kevin/prairiedog/Snakefile", line 90, in __rule_pangenome
  File "/home/kevin/prairiedog/prairiedog/subgraph_ref.py", line 75, in update_graph
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 64, in save
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 211, in commit
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 232, in _common_except_commit
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 199, in commit
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/client_stub.py", line 62, in commit_or_abort
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/grpc/_channel.py", line 565, in __call__
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/concurrent/futures/thread.py", line 57, in run
Exiting because a job execution failed. Look above for error message
2019-07-04 12:07:00 panther snakemake.logging[27694] ERROR Exiting because a job execution failed. Look above for error message
Exiting because a job execution failed. Look above for error message
2019-07-04 12:07:00 panther snakemake.logging[27816] ERROR Exiting because a job execution failed. Look above for error message
2019-07-04 12:07:10 panther prairiedog[27667] DEBUG 1000000/5077560, 19%
2019-07-04 12:07:14 panther prairiedog[27721] DEBUG 1000000/5063372, 19%
2019-07-04 12:07:15 panther prairiedog[28071] DEBUG 800000/5008804, 15%
2019-07-04 12:07:16 panther prairiedog[27748] DEBUG 1000000/5134657, 19%
kevinkle commented 5 years ago

This only happens randomly when parallel. Looks like its not an antipattern to just retry the mutation: https://github.com/dgraph-io/pydgraph#commit-a-transaction

kevinkle commented 5 years ago 
2019-07-04 16:19:43 panther prairiedog[30295] DEBUG 2300000/5077560, 45%
2019-07-04 16:19:44 panther prairiedog[30289] DEBUG 2300000/5008804, 45%
2019-07-04 16:19:45 panther prairiedog[30306] DEBUG 2300000/5050119, 45%
2019-07-04 16:19:45 panther prairiedog[30299] DEBUG 2300000/5063372, 45%
2019-07-04 16:20:06 panther prairiedog[30303] DEBUG 2400000/5008903, 47%
2019-07-04 16:20:11 panther prairiedog[30293] DEBUG 2400000/5134657, 46%
2019-07-04 16:20:13 panther prairiedog[30301] DEBUG 2400000/5445017, 44%
2019-07-04 16:20:21 panther prairiedog[30299] DEBUG Ran into exception <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Only leader can decide to commit or abort"
        debug_error_string = "{"created":"@1562278821.627877366","description":"Error received from peer ipv6:[::1]:9080","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Only leader can decide to commit or abort","grpc_status":2}"
>, retrying 1/3...
2019-07-04 16:20:21 panther prairiedog[30295] DEBUG Ran into exception <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Only leader can decide to commit or abort"
        debug_error_string = "{"created":"@1562278821.643890888","description":"Error received from peer ipv6:[::1]:9080","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Only leader can decide to commit or abort","grpc_status":2}"
>, retrying 1/3...
2019-07-04 16:20:21 panther prairiedog[30301] DEBUG Ran into exception <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Only leader can decide to commit or abort"
        debug_error_string = "{"created":"@1562278821.611564480","description":"Error received from peer ipv6:[::1]:9080","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Only leader can decide to commit or abort","grpc_status":2}"
>, retrying 1/3...
2019-07-04 16:20:21 panther prairiedog[30306] DEBUG Ran into exception <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Only leader can decide to commit or abort"
        debug_error_string = "{"created":"@1562278821.659265597","description":"Error received from peer ipv6:[::1]:9080","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Only leader can decide to commit or abort","grpc_status":2}"
>, retrying 1/3...
2019-07-04 16:20:21 panther prairiedog[30293] DEBUG Ran into exception <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Only leader can decide to commit or abort"
        debug_error_string = "{"created":"@1562278821.679374250","description":"Error received from peer ipv6:[::1]:9080","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Only leader can decide to commit or abort","grpc_status":2}"
>, retrying 1/3...
2019-07-04 16:20:21 panther prairiedog[30289] DEBUG Ran into exception <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Only leader can decide to commit or abort"
        debug_error_string = "{"created":"@1562278821.692348042","description":"Error received from peer ipv6:[::1]:9080","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Only leader can decide to commit or abort","grpc_status":2}"
>, retrying 1/3...
2019-07-04 16:20:22 panther prairiedog[30301] DEBUG Trying mutation attempt 2/3
2019-07-04 16:20:22 panther prairiedog[30295] DEBUG Trying mutation attempt 2/3
2019-07-04 16:20:22 panther prairiedog[30301] DEBUG Exception type was <class 'Exception'>
2019-07-04 16:20:22 panther prairiedog[30299] DEBUG Trying mutation attempt 2/3
2019-07-04 16:20:22 panther prairiedog[30295] DEBUG Exception type was <class 'Exception'>
2019-07-04 16:20:22 panther prairiedog[30299] DEBUG Exception type was <class 'Exception'>
2019-07-04 16:20:22 panther prairiedog[30306] DEBUG Trying mutation attempt 2/3
2019-07-04 16:20:22 panther prairiedog[30306] DEBUG Exception type was <class 'Exception'>
[Thu Jul  4 16:20:22 2019]
2019-07-04 16:20:22 panther snakemake.logging[30295] INFO [Thu Jul  4 16:20:22 2019]
Error in rule pangenome:
2019-07-04 16:20:22 panther snakemake.logging[30295] ERROR Error in rule pangenome:
[Thu Jul  4 16:20:22 2019]
    jobid: 0
2019-07-04 16:20:22 panther snakemake.logging[30301] INFO [Thu Jul  4 16:20:22 2019]
2019-07-04 16:20:22 panther snakemake.logging[30295] ERROR     jobid: 0
Error in rule pangenome:
    output: outputs/pangenome_SRR3664992.g
2019-07-04 16:20:22 panther snakemake.logging[30301] ERROR Error in rule pangenome:
2019-07-04 16:20:22 panther snakemake.logging[30295] ERROR     output: outputs/pangenome_SRR3664992.g
    jobid: 0

2019-07-04 16:20:22 panther snakemake.logging[30301] ERROR     jobid: 0
    output: outputs/pangenome_SRR3933051.g
2019-07-04 16:20:22 panther snakemake.logging[30295] ERROR
2019-07-04 16:20:22 panther snakemake.logging[30301] ERROR     output: outputs/pangenome_SRR3933051.g

2019-07-04 16:20:22 panther snakemake.logging[30301] ERROR
RuleException:
Exception in line 100 of /home/kevin/prairiedog/Snakefile:
Transaction has already been committed or discarded
  File "/home/kevin/prairiedog/Snakefile", line 100, in __rule_pangenome
  File "/home/kevin/prairiedog/prairiedog/subgraph_ref.py", line 88, in update_graph
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 106, in save
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 95, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 98, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 82, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 111, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 157, in _common_mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/concurrent/futures/thread.py", line 57, in run
2019-07-04 16:20:22 panther snakemake.logging[30301] ERROR RuleException:
Exception in line 100 of /home/kevin/prairiedog/Snakefile:
Transaction has already been committed or discarded
  File "/home/kevin/prairiedog/Snakefile", line 100, in __rule_pangenome
  File "/home/kevin/prairiedog/prairiedog/subgraph_ref.py", line 88, in update_graph
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 106, in save
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 95, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 98, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 82, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 111, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 157, in _common_mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/concurrent/futures/thread.py", line 57, in run
RuleException:
Exception in line 100 of /home/kevin/prairiedog/Snakefile:
Transaction has already been committed or discarded
  File "/home/kevin/prairiedog/Snakefile", line 100, in __rule_pangenome
  File "/home/kevin/prairiedog/prairiedog/subgraph_ref.py", line 88, in update_graph
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 106, in save
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 95, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 98, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 82, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 111, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 157, in _common_mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/concurrent/futures/thread.py", line 57, in run
[Thu Jul  4 16:20:22 2019]
2019-07-04 16:20:22 panther snakemake.logging[30295] ERROR RuleException:
Exception in line 100 of /home/kevin/prairiedog/Snakefile:
Transaction has already been committed or discarded
  File "/home/kevin/prairiedog/Snakefile", line 100, in __rule_pangenome
  File "/home/kevin/prairiedog/prairiedog/subgraph_ref.py", line 88, in update_graph
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 106, in save
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 95, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 98, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 82, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 111, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 157, in _common_mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/concurrent/futures/thread.py", line 57, in run
2019-07-04 16:20:22 panther snakemake.logging[30299] INFO [Thu Jul  4 16:20:22 2019]
Error in rule pangenome:
2019-07-04 16:20:22 panther snakemake.logging[30299] ERROR Error in rule pangenome:
    jobid: 0
2019-07-04 16:20:22 panther snakemake.logging[30299] ERROR     jobid: 0
    output: outputs/pangenome_SRR3664772.g
2019-07-04 16:20:22 panther snakemake.logging[30299] ERROR     output: outputs/pangenome_SRR3664772.g
Exiting because a job execution failed. Look above for error message

Exiting because a job execution failed. Look above for error message
2019-07-04 16:20:22 panther snakemake.logging[30299] ERROR
2019-07-04 16:20:22 panther snakemake.logging[30301] ERROR Exiting because a job execution failed. Look above for error message
2019-07-04 16:20:22 panther snakemake.logging[30295] ERROR Exiting because a job execution failed. Look above for error message
[Thu Jul  4 16:20:22 2019]
2019-07-04 16:20:22 panther snakemake.logging[30306] INFO [Thu Jul  4 16:20:22 2019]
Error in rule pangenome:
2019-07-04 16:20:22 panther snakemake.logging[30306] ERROR Error in rule pangenome:
    jobid: 0
2019-07-04 16:20:22 panther snakemake.logging[30306] ERROR     jobid: 0
    output: outputs/pangenome_SRR3295607.g
2019-07-04 16:20:22 panther snakemake.logging[30306] ERROR     output: outputs/pangenome_SRR3295607.g
RuleException:
Exception in line 100 of /home/kevin/prairiedog/Snakefile:
Transaction has already been committed or discarded
  File "/home/kevin/prairiedog/Snakefile", line 100, in __rule_pangenome
  File "/home/kevin/prairiedog/prairiedog/subgraph_ref.py", line 88, in update_graph
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 106, in save
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 95, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 98, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 82, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 111, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 157, in _common_mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/concurrent/futures/thread.py", line 57, in run

2019-07-04 16:20:22 panther snakemake.logging[30306] ERROR
2019-07-04 16:20:22 panther snakemake.logging[30299] ERROR RuleException:
Exception in line 100 of /home/kevin/prairiedog/Snakefile:
Transaction has already been committed or discarded
  File "/home/kevin/prairiedog/Snakefile", line 100, in __rule_pangenome
  File "/home/kevin/prairiedog/prairiedog/subgraph_ref.py", line 88, in update_graph
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 106, in save
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 95, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 98, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 82, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 111, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 157, in _common_mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/concurrent/futures/thread.py", line 57, in run
RuleException:
Exception in line 100 of /home/kevin/prairiedog/Snakefile:
Transaction has already been committed or discarded
  File "/home/kevin/prairiedog/Snakefile", line 100, in __rule_pangenome
  File "/home/kevin/prairiedog/prairiedog/subgraph_ref.py", line 88, in update_graph
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 106, in save
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 95, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 98, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 82, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 111, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 157, in _common_mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/concurrent/futures/thread.py", line 57, in run
2019-07-04 16:20:22 panther snakemake.logging[30306] ERROR RuleException:
Exception in line 100 of /home/kevin/prairiedog/Snakefile:
Transaction has already been committed or discarded
  File "/home/kevin/prairiedog/Snakefile", line 100, in __rule_pangenome
  File "/home/kevin/prairiedog/prairiedog/subgraph_ref.py", line 88, in update_graph
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 106, in save
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 95, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 98, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 82, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 111, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 157, in _common_mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/concurrent/futures/thread.py", line 57, in run
Exiting because a job execution failed. Look above for error message
2019-07-04 16:20:22 panther snakemake.logging[30299] ERROR Exiting because a job execution failed. Look above for error message
Exiting because a job execution failed. Look above for error message
2019-07-04 16:20:22 panther snakemake.logging[30306] ERROR Exiting because a job execution failed. Look above for error message
2019-07-04 16:20:22 panther prairiedog[30293] DEBUG Trying mutation attempt 2/3
2019-07-04 16:20:22 panther prairiedog[30293] DEBUG Exception type was <class 'Exception'>
[Thu Jul  4 16:20:22 2019]
2019-07-04 16:20:22 panther snakemake.logging[30293] INFO [Thu Jul  4 16:20:22 2019]
Error in rule pangenome:
2019-07-04 16:20:22 panther snakemake.logging[30293] ERROR Error in rule pangenome:
    jobid: 0
2019-07-04 16:20:22 panther snakemake.logging[30293] ERROR     jobid: 0
    output: outputs/pangenome_SRR3295582.g
2019-07-04 16:20:22 panther snakemake.logging[30293] ERROR     output: outputs/pangenome_SRR3295582.g

2019-07-04 16:20:22 panther snakemake.logging[30293] ERROR
RuleException:
Exception in line 100 of /home/kevin/prairiedog/Snakefile:
Transaction has already been committed or discarded
  File "/home/kevin/prairiedog/Snakefile", line 100, in __rule_pangenome
  File "/home/kevin/prairiedog/prairiedog/subgraph_ref.py", line 88, in update_graph
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 106, in save
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 95, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 98, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 82, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 111, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 157, in _common_mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/concurrent/futures/thread.py", line 57, in run
2019-07-04 16:20:22 panther snakemake.logging[30293] ERROR RuleException:
Exception in line 100 of /home/kevin/prairiedog/Snakefile:
Transaction has already been committed or discarded
  File "/home/kevin/prairiedog/Snakefile", line 100, in __rule_pangenome
  File "/home/kevin/prairiedog/prairiedog/subgraph_ref.py", line 88, in update_graph
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 106, in save
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 95, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 98, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 82, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 111, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 157, in _common_mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/concurrent/futures/thread.py", line 57, in run
2019-07-04 16:20:22 panther prairiedog[30289] DEBUG Trying mutation attempt 2/3
Exiting because a job execution failed. Look above for error message
2019-07-04 16:20:22 panther prairiedog[30289] DEBUG Exception type was <class 'Exception'>
2019-07-04 16:20:22 panther snakemake.logging[30293] ERROR Exiting because a job execution failed. Look above for error message
[Thu Jul  4 16:20:22 2019]
2019-07-04 16:20:22 panther snakemake.logging[30289] INFO [Thu Jul  4 16:20:22 2019]
Error in rule pangenome:
2019-07-04 16:20:22 panther snakemake.logging[30289] ERROR Error in rule pangenome:
    jobid: 0
2019-07-04 16:20:22 panther snakemake.logging[30289] ERROR     jobid: 0
    output: outputs/pangenome_SRR3664817.g
2019-07-04 16:20:22 panther snakemake.logging[30289] ERROR     output: outputs/pangenome_SRR3664817.g

2019-07-04 16:20:22 panther snakemake.logging[30289] ERROR
RuleException:
Exception in line 100 of /home/kevin/prairiedog/Snakefile:
Transaction has already been committed or discarded
  File "/home/kevin/prairiedog/Snakefile", line 100, in __rule_pangenome
  File "/home/kevin/prairiedog/prairiedog/subgraph_ref.py", line 88, in update_graph
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 106, in save
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 95, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 98, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 82, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 111, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 157, in _common_mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/concurrent/futures/thread.py", line 57, in run
2019-07-04 16:20:22 panther snakemake.logging[30289] ERROR RuleException:
Exception in line 100 of /home/kevin/prairiedog/Snakefile:
Transaction has already been committed or discarded
  File "/home/kevin/prairiedog/Snakefile", line 100, in __rule_pangenome
  File "/home/kevin/prairiedog/prairiedog/subgraph_ref.py", line 88, in update_graph
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 106, in save
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 95, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 98, in mutate
  File "/home/kevin/prairiedog/prairiedog/dgraph.py", line 82, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 111, in mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pydgraph/txn.py", line 157, in _common_mutate
  File "/home/kevin/.pyenv/versions/3.7.3/lib/python3.7/concurrent/futures/thread.py", line 57, in run
Exiting because a job execution failed. Look above for error message
2019-07-04 16:20:22 panther snakemake.logging[30289] ERROR Exiting because a job execution failed. Look above for error message
^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A2019-07-04 16:20:34 panther prairiedog[30285] DEBUG 2400000/5043852, 47%

with commit https://github.com/superphy/prairiedog/commit/9417252701a767a796032df7f7681a0c5a71982d

kevinkle commented 5 years ago

The 1 txn per function call fixed the retry, was probably some weird thing going on with storing the txn in state.

2019-07-05 09:22:48 panther prairiedog[8242] DEBUG 1000000/5050119, 19%
2019-07-05 09:22:48 panther prairiedog[8236] DEBUG 1000000/5008903, 19%
2019-07-05 09:22:48 panther prairiedog[8238] DEBUG 1000000/5063372, 19%
2019-07-05 09:22:48 panther prairiedog[8245] DEBUG 1000000/5008804, 19%
2019-07-05 09:22:49 panther prairiedog[8253] DEBUG 1000000/5134657, 19%
2019-07-05 09:23:14 panther prairiedog[8251] DEBUG Ran into exception <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Only leader can decide to commit or abort"
        debug_error_string = "{"created":"@1562340194.792058266","description":"Error received from peer ipv6:[::1]:9080","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Only leader can decide to commit or abort","grpc_status":2}"
>, retrying 1/3...
2019-07-05 09:23:14 panther prairiedog[8245] DEBUG Ran into exception <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Only leader can decide to commit or abort"
        debug_error_string = "{"created":"@1562340194.813666085","description":"Error received from peer ipv6:[::1]:9080","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Only leader can decide to commit or abort","grpc_status":2}"
>, retrying 1/3...
2019-07-05 09:23:14 panther prairiedog[8253] DEBUG Ran into exception <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Only leader can decide to commit or abort"
        debug_error_string = "{"created":"@1562340194.829032331","description":"Error received from peer ipv6:[::1]:9080","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Only leader can decide to commit or abort","grpc_status":2}"
>, retrying 1/3...
2019-07-05 09:23:14 panther prairiedog[8247] DEBUG Ran into exception <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Only leader can decide to commit or abort"
        debug_error_string = "{"created":"@1562340194.847250170","description":"Error received from peer ipv6:[::1]:9080","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Only leader can decide to commit or abort","grpc_status":2}"
>, retrying 1/3...
2019-07-05 09:23:16 panther prairiedog[8251] DEBUG Trying mutation attempt 2/3
2019-07-05 09:23:16 panther prairiedog[8245] DEBUG Trying mutation attempt 2/3
2019-07-05 09:23:16 panther prairiedog[8253] DEBUG Trying mutation attempt 2/3
2019-07-05 09:23:16 panther prairiedog[8247] DEBUG Trying mutation attempt 2/3
2019-07-05 09:23:22 panther prairiedog[8251] DEBUG Attempt 2/3 completed successfully
2019-07-05 09:23:22 panther prairiedog[8247] DEBUG Attempt 2/3 completed successfully
2019-07-05 09:23:22 panther prairiedog[8253] DEBUG Attempt 2/3 completed successfully
2019-07-05 09:23:22 panther prairiedog[8245] DEBUG Attempt 2/3 completed successfully
2019-07-05 09:24:35 panther prairiedog[8247] DEBUG Ran into exception <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNKNOWN
        details = "Only leader can decide to commit or abort"
        debug_error_string = "{"created":"@1562340275.127731556","description":"Error received from peer ipv6:[::1]:9080","file":"src/core/lib/surface/call.cc","file_line":1052,"grpc_message":"Only leader can decide to commit or abort","grpc_status":2}"
>, retrying 1/3...
2019-07-05 09:24:37 panther prairiedog[8247] DEBUG Trying mutation attempt 2/3
2019-07-05 09:24:40 panther prairiedog[8247] DEBUG Attempt 2/3 completed successfully

considering this fixed with commit https://github.com/superphy/prairiedog/commit/97187fa24c5043e74df43d3fee8317a821bd7c14