search

fluidex / fluidex-backend

Building the first permissionless layer2 orderbook DEX on Ethereum, powered by PLONK zk-rollup
https://www.fluidex.io/
38 stars 8 forks source link

restart with `export DX_CLEAN=false` may fail #84

Open 0xmountaintop opened 3 years ago

0xmountaintop commented 3 years ago

description

restart with export DX_CLEAN=false may fail for

how to reproduce

export DX_CLEAN=false
./stopsh
./run.sh

log

ubuntu@ip-172-31-18-140:~/repos/fluidex-backend$ make tail_log 
docker-compose --file orchestra/docker/docker-compose.yaml --project-name orchestra logs > orchestra/docker-compose.log
docker-compose --file regnbue-bridge/docker/docker-compose.yaml --project-name faucet logs > regnbue-bridge/docker-compose.log
ls rollup-state-manager/*.log prover-cluster/*.log dingir-exchange/*.log dingir-exchange/logs/*.log regnbue-bridge/*.log contracts/*.log orchestra/*.log | xargs tail -n 3
==> contracts/ganache.2021-09-23.log <==
9007199254740991

Listening on 127.0.0.1:8545

==> contracts/ticker.2021-09-23.log <==
        at emitErrorNT (node:internal/streams/destroy:193:8)
        at emitErrorCloseNT (node:internal/streams/destroy:158:3)
        at processTicksAndRejections (node:internal/process/task_queues:83:21)

==> dingir-exchange/logs/matchengine.2021-09-23.log <==

Sep 23 04:10:38.953  INFO dingir_exchange::message::producer: kafka producer disconnected    
Sep 23 04:10:38.953  INFO dingir_exchange::message::producer: kafka producer running terminated    

==> dingir-exchange/logs/persistor.2021-09-23.log <==
Sep 23 04:10:40.281  INFO dingir_exchange::message::persist: start auto commiting for topic internaltransfer    
Sep 23 04:10:40.281  INFO dingir_exchange::message::persist: start auto commiting for topic registeruser    
Sep 23 04:10:41.301  INFO dingir_exchange::message::consumer: start consuming topic ["trades", "internaltransfer", "registeruser", "balances", "orders"]    

==> dingir-exchange/logs/restapi.2021-09-23.log <==
  44: __libc_start_main
  45: _start

==> dingir-exchange/tick.2021-09-23.log <==
  details: 'No connection established',
  metadata: Metadata { internalRepr: Map(0) {}, options: {} }
}

==> orchestra/docker-compose.log <==
exchange_envoy        | [2021-09-23 04:19:48.260][19][debug][connection] [source/common/network/connection_impl.cc:243] [C49] closing socket: 0
exchange_envoy        | [2021-09-23 04:19:48.260][19][debug][client] [source/common/http/codec_client.cc:107] [C49] disconnect. resetting 0 pending requests
exchange_envoy        | [2021-09-23 04:19:48.260][19][debug][pool] [source/common/conn_pool/conn_pool_base.cc:407] [C49] client disconnected, failure reason: 

==> prover-cluster/client.2021-09-23.log <==
  15: __libc_start_main
  16: _start

==> prover-cluster/coordinator.2021-09-23.log <==
  status = $1
  and updated_time < current_timestamp - interval '172800 seconds'

==> regnbue-bridge/block_submitter.2021-09-23.log <==
    0: error trying to connect: tcp connect error: Connection refused (os error 111)
    1: tcp connect error: Connection refused (os error 111)
    2: Connection refused (os error 111)

==> regnbue-bridge/docker-compose.log <==
regnbue_bridge_pq | 2021-09-23 04:10:26.580 UTC [28] LOG:  TimescaleDB background worker launcher connected to shared catalogs
regnbue_bridge_pq | 2021-09-23 04:11:31.098 UTC [51] LOG:  the "timescaledb" extension is not up-to-date
regnbue_bridge_pq | 2021-09-23 04:11:31.098 UTC [51] HINT:  The most up-to-date version is 2.4.2, the installed version is 2.1.0.

==> regnbue-bridge/faucet.2021-09-23.log <==

Sep 23 04:19:50.229  INFO sqlx::query: COMMIT; rows: 0, elapsed: 93.348µs  
Sep 23 04:19:50.230  INFO sqlx::query: /* SQLx ping */; rows: 0, elapsed: 83.309µs  

==> rollup-state-manager/rollup_state_manager.2021-09-23.log <==
Sep 23 04:15:22.823  INFO rollup_state_manager: generate 0 blocks with block_size 2 in 240.00372s: average TPS: 0    
Sep 23 04:17:22.823  INFO rollup_state_manager: generate 0 blocks with block_size 2 in 360.00385s: average TPS: 0    
Sep 23 04:19:22.823  INFO rollup_state_manager: generate 0 blocks with block_size 2 in 480.00394s: average TPS: 0    
ubuntu@ip-172-31-18-140:~/repos/fluidex-backend$ tail -n 300 dingir-exchange/logs/restapi.2021-09-23.log
Sep 23 04:10:40.256 DEBUG restapi: Prepared DB connection: postgres://exchange:exchange_AA9944@127.0.0.1/exchange    
Sep 23 04:10:40.257  INFO restapi: Connect to manage channel http://0.0.0.0:50051    
Sep 23 04:10:41.666 ERROR fluidex_common::non_blocking_tracing: thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })))', src/bin/restapi.rs:34:18
   0: fluidex_common::non_blocking_tracing::get_backtrace
             at /home/ubuntu/.cargo/git/checkouts/common-rs-f10c3d305ff6aa0a/24293b3/src/non_blocking_tracing.rs:50:30
   1: fluidex_common::non_blocking_tracing::set_panic_hook::{{closure}}
             at /home/ubuntu/.cargo/git/checkouts/common-rs-f10c3d305ff6aa0a/24293b3/src/non_blocking_tracing.rs:39:13
   2: std::panicking::rust_panic_with_hook
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/panicking.rs:595:17
   3: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/panicking.rs:497:13
   4: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/sys_common/backtrace.rs:141:18
   5: rust_begin_unwind
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/panicking.rs:493:5
   6: core::panicking::panic_fmt
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/panicking.rs:92:14
   7: core::result::unwrap_failed
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/result.rs:1355:5
   8: core::result::Result<T,E>::unwrap
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/result.rs:1037:23
   9: restapi::main::{{closure}}
             at src/bin/restapi.rs:29:13
  10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/future/mod.rs:80:19
  11: <tokio::task::local::RunUntil<T> as core::future::future::Future>::poll::{{closure}}::{{closure}}
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/task/local.rs:668:65
  12: tokio::coop::with_budget::{{closure}}
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/coop.rs:106:9
  13: std::thread::local::LocalKey<T>::try_with
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/thread/local.rs:376:16
  14: std::thread::local::LocalKey<T>::with
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/thread/local.rs:352:9
  15: tokio::coop::with_budget
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/coop.rs:99:5
      tokio::coop::budget
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/coop.rs:76:5
      <tokio::task::local::RunUntil<T> as core::future::future::Future>::poll::{{closure}}
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/task/local.rs:668:42
  16: tokio::macros::scoped_tls::ScopedKey<T>::set
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/macros/scoped_tls.rs:61:9
  17: tokio::task::local::LocalSet::with
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/task/local.rs:573:9
  18: <tokio::task::local::RunUntil<T> as core::future::future::Future>::poll
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/task/local.rs:658:9
  19: tokio::task::local::LocalSet::run_until::{{closure}}
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/task/local.rs:516:9
  20: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/future/mod.rs:80:19
  21: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/future/future.rs:120:9
  22: tokio::runtime::basic_scheduler::Inner<P>::block_on::{{closure}}::{{closure}}
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/basic_scheduler.rs:208:62
  23: tokio::coop::with_budget::{{closure}}
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/coop.rs:106:9
  24: std::thread::local::LocalKey<T>::try_with
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/thread/local.rs:376:16
  25: std::thread::local::LocalKey<T>::with
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/thread/local.rs:352:9
  26: tokio::coop::with_budget
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/coop.rs:99:5
      tokio::coop::budget
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/coop.rs:76:5
      tokio::runtime::basic_scheduler::Inner<P>::block_on::{{closure}}
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/basic_scheduler.rs:208:39
  27: tokio::runtime::basic_scheduler::enter::{{closure}}
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/basic_scheduler.rs:299:29
  28: tokio::macros::scoped_tls::ScopedKey<T>::set
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/macros/scoped_tls.rs:61:9
  29: tokio::runtime::basic_scheduler::enter
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/basic_scheduler.rs:299:5
  30: tokio::runtime::basic_scheduler::Inner<P>::block_on
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/basic_scheduler.rs:197:9
  31: tokio::runtime::basic_scheduler::InnerGuard<P>::block_on
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/basic_scheduler.rs:452:9
  32: tokio::runtime::basic_scheduler::BasicScheduler<P>::block_on
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/basic_scheduler.rs:157:24
  33: tokio::runtime::Runtime::block_on
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/runtime/mod.rs:450:46
  34: tokio::task::local::LocalSet::block_on
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.9.0/src/task/local.rs:477:9
  35: actix_rt::runtime::Runtime::block_on
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-rt-2.1.0/src/runtime.rs:85:9
  36: actix_rt::system::SystemRunner::block_on
             at /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-rt-2.1.0/src/system.rs:186:9
  37: restapi::main
             at src/bin/restapi.rs:17:1
  38: core::ops::function::FnOnce::call_once
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/ops/function.rs:227:5
  39: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/sys_common/backtrace.rs:125:18
  40: std::rt::lang_start::{{closure}}
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/rt.rs:49:18
  41: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/core/src/ops/function.rs:259:13
      std::panicking::try::do_call
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/panicking.rs:379:40
      std::panicking::try
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/panicking.rs:343:19
      std::panic::catch_unwind
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/panic.rs:431:14
      std::rt::lang_start_internal
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/rt.rs:34:21
  42: std::rt::lang_start
             at /rustc/53cb7b09b00cbea8754ffb78e7e3cb521cb8af4b/library/std/src/rt.rs:48:5
  43: main
  44: __libc_start_main
  45: _start
0xmountaintop commented 3 years ago

I try again but cannot reproduce.

This time the binaries can start but the data looks weird: kline lose old data & we don't have new blocks