Create index fails after after inserting a lot of rows

[fritshoogland-yugabyte]

Jira Link: DB-1646

1. Yugabyte version 2.7.1.0 build 155.

2. RF=3, with 3 tservers. Testcase code:

   generate_data function:
   create or replace function generate_data( total_nr int )
   returns varchar as $$
   declare
   max_batch_nr int := 25000;
   begin_nr int := 0;
   end_nr int := 0;
   begin
   while end_nr < total_nr loop
   if begin_nr + max_batch_nr < total_nr then
     end_nr := begin_nr + max_batch_nr;
   else
     end_nr :=  total_nr;
   end if;
   insert into test_indexes 
     select s, 
               md5(random()::text)||md5(random()::text)||md5(random()::text),
               md5(random()::text)||md5(random()::text)||md5(random()::text) 
     from generate_series(begin_nr,end_nr) s;
   begin_nr := end_nr + 1;
   end loop;
   return 'inserted '||total_nr||' rows.';
   end;
   $$ language plpgsql;   

   table creation:

   create table test_indexes (
   id int primary key,
   f1 varchar(200),
   f2 varchar(200)
   );

   Testcase:

3. create table

4. insert 10,000,000 rows into the table: select generate_data(1000000);

5. create index: create index test_indexes_f1 on test_indexes(f1);

6. after some time, the create index start throwing messages in the postgresql.log file, and later is terminated:

The first line is indicating the creation of the index (which it calls transactional table). Then a connection refused and unable to pick leader error is printed, at level informal:

I0528 09:17:24.763717 21364 pg_ddl.cc:309] PgCreateTable: creating a transactional table: yugabyte.test_indexes_f1
I0528 09:19:21.240825 21369 tcp_stream.cc:308] { local: 172.158.33.218:36594 remote: 172.158.27.88:9100 }:  Recv failed: Network error (yb/util/net/socket.cc:537): recvmsg error: Connection refused (system error 111)
I0528 09:19:21.284657 21369 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.019s }}]

Then warnings are printed, these seem to say that a deadline has passed.

W0528 09:19:21.738869 21369 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504826759114752, retrier: { task_id: -1 state: kRunning deadline: 502242.503s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 25 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504826759114752, retrier: { task_id: -1 state: kRunning deadline: 502242.503s } passed its deadline 502242.503s (passed: 0.512s): Network error (yb/util/net/socket.cc:537): recvmsg error: Connection refused (system error 111)
W0528 09:19:21.738945 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504826759114752, retrier: { task_id: -1 state: kRunning deadline: 502242.503s } passed its deadline 502242.503s (passed: 0.512s): Network error (yb/util/net/socket.cc:537): recvmsg error: Connection refused (system error 111), state: kRunning
W0528 09:19:22.251821 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504828882792448, retrier: { task_id: -1 state: kRunning deadline: 502243.021s } passed its deadline 502243.021s (passed: 0.513s): Network error (yb/util/net/socket.cc:537): recvmsg error: Connection refused (system error 111), state: kRunning

The are further numerous logged lines, after which the connection is terminated:

W0528 09:20:05.316143 22051 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Operation expired (yb/tablet/transaction_coordinator.cc:1161): Transaction 44642ed9-95c2-4dfa-b49f-dcc3ee574820 expired or aborted by a conflict: 40001 (pgsql error 40001) (transaction error 1), state: kRunning
2021-05-28 09:21:04.260 UTC [21100] LOG:  received fast shutdown request
2021-05-28 09:21:04.266 UTC [21100] LOG:  aborting any active transactions
2021-05-28 09:21:04.270 UTC [21123] FATAL:  terminating background worker "YSQL webserver" due to administrator command
2021-05-28 09:21:04.343 UTC [21100] LOG:  background worker "YSQL webserver" (PID 21123) exited with exit code 1
I0528 09:22:36.481132 21364 table_creator.cc:326] Created index yugabyte.test_indexes_f1 of type PGSQL_TABLE_TYPE
2021-05-28 09:22:36.536 UTC [21364] LOG:  committing pg_index tuple with indislive=true
2021-05-28 09:22:36.536 UTC [21364] STATEMENT:  create index test_indexes_f1 on test_indexes(f1);
2021-05-28 09:22:36.537 UTC [21364] FATAL:  terminating connection due to administrator command

It leaves YSQL down, while the tserver seems to be still up.

Desired state:

• if background actions are still taking place on the table, and the create index must wait, it should wait until that has happened, unless explicitly configured differently.
• if configured differently, it should fail immediately
• YSQL should not go down or be interrupted, or worse kill tserver. It should only print a message if it fails.

This issue might be related to #8686

[fritshoogland-yugabyte]

postgresql.log output: I0528 09:17:24.763717 21364 pg_ddl.cc:309] PgCreateTable: creating a transactional table: yugabyte.test_indexes_f1

I0528 09:19:21.240825 21369 tcp_stream.cc:308] { local: 172.158.33.218:36594 remote: 172.158.27.88:9100 }: Recv failed: Network error (yb/util/net/socket.cc:537): recvmsg error: Connection refused (system error 111) I0528 09:19:21.284657 21369 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.019s }}] W0528 09:19:21.738869 21369 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504826759114752, retrier: { task_id: -1 state: kRunning deadline: 502242.503s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 25 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504826759114752, retrier: { task_id: -1 state: kRunning deadline: 502242.503s } passed its deadline 502242.503s (passed: 0.512s): Network error (yb/util/net/socket.cc:537): recvmsg error: Connection refused (system error 111) W0528 09:19:21.738945 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504826759114752, retrier: { task_id: -1 state: kRunning deadline: 502242.503s } passed its deadline 502242.503s (passed: 0.512s): Network error (yb/util/net/socket.cc:537): recvmsg error: Connection refused (system error 111), state: kRunning W0528 09:19:22.251821 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504828882792448, retrier: { task_id: -1 state: kRunning deadline: 502243.021s } passed its deadline 502243.021s (passed: 0.513s): Network error (yb/util/net/socket.cc:537): recvmsg error: Connection refused (system error 111), state: kRunning I0528 09:19:22.348852 21369 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.017s }}] [suppressed 11 similar messages] I0528 09:19:22.808774 21370 meta_cache.cc:809] 0x00007f7be801bbf0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502243.534s (passed: 0.078s) W0528 09:19:22.808813 21370 meta_cache.cc:872] 0x00007f7be801bbf0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502243.534s (passed: 0.078s) W0528 09:19:22.829097 21370 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504830983606272, retrier: { task_id: -1 state: kRunning deadline: 502243.534s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 20 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504830983606272, retrier: { task_id: -1 state: kRunning deadline: 502243.534s } passed its deadline 502243.534s (passed: 0.577s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:22.829119 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504830983606272, retrier: { task_id: -1 state: kRunning deadline: 502243.534s } passed its deadline 502243.534s (passed: 0.577s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning W0528 09:19:23.340405 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504833348116480, retrier: { task_id: -1 state: kRunning deadline: 502244.111s } passed its deadline 502244.111s (passed: 0.511s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:23.466071 21369 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.016s }}] [suppressed 8 similar messages] I0528 09:19:23.890336 21369 meta_cache.cc:809] 0x00007f7bf402e650 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502244.623s (passed: 0.111s) W0528 09:19:23.890364 21369 meta_cache.cc:872] 0x00007f7bf402e650 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502244.623s (passed: 0.111s) W0528 09:19:23.907582 21369 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504835442458624, retrier: { task_id: -1 state: kRunning deadline: 502244.623s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 15 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504835442458624, retrier: { task_id: -1 state: kRunning deadline: 502244.623s } passed its deadline 502244.623s (passed: 0.567s): Network error (yb/util/net/socket.cc:537): recvmsg error: Connection refused (system error 111) W0528 09:19:23.907605 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504835442458624, retrier: { task_id: -1 state: kRunning deadline: 502244.623s } passed its deadline 502244.623s (passed: 0.567s): Network error (yb/util/net/socket.cc:537): recvmsg error: Connection refused (system error 111), state: kRunning I0528 09:19:24.344611 21369 tcp_stream.cc:308] { local: 172.158.33.218:36696 remote: 172.158.27.88:9100 }: Recv failed: Network error (yb/util/net/socket.cc:537): recvmsg error: Connection refused (system error 111) I0528 09:19:24.515360 21369 meta_cache.cc:809] 0x00007f7bf402e650 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502245.190s (passed: 0.125s) W0528 09:19:24.530603 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504837765595136, retrier: { task_id: -1 state: kRunning deadline: 502245.190s } passed its deadline 502245.190s (passed: 0.623s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:24.533129 21369 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.002s }}] [suppressed 6 similar messages] W0528 09:19:25.033622 21369 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504840317460480, retrier: { task_id: -1 state: kRunning deadline: 502245.813s } to tablet 93eca372aa8c4f31b2298cc841261c8b on tablet server { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b" after 13 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504840317460480, retrier: { task_id: -1 state: kRunning deadline: 502245.813s } passed its deadline 502245.813s (passed: 0.503s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:25.033646 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504840317460480, retrier: { task_id: -1 state: kRunning deadline: 502245.813s } passed its deadline 502245.813s (passed: 0.503s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:25.554443 21369 meta_cache.cc:809] 0x00007f7bf4043eb0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502246.316s (passed: 0.150s) W0528 09:19:25.554476 21369 meta_cache.cc:872] 0x00007f7bf4043eb0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502246.316s (passed: 0.150s) W0528 09:19:25.567721 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504842377863168, retrier: { task_id: -1 state: kRunning deadline: 502246.316s } passed its deadline 502246.316s (passed: 0.534s): Network error (yb/util/net/socket.cc:537): recvmsg error: Connection refused (system error 111), state: kRunning I0528 09:19:25.573302 21369 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.005s }}] [suppressed 5 similar messages] I0528 09:19:26.106110 21369 meta_cache.cc:809] 0x00007f7bf4043eb0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502246.850s (passed: 0.161s) W0528 09:19:26.120362 21369 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504844565504000, retrier: { task_id: -1 state: kRunning deadline: 502246.850s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 11 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504844565504000, retrier: { task_id: -1 state: kRunning deadline: 502246.850s } passed its deadline 502246.850s (passed: 0.552s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:26.120383 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504844565504000, retrier: { task_id: -1 state: kRunning deadline: 502246.850s } passed its deadline 502246.850s (passed: 0.552s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:26.686988 21370 meta_cache.cc:809] 0x00007f7be801bbf0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502247.402s (passed: 0.172s) W0528 09:19:26.687016 21370 meta_cache.cc:872] 0x00007f7be801bbf0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502247.402s (passed: 0.172s) W0528 09:19:26.700253 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504846829129728, retrier: { task_id: -1 state: kRunning deadline: 502247.402s } passed its deadline 502247.402s (passed: 0.580s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:26.705742 21370 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.005s }}] [suppressed 5 similar messages] I0528 09:19:27.305944 21370 meta_cache.cc:809] 0x00007f7be801bbf0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502247.982s (passed: 0.184s) W0528 09:19:27.319200 21370 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504849204346880, retrier: { task_id: -1 state: kRunning deadline: 502247.982s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 11 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504849204346880, retrier: { task_id: -1 state: kRunning deadline: 502247.982s } passed its deadline 502247.982s (passed: 0.619s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:27.319221 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504849204346880, retrier: { task_id: -1 state: kRunning deadline: 502247.982s } passed its deadline 502247.982s (passed: 0.619s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:27.752537 21369 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.023s }}] [suppressed 4 similar messages] I0528 09:19:27.945204 21369 meta_cache.cc:809] 0x00007f7bf4043eb0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502248.601s (passed: 0.192s) W0528 09:19:27.945235 21369 meta_cache.cc:872] 0x00007f7bf4043eb0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502248.601s (passed: 0.192s) W0528 09:19:27.955466 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504851739574272, retrier: { task_id: -1 state: kRunning deadline: 502248.601s } passed its deadline 502248.601s (passed: 0.636s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:28.620540 21369 meta_cache.cc:809] 0x00007f7bf402e650 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502249.238s (passed: 0.203s) W0528 09:19:28.633785 21369 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504854345682944, retrier: { task_id: -1 state: kRunning deadline: 502249.238s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 11 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504854345682944, retrier: { task_id: -1 state: kRunning deadline: 502249.238s } passed its deadline 502249.238s (passed: 0.678s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:28.633805 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504854345682944, retrier: { task_id: -1 state: kRunning deadline: 502249.238s } passed its deadline 502249.238s (passed: 0.678s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:28.868837 21370 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.014s }}] [suppressed 4 similar messages] I0528 09:19:29.330773 21370 meta_cache.cc:809] 0x00007f7be801bbf0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502249.916s (passed: 0.212s) W0528 09:19:29.330806 21370 meta_cache.cc:872] 0x00007f7be801bbf0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502249.916s (passed: 0.212s) W0528 09:19:29.345032 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504857124110336, retrier: { task_id: -1 state: kRunning deadline: 502249.916s } passed its deadline 502249.916s (passed: 0.711s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:30.062855 21369 meta_cache.cc:809] 0x00007f7bf40262a0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502250.627s (passed: 0.223s) W0528 09:19:30.073104 21369 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504860037349376, retrier: { task_id: -1 state: kRunning deadline: 502250.627s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 11 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504860037349376, retrier: { task_id: -1 state: kRunning deadline: 502250.627s } passed its deadline 502250.627s (passed: 0.728s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:30.073125 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504860037349376, retrier: { task_id: -1 state: kRunning deadline: 502250.627s } passed its deadline 502250.627s (passed: 0.728s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:30.077849 21369 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.005s }}] [suppressed 4 similar messages] W0528 09:19:30.575273 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504863019569152, retrier: { task_id: -1 state: kRunning deadline: 502251.355s } passed its deadline 502251.355s (passed: 0.502s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:31.087677 21369 meta_cache.cc:809] 0x00007f7bf40262a0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502251.857s (passed: 0.238s) W0528 09:19:31.087709 21369 meta_cache.cc:872] 0x00007f7bf40262a0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502251.857s (passed: 0.238s) W0528 09:19:31.097939 21369 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504865076424704, retrier: { task_id: -1 state: kRunning deadline: 502251.857s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 8 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504865076424704, retrier: { task_id: -1 state: kRunning deadline: 502251.857s } passed its deadline 502251.857s (passed: 0.523s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:31.097962 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504865076424704, retrier: { task_id: -1 state: kRunning deadline: 502251.857s } passed its deadline 502251.857s (passed: 0.523s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:31.101554 21369 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.004s }}] [suppressed 3 similar messages] I0528 09:19:31.606405 21369 meta_cache.cc:809] 0x00007f7bf402e650 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502252.380s (passed: 0.243s) W0528 09:19:31.613672 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504867217301504, retrier: { task_id: -1 state: kRunning deadline: 502252.380s } passed its deadline 502252.380s (passed: 0.516s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:32.129465 21369 meta_cache.cc:809] 0x00007f7bf4043eb0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502252.896s (passed: 0.247s) W0528 09:19:32.129498 21369 meta_cache.cc:872] 0x00007f7bf4043eb0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502252.896s (passed: 0.247s) W0528 09:19:32.139731 21369 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504869329698816, retrier: { task_id: -1 state: kRunning deadline: 502252.896s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 7 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504869329698816, retrier: { task_id: -1 state: kRunning deadline: 502252.896s } passed its deadline 502252.896s (passed: 0.526s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:32.139751 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504869329698816, retrier: { task_id: -1 state: kRunning deadline: 502252.896s } passed its deadline 502252.896s (passed: 0.526s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:32.143303 21370 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.003s }}] [suppressed 3 similar messages] I0528 09:19:32.397694 21369 tcp_stream.cc:308] { local: 172.158.33.218:36796 remote: 172.158.27.88:9100 }: Recv failed: Network error (yb/util/net/socket.cc:537): recvmsg error: Connection refused (system error 111) I0528 09:19:32.668675 21369 meta_cache.cc:809] 0x00007f7bf402e650 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502253.422s (passed: 0.254s) W0528 09:19:32.678932 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504871484465152, retrier: { task_id: -1 state: kRunning deadline: 502253.422s } passed its deadline 502253.422s (passed: 0.539s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:33.220788 21369 meta_cache.cc:809] 0x00007f7bf4025170 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502253.961s (passed: 0.258s) W0528 09:19:33.220814 21369 meta_cache.cc:872] 0x00007f7bf4025170 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502253.961s (passed: 0.258s) W0528 09:19:33.229038 21369 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504873693003776, retrier: { task_id: -1 state: kRunning deadline: 502253.961s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 7 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504873693003776, retrier: { task_id: -1 state: kRunning deadline: 502253.961s } passed its deadline 502253.961s (passed: 0.550s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:33.229056 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504873693003776, retrier: { task_id: -1 state: kRunning deadline: 502253.961s } passed its deadline 502253.961s (passed: 0.550s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:33.233985 21369 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.004s }}] [suppressed 3 similar messages] I0528 09:19:33.773406 21369 meta_cache.cc:809] 0x00007f7bf4048800 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502254.511s (passed: 0.262s) W0528 09:19:33.783668 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504875946254336, retrier: { task_id: -1 state: kRunning deadline: 502254.511s } passed its deadline 502254.511s (passed: 0.555s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:34.342921 21369 meta_cache.cc:809] 0x00007f7bf403fa40 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502255.066s (passed: 0.266s) W0528 09:19:34.342948 21369 meta_cache.cc:872] 0x00007f7bf403fa40 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502255.066s (passed: 0.266s) W0528 09:19:34.351177 21369 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504878217998336, retrier: { task_id: -1 state: kRunning deadline: 502255.066s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 7 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504878217998336, retrier: { task_id: -1 state: kRunning deadline: 502255.066s } passed its deadline 502255.066s (passed: 0.567s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:34.351194 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504878217998336, retrier: { task_id: -1 state: kRunning deadline: 502255.066s } passed its deadline 502255.066s (passed: 0.567s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:34.355636 21369 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.004s }}] [suppressed 3 similar messages] I0528 09:19:34.918479 21370 meta_cache.cc:809] 0x00007f7be801bbf0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502255.633s (passed: 0.272s) W0528 09:19:34.928755 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504880542535680, retrier: { task_id: -1 state: kRunning deadline: 502255.633s } passed its deadline 502255.633s (passed: 0.578s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:35.502660 21369 meta_cache.cc:809] 0x00007f7bf403fa40 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502256.211s (passed: 0.278s) W0528 09:19:35.502699 21369 meta_cache.cc:872] 0x00007f7bf403fa40 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502256.211s (passed: 0.278s) W0528 09:19:35.512920 21369 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504882908307456, retrier: { task_id: -1 state: kRunning deadline: 502256.211s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 7 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504882908307456, retrier: { task_id: -1 state: kRunning deadline: 502256.211s } passed its deadline 502256.211s (passed: 0.584s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:35.512940 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504882908307456, retrier: { task_id: -1 state: kRunning deadline: 502256.211s } passed its deadline 502256.211s (passed: 0.584s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:35.516360 21369 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.003s }}] [suppressed 3 similar messages] I0528 09:19:36.100024 21370 meta_cache.cc:809] 0x00007f7be801bbf0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502256.795s (passed: 0.282s) W0528 09:19:36.106268 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504885301047296, retrier: { task_id: -1 state: kRunning deadline: 502256.795s } passed its deadline 502256.795s (passed: 0.593s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:36.696509 21369 meta_cache.cc:809] 0x00007f7bf4025170 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502257.388s (passed: 0.289s) W0528 09:19:36.696542 21369 meta_cache.cc:872] 0x00007f7bf4025170 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502257.388s (passed: 0.289s) W0528 09:19:36.706780 21369 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504887731367936, retrier: { task_id: -1 state: kRunning deadline: 502257.388s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 7 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504887731367936, retrier: { task_id: -1 state: kRunning deadline: 502257.388s } passed its deadline 502257.388s (passed: 0.601s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:36.706804 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504887731367936, retrier: { task_id: -1 state: kRunning deadline: 502257.388s } passed its deadline 502257.388s (passed: 0.601s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:36.711283 21370 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.004s }}] [suppressed 3 similar messages] I0528 09:19:37.318351 21369 meta_cache.cc:809] 0x00007f7bf403fa40 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502257.989s (passed: 0.290s) W0528 09:19:37.326582 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504890191110144, retrier: { task_id: -1 state: kRunning deadline: 502257.989s } passed its deadline 502257.989s (passed: 0.620s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:37.947775 21370 meta_cache.cc:809] 0x00007f7be801bbf0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502258.609s (passed: 0.299s) W0528 09:19:37.947813 21370 meta_cache.cc:872] 0x00007f7be801bbf0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502258.609s (passed: 0.299s) W0528 09:19:37.956051 21370 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504892729774080, retrier: { task_id: -1 state: kRunning deadline: 502258.609s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 7 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504892729774080, retrier: { task_id: -1 state: kRunning deadline: 502258.609s } passed its deadline 502258.609s (passed: 0.629s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:37.956077 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504892729774080, retrier: { task_id: -1 state: kRunning deadline: 502258.609s } passed its deadline 502258.609s (passed: 0.629s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:37.960496 21370 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.005s }}] [suppressed 3 similar messages] I0528 09:19:38.583716 21369 meta_cache.cc:809] 0x00007f7bf4025170 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502259.238s (passed: 0.305s) W0528 09:19:38.591029 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504895308144640, retrier: { task_id: -1 state: kRunning deadline: 502259.238s } passed its deadline 502259.238s (passed: 0.635s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:39.228669 21369 meta_cache.cc:809] 0x00007f7bf4048800 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502259.873s (passed: 0.308s) W0528 09:19:39.228698 21369 meta_cache.cc:872] 0x00007f7bf4048800 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502259.873s (passed: 0.308s) W0528 09:19:39.236923 21369 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504897908948992, retrier: { task_id: -1 state: kRunning deadline: 502259.873s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 7 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504897908948992, retrier: { task_id: -1 state: kRunning deadline: 502259.873s } passed its deadline 502259.873s (passed: 0.646s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:39.236945 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504897908948992, retrier: { task_id: -1 state: kRunning deadline: 502259.873s } passed its deadline 502259.873s (passed: 0.646s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:39.239456 21369 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.002s }}] [suppressed 3 similar messages] I0528 09:19:39.879503 21370 meta_cache.cc:809] 0x00007f7be801bbf0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502260.519s (passed: 0.313s) W0528 09:19:39.889890 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504900554571776, retrier: { task_id: -1 state: kRunning deadline: 502260.519s } passed its deadline 502260.519s (passed: 0.653s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:40.552683 21370 meta_cache.cc:809] 0x00007f7be801bbf0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502261.172s (passed: 0.316s) W0528 09:19:40.552717 21370 meta_cache.cc:872] 0x00007f7be801bbf0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502261.172s (passed: 0.316s) W0528 09:19:40.561936 21370 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504903229083648, retrier: { task_id: -1 state: kRunning deadline: 502261.172s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 7 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504903229083648, retrier: { task_id: -1 state: kRunning deadline: 502261.172s } passed its deadline 502261.172s (passed: 0.672s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:40.561959 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504903229083648, retrier: { task_id: -1 state: kRunning deadline: 502261.172s } passed its deadline 502261.172s (passed: 0.672s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:40.564455 21369 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.002s }}] [suppressed 3 similar messages] I0528 09:19:41.231042 21369 meta_cache.cc:809] 0x00007f7bf4025170 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502261.844s (passed: 0.324s) W0528 09:19:41.238306 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504905981825024, retrier: { task_id: -1 state: kRunning deadline: 502261.844s } passed its deadline 502261.844s (passed: 0.676s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:41.590168 21370 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.016s }}] [suppressed 2 similar messages] I0528 09:19:41.915848 21370 meta_cache.cc:809] 0x00007f7be801bbf0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502262.520s (passed: 0.326s) W0528 09:19:41.915879 21370 meta_cache.cc:872] 0x00007f7be801bbf0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502262.520s (passed: 0.326s) W0528 09:19:41.922091 21370 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504908752191488, retrier: { task_id: -1 state: kRunning deadline: 502262.520s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 7 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504908752191488, retrier: { task_id: -1 state: kRunning deadline: 502262.520s } passed its deadline 502262.520s (passed: 0.684s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:41.922111 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504908752191488, retrier: { task_id: -1 state: kRunning deadline: 502262.520s } passed its deadline 502262.520s (passed: 0.684s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:42.609844 21369 meta_cache.cc:809] 0x00007f7bf4025170 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502263.204s (passed: 0.332s) W0528 09:19:42.616106 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504911553011712, retrier: { task_id: -1 state: kRunning deadline: 502263.204s } passed its deadline 502263.204s (passed: 0.694s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:42.617558 21369 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.002s }}] [suppressed 2 similar messages] I0528 09:19:43.310142 21369 meta_cache.cc:809] 0x00007f7bf4048800 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502263.898s (passed: 0.337s) W0528 09:19:43.310173 21369 meta_cache.cc:872] 0x00007f7bf4048800 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502263.898s (passed: 0.337s) W0528 09:19:43.320405 21369 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504914395664384, retrier: { task_id: -1 state: kRunning deadline: 502263.898s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 7 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504914395664384, retrier: { task_id: -1 state: kRunning deadline: 502263.898s } passed its deadline 502263.898s (passed: 0.704s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:43.320426 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504914395664384, retrier: { task_id: -1 state: kRunning deadline: 502263.898s } passed its deadline 502263.898s (passed: 0.704s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:43.688647 21370 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.018s }}] [suppressed 2 similar messages] I0528 09:19:44.032472 21370 meta_cache.cc:809] 0x00007f7be8020ca0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502264.603s (passed: 0.344s) W0528 09:19:44.042737 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504917280550912, retrier: { task_id: -1 state: kRunning deadline: 502264.603s } passed its deadline 502264.603s (passed: 0.722s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:44.767212 21369 meta_cache.cc:809] 0x00007f7bf4048800 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502265.325s (passed: 0.350s) W0528 09:19:44.767246 21369 meta_cache.cc:872] 0x00007f7bf4048800 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502265.325s (passed: 0.350s) W0528 09:19:44.776501 21369 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504920239165440, retrier: { task_id: -1 state: kRunning deadline: 502265.325s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 7 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504920239165440, retrier: { task_id: -1 state: kRunning deadline: 502265.325s } passed its deadline 502265.325s (passed: 0.733s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:44.776533 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504920239165440, retrier: { task_id: -1 state: kRunning deadline: 502265.325s } passed its deadline 502265.325s (passed: 0.733s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:44.778029 21369 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.001s }}] [suppressed 2 similar messages] I0528 09:19:45.516726 21369 meta_cache.cc:809] 0x00007f7bf403fa40 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502266.059s (passed: 0.352s) W0528 09:19:45.524969 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504923244724224, retrier: { task_id: -1 state: kRunning deadline: 502266.059s } passed its deadline 502266.059s (passed: 0.748s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:45.903247 21369 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.010s }}] [suppressed 2 similar messages] I0528 09:19:46.259058 21369 meta_cache.cc:809] 0x00007f7bf4025170 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502266.807s (passed: 0.356s) W0528 09:19:46.259086 21369 meta_cache.cc:872] 0x00007f7bf4025170 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502266.807s (passed: 0.356s) W0528 09:19:46.265319 21369 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504926310359040, retrier: { task_id: -1 state: kRunning deadline: 502266.807s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 7 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504926310359040, retrier: { task_id: -1 state: kRunning deadline: 502266.807s } passed its deadline 502266.807s (passed: 0.740s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:46.265339 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504926310359040, retrier: { task_id: -1 state: kRunning deadline: 502266.807s } passed its deadline 502266.807s (passed: 0.740s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:47.011276 21370 meta_cache.cc:809] 0x00007f7be8020ca0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502267.547s (passed: 0.361s) W0528 09:19:47.019515 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504929342873600, retrier: { task_id: -1 state: kRunning deadline: 502267.547s } passed its deadline 502267.547s (passed: 0.755s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:47.022945 21369 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.003s }}] [suppressed 2 similar messages] I0528 09:19:47.775269 21369 meta_cache.cc:809] 0x00007f7bf4048800 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502268.302s (passed: 0.368s) W0528 09:19:47.775300 21369 meta_cache.cc:872] 0x00007f7bf4048800 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502268.302s (passed: 0.368s) W0528 09:19:47.783531 21369 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504932432031744, retrier: { task_id: -1 state: kRunning deadline: 502268.302s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 7 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504932432031744, retrier: { task_id: -1 state: kRunning deadline: 502268.302s } passed its deadline 502268.302s (passed: 0.764s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:47.783551 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504932432031744, retrier: { task_id: -1 state: kRunning deadline: 502268.302s } passed its deadline 502268.302s (passed: 0.764s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:48.179409 21369 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.014s }}] [suppressed 2 similar messages] I0528 09:19:48.553308 21369 meta_cache.cc:809] 0x00007f7bf403fa40 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502269.066s (passed: 0.373s) W0528 09:19:48.562520 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504935561478144, retrier: { task_id: -1 state: kRunning deadline: 502269.066s } passed its deadline 502269.066s (passed: 0.779s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:48.940673 21369 tcp_stream.cc:308] { local: 172.158.33.218:36898 remote: 172.158.27.88:9100 }: Recv failed: Network error (yb/util/net/socket.cc:537): recvmsg error: Connection refused (system error 111) I0528 09:19:49.338726 21369 meta_cache.cc:809] 0x00007f7bf4048800 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502269.845s (passed: 0.380s) W0528 09:19:49.338754 21369 meta_cache.cc:872] 0x00007f7bf4048800 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502269.845s (passed: 0.380s) W0528 09:19:49.347990 21369 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504938752180224, retrier: { task_id: -1 state: kRunning deadline: 502269.845s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 7 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504938752180224, retrier: { task_id: -1 state: kRunning deadline: 502269.845s } passed its deadline 502269.845s (passed: 0.785s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:49.348009 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504938752180224, retrier: { task_id: -1 state: kRunning deadline: 502269.845s } passed its deadline 502269.845s (passed: 0.785s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:49.350638 21369 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.002s }}] [suppressed 2 similar messages] I0528 09:19:50.132613 21370 meta_cache.cc:809] 0x00007f7be8020ca0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502270.630s (passed: 0.381s) W0528 09:19:50.138913 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504941969494016, retrier: { task_id: -1 state: kRunning deadline: 502270.630s } passed its deadline 502270.630s (passed: 0.791s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:50.552490 21370 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.014s }}] [suppressed 2 similar messages] I0528 09:19:50.940342 21370 meta_cache.cc:809] 0x00007f7be8020ca0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502271.421s (passed: 0.387s) W0528 09:19:50.940371 21370 meta_cache.cc:872] 0x00007f7be8020ca0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502271.421s (passed: 0.387s) W0528 09:19:50.946578 21370 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504945209085952, retrier: { task_id: -1 state: kRunning deadline: 502271.421s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 7 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504945209085952, retrier: { task_id: -1 state: kRunning deadline: 502271.421s } passed its deadline 502271.421s (passed: 0.808s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:50.946595 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504945209085952, retrier: { task_id: -1 state: kRunning deadline: 502271.421s } passed its deadline 502271.421s (passed: 0.808s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:51.757555 21370 meta_cache.cc:809] 0x00007f7be8020ca0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502272.229s (passed: 0.394s) W0528 09:19:51.767798 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504948517298176, retrier: { task_id: -1 state: kRunning deadline: 502272.229s } passed its deadline 502272.229s (passed: 0.821s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:51.771320 21369 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.003s }}] [suppressed 2 similar messages] I0528 09:19:52.588546 21369 meta_cache.cc:809] 0x00007f7bf403fa40 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502273.050s (passed: 0.397s) W0528 09:19:52.588578 21369 meta_cache.cc:872] 0x00007f7bf403fa40 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502273.050s (passed: 0.397s) W0528 09:19:52.595810 21369 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504951880998912, retrier: { task_id: -1 state: kRunning deadline: 502273.050s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 7 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504951880998912, retrier: { task_id: -1 state: kRunning deadline: 502273.050s } passed its deadline 502273.050s (passed: 0.828s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:52.595831 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504951880998912, retrier: { task_id: -1 state: kRunning deadline: 502273.050s } passed its deadline 502273.050s (passed: 0.828s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:53.019971 21370 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.015s }}] [suppressed 2 similar messages] I0528 09:19:53.423768 21370 meta_cache.cc:809] 0x00007f7be8020ca0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502273.878s (passed: 0.404s) W0528 09:19:53.432019 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504955272568832, retrier: { task_id: -1 state: kRunning deadline: 502273.878s } passed its deadline 502273.878s (passed: 0.836s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:54.272266 21370 meta_cache.cc:809] 0x00007f7be8020ca0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502274.714s (passed: 0.406s) W0528 09:19:54.272292 21370 meta_cache.cc:872] 0x00007f7be8020ca0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502274.714s (passed: 0.406s) W0528 09:19:54.279506 21370 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504958697644032, retrier: { task_id: -1 state: kRunning deadline: 502274.714s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 7 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504958697644032, retrier: { task_id: -1 state: kRunning deadline: 502274.714s } passed its deadline 502274.714s (passed: 0.848s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:54.279526 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504958697644032, retrier: { task_id: -1 state: kRunning deadline: 502274.714s } passed its deadline 502274.714s (passed: 0.848s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:54.280977 21369 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.001s }}] [suppressed 2 similar messages] I0528 09:19:55.128556 21370 meta_cache.cc:809] 0x00007f7be8020ca0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502275.562s (passed: 0.415s) W0528 09:19:55.135816 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504962168987648, retrier: { task_id: -1 state: kRunning deadline: 502275.562s } passed its deadline 502275.562s (passed: 0.856s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:55.576007 21370 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.016s }}] [suppressed 2 similar messages] I0528 09:19:55.992933 21370 meta_cache.cc:809] 0x00007f7be8020ca0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502276.418s (passed: 0.417s) W0528 09:19:55.992961 21370 meta_cache.cc:872] 0x00007f7be8020ca0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502276.418s (passed: 0.417s) W0528 09:19:56.003185 21370 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504965676412928, retrier: { task_id: -1 state: kRunning deadline: 502276.418s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 7 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504965676412928, retrier: { task_id: -1 state: kRunning deadline: 502276.418s } passed its deadline 502276.418s (passed: 0.867s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:56.003209 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504965676412928, retrier: { task_id: -1 state: kRunning deadline: 502276.418s } passed its deadline 502276.418s (passed: 0.867s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:56.871078 21369 meta_cache.cc:809] 0x00007f7bf403fa40 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502277.285s (passed: 0.424s) W0528 09:19:56.879357 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504969229213696, retrier: { task_id: -1 state: kRunning deadline: 502277.285s } passed its deadline 502277.285s (passed: 0.876s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:56.884855 21369 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.005s }}] [suppressed 2 similar messages] I0528 09:19:57.761144 21370 meta_cache.cc:809] 0x00007f7be8020ca0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502278.161s (passed: 0.426s) W0528 09:19:57.761175 21370 meta_cache.cc:872] 0x00007f7be8020ca0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502278.161s (passed: 0.426s) W0528 09:19:57.767410 21370 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504972817952768, retrier: { task_id: -1 state: kRunning deadline: 502278.161s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 7 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504972817952768, retrier: { task_id: -1 state: kRunning deadline: 502278.161s } passed its deadline 502278.161s (passed: 0.888s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:57.767450 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504972817952768, retrier: { task_id: -1 state: kRunning deadline: 502278.161s } passed its deadline 502278.161s (passed: 0.888s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:58.218672 21370 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.012s }}] [suppressed 2 similar messages] I0528 09:19:58.652683 21370 meta_cache.cc:809] 0x00007f7be8020ca0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502279.050s (passed: 0.434s) W0528 09:19:58.660933 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504976455528448, retrier: { task_id: -1 state: kRunning deadline: 502279.050s } passed its deadline 502279.050s (passed: 0.893s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:59.551095 21369 meta_cache.cc:809] 0x00007f7bf4048800 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502279.943s (passed: 0.435s) W0528 09:19:59.551124 21369 meta_cache.cc:872] 0x00007f7bf4048800 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502279.943s (passed: 0.435s) W0528 09:19:59.559342 21369 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504980115283968, retrier: { task_id: -1 state: kRunning deadline: 502279.943s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 7 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504980115283968, retrier: { task_id: -1 state: kRunning deadline: 502279.943s } passed its deadline 502279.943s (passed: 0.898s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:19:59.559376 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504980115283968, retrier: { task_id: -1 state: kRunning deadline: 502279.943s } passed its deadline 502279.943s (passed: 0.898s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:19:59.563788 21370 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.004s }}] [suppressed 2 similar messages] I0528 09:20:00.467051 21369 meta_cache.cc:809] 0x00007f7bf4025170 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502280.841s (passed: 0.444s) W0528 09:20:00.476300 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504983795249152, retrier: { task_id: -1 state: kRunning deadline: 502280.841s } passed its deadline 502280.841s (passed: 0.917s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:20:00.943328 21370 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.012s }}] [suppressed 2 similar messages] I0528 09:20:01.390439 21370 meta_cache.cc:809] 0x00007f7be8020ca0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502281.758s (passed: 0.447s) W0528 09:20:01.390475 21370 meta_cache.cc:872] 0x00007f7be8020ca0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502281.758s (passed: 0.447s) W0528 09:20:01.399716 21370 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504987551014912, retrier: { task_id: -1 state: kRunning deadline: 502281.758s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 7 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504987551014912, retrier: { task_id: -1 state: kRunning deadline: 502281.758s } passed its deadline 502281.758s (passed: 0.924s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:20:01.399742 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504987551014912, retrier: { task_id: -1 state: kRunning deadline: 502281.758s } passed its deadline 502281.758s (passed: 0.924s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:20:02.326946 21370 meta_cache.cc:809] 0x00007f7be8020ca0 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502282.682s (passed: 0.453s) W0528 09:20:02.335228 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504991333388288, retrier: { task_id: -1 state: kRunning deadline: 502282.682s } passed its deadline 502282.682s (passed: 0.935s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:20:02.336818 21370 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b"], followers: [{0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.001s }}] [suppressed 2 similar messages] I0528 09:20:03.286615 21369 meta_cache.cc:809] 0x00007f7bf4048800 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Failed: Timed out (yb/rpc/rpc.cc:211): LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502283.617s (passed: 0.458s) W0528 09:20:03.286653 21369 meta_cache.cc:872] 0x00007f7bf4048800 -> LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2): Timed out (yb/rpc/rpc.cc:211): timed out after deadline expired: LookupByIdRpc(tablet: 93eca372aa8c4f31b2298cc841261c8b, num_attempts: 2) passed its deadline 502283.617s (passed: 0.458s) W0528 09:20:03.294881 21369 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504995165200384, retrier: { task_id: -1 state: kRunning deadline: 502283.617s } to tablet 93eca372aa8c4f31b2298cc841261c8b (no tablet server available) after 7 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504995165200384, retrier: { task_id: -1 state: kRunning deadline: 502283.617s } passed its deadline 502283.617s (passed: 0.960s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:20:03.294901 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504995165200384, retrier: { task_id: -1 state: kRunning deadline: 502283.617s } passed its deadline 502283.617s (passed: 0.960s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning W0528 09:20:03.799593 21369 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644504999095975936, retrier: { task_id: -1 state: kRunning deadline: 502284.577s } passed its deadline 502284.577s (passed: 0.505s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:20:03.811231 21370 tablet_rpc.cc:140] Unable to pick leader for 93eca372aa8c4f31b2298cc841261c8b, replicas: [0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", 0x00007f7bd4003230 -> { uuid: c4e76266a97c47aaadbe99a1519a8990 private: [host: "172.158.55.189" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1c"], followers: [{0x00007f7bd4003230 -> { uuid: c4e76266a97c47aaadbe99a1519a8990 private: [host: "172.158.55.189" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1c", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.005s }}, {0x00007f7bd40025f0 -> { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b", { status: Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) time: 0.011s }}] [suppressed 2 similar messages] W0528 09:20:04.307476 21370 tablet_rpc.cc:436] Timed out (yb/rpc/rpc.cc:211): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644505001163251712, retrier: { task_id: -1 state: kRunning deadline: 502285.082s } to tablet 93eca372aa8c4f31b2298cc841261c8b on tablet server { uuid: c4e76266a97c47aaadbe99a1519a8990 private: [host: "172.158.55.189" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1c" after 7 attempt(s): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644505001163251712, retrier: { task_id: -1 state: kRunning deadline: 502285.082s } passed its deadline 502285.082s (passed: 0.507s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15) W0528 09:20:04.307502 21370 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Timed out (yb/rpc/rpc.cc:211): UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644505001163251712, retrier: { task_id: -1 state: kRunning deadline: 502285.082s } passed its deadline 502285.082s (passed: 0.507s): Illegal state (yb/consensus/consensus.cc:152): Not the leader (tablet server error 15), state: kRunning I0528 09:20:05.311367 22051 tablet_rpc.cc:434] Operation expired (yb/tablet/transaction_coordinator.cc:1161): Failed UpdateTransaction: tablet_id: "93eca372aa8c4f31b2298cc841261c8b" state { transaction_id: "Dd.\331\225\302M\372\264\237\334\303\356WH " status: PENDING } propagated_hybrid_time: 6644505007267934208, retrier: { task_id: -1 state: kIdle deadline: 502286.572s } to tablet 93eca372aa8c4f31b2298cc841261c8b on tablet server { uuid: 29a4260f535f4ed5b2b35a4331799eda private: [host: "172.158.33.218" port: 9100] cloud_info: placement_cloud: "aws" placement_region: "eu-central-1" placement_zone: "eu-central-1b" after 1 attempt(s): Transaction 44642ed9-95c2-4dfa-b49f-dcc3ee574820 expired or aborted by a conflict: 40001 (pgsql error 40001) (transaction error 1) W0528 09:20:05.316143 22051 transaction.cc:1015] 44642ed9-95c2-4dfa-b49f-dcc3ee574820: Send heartbeat failed: Operation expired (yb/tablet/transaction_coordinator.cc:1161): Transaction 44642ed9-95c2-4dfa-b49f-dcc3ee574820 expired or aborted by a conflict: 40001 (pgsql error 40001) (transaction error 1), state: kRunning 2021-05-28 09:21:04.260 UTC [21100] LOG: received fast shutdown request 2021-05-28 09:21:04.266 UTC [21100] LOG: aborting any active transactions 2021-05-28 09:21:04.270 UTC [21123] FATAL: terminating background worker "YSQL webserver" due to administrator command 2021-05-28 09:21:04.343 UTC [21100] LOG: background worker "YSQL webserver" (PID 21123) exited with exit code 1 I0528 09:22:36.481132 21364 table_creator.cc:326] Created index yugabyte.test_indexes_f1 of type PGSQL_TABLE_TYPE 2021-05-28 09:22:36.536 UTC [21364] LOG: committing pg_index tuple with indislive=true 2021-05-28 09:22:36.536 UTC [21364] STATEMENT: create index test_indexes_f1 on test_indexes(f1); 2021-05-28 09:22:36.537 UTC [21364] FATAL: terminating connection due to administrator command

[fritshoogland-yugabyte]

threadz dump all: <!DOCTYPE html>

• 
• Dashboards
• Tables
• Tablets
• Utilities

Thread Group: all

All Threads :

Thread name	Cumulative User CPU(s)	Cumulative Kernel CPU(s)	Cumulative IO-wait(s)
pg_supervisorxx-21101	0.000s	0.000s	0.000s	@ 0x7f576e04f11f (unknown) @ 0x7f576e9cd7c3 __waitpid @ 0x7f5773298fe4 yb::Subprocess::DoWait() @ 0x7f5773299492 yb::Subprocess::Wait() @ 0x7f57732994bf yb::Subprocess::Wait() @ 0x7f5777f68363 yb::pgwrapper::PgWrapper::Wait() @ 0x7f5777f7026e yb::pgwrapper::PgSupervisor::RunThread() @ 0x7f57732a7dce yb::Thread::SuperviseThread() @ 0x7f576e9c5693 start_thread @ 0x7f576e10241c __clone @ 0xffffffffffffffff (unknown) Total number of threads: 1
rpc_tp_TabletServer_2-21099	19.580s	0.000s	0.000s	@ 0x7f576e04f11e (unknown) @ 0x7f576e9cd592 __GI_pread64 @ 0x7f57731fb5d0 yb::PosixRandomAccessFile::Read() @ 0x7f5775f6ae35 rocksdb::RandomAccessFileReader::ReadAndValidate() @ 0x7f5775f16fec rocksdb::ReadBlockContents() @ 0x7f5775f1188c rocksdb::block_based_table::ReadBlockFromFile() @ 0x7f5775f0d64e rocksdb::BlockBasedTable::NewDataBlockIterator() @ 0x7f5775f4881f rocksdb::(anonymous namespace)::TwoLevelIterator::InitDataBlock() @ 0x7f5775f48ae3 rocksdb::(anonymous namespace)::TwoLevelIterator::Seek() @ 0x7f5775f2d654 rocksdb::MergingIterator::Seek() @ 0x7f5775e7cf34 rocksdb::DBIter::Seek() @ 0x7f577bdf0cfb yb::docdb::BoundedRocksDbIterator::Seek() @ 0x7f577be190fb yb::docdb::IntentToWriteRequest() @ 0x7f577be1f95c yb::docdb::PrepareApplyIntentsBatch() @ 0x7f577ce271cb yb::tablet::Tablet::ApplyIntents() @ 0x7f577cdf4e06 yb::tablet::ApplyIntentsTask::Run() Total number of threads: 2
rpc_tp_TabletServer_4-21337	19.530s	0.000s	0.000s
rpc_tp_TabletServer_3-21333	20.570s	0.000s	0.000s	@ 0x7f576e04f11e (unknown) @ 0x7f576e9cd592 __GI_pread64 @ 0x7f57731fb5d0 yb::PosixRandomAccessFile::Read() @ 0x7f5775f6ae35 rocksdb::RandomAccessFileReader::ReadAndValidate() @ 0x7f5775f16fec rocksdb::ReadBlockContents() @ 0x7f5775f1188c rocksdb::block_based_table::ReadBlockFromFile() @ 0x7f5775f0d64e rocksdb::BlockBasedTable::NewDataBlockIterator() @ 0x7f5775f4881f rocksdb::(anonymous namespace)::TwoLevelIterator::InitDataBlock() @ 0x7f5775f48ae3 rocksdb::(anonymous namespace)::TwoLevelIterator::Seek() @ 0x7f5775f2cdfe rocksdb::MergingIterator::Seek() @ 0x7f5775e7cf34 rocksdb::DBIter::Seek() @ 0x7f577bdf0cfb yb::docdb::BoundedRocksDbIterator::Seek() @ 0x7f577be190fb yb::docdb::IntentToWriteRequest() @ 0x7f577be1f95c yb::docdb::PrepareApplyIntentsBatch() @ 0x7f577ce271cb yb::tablet::Tablet::ApplyIntents() @ 0x7f577cdf4e06 yb::tablet::ApplyIntentsTask::Run() Total number of threads: 2
rpc_tp_TabletServer_5-21338	19.980s	0.000s	0.000s
CQLServer_reactor-21109	0.050s	0.000s	0.000s	@ 0x7f576e04f11f (unknown) @ 0x7f576e1029f2 __GI_epoll_wait @ 0x7f5771d97cf7 epoll_poll @ 0x7f5771d9a5d1 ev_run @ 0x7f57774cdf7b yb::rpc::Reactor::RunThread() @ 0x7f57732a7dce yb::Thread::SuperviseThread() @ 0x7f576e9c5693 start_thread @ 0x7f576e10241c __clone @ 0xffffffffffffffff (unknown) Total number of threads: 2
CQLServer_reactor-21108	0.040s	0.000s	0.000s
TabletServer_reactor-21082	6.850s	0.000s	0.000s	@ 0x7f576e04f11f (unknown) @ 0x7f576e1029f2 __GI_epoll_wait @ 0x7f5771d97cf7 epoll_poll @ 0x7f5771d9a5d1 ev_run @ 0x7f57774cdf7b yb::rpc::Reactor::RunThread() @ 0x7f57732a7dce yb::Thread::SuperviseThread() @ 0x7f576e9c5693 start_thread @ 0x7f576e10241c __clone @ 0xffffffffffffffff (unknown) Total number of threads: 2
TabletServer_reactor-21081	5.260s	0.000s	0.000s
acceptorxxxxxxx-21111	0.000s	0.000s	0.000s	@ 0x7f576e04f11f (unknown) @ 0x7f576e1029f2 __GI_epoll_wait @ 0x7f5771d97cf7 epoll_poll @ 0x7f5771d9a5d1 ev_run @ 0x7f5777496828 yb::rpc::Acceptor::RunThread() @ 0x7f57732a7dce yb::Thread::SuperviseThread() @ 0x7f576e9c5693 start_thread @ 0x7f576e10241c __clone @ 0xffffffffffffffff (unknown) Total number of threads: 2
acceptorxxxxxxx-21092	0.000s	0.000s	0.000s
heartbeatxxxxxx-21095	0.140s	0.000s	0.000s	@ 0x7f576e04f11f (unknown) @ 0x7f576e9ca3b7 __pthread_cond_timedwait @ 0x7f577319157c yb::ConditionVariable::TimedWait() @ 0x7f577d723f10 yb::tserver::Heartbeater::Thread::RunThread() @ 0x7f57732a7dce yb::Thread::SuperviseThread() @ 0x7f576e9c5693 start_thread @ 0x7f576e10241c __clone @ 0xffffffffffffffff (unknown) Total number of threads: 1
iotp_CQLServer_3-21106	0.000s	0.000s	0.000s	@ 0x7f576e04f11f (unknown) @ 0x7f576e9ca00c __pthread_cond_wait @ 0x7f57774b0991 yb::rpc::IoThreadPool::Impl::Execute() @ 0x7f57732a7dce yb::Thread::SuperviseThread() @ 0x7f576e9c5693 start_thread @ 0x7f576e10241c __clone @ 0xffffffffffffffff (unknown) Total number of threads: 3
iotp_CQLServer_2-21104	0.000s	0.000s	0.000s
iotp_CQLServer_1-21103	0.000s	0.000s	0.000s
iotp_CQLServer_0-21102	0.000s	0.000s	0.000s	@ 0x7f576e04f11f (unknown) @ 0x7f576e1029f2 __GI_epoll_wait @ 0x4151ee boost::asio::detail::epoll_reactor::run() @ 0x7f57774b07b0 yb::rpc::IoThreadPool::Impl::Execute() @ 0x7f57732a7dce yb::Thread::SuperviseThread() @ 0x7f576e9c5693 start_thread @ 0x7f576e10241c __clone @ 0xffffffffffffffff (unknown) Total number of threads: 1
iotp_TabletServer_2-21079	0.300s	0.000s	0.000s	@ 0x7f576e04f11f (unknown) @ 0x7f576e9ca00c __pthread_cond_wait @ 0x7f57774b0991 yb::rpc::IoThreadPool::Impl::Execute() @ 0x7f57732a7dce yb::Thread::SuperviseThread() @ 0x7f576e9c5693 start_thread @ 0x7f576e10241c __clone @ 0xffffffffffffffff (unknown) Total number of threads: 3
iotp_TabletServer_1-21078	0.350s	0.000s	0.000s
iotp_TabletServer_0-21077	0.310s	0.000s	0.000s
iotp_TabletServer_3-21080	0.300s	0.000s	0.000s	@ 0x7f576e04f11f (unknown) @ 0x7f576e1029f2 __GI_epoll_wait @ 0x4151ee boost::asio::detail::epoll_reactor::run() @ 0x7f57774b07b0 yb::rpc::IoThreadPool::Impl::Execute() @ 0x7f57732a7dce yb::Thread::SuperviseThread() @ 0x7f576e9c5693 start_thread @ 0x7f576e10241c __clone @ 0xffffffffffffffff (unknown) Total number of threads: 1
iotp_call_home_0-21098	0.000s	0.000s	0.000s	@ 0x7f576e04f11f (unknown) @ 0x7f576e1029f2 __GI_epoll_wait @ 0x4151ee boost::asio::detail::epoll_reactor::run() @ 0x7f57774b07b0 yb::rpc::IoThreadPool::Impl::Execute() @ 0x7f57732a7dce yb::Thread::SuperviseThread() @ 0x7f576e9c5693 start_thread @ 0x7f576e10241c __clone @ 0xffffffffffffffff (unknown) Total number of threads: 1
maintenance_scheduler-21097	0.710s	0.000s	0.000s	@ 0x7f576e04f11f (unknown) @ 0x7f576e9ca3b7 __pthread_cond_timedwait @ 0x7f577319157c yb::ConditionVariable::TimedWait() @ 0x7f577ceb8752 yb::MaintenanceManager::RunSchedulerThread() @ 0x7f57732a7dce yb::Thread::SuperviseThread() @ 0x7f576e9c5693 start_thread @ 0x7f576e10241c __clone @ 0xffffffffffffffff (unknown) Total number of threads: 1
priority-worker-21381	172.650s	0.000s	0.000s	@ 0x7f576e04f11f (unknown) @ 0x7f576e9ca00c __pthread_cond_wait @ 0x7f576f1ab85b std::condition_variable::wait() @ 0x7f577327af60 yb::(anonymous namespace)::PriorityThreadPoolWorker::Run() @ 0x7f57732a7dce yb::Thread::SuperviseThread() @ 0x7f576e9c5693 start_thread @ 0x7f576e10241c __clone @ 0xffffffffffffffff (unknown) Total number of threads: 1
rb-session-expx-21088	0.000s	0.000s	0.000s	@ 0x7f576e04f11f (unknown) @ 0x7f576e9ca3b7 __pthread_cond_timedwait @ 0x7f577319157c yb::ConditionVariable::TimedWait() @ 0x7f57731915c5 yb::ConditionVariable::WaitUntil() @ 0x7f57731917aa yb::CountDownLatch::WaitUntil() @ 0x7f5773191851 yb::CountDownLatch::WaitFor() @ 0x7f577d748f73 yb::tserver::RemoteBootstrapServiceImpl::EndExpiredSessions() @ 0x7f57732a7dce yb::Thread::SuperviseThread() @ 0x7f576e9c5693 start_thread @ 0x7f576e10241c __clone @ 0xffffffffffffffff (unknown) Total number of threads: 1
rocksdb:high:0x-21355	117.600s	0.000s	0.000s	@ 0x7f576e04f11f (unknown) @ 0x7f576e9ca00c __pthread_cond_wait @ 0x7f5775f65b5a rocksdb::ThreadPool::BGThread() @ 0x7f57732a7dce yb::Thread::SuperviseThread() @ 0x7f576e9c5693 start_thread @ 0x7f576e10241c __clone @ 0xffffffffffffffff (unknown) Total number of threads: 1
rpc_tp_CQLServer_1-21329	0.000s	0.000s	0.000s	@ 0x7f576e04f11f (unknown) @ 0x7f576e9ca00c __pthread_cond_wait @ 0x7f576f1ab85b std::condition_variable::wait() @ 0x7f577751bb7a yb::rpc::(anonymous namespace)::Worker::Execute() @ 0x7f57732a7dce yb::Thread::SuperviseThread() @ 0x7f576e9c5693 start_thread @ 0x7f576e10241c __clone @ 0xffffffffffffffff (unknown) Total number of threads: 37
rpc_tp_TabletServer-high-pri_8-21847	16.120s	0.000s	0.000s
rpc_tp_TabletServer-high-pri_12-22783	0.030s	0.000s	0.000s
rpc_tp_TabletServer_9-21748	15.980s	0.000s	0.000s
rpc_tp_CQLServer_2-21330	0.000s	0.000s	0.000s
rpc_tp_TabletServer-high-pri_2-21135	12.480s	0.000s	0.000s
rpc_tp_CQLServer_0-21328	0.000s	0.000s	0.000s
rpc_tp_TabletServer_8-21747	67.930s	0.000s	0.000s
rpc_tp_TabletServer-high-pri_6-21838	12.470s	0.000s	0.000s
rpc_tp_TabletServer_10-21749	15.650s	0.000s	0.000s
rpc_tp_TabletServer-high-pri_3-21194	12.830s	0.000s	0.000s
rpc_tp_TabletServer_11-21750	16.490s	0.000s	0.000s
rpc_tp_TabletServer_7-21746	16.380s	0.000s	0.000s
rpc_tp_TabletServer-high-pri_1-21096	16.540s	0.000s	0.000s
rpc_tp_TabletServer-high-pri_0-21093	12.490s	0.000s	0.000s
rpc_tp_TabletServer_1-21091	116.590s	0.000s	0.000s
rpc_tp_TabletServer_0-21089	63.610s	0.000s	0.000s
rpc_tp_TabletServer_13-22902	0.000s	0.000s	0.000s
rpc_tp_TabletServer-high-pri_5-21432	12.450s	0.000s	0.000s
rpc_tp_TabletServer-high-pri_15-22837	0.010s	0.000s	0.000s
rpc_tp_TabletServer-high-pri_14-22836	0.010s	0.000s	0.000s
rpc_tp_TabletServer-high-pri_13-22825	0.020s	0.000s	0.000s
rpc_tp_TabletServer-high-pri_11-22341	8.300s	0.000s	0.000s
rpc_tp_TabletServer-high-pri_9-21880	11.760s	0.000s	0.000s
rpc_tp_TabletServer-high-pri_10-22104	10.450s	0.000s	0.000s
rpc_tp_TabletServer-high-pri_7-21842	12.500s	0.000s	0.000s
rpc_tp_TabletServer_12-22901	0.000s	0.000s	0.000s
rpc_tp_TabletServer-high-pri_4-21360	16.470s	0.000s	0.000s
rpc_tp_TransactionManager_1-21830	0.000s	0.000s	0.000s
rpc_tp_TabletServer_6-21339	15.710s	0.000s	0.000s
rpc_tp_CQLServer_7-21336	0.000s	0.000s	0.000s
rpc_tp_CQLServer_6-21335	0.000s	0.000s	0.000s
rpc_tp_CQLServer_5-21334	0.000s	0.000s	0.000s
rpc_tp_CQLServer_4-21332	0.000s	0.000s	0.000s
rpc_tp_CQLServer_3-21331	0.000s	0.000s	0.000s
rpc_tp_TabletServer-high-pri_16-22839	0.010s	0.000s	0.000s
rpc_tp_TransactionManager_0-21829	0.000s	0.000s	0.000s
flush scheduler bgtask-21085	0.010s	0.000s	0.000s	@ 0x7f576e04f11f (unknown) @ 0x7f576e9ca00c __pthread_cond_wait @ 0x7f576f1ab85b std::condition_variable::wait() @ 0x7f577d7b6f2a yb::BackgroundTask::Run() @ 0x7f57732a7dce yb::Thread::SuperviseThread() @ 0x7f576e9c5693 start_thread @ 0x7f576e10241c __clone @ 0xffffffffffffffff (unknown) Total number of threads: 1
MaintenanceMgr [worker]-21076	0.000s	0.000s	0.000s	@ 0x7f576e04f11f (unknown) @ 0x7f576e9ca00c __pthread_cond_wait @ 0x7f57732ab9a0 yb::ThreadPool::DispatchThread() @ 0x7f57732a7dce yb::Thread::SuperviseThread() @ 0x7f576e9c5693 start_thread @ 0x7f576e10241c __clone @ 0xffffffffffffffff (unknown) Total number of threads: 4
log-alloc [worker]-21075	0.000s	0.000s	0.000s
append [worker]-21074	0.070s	0.000s	0.000s
prepare [worker]-21073	0.320s	0.000s	0.000s
consensus [worker]-21072	9.520s	0.000s	0.000s	@ 0x7f576e04f11f (unknown) @ 0x7f576e9ca00e __pthread_cond_wait @ 0x7f57732ab9a0 yb::ThreadPool::DispatchThread() @ 0x7f57732a7dce yb::Thread::SuperviseThread() @ 0x7f576e9c5693 start_thread @ 0x7f576e10241c __clone @ 0xffffffffffffffff (unknown) Total number of threads: 1
consensus [worker]-22495	0.800s	0.000s	0.000s	@ 0x7f576e04f11f (unknown) @ 0x7f576e9ca3b7 __pthread_cond_timedwait @ 0x7f577319157c yb::ConditionVariable::TimedWait() @ 0x7f57732aba42 yb::ThreadPool::DispatchThread() @ 0x7f57732a7dce yb::Thread::SuperviseThread() @ 0x7f576e9c5693 start_thread @ 0x7f576e10241c __clone @ 0xffffffffffffffff (unknown) Total number of threads: 8
consensus [worker]-21849	8.570s	0.000s	0.000s
consensus [worker]-21840	10.000s	0.000s	0.000s
consensus [worker]-21839	8.700s	0.000s	0.000s
consensus [worker]-21164	8.860s	0.000s	0.000s
consensus [worker]-22778	0.020s	0.000s	0.000s
consensus [worker]-21167	8.850s	0.000s	0.000s
consensus [worker]-21379	8.990s	0.000s	0.000s

 Congratulations on installing YugabyteDB. We'd like to welcome you to the community with a free t-shirt and pack of stickers! Please claim your reward here: https://www.yugabyte.com/community-rewards/

version 2.7.1.0 build 155 revision c9e64dfd95b0cc89d6592544703d75d4f3866a27 build_type RELEASE built at 26 Apr 2021 05:33:01 UTC            

[fritshoogland-yugabyte]

I gotten the following error after waiting some time after table insertion:

yugabyte=# create index test_indexes_f1 on test_indexes(f1);
ERROR:  Catalog Version Mismatch: A DDL occurred while processing this query. Try again.
DETAIL:  Internal error: Operation expired: Transaction b2f76c96-c1fd-498a-aa8a-e796b0e7a3ad expired or aborted by a conflict: 40001

Is that a separate issue, or is this the same issue reporting it in a different way?