search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
9.04k stars 1.08k forks source link

[YB] During upsert on a table, other read thread is blocked #4217

Open ryli17 opened 4 years ago

ryli17 commented 4 years ago

Jira Link: DB-1593 Yugabyte version: 2.1.3.0-b26

This related to https://github.com/yugabyte/yugabyte-db/issues/4216

Symptom:

This query stuck.

ndeodhar commented 4 years ago

I'm able to reproduce this behavior. Here's what happens: We query every tablet to find the first row. While the script is running, there are no committed records in the DB but there are a lot of provisional records. It seems like every tablet scan takes time because of provisional records. This is evident from these log messages (there are a number of such messages):

W0423 02:22:38.069784  2655 long_operation_tracker.cc:112] Read running for 1.000s:
    @     0x7f2c352b811e  (unknown)
    @     0x7f2c35367066  syscall
    @     0x7f2c36417746  std::__atomic_futex_unsigned_base::_M_futex_wait_until()
    @     0x7f2c429dd8ee  yb::docdb::TransactionStatusCache::DoGetCommitTime()
    @     0x7f2c429de5a2  yb::docdb::TransactionStatusCache::GetCommitTime()
    @     0x7f2c429df04d  yb::docdb::IntentAwareIterator::ProcessIntent()
    @     0x7f2c429e6c2e  yb::docdb::IntentAwareIterator::SeekToSuitableIntent<>()
    @     0x7f2c429dfc82  yb::docdb::IntentAwareIterator::SeekIntentIterIfNeeded()
    @     0x7f2c429dfcbb  yb::docdb::IntentAwareIterator::valid()
    @     0x7f2c429c4077  yb::docdb::DocRowwiseIterator::HasNext()
    @     0x7f2c429eaa15  yb::docdb::PgsqlReadOperation::Execute()
    @     0x7f2c434abd8b  yb::tablet::AbstractTablet::HandlePgsqlReadRequest()
    @     0x7f2c434c8326  yb::tablet::Tablet::HandlePgsqlReadRequest()
    @     0x7f2c43dd53d5  yb::tserver::TabletServiceImpl::DoRead()
    @     0x7f2c43dd656d  yb::tserver::TabletServiceImpl::CompleteRead()
    @     0x7f2c43dd7fe4  yb::tserver::TabletServiceImpl::Read()

Also, count_intents shows a high count of intents on all tservers:

$ ./tserver/bin/yb-ts-cli --server_address <IP1>:9100 count_intents
I0423 02:22:32.879046 10838 mem_tracker.cc:249] MemTracker: hard memory limit is 6.201147 GB
I0423 02:22:32.879210 10838 mem_tracker.cc:251] MemTracker: soft memory limit is 5.270975 GB
1000168
$ ./tserver/bin/yb-ts-cli --server_address <IP2>:9100 count_intents
I0423 02:22:35.455430 10851 mem_tracker.cc:249] MemTracker: hard memory limit is 6.201147 GB
I0423 02:22:35.455608 10851 mem_tracker.cc:251] MemTracker: soft memory limit is 5.270975 GB
1310352
$ ./tserver/bin/yb-ts-cli --server_address <IP3>:9100 count_intents
I0423 02:22:40.415132 10870 mem_tracker.cc:249] MemTracker: hard memory limit is 6.201147 GB
I0423 02:22:40.415275 10870 mem_tracker.cc:251] MemTracker: soft memory limit is 5.270975 GB
1933024

cc: @bmatican @spolitov

rthallamko3 commented 7 months ago

@ryli17 , Have you tried on the most recent release?