Open gapisback opened 1 year ago
Hi @gapisback , I also ran into the issue of this message while inserting 20M kv pairs (each 16 bytes large) in a single thread.
btree_pack(): req->num_tuples=6291456 exceeded output size limit, req->max_tuples=6291456 btree_pack failed: No space left on device
Any idea how to address this? Thanks :)
P.S. If I increase the size of kv to 128 bytes (8 bytes key and 120 bytes value), this issue could not be reproduced.
Thanks for reporting this @chrisxu333 -- I'm afraid that I don't have much more to add.
In your failing repro situation: while inserting 20M kv pairs (each 16 bytes large) in a single thread.
, can you clarify what were the sizes of the key and the value?
There were some set of known instabilities around trunk bundle mgmt, and at some point (~ 12 months ago) these were discussed internally with Splinter dev engineers. I have since moved on from that project and this repo, so am not able to provide any meaningful suggestions.
Cc:'ing @rtjohnso who is the gate-keepeer for this repo now and may have been doing some work to stabilize some of these areas.
@gapisback Thanks for your kindly reply and explanation. Regarding the key and value size for the failing scenario, I used 8 byte key and 8 byte value.
Moreover, I also opened a new issue about another likely deadlock bug regarding O_DIRECT that I encountered (https://github.com/vmware/splinterdb/issues/620). It would be really helpful if you or whoever is working on this take a look at your convenience :) Thank you!
I believe I've seen the issue with small kv-pairs before. It is due to an estimate of the maximum number of items that might be in a trunk node: https://github.com/vmware/splinterdb/blob/9359c9aa782dd56cfd7f0c2e20e6f92988fe2500/src/trunk.c#L9632 It assumes kv-pairs are at least 32 bytes.
You could try changing the divisor from 32 to 16, or you could just pad out your kv-pairs to 32 bytes.
This is a long-term item to fix due to limitations in other parts of the code.
The test case
splinterdb_stress_test.c:test_issue_458_mini_destroy_unused_debug_assert
is currently commented out.It was added as part of commit SHA f3c92ef6cc6 to fix issue #545 (under PR #561). The test case is a simple workload of a single client loading 100M short k/v pairs.
Repro has been provided as part of this branch: agurajada/570-filter-addr-ne-0-assert
When enabled, that test runs into the following assertion. Repro'ed on
/main
@ SHA b2245ac:A possibly related issue that should be investigated as part of this item is that while the workload is running, we see these messages:
These messages also do appear with release binary, but the test case seems to succeed. (Of course, it's a debug assert that is tripping.)
Historical note: This specific assertion has been reported and is mixed-up in the annals of issue #545 (bug in routing_filter_prefetch()). That bug has been fixed separately, so I'm peeling off this different failure to its own issue.