wrong msgs kept and returned after compaction

[rkrambovitis]

config

log.retention.secs = 120
log.roll.secs = 120
log.cleanup.policy = cleanup

Operation

1. I published a few messages with the same key. Then some more with another key.
2. 2 minutes later (see retention time) I set some more messages with both keys.
3. I issued a consume request from 0

What I got back was not what I expected.

• From first file (14 I think), I got back messages with sequence id 18 and 19.
• When requesting 20, I again got back messages 18 and 19, which caused my client to loop infinitely.
• The next file was 24.
• When requesting 24, I got message 24 as expected.

So there are 2 issues here.

1. After compaction, the last message in file 14 should have been 23, rather than 19.
2. How do I get message 24 without knowing where the new file starts ?

tank-cli also crashed when attempting to consume 0

#0  Buffer::Reserved (this=<optimized out>) at Switch/buffer.h:173
#1  TankClient::get_buffer (this=0x7fffee91da80) at tank_client.h:474
#2  TankClient::try_recv (this=this@entry=0x7fffee91da80, c=c@entry=0xdd5fa0) at client.cpp:2010
#3  0x000000000042338b in TankClient::poll (this=0x7fffee91da80, timeoutMS=<optimized out>) at client.cpp:2453
#4  0x0000000000404aaf in main (argc=1, argv=0x7fffee91f3a8) at cli.cpp:359

[markpapadakis]

I was not able to reproduce this problem. In fact, I got exactly what I should get back, using your configuration and following your instructions to the letter. Maybe I got something wrong though ? Did you use tank-cli set -K to set those messages?

[rkrambovitis]

no, I used Java to put. I'll try to reproduce using tank-cli

[markpapadakis]

If you preserved the partition segments please keep them around and give me path to the data directory and the tank-cli command you used so that I can reproduce the problem with tank cli crash.

[rkrambovitis]

I have been able to reproduce with tank-cli. I misreported originally,

• my "roll.secs" was half of the retention secs. So use this example config:

log.retention.secs = 20
log.roll.secs = 10
log.cleanup.policy = cleanup

Order of actions:

1. ./tank-cli -b localhost:11011 -p 0 -t foo set -K remain="some text"
2. (few iterations of this one): ./tank-cli -b localhost:11011 -p 0 -t foo set -K clean="some other text"

[markpapadakis]

Resolved.