tarantool / queue

Create task queues, add and take jobs, monitor failed tasks
Other
237 stars 52 forks source link

fifottl queue entries with an explicit ttl get stuck in an odd state #191

Open mannnick24 opened 2 years ago

mannnick24 commented 2 years ago

when this syntax is used to put data in the buffer

queue.tube.test_buffer:put(data, {ttr=60, ttl=15768000000000000})

You end up with entries that have a ttl of 0 and a large next_event time

issue after the test fails the buffer has entries in a state where the ttl is 0, but next event is large [[select "task_id" from "test_buffer" where "ttl" = 0;]] the entries are not returned from take as they are expired, but they should not have expired

using docker image tarantool/tarantool:2.8.4 and a nodejs client

Place tarantool --version output here. Tarantool 2.8.4-0-g47e6bd362 Target: Linux-x86_64-RelWithDebInfo Build options: cmake . -DCMAKE_INSTALL_PREFIX=/usr/local -DENABLE_BACKTRACE=ON Compiler: /usr/bin/cc /usr/bin/c++ C_FLAGS: -fexceptions -funwind-tables -fno-omit-frame-pointer -fno-stack-protector -fno-common -fopenmp -msse2 -std=c11 -Wall -Wextra -Wno-strict-aliasing -Wno-char-subscripts -Wno-format-truncation -Wno-gnu-alignof-expression -fno-gnu89-inline -Wno-cast-function-type CXX_FLAGS: -fexceptions -funwind-tables -fno-omit-frame-pointer -fno-stack-protector -fno-common -fopenmp -msse2 -std=c++11 -Wall -Wextra -Wno-strict-aliasing -Wno-char-subscripts -Wno-format-truncation -Wno-invalid-offsetof -Wno-gnu-alignof-expression -Wno-cast-function-type

Steps to reproduce see the project at https://github.com/mannnick24/tarantool-fifottl-explicit-ttl/blob/main/docker/Dockerfile this code uses node tarantool connection to concurrently call queue.tube.test_buffer:take, queue.tube.test_buffer:put and queue.tube.test_buffer:ack

How to run:

npm i

build.sh (in docker)

npm run test

Actual behavior after the test fails the buffer has entries in a state where the ttl is 0, but next event is large [[select "task_id" from "test_buffer" where "ttl" = 0;]] the entries are not returned from take as they are expired, but they should not have expired

e.g. box.execute([[select "task_id", "status", "created", "ttr", "ttl", "next_event" from "el_buffer" where "ttl" = 0 limit 20;]])

metadata:

name: task_id type: unsigned

name: status type: string

name: created type: unsigned

name: ttr type: unsigned

name: ttl type: unsigned

name: next_event type: unsigned rows:

[4, 'r', 1662662470225835, 60000000, 0, 18446744073709551615]

Expected behavior the buffer would be empty

mannnick24 commented 2 years ago

further experimenting: the value I was putting in for the ttl was huge queue.tube.test_buffer:put(data, {ttr=60, ttl=15768000000000000}) when I tried with a more reasonable large number queue.tube.test_buffer:put(data, {ttr=60, ttl=2000000000}) The test works as expected - so all that's needed is some input validation

LeonidVas commented 2 years ago

Hmm, looks like this issue is more related with https://github.com/tarantool/queue . So, unless I missed something, I'll move the issue to https://github.com/tarantool/queue .

LeonidVas commented 2 years ago

image