moonlibs / xqueue

Imbue tarantool space with power of queue
15 stars 12 forks source link

Runat contains bad value (monotonic clock) #2

Open Mons opened 5 years ago

Mons commented 5 years ago

I've failed with usage of monotonic clock on linux within distributed system.

In general monotonic clock specified as "monotonic time since some unspecified starting point". On some system it equals to unix timestamp. But on other it contains in fact uptime of system. So, just by machine reboot (or having several replicated nodes) time consistency became broken

Clock will be changed to realtime clock.

I will consider timestamps, earlier than 10 years ago as monotonic timestamps to avoid deletion of data with ttl tasks or early fireups of delayed tasks (I don't believe in hosts with 10y uptime running tarantool with xqueue)

It is recommeded to upgrage all runat fields.

for updating space the following snipped may be used

local sp = box.space.spacename
local ix = sp.index.runat
local fn = ix.parts[1].fieldno
local clock = require'clock'
local log = require'log'
while true do
    local c = 0
    for _,t in ix:pairs({ 0 }, { iterator='GT' }):take(100) do
        if t[fn] > 10*365*86400 then break end
        local newval = t[fn] - clock.monotonic() + clock.realtime()
        if newval < 10*365*86400 then error("Failed to correct runat") end
        log.info("Fix %s time %s to %s",t[1],t[fn], newval)
        sp:update({t[1]},{{'=',fn, newval }})
        c=c+1
    end
    if c == 0 then break end;
end
Mons commented 5 years ago

Upgrade policy:

  1. In branch master scm-1 rockspec will be disabled with an error, pointing to this issue
  2. In branch broken will be original version scm-1, which would be available to install, if required
  3. In branch monofix will be transitional version (scm-2), that will support live fix without downtime of queue
  4. In branch master will be version scm-2 with clock.realtime instead of clock.monotonic

So:

  1. If you want to upgrade, install version 2, update all tasks, then instal version 3
  2. If you don't want to upgrade, take rockspec from branch broken (not recommended)