Potential Kong ACL Plugin issue 403s occasionally for windows at a time

[jeremyjpj0916]

Summary

Interesting ticket came in today from a user. Seeing 403's on their proxy as well as 200's, the Consumer user is authenticated as per my HTTP Log plugin has their Consumer username set on these 403's short stopping at Kong . These are users that are intermittently getting sometimes 200's back from backends and sometimes 403's(from kong) in chunks. Checking webserver logs and seeing no webserver logs rules out this 403 option: https://github.com/Kong/kong/blob/master/kong/plugins/acl/handler.lua#L68 ,

which leaves me here: https://github.com/Kong/kong/blob/master/kong/plugins/acl/handler.lua#L117 .

I am noticing these 403's much more frequently in our Stage environment which has same number of consumers but wayyy more proxies in it than prod(think like 1,500+ proxies vs like 300). And each proxy represents an route id acl uuid group whitelist to add to certain consumers to gain access, some as many as 100+ acl groups(representing access to individual proxies). I am wondering if maybe there is something at scale during run time that has issues in the lua logic. Maybe when consumers are getting added to newer proxies and simultaneously calling proxies too actively? Something about the invalidation and cache management of a consumer and his various acl's? The fact I can see 200's and 403's for an Authenticated user in chunked windows tells me something has to be fishy on the Authorization cache side of the ACL plugin.

Will try to think of a good way I could add some lua logging of vars to this acl handler in our Stage to potentially capture the moment and then diff with the admin API which might prove the cache referenced at one point was lacking the ACL groups the user actually possessed. Might be good to review the code base and see if there are limitations to its design currently either in resource scale count or during active additions or subtractions on a users ACL group and the potential for lost groups in the users group list for extended amounts of time(because the 403's will not briefly pop up, they can persist for large chunks), likely assumed because of cache rebuilding.

If anyone has any theories and wants me to run a debug snippet for a few days I will drop it in too 👍 .

Steps To Reproduce

1. Not really sure at the moment. Large gateway with many proxies+consumers+acl groups per consumer and whitelisted by proxy route uuid.

The proxies in question use a route with the acl + oauth2 plugin enabled ontop of them.

Additional Details & Logs

• Kong 1.3.0

[bungle]

If it works on some proxies, but not all, I think issue is with invalidations / cluster events / worker events. It might be worth it to try if this has any effect: https://github.com/Kong/lua-resty-worker-events/pull/24

Edit: it might have nothing to do with this. But perhaps easy to test.

[jeremyjpj0916]

@bungle At what location in Kong's codebase would it be best to:

assert(ev.enable_burst_mode(0.020, 1000))

and then disable it too?

ev.disable_burst_mode()

Or can these be invoked specifically in the ACL plugin handler itself?

Speaking to the bug itself it seems when I rolling redeployed Kong nodes just now the 403's have gone away(team was in a panic about non-prod testing completion 💀 ). Glad seeing this in stage prior to noticeable prod impact though(saw potentially 8 403's in prod this week that may or may not be related to this, vs the like 2,000+ daily tx's in stage exhibiting this). Its nothing specific to 1.3 or anything either, checking global logs 3 months back it seems this has been sneaking around.

Edit - Uh oh more of tables getting too large with tombstones I imagine...

xxxx@cqlsh:kong_stage> select count(*) from kong_stage.cluster_events;
ReadFailure: Error from server: code=1300 [Replica(s) failed to execute read] message="Operation failed - received 0 responses and 1 failures" info={'failures': 1, 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}
xxxx@cqlsh:kong_stage>

I imagine when this table gets rekt with too many tombstones going over the tombstone threshold that could cause some funny Kong behavior? Although I would think this table having too many tombstones to be unable to support a query would mean ACLs persisting to a user in cache that should have already been invalidated, not the opposite(user unable to access something they should have access to):

tombstone_failure_threshold 100,000 default

I even reduced the gc grace periods to gc_grace_seconds to 86400 (1 day vs 10 day default!):

CREATE TABLE kong_stage.cluster_events (
    channel text,
    at timestamp,
    node_id uuid,
    id uuid,
    data text,
    nbf timestamp,
    PRIMARY KEY (channel, at, node_id, id)
) WITH CLUSTERING ORDER BY (at ASC, node_id ASC, id ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = 'Kong cluster events broadcasting and polling'
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 86400
    AND gc_grace_seconds = 86400
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

Maybe I should make cluster_events table specifically gc_grace_seconds something like an hour exclusively compared to other tables. Or would it really be that harmful if I am only running Kong on these C nodes to bump tombstone_failure_threshold from 100,000 to say 10,000,000? Would that be bad on C nodes that have specs of 2 CPU 8 GB RAM. Both have their negatives right? One side by lowering gc_grace_seconds opens you up for ghost records, and toombstone_failure_threshold bumps means you need more MEM to handle it eh? Maybe our capacity is good enough though bumping it up to 10,000,000 would not be such a bad thing though?

Last time I checked this table was flooded due to oauth2 client_credentials grant records.

[bungle]

@Tieske, did you have any plans on how to enable burst mode with Kong? And do you think it would be worth to try?

[Tieske]

I do not think burst-mode will help here. What it does is preventing out-of-memory errors (when writing to the shm) and "waiting for event data timed out" type errors.

If you do not see those in your logs, then it will not help.

[jeremyjpj0916]

Gotcha, yeah event logs reveal no such errors when this occurs. One would assume Kong is fully operational in the windows of impact.

[jeremyjpj0916]

Made some adjustments to the table to keep cluster_events accessible(and oauth2 as well for good measure):

ALTER TABLE oauth2_tokens  WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4', 'tombstone_compaction_interval':'3600'};
ALTER TABLE oauth2_tokens WITH gc_grace_seconds = 3600;
ALTER TABLE cluster_events  WITH compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4', 'tombstone_compaction_interval':'3600'};
ALTER TABLE cluster_events WITH gc_grace_seconds = 3600;

So far not seeing any improper 403's in stage last 24 hours but will keep watch and report back here if I see any pop up more. Otherwise I will close this in 3-4 weeks if I do not see any other occurrences. I can't think of why a cluster_events table over the tombstone failure threshold would cause this though.

[jeremyjpj0916]

Update on some relevant ACL stuff:

Hmm we had an ACL 403 event issue again we are tracking that hit us in prod once on consumer that should have not had that happen. We now have a Splunk query setup to capture it, looks something like this:

index=kong host!="*stage*" host!="*dev*" (HTTPStatus=200 OR (HTTPStatus=403 BackendLatency<0))
| eval okTime = if (HTTPStatus=200, _time, null())
| eval forbiddenTime = if (HTTPStatus=403, _time, null())
| stats dc(HTTPStatus) as stat_div,         
        count as Total,
        sum(eval(HTTPStatus==403)) as 403s,
        sum(eval(HTTPStatus==200)) as 200s, 
        latest(forbiddenTime) as "latest_403"
        earliest(okTime) as "earliest_200",
    by ServiceName, Consumer, GatewayHost
    | where latest_403>earliest_200
    | fieldformat latest_403 = strftime('latest_403', "%c")
    | fieldformat earliest_200 = strftime('earliest_200', "%c")
    | where stat_div>1

Which catches occurrences of consumers rightfully getting through to the service via 200's but then Kong sometimes stopping their access w a 403(even though they were not revoked).

Currently speculating some sort of acl cache poisoning because the prod occurrence was an older established proxy where the customer in question has the largest # of acl groups tied to their account(around 500+). At time of impact they were actively calling 11 proxies they have access to. and only 3 of them were still working during impact window and 8 were not. We correlated the impact window to when a self-service action was being done which does route/service/plugin/acl changes so cache actions were occurring. Its worth noting we run 4 instances of Kong and only 1 pod was the bad actor of the 403's on a legitimate customer. The suspected instance that received the CRUD self-service updates for the admin API calls was NOT the pod that caused the 403 failures either which was interesting, so through cluster events or its polling it musta had a cache rebuild invoked. The cure at the time of prod impact was to rolling redeploy the bad pod and then the problem resolved itself(the problem was present for 2 full hours of impact as well which is also interesting).

Worth noting no other consumers or proxies were impacted at this time. No std out logs revealing DB connection issues or cache lookup failures or OOM or anything.

I decided since we see blips of it to add some robust logging to Stage env within the ACL plugin, to hopefully catch the groups cache lookup value on the 403 situations:

ACL handler.lua

    else
      -- get the consumer groups, since we need those as cache-keys to make sure
      -- we invalidate properly if they change
      local consumer_groups, err = groups.get_consumer_groups(consumer_id)
      kong.ctx.plugin.currentgroups = consumer_groups
      if not consumer_groups then
        kong.log.err(err)
        return kong.response.exit(500, {
          message = "An unexpected error occurred"
        })
      end

.....

if to_be_blocked == true then -- NOTE: we only catch the boolean here!
      kong.log.err("AUTHENTICATED USER FAILED TO ACCESS PROXY '" .. ngx.ctx.service.name .. "'\nWITH CONSUMER `" .. ngx.ctx.authenticated_consumer.username .. "\nWITH GROUPS:")
      local aclGroups = dump(kong.ctx.plugin.currentgroups)
      if 3997 < string.len(aclGroups) then
        kong.log.err("-------- START OF ACL GROUPS --------")
        local i = 1
        local response_body_array = splitByChunk(aclGroups, 3986)
        for _,v in ipairs(response_body_array) do
           kong.log.err("Response Body Chunk #",tostring(i)," : ", v)
           i = i + 1
        end
        kong.log.err("-------- END OF ACL GROUPS --------")
      else
        kong.log.err(aclGroups)
      end

    return kong.response.exit(403, {
      message = "You cannot consume this service"
    })
  end

Which uses a few functions I dropped in too:

local function dump(o)
   if type(o) == 'table' then
      local s = '{ '
      for k,v in pairs(o) do
         if type(k) ~= 'number' then k = '"'..k..'"' end
         s = s .. '['..k..'] = ' .. dump(v) .. ','
      end
      return s .. '} '
   else
      return tostring(o)
   end
end

--Help break large strings into loggable chunks(seems actually ~4096 byte limitation)
local function splitByChunk(text, chunkSize)
    local s = {}
    for i=1, #text, chunkSize do
        s[#s+1] = text:sub(i,i+chunkSize - 1)
    end
    return s
end

Some of the logs are coming out like this interestingly(for some test acl's I setup in a sandbox to see if it was working):

2020/02/01 23:39:34 [error] 76#0: *8961 [kong] handler.lua:148 [acl] Response Body Chunk #1 : { [1] = unitTestGroup58,[2] = unitTestGroup89,[3] = unitTestGroup87,[4] = unitTestGroup36,[5] = unitTestGroup1,[6] = unitTestGroup40,[7] = unitTestGroup4,[8] = unitTestGroup106,[9] = unitTestGroup100,[10] = unitTestGroup88,[11] = unitTestGroup15,[12] = unitTestGroup21,[13] = unitTestGroup29,[14] = unitTestGroup17,[15] = unitTestGroup61,[16] = unitTestGroup35,[17] = unitTestGroup48,[18] = unitTestGroup18,[19] = unitTestGroup80,[20] = unitTestGroup84,[21] = unitTestGroup72,[22] = unitTestGroup56,[23] = unitTestGroup74,[24] = unitTestGroup22,[25] = unitTestGroup20,[26] = unitTestGroup90,[27] = unitTestGroup2,[28] = unitTestGroup52,[29] = unitTestGroup78,[30] = unitTestGroup49,[31] = unitTestGroup55,[32] = unitTestGroup27,[33] = unitTestGroup31,[34] = unitTestGroup9,[35] = unitTestGroup77,[36] = unitTestGroup60,[37] = unitTestGroup62,[38] = unitTestGroup63,[39] = unitTestGroup47,[40] = unitTestGroup8,[41] = unitTestGroup25,[42] = unitTestGroup107,[43] = unitTestGroup97,[44] = unitTestGroup83,[45] = unitTestGroup85,[46] = unitTestGroup5,[47] = unitTestGroup10,[48] = unitTestGroup66,[49] = unitTestGroup71,[50] = unitTestGroup70,[51] = unitTestGroup37,[52] = unitTestGroup46,[53] = unitTestGroup102,[54] = unitTestGroup108,[55] = unitTestGroup109,[56] = unitTestGroup91,[57] = unitTestGroup69,[58] = unitTestGroup24,[59] = unitTestGroup3,[60] = unitTestGroup57,[61] = unitTestGroup51,[62] = unitTestGroup39,[63] = 18fa4238-3a83-4da2-8969-303c7b65813a,[64] = unitTestGroup42,[65] = unitTestGroup99,[66] = unitTestGroup54,[67] = unitTestGroup96,[68] = unitTestGroup64,[69] = unitTestGroup94,[70] = unitTestGroup76,[71] = unitTestGroup82,[72] = unitTestGroup12,[73] = unitTestGroup11,[74] = unitTestGroup16,[75] = unitTestGroup92,[76] = unitTestGroup28,[77] = unitTestGroup14,[78] = unitTestGroup13,[79] = unitTestGroup30,[80] = unitTestGroup19,[81] = unitTestGroup23,[82] = unitTestGroup0,[83] = unitTestGroup59,[84] = unitTestGroup41,[85] = unitTestGroup73,[86] = unitTestGroup53,[87] = unitTestGroup6,[88] = unitTestGroup67,[89] = unitTestGroup50,[90] = unitTestGroup45,[91] = unitTestGroup26,[92] = unitTestGroup7,[93] = unitTestGroup75,[94] = unitTestGroup34,[95] = unitTestGroup104,[96] = unitTestGroup103,[97] = unitTestGroup81,[98] = unitTestGroup68,[99] = unitTestGroup44,[100] = unitTestGroup32,[101] = unitTestGroup38,[102] = unitTestGroup33,[103] = unitTestGroup105,[104] = unitTestGroup93,[105] = unitTestGroup65,[106] = unitTestGroup98,[107] = unitTestGroup86,[108] = unitTestGroup43,[109] = unitTestGroup79,[110] = unitTestGroup95,[111] = unitTestGroup101,["unitTestGroup8"] = unitTestGroup8,["unitTestGroup9"] = unitTestGroup9,["unitTestGroup79"] = unitTestGroup79,["unitTestGroup78"] = unitTestGroup78,["unitTestGroup27"] = unitTestGroup27,["unitTestGroup100"] = unitTestGroup100,["unitTestGroup12"] = unitTestGroup12,["unitTestGroup99"] = unitTestGroup99,["unitTestGroup17"] = unitTestGroup17,["18fa4238-3a83-4da2-8969-303c7b65813a"] = 18fa4238-3a83-4da2-8969-303c7b65813a,["unitTestGroup22"] = unitTestGroup22,["unitTestGroup31"] = unitTestGroup31,["unitTestGroup108"] = unitTestGroup108,["unitTestGroup49"] = unitTestGroup49,["unitTestGroup48"] = unitTestGroup48,["unitTestGroup64"] = unitTestGroup64,["unitTestGroup86"] = unitTestGroup86,["unitTestGroup95"] = unitTestGroup95,["unitTestGroup77"] = unitTestGroup77,["unitTestGroup43"] = unitTestGroup43,["unitTestGroup51"] = unitTestGroup51,["unitTestGroup42"] = unitTestGroup42,["unitTestGroup101"] = unitTestGroup101,["unitTestGroup98"] = unitTestGroup98,["unitTestGroup85"] = unitTestGroup85,["unitTestGroup84"] = unitTestGroup84,["unitTestGroup65"] = unitTestGroup65,["unitTestGroup28"] = unitTestGroup28,["unitTestGroup93"] = unitTestGroup93,["unitTestGroup105"] = unitTestGroup105,["unitTestGroup29"] = unitTestGroup29,["unitTestGroup33"] = unitTestGroup33,["unitTestGroup96"] = unitTestGroup96,["unitTestGroup32"] = unitTestGroup32,["unitTestGroup44"] = unitTestGroup44,["unitTestGroup18"] = unitTestG, client: 10.94
2020/02/01 23:39:34 [error] 76#0: *8961 [kong] handler.lua:148 [acl] Response Body Chunk #2 : roup18,["unitTestGroup80"] = unitTestGroup80,["unitTestGroup109"] = unitTestGroup109,["unitTestGroup81"] = unitTestGroup81,["unitTestGroup103"] = unitTestGroup103,["unitTestGroup2"] = unitTestGroup2,["unitTestGroup47"] = unitTestGroup47,["unitTestGroup104"] = unitTestGroup104,["unitTestGroup61"] = unitTestGroup61,["unitTestGroup34"] = unitTestGroup34,["unitTestGroup75"] = unitTestGroup75,["unitTestGroup72"] = unitTestGroup72,["unitTestGroup7"] = unitTestGroup7,["unitTestGroup83"] = unitTestGroup83,["unitTestGroup90"] = unitTestGroup90,["unitTestGroup26"] = unitTestGroup26,["unitTestGroup45"] = unitTestGroup45,["unitTestGroup50"] = unitTestGroup50,["unitTestGroup0"] = unitTestGroup0,["unitTestGroup6"] = unitTestGroup6,["unitTestGroup70"] = unitTestGroup70,["unitTestGroup53"] = unitTestGroup53,["unitTestGroup73"] = unitTestGroup73,["unitTestGroup63"] = unitTestGroup63,["unitTestGroup41"] = unitTestGroup41,["unitTestGroup57"] = unitTestGroup57,["unitTestGroup56"] = unitTestGroup56,["unitTestGroup107"] = unitTestGroup107,["unitTestGroup1"] = unitTestGroup1,["unitTestGroup52"] = unitTestGroup52,["unitTestGroup5"] = unitTestGroup5,["unitTestGroup102"] = unitTestGroup102,["unitTestGroup67"] = unitTestGroup67,["unitTestGroup23"] = unitTestGroup23,["unitTestGroup19"] = unitTestGroup19,["unitTestGroup74"] = unitTestGroup74,["unitTestGroup39"] = unitTestGroup39,["unitTestGroup30"] = unitTestGroup30,["unitTestGroup13"] = unitTestGroup13,["unitTestGroup38"] = unitTestGroup38,["unitTestGroup92"] = unitTestGroup92,["unitTestGroup94"] = unitTestGroup94,["unitTestGroup87"] = unitTestGroup87,["unitTestGroup16"] = unitTestGroup16,["unitTestGroup21"] = unitTestGroup21,["unitTestGroup11"] = unitTestGroup11,["unitTestGroup82"] = unitTestGroup82,["unitTestGroup20"] = unitTestGroup20,["unitTestGroup76"] = unitTestGroup76,["unitTestGroup14"] = unitTestGroup14,["unitTestGroup15"] = unitTestGroup15,["unitTestGroup54"] = unitTestGroup54,["unitTestGroup25"] = unitTestGroup25,["unitTestGroup58"] = unitTestGroup58,["unitTestGroup59"] = unitTestGroup59,["unitTestGroup24"] = unitTestGroup24,["unitTestGroup91"] = unitTestGroup91,["unitTestGroup3"] = unitTestGroup3,["unitTestGroup46"] = unitTestGroup46,["unitTestGroup55"] = unitTestGroup55,["unitTestGroup68"] = unitTestGroup68,["unitTestGroup71"] = unitTestGroup71,["unitTestGroup10"] = unitTestGroup10,["unitTestGroup69"] = unitTestGroup69,["unitTestGroup97"] = unitTestGroup97,["unitTestGroup62"] = unitTestGroup62,["unitTestGroup60"] = unitTestGroup60,["unitTestGroup106"] = unitTestGroup106,["unitTestGroup66"] = unitTestGroup66,["unitTestGroup89"] = unitTestGroup89,["unitTestGroup88"] = unitTestGroup88,["unitTestGroup40"] = unitTestGroup40,["unitTestGroup36"] = unitTestGroup36,["unitTestGroup4"] = unitTestGroup4,["unitTestGroup35"] = unitTestGroup35,["unitTestGroup37"] = unitTestGroup37,} , client: 10.94.77.235, server: kong, request: "POST /api/unittest/route499 HTTP/1.1", host: "gateway.company.com"
2020/02/01 23:39:34 [error] 76#0: *8961 [kong] handler.lua:151 [acl] -------- END OF ACL GROUPS --------, client: 10.94.77.235, server: kong, request: "POST /api/unittest/route499 HTTP/1.1", host: "gateway.company.com"

Isn't it a bit odd how the logging goes from keys being int's [1], [2], ex: [1] = unitTestGroup58,[2] = unitTestGroup89,. to table keys being strings of the values itself ,ex: ["unitTestGroup35"] = unitTestGroup35, in the print out? I found that to be pretty strange myself, why is that happening?

Ultimately this logging is only going to 100% prove the cache lookup at the time was invalid/bad (when the groups returned != the groups the user actually has within the db which grants him access on these older established proxies). It's no cure to where the actual culprit lives, but hopefully will help us start to speculate. Maybe ACL cache related rebuilding needs some sort of eventual consistency/use the stale cache while new cache rebuilds logic(similar to router/service rebuilds)? Unsure yet.

We tried reproducing in dev by making tons of proxies, tons of consumers each with 100+ acl groups and then we scripted them calling the proxies and then adding/removing an arbitrary group not relevant to the services taking traffic in question and could not reproduce this problem. Then again its really hard trying to reproduce in a vacuum without all kinda customers calling services and self-service integrations doing real admin CRUD on the gateway real time.

Occurrences of this issue over last 3 months: 4 Identified prod occurrences - 1 big one 7 Identified non-prod occurrences - 1 big one

My colleague holds out hope Kong 2.0 and a separation of the Kong core cache and Kong plugins cache might help. I am slightly skeptical it will and think this has probably been a small problem lurking around that rarely rears its head across a # of Kong versions and only can be seen in realtime gateways that get used by a variety of folks and a lot of proxies.

Kong version 1.4.2 + 1.4.3.

[jeremyjpj0916]

All right, today it hit us in stage again and we had proper logging in the ACL plugin injected to capture values and confirm on it.

Notes:

Correct ACL Group: 9dafcd13-f3e2-4bf1-bdf4-d47725278352 (as seen in the database and admin API) Confirmed, call them XYZ consumer DOES have group as part of the ACL whitelist for the proxy.

XYZ has in total 300-400 ACL groups full of kong route resource uuid's.

Custom debug logging printed groups at time of impact (which are route UUIDS) from cache, NOTE the missing UUID from above being present:

2020/02/11 18:00:40 [error] 72#0: *10217680 [kong] handler.lua:139 [acl] AUTHENTICATED USER FAILED TO ACCESS PROXY 'eligibilities.v2.alpha'
WITH CONSUMER `XYZ
WITH GROUPS:, client: 10.48.104.59, server: kong, request: "POST /api/eligibilities/v2/read HTTP/1.1", host: "gateway-stage-core.company.com"
2020/02/11 18:00:40 [error] 72#0: *10217680 [kong] handler.lua:142 [acl] -------- START OF ACL GROUPS --------, client: 10.48.104.59, server: kong, request: "POST /api/eligibilities/v2/read HTTP/1.1", host: "gateway-stage-core.company.com"
2020/02/11 18:00:40 [error] 72#0: *10217680 [kong] handler.lua:146 [acl] Response Body Chunk #1 : { [1] = b4223cb7-1bd1-47a9-917a-4a2ff3e501f2,[2] = 89ab79bf-1992-4589-9093-0de146a35911,[3] = 8ae89228-7802-49a3-9a47-79e5e83c81ea,[4] = 9b1f5550-899d-4dc8-8a81-8da1f2d738cb,[5] = 22608af0-a841-4f18-8547-e8282e9f7ab5,[6] = 694093c1-b139-4fd9-9a51-ee1b8a7f6278,[7] = 86b8e7dd-c684-4bf7-88f7-abacd0098db4,[8] = 40d00a2f-fedc-4bfd-bb6c-978ae82e6018,[9] = 70a35be0-7dc5-4397-91e9-2c776e16baa7,[10] = 461e58b1-2af3-4a51-95bf-ca45bcf929db,[11] = 5786dbed-308c-4306-a894-2824d8dc33a8,[12] = 16354b2b-1997-4473-82ad-04d23062361c,[13] = 30e8f43e-9f26-4872-99ee-a2fff988d87c,[14] = b605f115-1a9b-4e8c-8d0b-182a78e62b55,[15] = c7ec6939-701d-4ccd-9fc1-c9feae839078,[16] = b400ef30-46f3-4d60-82b9-febcf1902a60,[17] = 783ac4ed-de8d-4fa8-985b-08b56d5564d4,[18] = b2295405-7e14-4e84-b521-179c30aeb057,[19] = dcaa38f1-2946-40a8-91eb-6ba1b4774dea,[20] = 3249bd7a-390c-4930-a58d-5147d14224d2,[21] = d2c68c7b-7f84-47b8-8d7e-c6dbdba6c21e,[22] = decb0383-f5f8-41b5-879b-1f06a46c2d4c,[23] = 0ccc8eb7-4f4a-45f5-9ad0-c07c332290bd,[24] = f2a5f291-f395-40a0-99fa-b01fadbf319d,[25] = 497f883f-17bc-49dc-a1db-eec395669ebc,[26] = f310d5c2-dff5-4fd7-82f5-76aefcd3a835,[27] = 38107061-795d-4c91-86f4-fe10ac7b398a,[28] = 3745ca8f-eadf-4833-aafe-3b445dcf0c55,[29] = a76917ec-482f-49fe-ac35-15d0da0f5db7,[30] = ca3f6847-d142-4d0e-8288-d06646c4d7bf,[31] = 8771053d-9c60-4b68-ab38-16546c16ceb1,[32] = 0f74c14f-a7d7-4607-8f3e-8a94b65ef722,[33] = 1e0145a8-92d2-4505-9026-497d4d97de41,[34] = 0291e545-1057-455d-87fb-a9b564adc2c6,[35] = f4161095-3c36-4db0-b7cc-133a16840715,[36] = b02a12e3-f86b-4833-bb79-f44edf454086,[37] = 1283d9bf-b31f-4559-974f-fd0cba00237e,[38] = 8e711e06-433b-4d94-b876-25d8db940062,[39] = 53f48477-2219-45db-a09d-4a0bbfcad4a0,[40] = c366c0a2-7432-4781-a71a-129df4c07919,[41] = ad6abf1d-124d-453d-b314-834958bf3bc0,[42] = 232b34bf-85bf-4530-9283-d0ff857718db,[43] = 354da00e-e0c7-484e-a2db-8e3cb1cb7a87,[44] = 6ce58232-224a-4d5a-88e2-0a81b3457b98,[45] = b7f954ca-cb9d-
4aab-a770-1b4a4ec2143b,[46] = 00918cda-39bf-4d95-9ea6-82ca542a25fd,[47] = 41625273-9839-45c6-812b-936136e28d9d,[48] = f0d0a4ca-2e29-4535-b845-36223eaf7642,[49] = 2e86401e-d12f-4e5f-8afc-7a2f7bc941b9,[50] = d258fa9b-80a5-4831-bced-fc1c77c8c7e9,[51] = f8641a40-9aea-4f75-9e92-60f1abb83b7f,[52] = 18329d50-9d61-477a-a55f-8d0cf7938dd2,[53] = cd227d94-bc12-4fb0-b37d-b18938562691,[54] = dfa0c8c2-333d-4ff0-82c8-e4a8f1b208a1,[55] = c1ece03c-8bc3-4061-89f9-f555a68b1dd9,[56] = 86814b19-66c3-4941-a41e-ac481ea503cf,[57] = 8ef8a307-80dd-4263-86aa-6f62a0b9ca4b,[58] = cd49631a-fcee-4a06-9018-122c9875c2bc,[59] = bf5f7bb5-e53e-4010-bb79-bafa45963a44,[60] = 7b043ff6-c956-40d0-a5f4-86355c1a1501,[61] = 4634bdc0-e408-421c-94a3-c46e0ea2cfd2,[62] = 2f47f549-fcc4-4ef3-b90e-ff2c13abd577,[63] = 07a3848e-5f50-47fd-ae67-0944d05d5545,[64] = 7abb5fc0-2efd-4db4-a174-c9e6cf63ff45,[65] = ac471188-1a7b-4c77-9bd2-4f8911f18e69,[66] = 1daa79d3-03cf-4976-9290-710e2393a0cc,[67] = 4a8d353d-7e5b-44c0-bf37-adbb95199302,[68] = 9e4e6d98-37ae-45cf-b82e-5c31574a5b23,[69] = 3b4adc69-3d4c-457a-8469-a0a36f27e28d,[70] = b2d60265-b250-402f-8ea4-045a846b436f,[71] = 4e0c36a5-cc73-415e-8835-82e8bebbd452,[72] = 6092ac20-1204-4906-b1dd-6cdcb49882ba,[73] = 6089bd00-1215-40d3-bc02-03b38afe318f,[74] = 9e409f06-6c62-4768-a06a-ddac44526a6b,[75] = b96ace91-6ff0-4020-9c62-aa5d1717f4f5,[76] = 57359db2-416e-47f6-b7c9-ee0c552f6110,[77] = acd1c7f3-1ad6-4bed-948a-c00ee4c7a908,[78] = 8594fea5-6f9d-4bd4-8ecb-1b4fb6aa4dae,[79] = d15f88ba-7601-4d0b-8441-fd57578d80e2,[80] = 1b627735-387d-462f-9e9c-025a283ece23,[81] = 5313b0c2-28e9-42a7-9878-e94eba2f8345,[82] = 65b96331-2d7f-49bf-92bf-ce770d61d78b,[83] = 3c8a3db0-9f12-49fe-81f4-e857ce17ae0c,[84] = 632a06e2-7eff-468b-871a-bb46ab5fcfe3,[85] = 74e217b4-ce7f-4ec4-8182-cdcedbd170dd,[86] = edf94ea4-b833-49f9-9d12-d76519fc8d7e,[87] = af758bef-cbe8-4d92-b981-368cd2875646,[88] = 7c5a9fb6-4f8e-4948-9211-345d0fc236e9,[89] = 0dd5f21f-5ffd-4dd2-9af7-d6f8a2ad4d03,[90] = e7e03082-4f2a-4a42-95b5-8f1099e38436,[91] = 099dd004-c1f6-49c4-8af9-11, client: 1
2020/02/11 18:00:40 [error] 72#0: *10217680 [kong] handler.lua:146 [acl] Response Body Chunk #2 : 3559d9407c,[92] = 12e967d8-4cb2-4f0a-a78f-d00b6c356542,[93] = 33b5f5fa-b132-418a-9b30-764421ba48fd,[94] = 034dd659-8dbf-4bbb-a786-3b93e48e83c3,[95] = 749c6a72-430d-4e6e-8e5c-7eea463110a6,[96] = a57ee5d3-2b00-4d55-9d87-32f7cdaea9a3,[97] = ac606c1c-b549-4242-b337-29efc071592b,[98] = 30e72c94-e14a-43c6-ac64-803e77beea09,[99] = 2d7f105b-2b67-427f-a7ec-ac3bce006c54,[100] = 91c4c862-d59a-4951-9b1b-310b9c7fcfeb,[101] = 3b456311-5e40-4dcc-85f8-aee4cc0140f5,[102] = 3d226c0c-408d-4ce3-9c1b-0983f25e3037,[103] = b816509a-5b18-4a0b-abc7-236da8183a2b,[104] = 31df8801-9716-47a2-b947-039c04f46a58,[105] = ef71ea26-c3cc-4d86-aafc-4cc74d793ca5,[106] = 00122cd4-714e-4ef5-b5e8-b6dd1bdcfff4,[107] = f651f949-ebdf-492a-9c27-dee79dcc4d1a,[108] = 1709075c-f52b-4c4a-bfa0-6102acc2a839,[109] = 3c3ac134-46b4-44b6-b124-2dc064a52d0d,[110] = 2a75879b-8026-4bee-a54a-f23da4ea8fab,[111] = b2a9a2e7-35b8-46ca-b4e5-e37062be901a,[112] = 94ea8a53-5a3f-47a6-998a-4120beed1146,[113] = 032fb99e-7019-4ed6-8b76-43db343bfe74,[114] = 4ffe3036-20c7-41b2-9690-880f582942fe,[115] = c5a32e1f-3ce4-4472-b075-c80e4101d506,[116] = 1bc222da-b5a6-48cc-86f2-1e6d2059517c,[117] = bf9eb7d2-4000-4bc5-a0aa-9ed723446670,[118] = 70211ed0-1c14-4404-81cb-c32cd07c6aea,[119] = 2da824ad-5b7b-4549-9951-668ac2716e28,[120] = 34f8f947-727f-49dd-942c-11437dae1069,[121] = e090b785-4e91-4242-bd81-bb8c05aadff7,[122] = 9f7f684e-3772-48e4-8d9e-f06a2a309143,[123] = dc82884b-47f7-431a-9f44-b4bda2087042,[124] = 5521d4a6-dfb1-406e-bfbc-8c8aaadaa167,[125] = b0a223ba-5b03-48c1-9908-dfd9c2b6fa7e,[126] = 3f848bfc-25d1-42af-8cdf-0f96cb09969b,[127] = 3ab73648-2507-4f4c-b5eb-607a920eb617,[128] = 80e87915-c764-4997-9d36-bf16c557a0de,[129] = 385c8386-9495-41ab-9ec3-87da6fd71357,[130] = 25c936e6-c4d3-4365-91b1-895fb5df0a44,[131] = 419aab86-bdbb-419d-af92-246bfb300a52,[132] = 0aa3671c-a12a-46d2-bf28-44e6214c62a1,[133] = ec400470-c360-41c1-9200-a7c8421864a8,[134] = 95fc50d5-339a-40d6-ace0-088e442d1e35,[135] = be12
0696-83bb-412d-a40b-841321bde8ed,[136] = e78e79fc-1961-458d-9f37-27227f74212f,[137] = 4dcb8c3d-ecd9-4477-a6d6-e0874b43e18b,[138] = 58b259f2-fe1d-4e9a-89d9-5883a8ff7f55,[139] = 72b1dd45-0808-4320-b4d8-cc16870c754c,[140] = 22e868cd-19e6-410c-bc27-fb4aacc1f804,[141] = ea9524a0-fc97-4179-81f1-1911848b6ed3,[142] = 03281ce1-f689-42c8-a468-f3246086f768,[143] = 321a6cb7-5a98-43a6-8a54-3be98e14b373,[144] = 95675da0-0ef1-479f-81ca-7579ee808426,[145] = b8c8fdec-cc3e-49a6-8dab-af232a3032e6,[146] = 20ae9654-7393-47d3-b0f5-7266465f254d,[147] = 91fad533-a33c-4514-b147-25b7e14d25a2,[148] = b70e0a33-a91a-4c72-b07b-293c3d689147,[149] = 6853f129-c501-41cd-afe5-ad008c0701f2,[150] = cd8b60dd-1087-4cae-a352-fd4b094c1f0d,[151] = fbd20b17-f83f-43f9-8e16-e70fd87d8e9b,[152] = a7b1f5e0-59d7-46be-aae6-106b361615da,[153] = c49bc6b4-513f-4bbe-a7e0-2640aca97a9d,[154] = fe6f4b16-2b44-4560-88ab-b0634dddc03a,[155] = b149ffdc-6afb-4a69-b6e1-0326a3668021,[156] = ef50c0e9-0745-40ca-a360-a95bb28c311c,[157] = 2d6c2cd9-587a-45f5-a5d6-04aae4082a20,[158] = 759cb737-0c03-4869-ac48-827946c5e1aa,[159] = 8cf7a075-a998-4691-a45b-663d3625d485,[160] = 32294715-ddef-46da-94d7-b16a05ba5fff,[161] = 532dca3b-e023-4324-b6e3-95872974285a,[162] = 8b9482da-f8c9-40e9-a92b-98410bc54611,[163] = 142dc95a-53ae-4edf-81f1-3e0bfb7371a1,[164] = 8ea4a540-30b0-4a37-8122-869cdd93eb53,[165] = 4d404a0a-f8bb-40c6-a3ad-acea5ff00e65,[166] = 79ec26a5-ea34-40d6-ab3a-a8995144bf66,["783ac4ed-de8d-4fa8-985b-08b56d5564d4"] = 783ac4ed-de8d-4fa8-985b-08b56d5564d4,["461e58b1-2af3-4a51-95bf-ca45bcf929db"] = 461e58b1-2af3-4a51-95bf-ca45bcf929db,["5313b0c2-28e9-42a7-9878-e94eba2f8345"] = 5313b0c2-28e9-42a7-9878-e94eba2f8345,["38107061-795d-4c91-86f4-fe10ac7b398a"] = 38107061-795d-4c91-86f4-fe10ac7b398a,["bf5f7bb5-e53e-4010-bb79-bafa45963a44"] = bf5f7bb5-e53e-4010-bb79-bafa45963a44,["2f47f549-fcc4-4ef3-b90e-ff2c13abd577"] = 2f47f549-fcc4-4ef3-b90e-ff2c13abd577,["8e711e06-433b-4d94-b876-25d8db940062"] = 8e711e06-433b-4d94-b876-25d8db940062,["c7ec6939-701d-4ccd-9fc1-c9feae839078"] = c7ec6, client: 1
2020/02/11 18:00:40 [error] 72#0: *10217680 [kong] handler.lua:146 [acl] Response Body Chunk #3 : 939-701d-4ccd-9fc1-c9feae839078,["07a3848e-5f50-47fd-ae67-0944d05d5545"] = 07a3848e-5f50-47fd-ae67-0944d05d5545,["1bc222da-b5a6-48cc-86f2-1e6d2059517c"] = 1bc222da-b5a6-48cc-86f2-1e6d2059517c,["ca3f6847-d142-4d0e-8288-d06646c4d7bf"] = ca3f6847-d142-4d0e-8288-d06646c4d7bf,["8b9482da-f8c9-40e9-a92b-98410bc54611"] = 8b9482da-f8c9-40e9-a92b-98410bc54611,["694093c1-b139-4fd9-9a51-ee1b8a7f6278"] = 694093c1-b139-4fd9-9a51-ee1b8a7f6278,["b2295405-7e14-4e84-b521-179c30aeb057"] = b2295405-7e14-4e84-b521-179c30aeb057,["1e0145a8-92d2-4505-9026-497d4d97de41"] = 1e0145a8-92d2-4505-9026-497d4d97de41,["3c8a3db0-9f12-49fe-81f4-e857ce17ae0c"] = 3c8a3db0-9f12-49fe-81f4-e857ce17ae0c,["6ce58232-224a-4d5a-88e2-0a81b3457b98"] = 6ce58232-224a-4d5a-88e2-0a81b3457b98,["31df8801-9716-47a2-b947-039c04f46a58"] = 31df8801-9716-47a2-b947-039c04f46a58,["9e409f06-6c62-4768-a06a-ddac44526a6b"] = 9e409f06-6c62-4768-a06a-ddac44526a6b,["fe6f4b16-2b44-4560-88ab-b0634dddc03a"] = fe6f4b16-2b44-4560-88ab-b0634dddc03a,["12e967d8-4cb2-4f0a-a78f-d00b6c356542"] = 12e967d8-4cb2-4f0a-a78f-d00b6c356542,["cd227d94-bc12-4fb0-b37d-b18938562691"] = cd227d94-bc12-4fb0-b37d-b18938562691,["72b1dd45-0808-4320-b4d8-cc16870c754c"] = 72b1dd45-0808-4320-b4d8-cc16870c754c,["8ef8a307-80dd-4263-86aa-6f62a0b9ca4b"] = 8ef8a307-80dd-4263-86aa-6f62a0b9ca4b,["b4223cb7-1bd1-47a9-917a-4a2ff3e501f2"] = b4223cb7-1bd1-47a9-917a-4a2ff3e501f2,["70211ed0-1c14-4404-81cb-c32cd07c6aea"] = 70211ed0-1c14-4404-81cb-c32cd07c6aea,["8ea4a540-30b0-4a37-8122-869cdd93eb53"] = 8ea4a540-30b0-4a37-8122-869cdd93eb53,["f0d0a4ca-2e29-4535-b845-36223eaf7642"] = f0d0a4ca-2e29-4535-b845-36223eaf7642,["d258fa9b-80a5-4831-bced-fc1c77c8c7e9"] = d258fa9b-80a5-4831-bced-fc1c77c8c7e9,["95675da0-0ef1-479f-81ca-7579ee808426"] = 95675da0-0ef1-479f-81ca-7579ee808426,["16354b2b-1997-4473-82ad-04d23062361c"] = 16354b2b-1997-4473-82ad-04d23062361c,["53f48477-2219-45db-a09d-4a0bbfcad4a0"] = 53f48477-2219-45db-a09d-4a0bbfcad4a
0,["032fb99e-7019-4ed6-8b76-43db343bfe74"] = 032fb99e-7019-4ed6-8b76-43db343bfe74,["3b456311-5e40-4dcc-85f8-aee4cc0140f5"] = 3b456311-5e40-4dcc-85f8-aee4cc0140f5,["c1ece03c-8bc3-4061-89f9-f555a68b1dd9"] = c1ece03c-8bc3-4061-89f9-f555a68b1dd9,["22608af0-a841-4f18-8547-e8282e9f7ab5"] = 22608af0-a841-4f18-8547-e8282e9f7ab5,["1283d9bf-b31f-4559-974f-fd0cba00237e"] = 1283d9bf-b31f-4559-974f-fd0cba00237e,["86b8e7dd-c684-4bf7-88f7-abacd0098db4"] = 86b8e7dd-c684-4bf7-88f7-abacd0098db4,["30e72c94-e14a-43c6-ac64-803e77beea09"] = 30e72c94-e14a-43c6-ac64-803e77beea09,["00918cda-39bf-4d95-9ea6-82ca542a25fd"] = 00918cda-39bf-4d95-9ea6-82ca542a25fd,["2da824ad-5b7b-4549-9951-668ac2716e28"] = 2da824ad-5b7b-4549-9951-668ac2716e28,["0aa3671c-a12a-46d2-bf28-44e6214c62a1"] = 0aa3671c-a12a-46d2-bf28-44e6214c62a1,["a7b1f5e0-59d7-46be-aae6-106b361615da"] = a7b1f5e0-59d7-46be-aae6-106b361615da,["497f883f-17bc-49dc-a1db-eec395669ebc"] = 497f883f-17bc-49dc-a1db-eec395669ebc,["9f7f684e-3772-48e4-8d9e-f06a2a309143"] = 9f7f684e-3772-48e4-8d9e-f06a2a309143,["b2a9a2e7-35b8-46ca-b4e5-e37062be901a"] = b2a9a2e7-35b8-46ca-b4e5-e37062be901a,["8594fea5-6f9d-4bd4-8ecb-1b4fb6aa4dae"] = 8594fea5-6f9d-4bd4-8ecb-1b4fb6aa4dae,["fbd20b17-f83f-43f9-8e16-e70fd87d8e9b"] = fbd20b17-f83f-43f9-8e16-e70fd87d8e9b,["a76917ec-482f-49fe-ac35-15d0da0f5db7"] = a76917ec-482f-49fe-ac35-15d0da0f5db7,["6853f129-c501-41cd-afe5-ad008c0701f2"] = 6853f129-c501-41cd-afe5-ad008c0701f2,["41625273-9839-45c6-812b-936136e28d9d"] = 41625273-9839-45c6-812b-936136e28d9d,["b02a12e3-f86b-4833-bb79-f44edf454086"] = b02a12e3-f86b-4833-bb79-f44edf454086,["cd49631a-fcee-4a06-9018-122c9875c2bc"] = cd49631a-fcee-4a06-9018-122c9875c2bc,["d15f88ba-7601-4d0b-8441-fd57578d80e2"] = d15f88ba-7601-4d0b-8441-fd57578d80e2,["2d7f105b-2b67-427f-a7ec-ac3bce006c54"] = 2d7f105b-2b67-427f-a7ec-ac3bce006c54,["0f74c14f-a7d7-4607-8f3e-8a94b65ef722"] = 0f74c14f-a7d7-4607-8f3e-8a94b65ef722,["ad6abf1d-124d-453d-b314-834958bf3bc0"] = ad6abf1d-124d-453d-b314-834958bf3bc0,["e7e03082-4f2a-4a42-95b5-8f1099e3, client: 1
2020/02/11 18:00:40 [error] 72#0: *10217680 [kong] handler.lua:146 [acl] Response Body Chunk #4 : 8436"] = e7e03082-4f2a-4a42-95b5-8f1099e38436,["6089bd00-1215-40d3-bc02-03b38afe318f"] = 6089bd00-1215-40d3-bc02-03b38afe318f,["89ab79bf-1992-4589-9093-0de146a35911"] = 89ab79bf-1992-4589-9093-0de146a35911,["70a35be0-7dc5-4397-91e9-2c776e16baa7"] = 70a35be0-7dc5-4397-91e9-2c776e16baa7,["2e86401e-d12f-4e5f-8afc-7a2f7bc941b9"] = 2e86401e-d12f-4e5f-8afc-7a2f7bc941b9,["4e0c36a5-cc73-415e-8835-82e8bebbd452"] = 4e0c36a5-cc73-415e-8835-82e8bebbd452,["b2d60265-b250-402f-8ea4-045a846b436f"] = b2d60265-b250-402f-8ea4-045a846b436f,["0291e545-1057-455d-87fb-a9b564adc2c6"] = 0291e545-1057-455d-87fb-a9b564adc2c6,["1709075c-f52b-4c4a-bfa0-6102acc2a839"] = 1709075c-f52b-4c4a-bfa0-6102acc2a839,["b8c8fdec-cc3e-49a6-8dab-af232a3032e6"] = b8c8fdec-cc3e-49a6-8dab-af232a3032e6,["91c4c862-d59a-4951-9b1b-310b9c7fcfeb"] = 91c4c862-d59a-4951-9b1b-310b9c7fcfeb,["8ae89228-7802-49a3-9a47-79e5e83c81ea"] = 8ae89228-7802-49a3-9a47-79e5e83c81ea,["dcaa38f1-2946-40a8-91eb-6ba1b4774dea"] = dcaa38f1-2946-40a8-91eb-6ba1b4774dea,["f8641a40-9aea-4f75-9e92-60f1abb83b7f"] = f8641a40-9aea-4f75-9e92-60f1abb83b7f,["759cb737-0c03-4869-ac48-827946c5e1aa"] = 759cb737-0c03-4869-ac48-827946c5e1aa,["3ab73648-2507-4f4c-b5eb-607a920eb617"] = 3ab73648-2507-4f4c-b5eb-607a920eb617,["354da00e-e0c7-484e-a2db-8e3cb1cb7a87"] = 354da00e-e0c7-484e-a2db-8e3cb1cb7a87,["58b259f2-fe1d-4e9a-89d9-5883a8ff7f55"] = 58b259f2-fe1d-4e9a-89d9-5883a8ff7f55,["33b5f5fa-b132-418a-9b30-764421ba48fd"] = 33b5f5fa-b132-418a-9b30-764421ba48fd,["e090b785-4e91-4242-bd81-bb8c05aadff7"] = e090b785-4e91-4242-bd81-bb8c05aadff7,["4d404a0a-f8bb-40c6-a3ad-acea5ff00e65"] = 4d404a0a-f8bb-40c6-a3ad-acea5ff00e65,["142dc95a-53ae-4edf-81f1-3e0bfb7371a1"] = 142dc95a-53ae-4edf-81f1-3e0bfb7371a1,["532dca3b-e023-4324-b6e3-95872974285a"] = 532dca3b-e023-4324-b6e3-95872974285a,["f4161095-3c36-4db0-b7cc-133a16840715"] = f4161095-3c36-4db0-b7cc-133a16840715,["ac606c1c-b549-4242-b337-29efc071592b"] = ac606c1c-b549-4242-b3
37-29efc071592b,["30e8f43e-9f26-4872-99ee-a2fff988d87c"] = 30e8f43e-9f26-4872-99ee-a2fff988d87c,["1daa79d3-03cf-4976-9290-710e2393a0cc"] = 1daa79d3-03cf-4976-9290-710e2393a0cc,["74e217b4-ce7f-4ec4-8182-cdcedbd170dd"] = 74e217b4-ce7f-4ec4-8182-cdcedbd170dd,["6092ac20-1204-4906-b1dd-6cdcb49882ba"] = 6092ac20-1204-4906-b1dd-6cdcb49882ba,["8cf7a075-a998-4691-a45b-663d3625d485"] = 8cf7a075-a998-4691-a45b-663d3625d485,["57359db2-416e-47f6-b7c9-ee0c552f6110"] = 57359db2-416e-47f6-b7c9-ee0c552f6110,["0ccc8eb7-4f4a-45f5-9ad0-c07c332290bd"] = 0ccc8eb7-4f4a-45f5-9ad0-c07c332290bd,["4a8d353d-7e5b-44c0-bf37-adbb95199302"] = 4a8d353d-7e5b-44c0-bf37-adbb95199302,["ef50c0e9-0745-40ca-a360-a95bb28c311c"] = ef50c0e9-0745-40ca-a360-a95bb28c311c,["b149ffdc-6afb-4a69-b6e1-0326a3668021"] = b149ffdc-6afb-4a69-b6e1-0326a3668021,["af758bef-cbe8-4d92-b981-368cd2875646"] = af758bef-cbe8-4d92-b981-368cd2875646,["d2c68c7b-7f84-47b8-8d7e-c6dbdba6c21e"] = d2c68c7b-7f84-47b8-8d7e-c6dbdba6c21e,["c49bc6b4-513f-4bbe-a7e0-2640aca97a9d"] = c49bc6b4-513f-4bbe-a7e0-2640aca97a9d,["7abb5fc0-2efd-4db4-a174-c9e6cf63ff45"] = 7abb5fc0-2efd-4db4-a174-c9e6cf63ff45,["cd8b60dd-1087-4cae-a352-fd4b094c1f0d"] = cd8b60dd-1087-4cae-a352-fd4b094c1f0d,["ac471188-1a7b-4c77-9bd2-4f8911f18e69"] = ac471188-1a7b-4c77-9bd2-4f8911f18e69,["232b34bf-85bf-4530-9283-d0ff857718db"] = 232b34bf-85bf-4530-9283-d0ff857718db,["4dcb8c3d-ecd9-4477-a6d6-e0874b43e18b"] = 4dcb8c3d-ecd9-4477-a6d6-e0874b43e18b,["91fad533-a33c-4514-b147-25b7e14d25a2"] = 91fad533-a33c-4514-b147-25b7e14d25a2,["40d00a2f-fedc-4bfd-bb6c-978ae82e6018"] = 40d00a2f-fedc-4bfd-bb6c-978ae82e6018,["20ae9654-7393-47d3-b0f5-7266465f254d"] = 20ae9654-7393-47d3-b0f5-7266465f254d,["f2a5f291-f395-40a0-99fa-b01fadbf319d"] = f2a5f291-f395-40a0-99fa-b01fadbf319d,["321a6cb7-5a98-43a6-8a54-3be98e14b373"] = 321a6cb7-5a98-43a6-8a54-3be98e14b373,["03281ce1-f689-42c8-a468-f3246086f768"] = 03281ce1-f689-42c8-a468-f3246086f768,["3249bd7a-390c-4930-a58d-5147d14224d2"] = 3249bd7a-390c-4930-a58d-5147d14224d2,["c366c0a2-7432-4781, client: 1
2020/02/11 18:00:40 [error] 72#0: *10217680 [kong] handler.lua:146 [acl] Response Body Chunk #5 : -a71a-129df4c07919"] = c366c0a2-7432-4781-a71a-129df4c07919,["00122cd4-714e-4ef5-b5e8-b6dd1bdcfff4"] = 00122cd4-714e-4ef5-b5e8-b6dd1bdcfff4,["95fc50d5-339a-40d6-ace0-088e442d1e35"] = 95fc50d5-339a-40d6-ace0-088e442d1e35,["b816509a-5b18-4a0b-abc7-236da8183a2b"] = b816509a-5b18-4a0b-abc7-236da8183a2b,["22e868cd-19e6-410c-bc27-fb4aacc1f804"] = 22e868cd-19e6-410c-bc27-fb4aacc1f804,["3b4adc69-3d4c-457a-8469-a0a36f27e28d"] = 3b4adc69-3d4c-457a-8469-a0a36f27e28d,["b70e0a33-a91a-4c72-b07b-293c3d689147"] = b70e0a33-a91a-4c72-b07b-293c3d689147,["419aab86-bdbb-419d-af92-246bfb300a52"] = 419aab86-bdbb-419d-af92-246bfb300a52,["e78e79fc-1961-458d-9f37-27227f74212f"] = e78e79fc-1961-458d-9f37-27227f74212f,["b400ef30-46f3-4d60-82b9-febcf1902a60"] = b400ef30-46f3-4d60-82b9-febcf1902a60,["f310d5c2-dff5-4fd7-82f5-76aefcd3a835"] = f310d5c2-dff5-4fd7-82f5-76aefcd3a835,["be120696-83bb-412d-a40b-841321bde8ed"] = be120696-83bb-412d-a40b-841321bde8ed,["ec400470-c360-41c1-9200-a7c8421864a8"] = ec400470-c360-41c1-9200-a7c8421864a8,["25c936e6-c4d3-4365-91b1-895fb5df0a44"] = 25c936e6-c4d3-4365-91b1-895fb5df0a44,["385c8386-9495-41ab-9ec3-87da6fd71357"] = 385c8386-9495-41ab-9ec3-87da6fd71357,["79ec26a5-ea34-40d6-ab3a-a8995144bf66"] = 79ec26a5-ea34-40d6-ab3a-a8995144bf66,["ea9524a0-fc97-4179-81f1-1911848b6ed3"] = ea9524a0-fc97-4179-81f1-1911848b6ed3,["b96ace91-6ff0-4020-9c62-aa5d1717f4f5"] = b96ace91-6ff0-4020-9c62-aa5d1717f4f5,["7b043ff6-c956-40d0-a5f4-86355c1a1501"] = 7b043ff6-c956-40d0-a5f4-86355c1a1501,["b0a223ba-5b03-48c1-9908-dfd9c2b6fa7e"] = b0a223ba-5b03-48c1-9908-dfd9c2b6fa7e,["c5a32e1f-3ce4-4472-b075-c80e4101d506"] = c5a32e1f-3ce4-4472-b075-c80e4101d506,["f651f949-ebdf-492a-9c27-dee79dcc4d1a"] = f651f949-ebdf-492a-9c27-dee79dcc4d1a,["749c6a72-430d-4e6e-8e5c-7eea463110a6"] = 749c6a72-430d-4e6e-8e5c-7eea463110a6,["32294715-ddef-46da-94d7-b16a05ba5fff"] = 32294715-ddef-46da-94d7-b16a05ba5fff,["5786dbed-308c-4306-a894-2824d8dc33a8"] = 5786dbe
d-308c-4306-a894-2824d8dc33a8,["3f848bfc-25d1-42af-8cdf-0f96cb09969b"] = 3f848bfc-25d1-42af-8cdf-0f96cb09969b,["dc82884b-47f7-431a-9f44-b4bda2087042"] = dc82884b-47f7-431a-9f44-b4bda2087042,["b7f954ca-cb9d-4aab-a770-1b4a4ec2143b"] = b7f954ca-cb9d-4aab-a770-1b4a4ec2143b,["ef71ea26-c3cc-4d86-aafc-4cc74d793ca5"] = ef71ea26-c3cc-4d86-aafc-4cc74d793ca5,["632a06e2-7eff-468b-871a-bb46ab5fcfe3"] = 632a06e2-7eff-468b-871a-bb46ab5fcfe3,["a57ee5d3-2b00-4d55-9d87-32f7cdaea9a3"] = a57ee5d3-2b00-4d55-9d87-32f7cdaea9a3,["7c5a9fb6-4f8e-4948-9211-345d0fc236e9"] = 7c5a9fb6-4f8e-4948-9211-345d0fc236e9,["2a75879b-8026-4bee-a54a-f23da4ea8fab"] = 2a75879b-8026-4bee-a54a-f23da4ea8fab,["0dd5f21f-5ffd-4dd2-9af7-d6f8a2ad4d03"] = 0dd5f21f-5ffd-4dd2-9af7-d6f8a2ad4d03,["decb0383-f5f8-41b5-879b-1f06a46c2d4c"] = decb0383-f5f8-41b5-879b-1f06a46c2d4c,["3d226c0c-408d-4ce3-9c1b-0983f25e3037"] = 3d226c0c-408d-4ce3-9c1b-0983f25e3037,["18329d50-9d61-477a-a55f-8d0cf7938dd2"] = 18329d50-9d61-477a-a55f-8d0cf7938dd2,["acd1c7f3-1ad6-4bed-948a-c00ee4c7a908"] = acd1c7f3-1ad6-4bed-948a-c00ee4c7a908,["034dd659-8dbf-4bbb-a786-3b93e48e83c3"] = 034dd659-8dbf-4bbb-a786-3b93e48e83c3,["3c3ac134-46b4-44b6-b124-2dc064a52d0d"] = 3c3ac134-46b4-44b6-b124-2dc064a52d0d,["edf94ea4-b833-49f9-9d12-d76519fc8d7e"] = edf94ea4-b833-49f9-9d12-d76519fc8d7e,["4634bdc0-e408-421c-94a3-c46e0ea2cfd2"] = 4634bdc0-e408-421c-94a3-c46e0ea2cfd2,["099dd004-c1f6-49c4-8af9-113559d9407c"] = 099dd004-c1f6-49c4-8af9-113559d9407c,["9b1f5550-899d-4dc8-8a81-8da1f2d738cb"] = 9b1f5550-899d-4dc8-8a81-8da1f2d738cb,["65b96331-2d7f-49bf-92bf-ce770d61d78b"] = 65b96331-2d7f-49bf-92bf-ce770d61d78b,["3745ca8f-eadf-4833-aafe-3b445dcf0c55"] = 3745ca8f-eadf-4833-aafe-3b445dcf0c55,["34f8f947-727f-49dd-942c-11437dae1069"] = 34f8f947-727f-49dd-942c-11437dae1069,["dfa0c8c2-333d-4ff0-82c8-e4a8f1b208a1"] = dfa0c8c2-333d-4ff0-82c8-e4a8f1b208a1,["1b627735-387d-462f-9e9c-025a283ece23"] = 1b627735-387d-462f-9e9c-025a283ece23,["80e87915-c764-4997-9d36-bf16c557a0de"] = 80e87915-c764-4997-9d36-bf16c557a0de,["9e4e, client: 1
2020/02/11 18:00:40 [error] 72#0: *10217680 [kong] handler.lua:146 [acl] Response Body Chunk #6 : 6d98-37ae-45cf-b82e-5c31574a5b23"] = 9e4e6d98-37ae-45cf-b82e-5c31574a5b23,["2d6c2cd9-587a-45f5-a5d6-04aae4082a20"] = 2d6c2cd9-587a-45f5-a5d6-04aae4082a20,["94ea8a53-5a3f-47a6-998a-4120beed1146"] = 94ea8a53-5a3f-47a6-998a-4120beed1146,["5521d4a6-dfb1-406e-bfbc-8c8aaadaa167"] = 5521d4a6-dfb1-406e-bfbc-8c8aaadaa167,["4ffe3036-20c7-41b2-9690-880f582942fe"] = 4ffe3036-20c7-41b2-9690-880f582942fe,["86814b19-66c3-4941-a41e-ac481ea503cf"] = 86814b19-66c3-4941-a41e-ac481ea503cf,["b605f115-1a9b-4e8c-8d0b-182a78e62b55"] = b605f115-1a9b-4e8c-8d0b-182a78e62b55,["bf9eb7d2-4000-4bc5-a0aa-9ed723446670"] = bf9eb7d2-4000-4bc5-a0aa-9ed723446670,["8771053d-9c60-4b68-ab38-16546c16ceb1"] = 8771053d-9c60-4b68-ab38-16546c16ceb1,} , client: 10.48.104.59, server: kong, request: "POST /api/eligibilities/v2/read HTTP/1.1", host: "gateway-stage-core.company.com"
2020/02/11 18:00:40 [error] 72#0: *10217680 [kong] handler.lua:149 [acl] -------- END OF ACL GROUPS --------, client: 10.48.104.59, server: kong, request: "POST /api/eligibilities/v2/read HTTP/1.1", host: "gateway-stage-core.company.com"

The above print was produced by this acl plugin handler.lua file to help capture and debug:

local constants = require "kong.constants"
local tablex = require "pl.tablex"
local groups = require "kong.plugins.acl.groups"

local setmetatable = setmetatable
local concat = table.concat
local kong = kong

local EMPTY = tablex.readonly {}
local BLACK = "BLACK"
local WHITE = "WHITE"

local mt_cache = { __mode = "k" }
local config_cache = setmetatable({}, mt_cache)

local function dump(o)
   if type(o) == 'table' then
      local s = '{ '
      for k,v in pairs(o) do
         if type(k) ~= 'number' then k = '"'..k..'"' end
         s = s .. '['..k..'] = ' .. dump(v) .. ','
      end
      return s .. '} '
   else
      return tostring(o)
   end
end

--Help break large strings into loggable chunks(seems actually ~4096 byte limitation)
local function splitByChunk(text, chunkSize)
    local s = {}
    for i=1, #text, chunkSize do
        s[#s+1] = text:sub(i,i+chunkSize - 1)
    end
    return s
end

local function get_to_be_blocked(config, groups, in_group)
  local to_be_blocked
  if config.type == BLACK then
    to_be_blocked = in_group
  else
    to_be_blocked = not in_group
  end

  if to_be_blocked == false then
    -- we're allowed, convert 'false' to the header value, if needed
    -- if not needed, set dummy value to save mem for potential long strings
    to_be_blocked = config.hide_groups_header and "" or concat(groups, ", ")
  end

  return to_be_blocked
end

local ACLHandler = {}

ACLHandler.PRIORITY = 950
ACLHandler.VERSION = "2.1.0"

function ACLHandler:access(conf)
  -- simplify our plugins 'conf' table
  local config = config_cache[conf]
  if not config then
    local config_type = (conf.blacklist or EMPTY)[1] and BLACK or WHITE

    config = {
      hide_groups_header = conf.hide_groups_header,
      type = config_type,
      groups = config_type == BLACK and conf.blacklist or conf.whitelist,
      cache = setmetatable({}, mt_cache),
    }

    config_cache[conf] = config
  end

  local to_be_blocked

  -- get the consumer/credentials
  local consumer_id = groups.get_current_consumer_id()
  if not consumer_id then
    local authenticated_groups = groups.get_authenticated_groups()
    if not authenticated_groups then
      kong.log.err("Cannot identify the consumer, add an authentication ",
                   "plugin to use the ACL plugin")

      return kong.response.exit(403, {
        message = "You cannot consume this service"
      })
    end

    local in_group = groups.group_in_groups(config.groups, authenticated_groups)
    to_be_blocked = get_to_be_blocked(config, authenticated_groups, in_group)

  else
    local authenticated_groups
    if not kong.client.get_credential() then
      -- authenticated groups overrides anonymous groups
      authenticated_groups = groups.get_authenticated_groups()
    end

    if authenticated_groups then
      consumer_id = nil

      local in_group = groups.group_in_groups(config.groups, authenticated_groups)
      to_be_blocked = get_to_be_blocked(config, authenticated_groups, in_group)

    else
      -- get the consumer groups, since we need those as cache-keys to make sure
      -- we invalidate properly if they change
      local consumer_groups, err = groups.get_consumer_groups(consumer_id)
      kong.ctx.plugin.currentgroups = consumer_groups
      if not consumer_groups then
        kong.log.err(err)
        return kong.response.exit(500, {
          message = "An unexpected error occurred"
        })
      end

      -- 'to_be_blocked' is either 'true' if it's to be blocked, or the header
      -- value if it is to be passed
      to_be_blocked = config.cache[consumer_groups]
      if to_be_blocked == nil then
        local in_group = groups.consumer_in_groups(config.groups, consumer_groups)
        to_be_blocked = get_to_be_blocked(config, consumer_groups, in_group)

        -- update cache
        config.cache[consumer_groups] = to_be_blocked
      end
    end
  end

  if to_be_blocked == true then -- NOTE: we only catch the boolean here!
      kong.log.err("AUTHENTICATED USER FAILED TO ACCESS PROXY '" .. ngx.ctx.service.name .. "'\nWITH CONSUMER `" .. ngx.ctx.authenticated_consumer.username .. "\nWITH GROUPS:")
      local aclGroups = dump(kong.ctx.plugin.currentgroups)
      if 3997 < string.len(aclGroups) then
        kong.log.err("-------- START OF ACL GROUPS --------")
        local i = 1
        local response_body_array = splitByChunk(aclGroups, 3986)
        for _,v in ipairs(response_body_array) do
           kong.log.err("Response Body Chunk #",tostring(i)," : ", v)
           i = i + 1
        end
        kong.log.err("-------- END OF ACL GROUPS --------")
      else
        kong.log.err(aclGroups)
      end

    return kong.response.exit(403, {
      message = "You cannot consume this service"
    })
  end

  if not conf.hide_groups_header and to_be_blocked then
    kong.service.request.set_header(consumer_id and
                                    constants.HEADERS.CONSUMER_GROUPS or
                                    constants.HEADERS.AUTHENTICATED_GROUPS,
                                    to_be_blocked)
  end
end

return ACLHandler

Admin API response to getting all ACL cache values(Done slightly later in the day today):

curl localhost:8001/consumers/XYZ/acls?size=1000 , note the present UUID from top of post

{"next":null,"data":[{"created_at":1567693365,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"76623373-386e-4395-95a5-cd8166f41421","group":"d5d9a44c-36ed-4a72-8038-8128f2bb52ab","tags":null},{"created_at":1565117200,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"ba02f33c-74f8-46b9-939d-470cfab58ade","group":"ce4ef832-c272-442e-bfd7-7d46eee37ce0","tags":null},{"created_at":1579461879,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"b5cad852-c0fd-467a-8d22-7b55bdc6df6c","group":"e0b2b463-f16b-4b87-b556-e1785a28e76b","tags":null},{"created_at":1578209876,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"f63a6071-a978-4810-a411-47cbbaa69717","group":"b8d351c3-4bc6-4a44-8496-a5bf14153262","tags":null},{"created_at":1574083614,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"5c4ffb02-43f7-4113-a208-ee0d8dcb1c94","group":"95d5db20-8aa4-4c24-86e0-ae54f80d664a","tags":null},{"created_at":1562695356,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"6951c828-23e5-44d0-b747-ba5f83050baa","group":"f14a8abb-2be2-4743-9ef7-0f80950edb34","tags":null},{"created_at":1578210229,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"7e904892-0420-417f-8c85-1a886c44f981","group":"e55a9927-c940-4dba-b52a-b62d3cba05f1","tags":null},{"created_at":1568645933,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"c11cf74a-ccff-44b3-8fa3-fbe08d88ef8b","group":"9dafcd13-f3e2-4bf1-bdf4-d47725278352","tags":null},{"created_at":1566315495,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"e396569f-1a02-4bb0-92ba-26d2fd9413e5","group":"a0369c12-bb2f-4b0c-a01e-a36ca579e2c9","tags":null},{"created_at":1564085499,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"25d4313b-67d8-4a02-8071-728eb93fdebd","group":"5f9fa557-2cc3-46c5-8058-00a499f32b75","tags":null},{"created_at":1576855231,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"46fd65bd-92f1-4131-a2db-057431984e2b","group":"ab4ff74a-8249-4887-9376-6ffd068aceaa","tags":null},{"created_at":1576855228,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"4cdaf9c8-8eea-4bf4-ac64-c4e05de0e020","group":"75b8ced1-5661-4dbb-8a8c-401c4a826ea7","tags":null},{"created_at":1560931394,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"6f04f469-fc4d-4d90-b8d8-cff30be61379","group":"2515afbf-364e-44ed-84d6-54fd0f723702","tags":null},{"created_at":1579459362,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"7672476b-0c27-4a41-9fc5-5ab8e1feacd1","group":"4aecdca8-1393-4c18-bd57-798af927ee1a","tags":null},{"created_at":1579706294,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"4b006bcf-4b0c-42b7-901e-6b268bd170e6","group":"6b124677-9bb9-45f1-997e-5c7c085b0519","tags":null},{"created_at":1549538888,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"606304e7-251c-4d1f-9416-bc67de6d4b5b","group":"e4d295f9-301e-4bd2-b52f-16c830026113","tags":null},{"created_at":1550664456,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"f9fe7152-49ab-4596-bdc0-1dd518274c9e","group":"18f016e8-dd5b-4137-af7d-15f804167c06","tags":null},{"created_at":1564660589,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"69114264-d23f-4357-a6ca-edcfff053c88","group":"859b1aa9-d35c-43ae-9f3b-71c664615cad","tags":null},{"created_at":1578209427,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"551c930b-e521-46cf-96ed-476bdd23e849","group":"84bf84f2-d2aa-4cd5-b821-f232ccbec487","tags":null},{"created_at":1579459372,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"05c6e50f-5bb1-4f9c-9ff2-3a7c8cfa9caa","group":"c630e577-26ed-49d5-8637-2a218ba01628","tags":null},{"created_at":1551958726,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"03f2bf1f-df15-4d53-953c-0033e5ec632d","group":"d7ecb31f-a293-4cd6-a1e9-87226e3d0243","tags":null},{"created_at":1572879128,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"1c32cf0b-3bc6-467b-b480-44c12dd3dbc3","group":"d2ef91db-ec3d-4e97-a2d1-a11d68095ee9","tags":null},{"created_at":1564675662,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"35f6dbe3-771c-4bd3-bcb4-36f77d4ca3db","group":"92d664a1-6883-4c3e-a638-d9f9b91542c4","tags":null},{"created_at":1578204774,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"4d002145-21f7-4e05-b9df-61aff22d253e","group":"795e94a4-a5f1-4102-b535-54fc5fdd4db4","tags":null},{"created_at":1578516887,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"14f860a7-c02c-4b96-9f37-f4642ed86266","group":"e54c1128-5732-4833-a3a9-91634f64c5c9","tags":null},{"created_at":1566577660,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"f43a47d8-ed73-4d85-aef9-974235131cd8","group":"a1207415-dc31-45f4-903c-e9594643a004","tags":null},{"created_at":1562169699,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"c6d2e94f-62fd-437b-ae4c-acd159e92ce8","group":"ddca3da5-b456-4fac-892d-857a49420758","tags":null},{"created_at":1571154817,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"fc104824-ab22-4a9c-92ef-32e12750d20b","group":"5019a582-7407-45ea-94bc-429b17fcf682","tags":null},{"created_at":1564085500,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"f4f86e3c-5fc5-45d4-bab1-228d65a1d361","group":"604e9bc1-aa3e-48fa-85b6-1612d3e0236b","tags":null},{"created_at":1571154828,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"44e660e4-de28-4842-9257-33dee14ffddc","group":"3743f861-f95a-41f0-b96a-23cd18436024","tags":null},{"created_at":1575455821,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"4e5537f7-55d8-4936-9b35-dccb125229ca","group":"8bb7bf97-0b7e-44cb-937f-0f423912bff2","tags":null},{"created_at":1579892174,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"9d827f1a-1349-4c03-b123-0335c57b305f","group":"4f6c3d6e-aae3-433d-992a-7618d6ae0388","tags":null},{"created_at":1578205676,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"a9544209-09c5-41db-a2f4-fceb6a6056d5","group":"92cbaf55-650b-4f77-96b6-b1a78a88f1a5","tags":null},{"created_at":1565293100,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"ac45a7f8-6189-49db-90cc-2f2f99aa8676","group":"ef5ee2c8-08a4-4060-920c-3e03b65e8fb3","tags":null},{"created_at":1579892172,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"9103366b-80ef-43a8-9493-5baf89cf7683","group":"17df1637-953c-476a-89ed-5a76376a1d25","tags":null},{"created_at":1576176047,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"8c685816-a813-494b-b032-4bf116f0d17a","group":"3d6fde28-bb0a-4b75-9284-3e84eff0b86c","tags":null},{"created_at":1562091066,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"2770ca0f-1d59-48d7-8edd-393b3da9c582","group":"604423a4-f501-4ea0-909c-5d24496dc18a","tags":null},{"created_at":1571154815,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"b19c82e1-848c-45c0-83a3-99791a9f123c","group":"28c7977d-dc9d-4534-a8b8-a0b3a91f2d79","tags":null},{"created_at":1576212532,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"06391822-5497-46e5-ad0d-66997231980c","group":"bbba6e46-9990-4fdb-975f-22242f6ecf43","tags":null},{"created_at":1564660590,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"7288d8b6-53b6-41f4-961c-b0b3a6ad24d7","group":"7bfe5902-edc8-41d8-9457-61c39438adeb","tags":null},{"created_at":1581442454,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"9fb5d319-f548-4e0e-ab68-1df869cd0a74","group":"982b3dac-6d84-4026-95c9-119a9dd9ca5b","tags":null},{"created_at":1566396734,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"354e77c6-c842-4c5a-88f2-ce4f7d1143b9","group":"632e0f60-66e9-461f-b728-f35619c5ebb8","tags":null},{"created_at":1539360136,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"34c43109-75eb-4087-8204-4eb49d10e4d8","group":"61eb2b68-d2be-4227-a646-3e231442a7a1","tags":null},{"created_at":1551349162,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"70f85955-fc88-4258-a311-2abd708cb86a","group":"1347e3ae-a21f-42be-9530-2c6e660fe3e9","tags":null},{"created_at":1565835916,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"6054b457-f4d9-4d27-af22-35244375c3fa","group":"98408add-a553-4f7b-804e-50e39cd732e4","tags":null},{"created_at":1577428151,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"46386eff-89aa-4325-a2b7-e008a4b26e54","group":"42e3ade8-67f6-4915-a37f-b79eb8ff860b","tags":null},{"created_at":1548411522,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"0b753904-fb4c-4bf6-9536-948d31c42331","group":"c12f5ce1-ea06-4764-97af-02736641c0b3","tags":null},{"created_at":1554461298,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"00b34bd4-6a22-43ca-9c6c-211691b494ad","group":"a9e2e860-585e-4375-b7dc-fe8fc1f8a60c","tags":null},{"created_at":1551959008,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"48c65762-4eb0-4d47-9a70-d6d620530846","group":"2e827f26-19d7-41d9-ac8d-cd84f9844ae1","tags":null},{"created_at":1564675660,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"3ac2cd5c-cc13-4828-ad80-daa1a86913e9","group":"677b04ec-52d1-4a7a-9128-8558fd51d926","tags":null},{"created_at":1572227348,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"6445347a-a99b-4a5c-a1d7-826d27e40c33","group":"20f2e001-34e6-4e63-9dfb-a46cf68c05a3","tags":null},{"created_at":1563962679,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"07d9b713-d534-4b30-b539-57b8175497f9","group":"9c0979e2-eb91-4ce1-8fcf-2a2ec3f93a26","tags":null},{"created_at":1579879581,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"68beffff-488a-4a70-941d-e55d51787f33","group":"faf9d98f-e6f6-4457-99c8-ef374b28bbde","tags":null},{"created_at":1568645930,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"45188f35-80b4-442c-931f-1245652d552d","group":"3e2941ce-ff47-49e0-851b-f82175122b3e","tags":null},{"created_at":1566316270,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"6cbaaf23-efcf-41a6-a2e4-ff8db15d0ee3","group":"bd0ea97d-5973-4e4f-8cce-2bf212b06b86","tags":null},{"created_at":1567523192,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"2848190a-7287-4070-9ba8-a4a4dfec69d7","group":"df5e7081-a66c-46b6-893d-cd2660829bf1","tags":null},{"created_at":1564675658,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"44f97664-3efe-487e-8009-5bce0049eaec","group":"37f1a8c3-efde-47f9-bfe5-d09642bcaa1b","tags":null},{"created_at":1578422571,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"d0fd1545-3dfa-4559-b6a3-475f35730566","group":"a735777f-10b7-43bf-8ccf-80ae675ef432","tags":null},{"created_at":1562361700,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"25febb01-54c7-4ad4-8e56-1b4f50675566","group":"65ef6324-601a-4433-8567-c9ba7506c8f4","tags":null},{"created_at":1574851331,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"7f1133d4-7c51-417b-a068-0edc0d24188a","group":"c3337429-65aa-49e7-99fe-c016d6150555","tags":null},{"created_at":1578209146,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"9196a7fa-497e-40ec-b436-d553157bd0e8","group":"b00f0536-ffdf-4130-a740-871d3695b6ed","tags":null},{"created_at":1555348279,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"d1f67d42-cf53-4149-a711-ff9171f33c9d","group":"9abbddd5-87e9-4acc-b017-2f65b4f1430d","tags":null},{"created_at":1570641386,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"089d6bc4-e75d-423d-9916-78f75010543e","group":"4f0879ea-50c3-4629-b7bd-6a867f0bc647","tags":null},{"created_at":1568662433,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"425d49a0-ca10-436c-9bee-9dbf8f69be99","group":"bd986d66-007e-4584-8a0b-6d71c390ea1c","tags":null},{"created_at":1566577005,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"8bfc5a78-d40c-404c-89b0-f34d4045f8d5","group":"e88f8ccc-3915-44a4-962c-6add78033b1b","tags":null},{"created_at":1578479186,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"d22f9df2-3794-462b-ab19-d78c7ecd49a6","group":"deaed621-6d06-46b1-b5c4-8723a00f6800","tags":null},{"created_at":1562169149,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"dea9024d-5631-4603-9594-f518fc1dfd74","group":"f4204ad6-b629-46ac-86e7-4ba5af13e78c","tags":null},{"created_at":1568720761,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"324226c9-2314-4b91-9eec-7eda011f1156","group":"4ff8354b-08cb-4a14-8623-7562565d56aa","tags":null},{"created_at":1567479524,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"9b53ed80-13ab-4ab8-9197-f65a1298e560","group":"b70a0e47-cd51-474b-ad6a-e27eef5b31bb","tags":null},{"created_at":1550757834,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"4882ec5f-b630-4004-8c32-1107e3653340","group":"c4c58f5a-39c2-4ae3-8550-3b273b3f6b17","tags":null},{"created_at":1579892173,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"f498f51c-c918-4f8e-b398-cd6bae16bc5a","group":"c725c964-7752-4b67-a2d9-6a135cd284c6","tags":null},{"created_at":1579461889,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"21c44c70-d3c7-44c8-a4d7-b882559284aa","group":"1174f7f9-191e-4c03-a2a3-4f0886afe661","tags":null},{"created_at":1567705394,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"e2bd043d-f3de-49cb-afe0-6024d20a3510","group":"22d82c0e-4363-441d-9b31-69acff58e11f","tags":null},{"created_at":1574928528,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"94d591c7-5ca9-4eb5-9d2f-6d68227b6363","group":"f0df490d-7af9-4877-acbf-e400d6200ca6","tags":null},{"created_at":1576747379,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"3f4cfcc3-158d-4773-9487-0b45c592a33c","group":"709672c2-d388-4e28-a111-6f39b1aa5d24","tags":null},{"created_at":1555348271,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"fb7fe1dc-eab6-4285-a106-5f2a44a767c6","group":"9796ad12-3467-4b84-8f50-b9ae9befde84","tags":null},{"created_at":1560988534,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"32c6be5a-e287-4ccb-b0ee-91a6db65e106","group":"55c9959f-f67f-4c1c-86d2-200010b915dd","tags":null},{"created_at":1561554751,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"63ee0c87-dcc9-4291-87da-75df9030bcb0","group":"3aaa1c8c-732f-4df7-af5a-d7cc8b502227","tags":null},{"created_at":1549538896,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"11064d95-c7ff-4f38-93ca-420ad74a0fb6","group":"59b557a4-0957-417b-8b98-7e43518e970d","tags":null},{"created_at":1553684035,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"0ae0c840-b7bf-43e0-bf0c-5819868b5406","group":"c9e09648-e70e-40ad-a816-7e23885616ff","tags":null},{"created_at":1575399752,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"1dd5ec29-4c4e-4253-a061-44834de2137f","group":"5e11210b-f9ba-42a7-9a7b-2ce74dde4296","tags":null},{"created_at":1554478069,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"3aff2dc0-ea70-43ff-bc07-0b1943418ea9","group":"a69ee7b1-9fd6-479a-a846-7b0670d6fd73","tags":null},{"created_at":1572227344,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"0a29ca29-44ac-43e4-bbf2-4a8c04b43ed7","group":"dbd85f51-89a9-499c-b388-05509212df93","tags":null},{"created_at":1578209287,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"45ed6490-9205-4be5-941e-598191bd99e2","group":"e1b7a0b9-9cf5-40f7-980e-f79ec83844ca","tags":null},{"created_at":1546444480,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"7b208539-d019-4ef4-8cf8-ef580817dbdf","group":"7b7d8537-ca8b-41a4-a2a8-3b06cb4c0752","tags":null},{"created_at":1578209775,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"67c9358f-9613-4f6f-b8ed-c8d22a6e2552","group":"7e801dec-d886-48dc-84f4-402857bf3d87","tags":null},{"created_at":1550584059,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"3db79631-df41-428c-873f-a7c44a5ea3cf","group":"0fdc936c-f5f9-4776-9a61-21d9ded46610","tags":null},{"created_at":1578206303,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"5be5d5c1-3bc7-414d-a5b2-86fea0e5fc1d","group":"0ad6aedb-aff6-48a3-bced-bcb1c8295c23","tags":null},{"created_at":1573589562,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"9f7ff68f-a2f6-479b-a7de-adeb78fe9085","group":"fce2d0aa-9522-4a52-bc60-611d62b1f222","tags":null},{"created_at":1550867371,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"d8957721-4086-45fe-8213-43364400667f","group":"8e990aee-5402-43f0-b4f5-5c07eeb1caf9","tags":null},{"created_at":1560365058,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"fe8c766e-37f1-48ba-9d78-9dfea54358f9","group":"bfe42db6-57a0-4cef-a896-99c9404e52be","tags":null},{"created_at":1564396414,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"c49c8a7d-dfd9-480d-a798-a86fa6d0d526","group":"f712c79c-383a-46f6-b59e-e311ab71a168","tags":null},{"created_at":1567706026,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"e8dc16b1-2f98-429f-bb69-abfbf495effa","group":"a03146f8-f4d7-4e66-ad15-3b96c7ed8bfb","tags":null},{"created_at":1554720073,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"4c50b98d-4f19-4b40-936f-c654eaa04bd6","group":"ccd95e0a-3b7f-44e8-976a-15b49d1227fa","tags":null},{"created_at":1579706297,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"7c40ffcc-d871-4331-9d8b-da8d0dadd7c3","group":"af391cf3-c498-4767-9cd3-903bf315076d","tags":null},{"created_at":1567703420,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"35e8ea43-50db-45b5-b8fb-ea8e5a45a28f","group":"c9003925-6da5-4d2b-8ff9-8e4a093d0224","tags":null},{"created_at":1579459371,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"983ec6d9-29d4-49e5-8671-bd4b80229f8c","group":"1337c4ba-fdbb-455c-b98a-94c8ad6f1b2d","tags":null},{"created_at":1575399753,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"d0cd2326-e53c-443d-a836-9c4cd1451196","group":"0bb5cbf9-f4c6-42c4-a54a-1622c9aa92a2","tags":null},{"created_at":1578519167,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"02e30e52-b63c-4c9c-85f8-16297b1cddc1","group":"4d2242ac-1451-4353-8da6-4d226f8c0ad1","tags":null},{"created_at":1538067397,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"bd6d2ce8-6b49-4da2-be86-fdaa3d70ee8b","group":"9e3662b7-636c-4657-8bb9-2a26e8f7eddb","tags":null},{"created_at":1570726798,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"2227f16a-25f3-4cc5-a15c-078c8bfaa754","group":"4db237b9-3153-4d32-a848-52ef6f8b2fe3","tags":null},{"created_at":1573838364,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"1b7a7548-471c-4bbc-ae70-9e5919bf2bee","group":"1e12d85e-5c2b-4db3-9ba8-b4cc31fcefa2","tags":null},{"created_at":1579461880,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"8cf69884-7a2e-4d2a-9378-6245d44238c8","group":"82078998-af82-4e80-be3e-310179c9f8b9","tags":null},{"created_at":1537511374,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"9071cfe9-9819-4027-9e55-625d4a1d7e37","group":"badda3fd-8ab0-482b-8125-950809177b2b","tags":null},{"created_at":1572438436,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"2d708719-ffd6-474d-ae86-9db4af4eb2e4","group":"a2933491-9423-466b-9f06-324422688bf2","tags":null},{"created_at":1565382161,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"b3d29239-e83c-492e-9925-6128fa1b30eb","group":"0d8cf876-b8fd-4c47-b8bb-7121321fa17c","tags":null},{"created_at":1537510536,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"63f39df8-01b4-4a3e-87fc-a266ba167534","group":"4b3ef797-ad9c-4085-bdac-72d55fe9e0b8","tags":null},{"created_at":1579875811,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"68ef610d-b632-40c8-88bc-376e742f5691","group":"79944b2c-0b96-4979-9e6e-c7319fca9773","tags":null},{"created_at":1578206035,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"2386c698-9bed-49d2-95cf-9ad2d855462f","group":"086b694f-344a-461f-b969-78f778f11bcf","tags":null},{"created_at":1570641383,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"b97f85a4-a94b-4224-9fa9-00c38b4fe227","group":"feb60557-b87c-4865-b46c-b8af8c941710","tags":null},{"created_at":1571813622,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"b374f19b-821d-48f6-b462-ac23342e07d6","group":"8b29a1af-66b9-468c-b234-2eaec30365b2","tags":null},{"created_at":1567703769,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"17a96bba-aced-46d8-bb93-f78ec70b8443","group":"e164b0ac-7283-480f-bd96-7dbb622c252f","tags":null},{"created_at":1551375675,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"d5d92d0d-c81a-426a-8707-3113fd8c671d","group":"35ceec9f-7de6-4c6d-9376-48ed64b70401","tags":null},{"created_at":1567700118,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"bb143e03-b43f-4898-9e19-edc97d5b95a9","group":"3a33cd26-530d-4361-93b2-9b7bc58d3c70","tags":null},{"created_at":1563797106,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"5773e9e7-8689-41a5-a27f-26f01d307bde","group":"aacf60de-ef65-43e9-92a5-3451a33c3029","tags":null},{"created_at":1578208599,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"e7ef2a83-e735-4f33-82a5-7115a03e1d89","group":"d4a4d258-02c0-448e-96a8-0ad56ed9fbd0","tags":null},{"created_at":1580449147,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"ecfff266-0ad2-47b9-baa2-993aa2c6b413","group":"3af4efe7-9f9d-4cf8-ba2a-cf85920ced3d","tags":null},{"created_at":1552318288,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"54ecaecd-98ee-4f9b-ad50-90b985c25486","group":"7f8dab2d-b848-4fe6-a952-0f013dd2406d","tags":null},{"created_at":1576855241,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"daa07e7c-5a60-43bd-8b98-c597db430e14","group":"01fb16ae-ce2c-4186-b350-b1c82a37a456","tags":null},{"created_at":1579875806,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"fb130dd9-5f99-4614-8762-8fec63e56da3","group":"47b2d093-8b9c-4ad8-a39e-953f70d35e9b","tags":null},{"created_at":1550664097,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"6be069a1-f241-4300-b9aa-d414d9af6334","group":"2baf47b6-cf12-4797-888d-b5f075c04e72","tags":null},{"created_at":1578210121,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"8bcafb68-37ad-4702-ba55-e8375d92dfb5","group":"c5efa87d-bf63-4f3f-997e-5c29183bddae","tags":null},{"created_at":1558380488,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"1a7134b2-8e23-4d21-a771-efc05d9bd61d","group":"5cd59475-ed09-4d65-b8af-577bb7d5a8a4","tags":null},{"created_at":1533848689,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"7939ae57-78e7-4b6d-9522-5de37c6a71a3","group":"2fd14860-bc49-40ba-998d-94ced69ddbcf","tags":null},{"created_at":1578205315,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"33c8fa59-d5fa-4e8f-8130-3fb8311a0d6f","group":"32985818-3f20-4d8a-9688-4be1b54564dd","tags":null},{"created_at":1566381561,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"6cc737fa-1883-4ab2-a5e4-188201d9abbc","group":"6654306f-f829-4286-9e98-ab721e84131b","tags":null},{"created_at":1571940591,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"640056e1-b87b-4d88-8d87-f5158c278ad3","group":"dafca5bf-fec0-4c42-b2a7-92905dad32fa","tags":null},{"created_at":1566577664,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"b8e57567-5c26-4f56-be27-44e18dd27fe8","group":"fc390a4e-c638-4571-970f-b03d3fc4b377","tags":null},{"created_at":1571154808,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"b3d634f5-164d-4840-895a-67bc7989ad9d","group":"e5c08f4b-ffe4-41f8-932a-e889358a8f4d","tags":null},{"created_at":1581313819,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"d10680c9-05a5-404d-ae4a-b9e2a3789be3","group":"3d0a3751-25b9-443f-b67a-63b3c71bac1a","tags":null},{"created_at":1563229607,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"2c610128-277a-446a-9a6c-076532fa37a3","group":"a74fb4a3-d3f0-43f6-8897-620bb507ad0e","tags":null},{"created_at":1548690786,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"c41da97d-619c-452c-85d6-b3c856c70be3","group":"9994156d-2cec-4b4a-b276-23e3a379ffd4","tags":null},{"created_at":1563228914,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"0a41373f-f6e6-4fcc-a577-8e436f0915d8","group":"f5ab0a0c-3a49-4a57-bd80-aaa8e1cd7071","tags":null},{"created_at":1572635148,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"6c09b026-c630-4feb-89ea-a3c74da1edd5","group":"af1df76c-d55e-44a1-88da-4a3185fcffe8","tags":null},{"created_at":1560365062,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"fd9956a5-12e2-4cbe-99e3-01af95a9c6c1","group":"de208a16-8457-4717-b5d4-358a42512145","tags":null},{"created_at":1568662431,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"1c6dfaee-6aac-4316-b19b-4eee68b339fe","group":"a271f6dd-0323-49b7-930c-e77f8a247719","tags":null},{"created_at":1569925585,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"17987403-dcc2-42ad-8aee-883716f99a89","group":"11964691-8b6f-4198-8476-05ffd7fb0f2b","tags":null},{"created_at":1562091070,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"19ac9522-9000-4327-a92c-564f9fe77a94","group":"ae578a49-2216-48fa-b588-0dd2f1c35814","tags":null},{"created_at":1581313818,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"347cea5c-aa04-4297-8434-4437f4688148","group":"21dbd4ed-bc78-4746-b3af-c10e2be0fbcf","tags":null},{"created_at":1553684476,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"ae71c9a9-4012-4c86-9b69-2ae8ef1ad0b5","group":"d29adb29-3e47-40d0-a0d6-886d0a059787","tags":null},{"created_at":1577338453,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"42a7aa85-2290-4514-a5a7-c2c5d51275c1","group":"c829143a-a282-4c7a-adb3-b2aa5c7d03b8","tags":null},{"created_at":1580149017,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"e89f4357-0073-4828-95c2-082ae950e2b2","group":"562eac54-885a-429c-92d0-3aadfb0125bd","tags":null},{"created_at":1576212530,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"be095922-0218-41fe-b157-a244ea4e3169","group":"1690951f-0a9e-4bca-8575-539e0c740f05","tags":null},{"created_at":1569925587,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"9b076618-6373-4346-ae2c-0f05833259af","group":"cb5f7a9d-8e5f-411f-984a-0b745930b00d","tags":null},{"created_at":1581372835,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"022f5c0a-35e3-48c1-9ad1-51d86ec4db40","group":"9f352b44-91c3-4933-9ea0-03434b21745c","tags":null},{"created_at":1533848674,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"c888c099-cbc0-4228-9425-0279fa648e8f","group":"03ea6ec4-1d82-4da3-ae1c-a8f160551c1e","tags":null},{"created_at":1551972585,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"8a40e6e2-897f-45ef-9462-cec61aba7879","group":"cdf95bc3-c994-467b-b836-92eb6dd6aadf","tags":null},{"created_at":1565835790,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"6d6901b6-9cfb-4a4d-999b-e36e12c6db33","group":"b4752aef-1af3-4dea-bfdd-339d8bfa8966","tags":null},{"created_at":1572626908,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"740bf04c-0469-440b-af26-e081ac1fd21c","group":"e0218569-3a69-404f-8ad3-3e9f9a7592b1","tags":null},{"created_at":1576131227,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"df198d10-422b-4815-adb1-4f78d3eab56f","group":"abe2f185-63df-4c49-9025-61d42e1d05c6","tags":null},{"created_at":1537981013,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"c1e91f2a-e1b5-4c34-bc0f-d0e38184e7f7","group":"3f615564-6563-4c8c-89e3-d9e022b25334","tags":null},{"created_at":1572227351,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"610ec69e-d105-4daa-956e-42f686058fcd","group":"13693502-9537-4a9c-99b5-8952e2d18223","tags":null},{"created_at":1578203466,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"45a6c06d-109c-45e0-b3ea-4e30c757cefb","group":"ea50feba-6ffd-4562-ac35-8e5a19a5fc1e","tags":null},{"created_at":1564085502,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"ff1da08d-4ca3-46bf-849c-af2aa285febb","group":"9eeac000-6794-4c6e-9766-5c6b967b6b28","tags":null},{"created_at":1567528097,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"ccdbfdb8-dec9-425d-bb1b-04bb3b7080d9","group":"578466bf-0831-4d93-b698-d64f7432fa4b","tags":null},{"created_at":1571154825,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"465efa97-3e22-4a7d-a756-6cf248e82048","group":"cfa4550a-ce35-47f4-ae85-409a0b4ba0b2","tags":null},{"created_at":1578422577,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"7de69b9b-64af-4c61-9a52-b63dba5db40a","group":"689b1af9-d570-4a22-ac04-84d02cd355a3","tags":null},{"created_at":1567421546,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"51868dd9-db9d-4c07-8916-cdcdcabc121e","group":"0db08497-3379-46e0-a1de-f701f5fe7011","tags":null},{"created_at":1562239685,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"17b02f07-b87c-4ce1-968f-4dc3f0998f04","group":"f59469db-e2bc-47ff-851b-a7e20b0e0a1c","tags":null},{"created_at":1572588488,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"2f375eae-985e-46c8-9998-ffe6cc343965","group":"851972ce-4332-4a0a-a02d-28b0d611ad9c","tags":null},{"created_at":1572879332,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"da22a3a2-d958-4dd8-920c-720e46bdf06e","group":"13a5f977-baf9-4b30-9989-3dd2641cfdc4","tags":null},{"created_at":1579875808,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"3b6e59a4-62dc-4a4b-b704-783e5db8220d","group":"027471fa-bd65-434c-b3ea-c689186dd2f7","tags":null},{"created_at":1578475344,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"216db336-b6fc-41d9-94d2-777ce80320d3","group":"bc14a61e-b271-426c-a1e4-d369778292e4","tags":null},{"created_at":1570733066,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"9c95f228-7f32-4487-ae27-052fb753914d","group":"100f80dc-a4e1-437e-9a52-ef97bd700b5e","tags":null},{"created_at":1533677901,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"93523576-bbc6-4062-9b9c-1092069cbe97","group":"53d54717-11fe-4569-b5c8-13af5ac5cd91","tags":null},{"created_at":1537510680,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"7c655e9c-0ee0-442e-8818-5b4ea97ce0b1","group":"b906e359-374c-4dcf-aae2-10993c34bbf1","tags":null},{"created_at":1548690779,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"86e1ece7-7a69-4f2c-80c8-7fe9a3776890","group":"027b0002-fc80-47af-8685-5508e48d5637","tags":null},{"created_at":1566317139,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"dc66a401-5d9b-4fec-a4a8-9d10057f27f4","group":"214dabd1-b9f2-47ae-be33-81f317a8a9f7","tags":null},{"created_at":1567702166,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"984bfe5e-484c-493c-8fb8-cceb4ee77450","group":"40dd95a8-6909-4545-be72-ea8b4df02f20","tags":null},{"created_at":1578917048,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"c9887f94-b88e-42ea-a63f-1295904f21ec","group":"74b2d144-621c-4198-bca4-d5606ccba076","tags":null},{"created_at":1579876949,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"476ecfb3-b049-4952-b8de-77367aa1f895","group":"668e0a16-4565-4396-abd7-1de4d881835e","tags":null},{"created_at":1566396731,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"12b53adb-eb1d-4f74-a184-24fa0166c59a","group":"326ac88d-164d-424c-843f-492282c776c9","tags":null},{"created_at":1578653644,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"b5679a25-7ba5-4917-b007-c576b434ec23","group":"819b7564-9d02-4479-b385-9e5b4b2649b5","tags":null},{"created_at":1579892181,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"dccc9dba-67ea-48f4-b5ed-ffe1fd69c6e0","group":"b4223cb7-1bd1-47a9-917a-4a2ff3e501f2","tags":null},{"created_at":1560931396,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"4d81e385-827f-4e92-9ccc-b72e0146cc15","group":"89ab79bf-1992-4589-9093-0de146a35911","tags":null},{"created_at":1553683696,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"655ccf15-930c-4647-8fd9-28a343f9e5a3","group":"8ae89228-7802-49a3-9a47-79e5e83c81ea","tags":null},{"created_at":1573745167,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"38288d6c-1c7b-4506-8999-cbbb7870df82","group":"9b1f5550-899d-4dc8-8a81-8da1f2d738cb","tags":null},{"created_at":1555348286,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"dad478aa-1cf0-4abd-9be3-ee544cd04f62","group":"22608af0-a841-4f18-8547-e8282e9f7ab5","tags":null},{"created_at":1576747376,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"cd572e90-3960-4d91-9ba7-743ba9187019","group":"694093c1-b139-4fd9-9a51-ee1b8a7f6278","tags":null},{"created_at":1568720307,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"68fd3e9a-73e5-4940-a30a-a6c14a69f8e9","group":"86b8e7dd-c684-4bf7-88f7-abacd0098db4","tags":null},{"created_at":1562311374,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"03e8f900-c178-4c2c-a9cd-ea1c0cac5360","group":"40d00a2f-fedc-4bfd-bb6c-978ae82e6018","tags":null},{"created_at":1560988535,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"68c93936-37a3-43c6-a444-dd0b4b4a4cbc","group":"70a35be0-7dc5-4397-91e9-2c776e16baa7","tags":null},{"created_at":1578517410,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"fb810370-424f-4f91-9759-ed364009549c","group":"461e58b1-2af3-4a51-95bf-ca45bcf929db","tags":null},{"created_at":1553510651,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"d97d7983-1775-408d-8061-11d455873a30","group":"5786dbed-308c-4306-a894-2824d8dc33a8","tags":null},{"created_at":1568407793,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"5fae0276-918d-4d5d-a7ed-5e5bcc9bc5a6","group":"16354b2b-1997-4473-82ad-04d23062361c","tags":null},{"created_at":1581092660,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"f987a499-9f83-437a-a3e1-87227f9cd33f","group":"30e8f43e-9f26-4872-99ee-a2fff988d87c","tags":null},{"created_at":1567105890,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"eb344317-a187-4705-95cb-caefa1291da8","group":"b605f115-1a9b-4e8c-8d0b-182a78e62b55","tags":null},{"created_at":1565606806,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"f51e13ab-0360-4aee-84b2-bf2a99e9f241","group":"c7ec6939-701d-4ccd-9fc1-c9feae839078","tags":null},{"created_at":1580817216,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"921ceb87-90f2-4dba-9124-759b231bfadd","group":"b400ef30-46f3-4d60-82b9-febcf1902a60","tags":null},{"created_at":1567704956,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"18900460-1638-417c-960e-25eb724837fd","group":"783ac4ed-de8d-4fa8-985b-08b56d5564d4","tags":null},{"created_at":1579459363,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"8dbc3f78-0166-4c0e-8776-bcb11771e8d5","group":"b2295405-7e14-4e84-b521-179c30aeb057","tags":null},{"created_at":1560257829,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"72e82b52-1579-46dc-8556-abbb9cce089f","group":"dcaa38f1-2946-40a8-91eb-6ba1b4774dea","tags":null},{"created_at":1569925581,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"4477f31d-0faf-4972-a443-ab448611a55f","group":"3249bd7a-390c-4930-a58d-5147d14224d2","tags":null},{"created_at":1551959508,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"74282b93-53e4-4904-b5ac-492d3ff8638a","group":"d2c68c7b-7f84-47b8-8d7e-c6dbdba6c21e","tags":null},{"created_at":1561470669,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"edc5bd63-f00b-40c3-bb41-f4fec999b062","group":"decb0383-f5f8-41b5-879b-1f06a46c2d4c","tags":null},{"created_at":1572626598,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"559ce7d8-237c-40e2-be3a-0df91fd7755e","group":"0ccc8eb7-4f4a-45f5-9ad0-c07c332290bd","tags":null},{"created_at":1573044119,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"17fac4df-1ede-428f-8de7-225242808532","group":"f2a5f291-f395-40a0-99fa-b01fadbf319d","tags":null},{"created_at":1566396737,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"33b3389c-7d5a-4b2a-ae36-09097d1adeea","group":"497f883f-17bc-49dc-a1db-eec395669ebc","tags":null},{"created_at":1564518036,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"14a347ed-279f-4aa8-99eb-5bfb2c29b034","group":"f310d5c2-dff5-4fd7-82f5-76aefcd3a835","tags":null},{"created_at":1580149004,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"488d34c3-7ef6-4ee4-8bd3-945be0ed9791","group":"38107061-795d-4c91-86f4-fe10ac7b398a","tags":null},{"created_at":1567528899,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"230e409e-427c-424b-bb28-95dc4a2c91cb","group":"3745ca8f-eadf-4833-aafe-3b445dcf0c55","tags":null},{"created_at":1567706842,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"5d08e23d-add4-43b1-873c-cd9b5926be33","group":"a76917ec-482f-49fe-ac35-15d0da0f5db7","tags":null},{"created_at":1563962680,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"b0219364-ddfb-4e4e-a0f2-503e697c0ed5","group":"ca3f6847-d142-4d0e-8288-d06646c4d7bf","tags":null},{"created_at":1569502256,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"d6a9d78b-3103-4c09-9f01-b690565b34ed","group":"8771053d-9c60-4b68-ab38-16546c16ceb1","tags":null},{"created_at":1561711332,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"564a0fd5-8fce-45aa-a560-b4ec16839be6","group":"0f74c14f-a7d7-4607-8f3e-8a94b65ef722","tags":null},{"created_at":1560491628,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"8a2eeae5-ec57-48a8-a3ee-99f2047fb0e4","group":"1e0145a8-92d2-4505-9026-497d4d97de41","tags":null},{"created_at":1533677912,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"ce51dbe6-b52b-4498-9029-341a81ee94ac","group":"0291e545-1057-455d-87fb-a9b564adc2c6","tags":null},{"created_at":1533848660,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"7726b59a-94f1-4c04-8412-66c5b606afda","group":"f4161095-3c36-4db0-b7cc-133a16840715","tags":null},{"created_at":1578203469,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"4d3ccb6b-effc-4c19-9357-11bf039c2a37","group":"b02a12e3-f86b-4833-bb79-f44edf454086","tags":null},{"created_at":1577708736,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"e0c6f03d-3244-4242-898e-ca6c9916d181","group":"1283d9bf-b31f-4559-974f-fd0cba00237e","tags":null},{"created_at":1566396736,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"8a8f841c-244d-46f1-b982-f45e0c01d9bc","group":"8e711e06-433b-4d94-b876-25d8db940062","tags":null},{"created_at":1576788081,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"ef5cdb1f-3058-4fff-9e18-c2b1c6df75dc","group":"53f48477-2219-45db-a09d-4a0bbfcad4a0","tags":null},{"created_at":1571813621,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"555c9167-7d12-4efe-b92d-c8f27fb97376","group":"c366c0a2-7432-4781-a71a-129df4c07919","tags":null},{"created_at":1572588489,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"80da12ba-ef37-40f2-824f-7489d9ac4210","group":"ad6abf1d-124d-453d-b314-834958bf3bc0","tags":null},{"created_at":1580296296,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"0994f5b4-6c29-409a-a343-acd637ccaf89","group":"232b34bf-85bf-4530-9283-d0ff857718db","tags":null},{"created_at":1566396733,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"c787596a-64d2-4ab8-b7a8-f716d83b5f2c","group":"354da00e-e0c7-484e-a2db-8e3cb1cb7a87","tags":null},{"created_at":1561711334,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"ecdf5be3-d5d9-42ba-9ac1-78f0f35aabaf","group":"6ce58232-224a-4d5a-88e2-0a81b3457b98","tags":null},{"created_at":1576747378,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"5406fcab-6d37-40af-8a39-f18888a255a4","group":"b7f954ca-cb9d-4aab-a770-1b4a4ec2143b","tags":null},{"created_at":1576855232,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"515dd1bb-6a25-4dbb-aa63-794d3d6a9209","group":"00918cda-39bf-4d95-9ea6-82ca542a25fd","tags":null},{"created_at":1572633203,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"5af3ceaf-eab4-422d-b8ba-0ae999b0e02b","group":"41625273-9839-45c6-812b-936136e28d9d","tags":null},{"created_at":1581442048,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"3b472e39-1627-483e-a182-6b22d8341492","group":"f0d0a4ca-2e29-4535-b845-36223eaf7642","tags":null},{"created_at":1570063089,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"e75d97ee-28cb-4c01-9964-4e29b4618253","group":"2e86401e-d12f-4e5f-8afc-7a2f7bc941b9","tags":null},{"created_at":1572587889,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"827f6581-8d83-4972-95c5-27d11558ed2b","group":"d258fa9b-80a5-4831-bced-fc1c77c8c7e9","tags":null},{"created_at":1569940035,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"3bcbfc59-973c-4121-8f3d-2ee6e304967c","group":"f8641a40-9aea-4f75-9e92-60f1abb83b7f","tags":null},{"created_at":1568714836,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"75ec72b0-c0db-47ec-84f6-a1dea9219e58","group":"18329d50-9d61-477a-a55f-8d0cf7938dd2","tags":null},{"created_at":1565360605,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"66fe334b-89bc-4189-b3d3-72d59f86914e","group":"cd227d94-bc12-4fb0-b37d-b18938562691","tags":null},{"created_at":1575442633,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"ec85c209-97ec-4b8f-ad1f-72c599fd89e7","group":"dfa0c8c2-333d-4ff0-82c8-e4a8f1b208a1","tags":null},{"created_at":1555444158,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"4b62f5cd-06e0-4646-8653-5f27da2bcefd","group":"c1ece03c-8bc3-4061-89f9-f555a68b1dd9","tags":null},{"created_at":1548411532,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"b1eaee2d-17d0-431e-bcb5-70e5df1be0a2","group":"86814b19-66c3-4941-a41e-ac481ea503cf","tags":null},{"created_at":1572438702,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"f2d30aa0-0c6e-44b6-8bee-14e36e2b80f5","group":"8ef8a307-80dd-4263-86aa-6f62a0b9ca4b","tags":null},{"created_at":1581369953,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"48810fe9-a650-47b4-b0da-c3ced1ec79f2","group":"cd49631a-fcee-4a06-9018-122c9875c2bc","tags":null},{"created_at":1565367317,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"0d1ef678-45da-40d9-b262-ac1646950aca","group":"bf5f7bb5-e53e-4010-bb79-bafa45963a44","tags":null},{"created_at":1579879583,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"7a6d7aba-fc59-45a4-b3f2-0cc7c725ce85","group":"7b043ff6-c956-40d0-a5f4-86355c1a1501","tags":null},{"created_at":1533848695,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"dc233dca-cbb5-4531-9d5b-241fb63d1481","group":"4634bdc0-e408-421c-94a3-c46e0ea2cfd2","tags":null},{"created_at":1579876950,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"77e107d4-409d-4a8e-a9a0-d8c6652863b5","group":"2f47f549-fcc4-4ef3-b90e-ff2c13abd577","tags":null},{"created_at":1578422580,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"00146a4e-6798-4cad-b84e-b962fc5bd181","group":"07a3848e-5f50-47fd-ae67-0944d05d5545","tags":null},{"created_at":1558379713,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"bd297ef0-b31d-4b56-b2e3-7cae87f4893a","group":"7abb5fc0-2efd-4db4-a174-c9e6cf63ff45","tags":null},{"created_at":1567626703,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"f610c517-718f-482e-a61a-09b71d14010b","group":"ac471188-1a7b-4c77-9bd2-4f8911f18e69","tags":null},{"created_at":1551958511,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"fc53f117-371d-4840-8f82-ce358fab0a4f","group":"1daa79d3-03cf-4976-9290-710e2393a0cc","tags":null},{"created_at":1575397259,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"2218e6b8-06c0-410f-bd1f-407567f2c3f9","group":"4a8d353d-7e5b-44c0-bf37-adbb95199302","tags":null},{"created_at":1566577662,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"45ca82bf-4da3-40c6-8f55-cbd9bacd9f96","group":"9e4e6d98-37ae-45cf-b82e-5c31574a5b23","tags":null},{"created_at":1578518285,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"45a0e903-f8ac-42f0-9bca-66d3cffe3aae","group":"3b4adc69-3d4c-457a-8469-a0a36f27e28d","tags":null},{"created_at":1560988536,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"24aaf9a9-1ffc-4687-9d17-515238c73e8d","group":"b2d60265-b250-402f-8ea4-045a846b436f","tags":null},{"created_at":1544624882,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"7413bdfa-60bb-4c28-adcd-dc1d784c4141","group":"4e0c36a5-cc73-415e-8835-82e8bebbd452","tags":null},{"created_at":1579706301,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"28ea6ba6-783d-4bc9-b7e8-27389d320ffb","group":"6092ac20-1204-4906-b1dd-6cdcb49882ba","tags":null},{"created_at":1579008331,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"ecf2a1d2-b5ed-4dc3-a497-085ee208e810","group":"6089bd00-1215-40d3-bc02-03b38afe318f","tags":null},{"created_at":1578209555,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"0265b0af-4bf9-4575-829b-04fd58cf3857","group":"9e409f06-6c62-4768-a06a-ddac44526a6b","tags":null},{"created_at":1548411510,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"31571456-c820-48c2-88c6-120c822cdd2f","group":"b96ace91-6ff0-4020-9c62-aa5d1717f4f5","tags":null},{"created_at":1568054207,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"3bd59f01-4c5d-4ba8-adae-0782c3cc5956","group":"57359db2-416e-47f6-b7c9-ee0c552f6110","tags":null},{"created_at":1577346396,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"b7d3971e-ba69-4f94-8a0f-c60576251557","group":"acd1c7f3-1ad6-4bed-948a-c00ee4c7a908","tags":null},{"created_at":1578206299,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"4485d883-e9a3-4628-ab65-024d6cc77340","group":"8594fea5-6f9d-4bd4-8ecb-1b4fb6aa4dae","tags":null},{"created_at":1571154805,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"c0c15672-548f-4cc6-8748-08ec6d979b35","group":"d15f88ba-7601-4d0b-8441-fd57578d80e2","tags":null},{"created_at":1571826448,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"a4d20874-0cd4-4913-93fa-f30eed7e753a","group":"1b627735-387d-462f-9e9c-025a283ece23","tags":null},{"created_at":1550664311,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"443b09f7-6236-4091-9e09-124ff1c4405a","group":"5313b0c2-28e9-42a7-9878-e94eba2f8345","tags":null},{"created_at":1568311735,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"cb5d6e77-1140-41dd-aea7-c1080789ce49","group":"65b96331-2d7f-49bf-92bf-ce770d61d78b","tags":null},{"created_at":1578206494,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"0a989d79-ebaf-438d-8dee-e512772236d0","group":"3c8a3db0-9f12-49fe-81f4-e857ce17ae0c","tags":null},{"created_at":1537511521,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"edf9bfb0-4f02-4f08-9ea8-9f0b81f8966b","group":"632a06e2-7eff-468b-871a-bb46ab5fcfe3","tags":null},{"created_at":1569939753,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"744e9bd8-0cd0-4390-9486-efa382b55886","group":"74e217b4-ce7f-4ec4-8182-cdcedbd170dd","tags":null},{"created_at":1564389035,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"b5076c6f-5628-4c5f-bb9d-da232cd551ca","group":"edf94ea4-b833-49f9-9d12-d76519fc8d7e","tags":null},{"created_at":1564085498,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"4803f7e2-0fc3-4afa-88eb-6cd9c96cdc0f","group":"af758bef-cbe8-4d92-b981-368cd2875646","tags":null},{"created_at":1575019441,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"b16d5722-4c21-4c6e-95ff-6e31b2ebd097","group":"7c5a9fb6-4f8e-4948-9211-345d0fc236e9","tags":null},{"created_at":1573830117,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"b98f4b59-e882-41c0-b218-45d31fa0bddc","group":"0dd5f21f-5ffd-4dd2-9af7-d6f8a2ad4d03","tags":null},{"created_at":1539115436,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"2be6cb61-a659-4fc9-82d7-f1914ec14825","group":"e7e03082-4f2a-4a42-95b5-8f1099e38436","tags":null},{"created_at":1571154801,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"9a35bf3a-4efd-466a-9c15-a6c40198a4b4","group":"099dd004-c1f6-49c4-8af9-113559d9407c","tags":null},{"created_at":1571813630,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"49ec2c87-4cec-4c29-947b-01ef85366265","group":"12e967d8-4cb2-4f0a-a78f-d00b6c356542","tags":null},{"created_at":1580149007,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"45bf75e4-9ed5-41aa-9601-aa8f739050b5","group":"33b5f5fa-b132-418a-9b30-764421ba48fd","tags":null},{"created_at":1569925583,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"d62a7f32-e81c-4dc5-a330-63b4ff9b56aa","group":"034dd659-8dbf-4bbb-a786-3b93e48e83c3","tags":null},{"created_at":1571154798,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"969c71d8-b522-4e47-b793-e55af7290101","group":"749c6a72-430d-4e6e-8e5c-7eea463110a6","tags":null},{"created_at":1553783792,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"cd54a8b4-070d-44fc-939b-cee9dfebea40","group":"a57ee5d3-2b00-4d55-9d87-32f7cdaea9a3","tags":null},{"created_at":1552995661,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"e11d5be1-195f-466d-ab3c-14dde58aac00","group":"ac606c1c-b549-4242-b337-29efc071592b","tags":null},{"created_at":1538067521,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"3b559f90-c6fd-41e5-9365-3010bd4e7d15","group":"30e72c94-e14a-43c6-ac64-803e77beea09","tags":null},{"created_at":1577346394,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"cf590c51-8dee-43ce-bb45-b1c557d4afda","group":"2d7f105b-2b67-427f-a7ec-ac3bce006c54","tags":null},{"created_at":1558635877,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"16450d12-9821-458f-8222-309fd1a5e4ab","group":"91c4c862-d59a-4951-9b1b-310b9c7fcfeb","tags":null},{"created_at":1560988533,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"e7970a30-b3b2-4b01-9dfc-2f375d7e43dc","group":"3b456311-5e40-4dcc-85f8-aee4cc0140f5","tags":null},{"created_at":1555348252,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"15ec827b-92af-4c3e-964b-3637700204bd","group":"3d226c0c-408d-4ce3-9c1b-0983f25e3037","tags":null},{"created_at":1577930881,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"efa0a8af-efe2-463e-9dad-5ffb7ffc78c9","group":"b816509a-5b18-4a0b-abc7-236da8183a2b","tags":null},{"created_at":1578510569,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"10dd2322-54e1-4288-9192-49d6651a6ab6","group":"31df8801-9716-47a2-b947-039c04f46a58","tags":null},{"created_at":1533848683,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"58b3cffa-ece2-41c2-aeaf-8c7c33b021cf","group":"ef71ea26-c3cc-4d86-aafc-4cc74d793ca5","tags":null},{"created_at":1572227346,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"7e620450-c0ee-4bf2-a09a-3ebb34aec58f","group":"00122cd4-714e-4ef5-b5e8-b6dd1bdcfff4","tags":null},{"created_at":1567525740,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"9d6f4478-a159-4173-9dc7-0cce397b3384","group":"f651f949-ebdf-492a-9c27-dee79dcc4d1a","tags":null},{"created_at":1565279649,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"3ea3d49c-e53e-496c-aa5f-6a03a4e507e9","group":"1709075c-f52b-4c4a-bfa0-6102acc2a839","tags":null},{"created_at":1551349860,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"e8948f24-4a73-439e-b713-d9ec12bcae1c","group":"3c3ac134-46b4-44b6-b124-2dc064a52d0d","tags":null},{"created_at":1578205144,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"5b24a59e-e89c-40ad-9b5c-f52ced5258f6","group":"2a75879b-8026-4bee-a54a-f23da4ea8fab","tags":null},{"created_at":1567705152,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"2b8ea323-89c0-44ee-9f1f-70f5a52d262e","group":"b2a9a2e7-35b8-46ca-b4e5-e37062be901a","tags":null},{"created_at":1560489926,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"62e584dc-eb03-4bfe-9d19-c82959017554","group":"94ea8a53-5a3f-47a6-998a-4120beed1146","tags":null},{"created_at":1560365054,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"2dfdc49d-355f-472b-8f19-bfca39c7f295","group":"032fb99e-7019-4ed6-8b76-43db343bfe74","tags":null},{"created_at":1579706305,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"536bd716-102d-425d-aa4e-647d8f48b485","group":"4ffe3036-20c7-41b2-9690-880f582942fe","tags":null},{"created_at":1553684244,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"501839d8-b508-4907-b900-84e07c7a5681","group":"c5a32e1f-3ce4-4472-b075-c80e4101d506","tags":null},{"created_at":1578209988,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"80ed8264-2234-4e17-bc10-f0b4698e79dc","group":"1bc222da-b5a6-48cc-86f2-1e6d2059517c","tags":null},{"created_at":1550761476,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"df3a6c3e-6284-40d0-8bc0-ddebc286b2bf","group":"bf9eb7d2-4000-4bc5-a0aa-9ed723446670","tags":null},{"created_at":1570194868,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"af9b886d-ea9f-4c75-b584-1236bfbfb5d3","group":"70211ed0-1c14-4404-81cb-c32cd07c6aea","tags":null},{"created_at":1581313820,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"f18dbeee-7a22-420b-ad7d-b3d2ee5edf86","group":"2da824ad-5b7b-4549-9951-668ac2716e28","tags":null},{"created_at":1579706292,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"dca093c8-c21b-46ed-97cd-7e412b3b03ca","group":"34f8f947-727f-49dd-942c-11437dae1069","tags":null},{"created_at":1569502257,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"bf791770-eb84-4f50-8ae7-8e109af7ab3c","group":"e090b785-4e91-4242-bd81-bb8c05aadff7","tags":null},{"created_at":1554723504,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"5c3946dd-8c1e-4c8e-b9fc-7fed19f56047","group":"9f7f684e-3772-48e4-8d9e-f06a2a309143","tags":null},{"created_at":1562695467,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"5692d916-e131-4574-b829-0c73ccf064e9","group":"dc82884b-47f7-431a-9f44-b4bda2087042","tags":null},{"created_at":1573671088,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"fad75f4d-0d94-4bcd-811f-1af0024e091c","group":"5521d4a6-dfb1-406e-bfbc-8c8aaadaa167","tags":null},{"created_at":1567704217,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"b7013e5d-2277-49fe-bd89-8ba915e7e1bb","group":"b0a223ba-5b03-48c1-9908-dfd9c2b6fa7e","tags":null},{"created_at":1567704510,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"c661946e-4b17-4812-aa14-6bc4850a8101","group":"3f848bfc-25d1-42af-8cdf-0f96cb09969b","tags":null},{"created_at":1538067272,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"20466e9b-4980-4883-8cd9-dc47bd6db826","group":"3ab73648-2507-4f4c-b5eb-607a920eb617","tags":null},{"created_at":1571154803,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"e3f3e40e-e425-4d4a-a9e6-ea87cca5ff68","group":"80e87915-c764-4997-9d36-bf16c557a0de","tags":null},{"created_at":1560532300,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"8b592d26-ed8d-42fb-8e15-32cdeec6f4bb","group":"385c8386-9495-41ab-9ec3-87da6fd71357","tags":null},{"created_at":1565183995,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"84f6c67f-6f9c-4936-8288-eba1595e7b78","group":"25c936e6-c4d3-4365-91b1-895fb5df0a44","tags":null},{"created_at":1558380999,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"28be9f40-97c3-4d05-aedd-4827af201366","group":"419aab86-bdbb-419d-af92-246bfb300a52","tags":null},{"created_at":1558381000,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"ce06c069-15e1-4e89-9f58-1d310ee76ee1","group":"0aa3671c-a12a-46d2-bf28-44e6214c62a1","tags":null},{"created_at":1539873792,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"5658c29f-0ecc-4e09-b2a9-ba457440bbf3","group":"ec400470-c360-41c1-9200-a7c8421864a8","tags":null},{"created_at":1555348262,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"7ac65963-bf7b-4ed7-9947-3d0c3abfe16f","group":"95fc50d5-339a-40d6-ace0-088e442d1e35","tags":null},{"created_at":1562091068,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"4fb93533-c4b0-46df-8e15-14bdb6e0d928","group":"be120696-83bb-412d-a40b-841321bde8ed","tags":null},{"created_at":1564396412,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"9d82e183-74a8-4bfa-951c-c21168eab44a","group":"e78e79fc-1961-458d-9f37-27227f74212f","tags":null},{"created_at":1567525339,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"cd051a3e-da38-4537-a2ab-0b5d772dd24e","group":"4dcb8c3d-ecd9-4477-a6d6-e0874b43e18b","tags":null},{"created_at":1560532587,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"d16b9d9a-3c24-44fe-b0b1-22f4f22edbab","group":"58b259f2-fe1d-4e9a-89d9-5883a8ff7f55","tags":null},{"created_at":1572588486,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"feb6ee10-2265-4c1b-a18e-6611c72befc5","group":"72b1dd45-0808-4320-b4d8-cc16870c754c","tags":null},{"created_at":1560529498,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"2940acd2-e7a2-41f3-95b3-6d57f987a4e8","group":"22e868cd-19e6-410c-bc27-fb4aacc1f804","tags":null},{"created_at":1576156835,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"44ecb5cf-2d13-4978-b6a6-210769d895dc","group":"ea9524a0-fc97-4179-81f1-1911848b6ed3","tags":null},{"created_at":1568662429,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"49fc62b4-22ef-41e7-be96-3d60baea4581","group":"03281ce1-f689-42c8-a468-f3246086f768","tags":null},{"created_at":1581442288,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"2fc0a3af-f627-4a67-a149-7a7d33df71ab","group":"321a6cb7-5a98-43a6-8a54-3be98e14b373","tags":null},{"created_at":1579874503,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"2d6e594d-860b-4485-94c5-05bb40edf238","group":"95675da0-0ef1-479f-81ca-7579ee808426","tags":null},{"created_at":1579703198,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"778db8b5-746e-486d-b7f0-57371b377473","group":"b8c8fdec-cc3e-49a6-8dab-af232a3032e6","tags":null},{"created_at":1566316883,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"7ad5478a-a075-4ebd-96ec-cc49e6b00823","group":"20ae9654-7393-47d3-b0f5-7266465f254d","tags":null},{"created_at":1571819074,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"cefeb4da-721e-4e18-86c8-2f17531c3036","group":"91fad533-a33c-4514-b147-25b7e14d25a2","tags":null},{"created_at":1567522387,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"ad0b4fe5-c14a-4abb-b287-99776a59a1dd","group":"b70e0a33-a91a-4c72-b07b-293c3d689147","tags":null},{"created_at":1571154822,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"5dd8e9fc-c1d4-4aeb-9833-c682c5207be3","group":"6853f129-c501-41cd-afe5-ad008c0701f2","tags":null},{"created_at":1579461888,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"27f52a8a-1335-406c-9276-fff8b293ec89","group":"cd8b60dd-1087-4cae-a352-fd4b094c1f0d","tags":null},{"created_at":1562091069,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"754472db-440b-44ba-92f7-463804820f24","group":"fbd20b17-f83f-43f9-8e16-e70fd87d8e9b","tags":null},{"created_at":1578006675,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"d3ce9572-bf3a-41b5-b7c2-381be4736223","group":"a7b1f5e0-59d7-46be-aae6-106b361615da","tags":null},{"created_at":1579879168,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"7c5557f5-22ab-450e-9672-89326df5d480","group":"c49bc6b4-513f-4bbe-a7e0-2640aca97a9d","tags":null},{"created_at":1564396411,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"314de0b0-0c92-40a3-96bd-afa21ad47057","group":"fe6f4b16-2b44-4560-88ab-b0634dddc03a","tags":null},{"created_at":1539360132,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"a857e6cc-d985-47c6-9c75-2fe96a439ce2","group":"b149ffdc-6afb-4a69-b6e1-0326a3668021","tags":null},{"created_at":1551972624,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"6b2bd8ac-7050-41b4-ba74-82946ae9600b","group":"ef50c0e9-0745-40ca-a360-a95bb28c311c","tags":null},{"created_at":1579879172,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"973344f9-a243-4f7f-aeb4-cdedf0e37fb3","group":"2d6c2cd9-587a-45f5-a5d6-04aae4082a20","tags":null},{"created_at":1568645932,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"4c94a46d-bb53-4cf0-864b-988d8f4d6fa0","group":"759cb737-0c03-4869-ac48-827946c5e1aa","tags":null},{"created_at":1565836027,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"4e896b5b-7d07-41a8-a7a3-4eac01437958","group":"8cf7a075-a998-4691-a45b-663d3625d485","tags":null},{"created_at":1571826447,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"c0b34cc0-b7a6-43ad-969a-05ccb9297308","group":"32294715-ddef-46da-94d7-b16a05ba5fff","tags":null},{"created_at":1578205871,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"87c64d7c-aa4a-4f10-bba0-efdd83f3c779","group":"532dca3b-e023-4324-b6e3-95872974285a","tags":null},{"created_at":1567529249,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"2fd987d2-ef47-4a95-9668-c43ca6d8365a","group":"8b9482da-f8c9-40e9-a92b-98410bc54611","tags":null},{"created_at":1548690770,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"3e5fdb1a-6bc6-4b7e-9e9f-b6255d1e64ec","group":"142dc95a-53ae-4edf-81f1-3e0bfb7371a1","tags":null},{"created_at":1565387302,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"f101dbd7-925a-419a-bb99-022edb54ac6e","group":"8ea4a540-30b0-4a37-8122-869cdd93eb53","tags":null},{"created_at":1577346397,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"d34f0a4f-a539-4392-8c60-9da486caad62","group":"4d404a0a-f8bb-40c6-a3ad-acea5ff00e65","tags":null},{"created_at":1572878654,"consumer":{"id":"97b8b271-da5e-4aff-afca-bfa0c1dad70f"},"id":"3f72cbe5-4859-417a-a348-a0e38b58075f","group":"79ec26a5-ea34-40d6-ab3a-a8995144bf66","tags":null}]}

With 100% confirmation that there must be some edge cases where the acl cache somehow responds lacking certain groups in a certain scenario, @bungle have any bandwidth to speculate where the issue may lie, or code snippets to add/try, as our snippet only confirms the cache lacking a group to the consumer that the db and admin API call showed they possess(after we had already bounced the afflicted server)? We tried to reproduce in dev by creating a ton of consumers and proxies and 100+ acl groups on consumers then sat there and tried to add + remove groups from a singular consumer while calling a proxy they had access to. We were unable to reproduce the behavior in doing so but it's also really hard to mimic all the CUD actions going on in our live customer facing environments in a sandbox at this point, and worst of all it only happens once or twice every week or two. It's not crazy frequent or predictable like the router rebuild issue, and a redeploy of Kong fresh startup does resolve the issue when we catch it quickly.

No logs of DB issues at times of impact, no lua OOM errors reported. Kong 1.4.3. Need some bungle mania magic like when you dominated the elusive router rebuilding issues earlier Kong faced. Happy to add more debug logging than the above if you think up a snippet that would give even more insight(maybe something logging with the acl cache is too big perhaps?) or something the acl plugin is not accounting for. Its worth noting we have only seen this error occur with consumers that have say 200+ group entries, so our larger users.

The other question I have resolves around why the table prints like it does in our debugging logic, where keys start as ints in key->value pairs and then eventually start printing as the uuids being both the key and value. Seems really odd? Some lua behavior I am not aware of?

Some configs(probably not super relevant for the situation at hand though):

KONG_MEM_CACHE_SIZE 1024m
KONG_DB_CACHE_TTL 0
KONG_DB_UPDATE_FREQUENCY 5
KONG_DB_UPDATE_PROPAGATION 5

[Tieske]

Isn't it a bit odd how the logging goes from keys being int's [1], [2], ex: [1] = unitTestGroup58,[2] = unitTestGroup89,. to table keys being strings of the values itself ,ex: ["unitTestGroup35"] = unitTestGroup35, in the print out? I found that to be pretty strange myself, why is that happening?

This is quite a common pattern in Lua, it provides a structure that one can iterate over (using the numerical indices), as well do a quick check whether an entry is present without having to loop through the whole array. In most languages one would use a separate array and map, but with Lua the table-type provides the versatility to do it all in one.

-- loop over all entries
for _, uuid in ipairs(tbl) do
  print(uuid)
end

-- check a single entry
if tbl[my_uuid] then
  -- 'my_uuid' actually exists
end

[Tieske]

TLDR so far:

Version:

Kong 1.3.0 Kong version 1.4.2 + 1.4.3.

Checking webserver logs and seeing no webserver logs rules out this 403 option: https://github.com/Kong/kong/blob/master/kong/plugins/acl/handler.lua#L68 , which leaves me here: https://github.com/Kong/kong/blob/master/kong/plugins/acl/handler.lua#L117 .

We correlated the impact window to when a self-service action was being done which does route/service/plugin/acl changes so cache actions were occurring. Its worth noting we run 4 instances of Kong and only 1 pod was the bad actor of the 403's on a legitimate customer. The suspected instance that received the CRUD self-service updates for the admin API calls was NOT the pod that caused the 403 failures either which was interesting, so through cluster events or its polling it musta had a cache rebuild invoked.

Its worth noting we have only seen this error occur with consumers that have say 200+ group entries, so our larger users.

[Tieske]

thinking out loud; this very much smells like a caching issue. Considering the large number of groups, could it be pagination related? updates in between pages?

[Tieske]

@jeremyjpj0916 just verifying; when a failure happens, it does not happen on all Kong nodes. But the ones on which it happens, do they consistently return the bad 403 response?

underlying question: do all workers in that Kong node respond the same? if the Kong node responds intermittently bad, then it means that some workers are ok, others not. If all workers respond bad then the conclusion must be that it is DB/DAO caching related. Since the workers share data only on that level.

Note: might still be that the plugin makes a bad query, since only 1 will make the query, the others will share the query results.

[Tieske]

from the snippets above, it seems you're using Cassandra. So maybe it is Cassandra eventual-consistency at play here?

[jeremyjpj0916]

@Tieske Thanks for dropping in!

thinking out loud; this very much smells like a caching issue. Considering the large number of groups, could it be pagination related? updates in between pages?

Same thoughts I have, somehow some way, the cache reference is missing all the consumers existing groups from the db. It does not lack all the entries(as shown in above debug output), but some 100% for sure are missing causing the acl plugin to 403 block them saying they don't have a group that the underlying db would show they rightfully do.

underlying question: do all workers in that Kong node respond the same? if the Kong node responds intermittently bad, then it means that some workers are ok, others not. If all workers respond bad then the conclusion must be that it is DB/DAO caching related. Since the workers share data only on that level.

Sadly we were gonna use the correlation_id log we keep to confirm that by the worker uuid assigned, but the ACL termination early means we are not getting an id on these tx's in our Splunk logs. What we did do is for the 1 bad node that started exhibiting this behavior in prod we looked to see if the node ever returned a 200 during the impact window for the customer(the happy path that should be occurring and was on other Kong nodes) on a given proxy and I can report for the 2+ hours on a given endpoint they all failed with the acl 403 response for that given proxy. So that means every worker way failing very likely(we only run 6 workers) and it was for 50,000+ tx's. Worth noting however that even on the node in production that was failing for a given consumer, some of the proxies they actively called were fine and continued throwing 200's, which supports the theory and debug logs that show a given customer still has some of the cached groups correctly, just not all of them.

from the snippets above, it seems you're using Cassandra. So maybe it is Cassandra eventual-consistency at play here?

The problem is rearing itself with established proxies and and older acl group associated to the consumer(we use the route resource uuid as the group). I would think eventual consistency at play would only be dealing with newer proxies and newer access additions. Not older established proxies they have been calling for months. But you know better than I most likely if eventual consistency around CUD actions maybe that are taking place on the gateway via admin api calls could play a role too.

I suppose first steps are understanding how the acl cache mechanism works, which I have not dug deep enough into just yet. Does every CUD action on a given consumers acl cache via admin api then work its way through the cluster_events table to invalidate the existing consumers acl group cache on other nodes. What happens if mid invalidation a call comes through to a given node trying to purge and re-write the in mem cache from an event, is it possible we did a cache read when it was half deleted in a hotpath and then that given node caches that 403 block and since our customer keeps calling that proxy that 403 response gets cached to speed up lookups and they get stuck with 403's afterward(since our cache records are set to not expire ever 0 and never gets LRU'ed because its used frequently)? Just some of the crazy ideas I am pondering with a goal of isolating all the possibilities for where a problem could occur that could produce the seen behavior. If it was like C issues I have seen in the past with paging and such I would expect the lua driver for C Tibo wrote throwing that nil pointer exception response I have seen in the past but the db's are healthy during impact window as far as monitoring goes and Kong logs have no db failures printed.

And the remedy has been just to search and isolate the bad running pod right now and rolling redeploy it so Kong starts up fresh and then that solves the problem for us. I wish I was able to reproduce in a vacuum like our dev environment but acl cache churn alone without the hustle and bustle of other proxies/auths/customers and CUD actions going on and activities must play a role in hitting this rare case.

[jeremyjpj0916]

Ooo I also had what I thought was a pretty good band-aid idea, is there a simple snippit I could add in the ACL 403 block that IF the case was the cache was poisoned and incorrect and this consumer is hitting the 403 to actually within the local node re-cache that specific consumers ACL's(might help me sleep at night until maybe more can be discerned from this scenario)? That way the first time this actual failure occurs the problem node will spit out logs of it in the 403 block again and alert us but then likely fix that consumers acl cache internally to help us avoid a likely waroom with stuck 403's? Some kinda local worker events call 1 liner I can spawn after my debug print statements in the code.

Then I would expect we still see the issue, but only super briefly and the node self-remedies(for now) until something more proper can be done. Real valid 403's(consumer was authenticated but not authorized to our proxy) are quite rare so I am not worried about hit taken to spend to potentially "fix" a consumers acls getting a 403 incorrectly on a local node. I see usually like 100 or less authentic 403's a day in prod.

[Tieske]

Could this be related to #5564 ?

Ping @bungle

[bungle]

@Tieske, it certainly could, as you questioned above could it be pagination related, and that fix is all about Cassandra pagination.

[jeremyjpj0916]

Yall have me all excited after reading that PR, sounds like the issue I have been seeing all along at scale w paging in Kong may have finally been fixed! I can probably take snippits of the PR too and just apply it on my 1.4.3/1.4.2 versions running now in non-prod to confirm if it helps or not.

[jeremyjpj0916]

Not sure it will fix the ACL issue I have though because every time I have hit C paging issues I still get std out logs of the problem and the nil pointer exception from C(which when this ACL bug occurs no such logs are present in the node under impact, maybe if another node recieved the admin api call and it saw nil on paging the cluster_event it produced would be malformed though potentially?). But that PR definitely is hitting on a longstanding problem I have seen in Kong, and the fix wasn't even a huge change! simple little tweak to 1 file hah.

But I will apply it in our non-prod and continue to monitor this ACL cache bug we have managed to create good monitoring and alerting/debugging on.

[bungle]

I can probably take snippits of the PR too and just apply it on my 1.4.3/1.4.2 versions running now in non-prod to confirm if it helps or not.

Yes, safe to backport that.

[jeremyjpj0916]

Backported cassandra/init.lua for 1.4.2/1.4.3 as such, had a few other differences than just the 5564 PR in file I included too, looks like field setting improvements and field/entity checks lower got moved around some at 2 second glance, but didn't do the refactor that got rid of kong.constants just yet, woulda been a lot more files to mess with to grab that refactor:

edit: moved file to attachment: code.txt

Noted extra diffs besides just the opts adjustment:

Hopefully these extras are safe and good adjustments, just seems like enhancements for better handling, not dependent on major refactors outside this file.

[jeremyjpj0916]

@bungle @Tieske Here is some followup general testing after applying the new C* patch from the PR to fix paging and mentioned in my prior post:

Ran a 4k + routes/services/consumers creation(with creds) and I think it made something like 20K + plugins because I put 5-6 plugins on each route. Then I also decided to tear them all down by calling delete on the service/route/consumers by all id's returned. HTTP Status breakdown was like this:

401's I would not worry about, thats just my admin user calling with an expired jwt token in the loopback proxy call and needing to renew.

The 409, 403 and 404 could be interesting.

The 4 409's seem to have occurred on the PUT creations of consumers/services in my build up of the Gateway ex, there were a few times my lb or ingress to cloud platform threw a con reset so when I reran my script it tried to PUT with a different UUID for a consumer name/service name value that already existed if I forgot to change the starting index in my for loop creation logic, not sure if that would explain the 409 scenario, I think that is what would happen:

{ [-] BackendLatency: 15 CID: 69ed5e17-fb5e-4595-8684-d51a6732cfb2#12249 ClientIP: 10.96.25.68 Consumer: admin GatewayHost: gateway.company.com GatewayPort: 443 HTTPMethod: PUT HTTPStatus: 409 KongLatency: { [+] } RequestSize: 474 ResponseSize: 482 RoutingURL: localhost:8001/consumers/e7510e53-0f86-4376-bdac-9fe5693583dc ServiceName: AdminService TotalLatency: 29 Tries: [ [+] ] URI: /admin-api/consumers/e7510e53-0f86-4376-bdac-9fe5693583dc }

{ [-] BackendLatency: 8 CID: f01147a3-627c-4330-ac46-0bf4ea9b3831#195 ClientIP: 10.96.25.68 Consumer: admin GatewayHost: gateway.company.com GatewayPort: 443 HTTPMethod: PUT HTTPStatus: 409 KongLatency: { [+] } RequestSize: 624 ResponseSize: 469 RoutingURL: localhost:8001/services/ec9607d3-c54e-4d2e-986d-c36ce0e210ab ServiceName: AdminService TotalLatency: 17 Tries: [ [+] ] URI: /admin-api/services/ec9607d3-c54e-4d2e-986d-c36ce0e210ab }

403 I already know the issue there, ignore that. Not ACL related but ModSec WAF related on JWT plugin creation calls I ran into early in testing, and resolved it.

404 I think it was during build up of resources I had a call that ran after creating consumers that did another GET on the admin api and maybe it 404'ed cause it calls sooner than C* has time to establish a local quorum the user existed:

{ [-] BackendLatency: 4 CID: 577cdb01-9021-4467-9c76-b5e2e933ad96#11596 ClientIP: 10.96.25.68 Consumer: admin GatewayHost: gateway.company.com GatewayPort: 443 HTTPMethod: GET HTTPStatus: 404 KongLatency: { [+] } RequestSize: 372 ResponseSize: 331 RoutingURL: localhost:8001/consumers/e7510e53-0f86-4376-bdac-9fe5693583dc ServiceName: AdminService TotalLatency: 12 Tries: [ [+] ] URI: /admin-api/consumers/e7510e53-0f86-4376-bdac-9fe5693583dc }

Looks like currently logging is NOT showing anything specific regarding C* nil pointer exceptions though, so I believe that PR has indeed cured the paging issues. There are a few occurs of a log I can see That had to have occurred during my resource deletion tear down phase I find the most interesting:

2020/02/15 09:47:22 [error] 75#0: in lua semaphore gc wait queue is not empty while the 
semaphore 00007F8B6B925498 is being destroyed

2020/02/15 08:42:11 [error] 72#0: in lua semaphore gc wait queue is not 
empty while the semaphore 00007F8B6B932428 is being destroyed

^ Might be related to the change potentially? Can't say I have ever seen this log before. I can only view so many logs back in std out this AM from when my test ran overnight because our kibana instance is down in our cloud sandbox env right now but I see these 2 so far, so unsure how many times it occurred during the full window of testing.

Overall I would say there was certainly a huge positive difference though in stability and I will be pushing this to our Stage immediately for further testing. I think I will also run another big test in non-prod today too to see if I can get more of those GC errors and correlate them to any odd admin API HTTP Statuses(they probably won't since it seems like underlying lua gc stuff though).

[jeremyjpj0916]

Ran a fresh tear up and tear down again today up to about 4kroute/service/consumers(each with their own jwt/oauth cred pair) + like 16k plugins total(on on the routes each). Some odd logs this time around but never saw the lua gc wait queue error I saw before in this testing which I was hoping to reproduce (but saw odd new things like worker process failures and a few others errors that were new to me).

Few times during the teardown it failed on a few calls, looks like I could correlate them a couple times to when the rebuild failed due to what looks like C timeouts? I suppose Kong just kills off and does not retry if a rebuild fails due to C timeout query:

2020/02/15 20:39:27 [warn] 1#0: [lua] cluster.lua:597: refresh(): [lua-cassandra] found contact point with '0.0.0.0' as rpc_address, using '10.86.168.39' to contact it instead. If this is incorrect you should avoid using '0.0.0.0' in rpc_address
nginx: [warn] [lua] cluster.lua:597: refresh(): [lua-cassandra] found contact point with '0.0.0.0' as rpc_address, using '10.86.168.39' to contact it instead. If this is incorrect you should avoid using '0.0.0.0' in rpc_address
2020/02/15 20:39:29 [notice] 1#0: using the "epoll" event method
2020/02/15 20:39:29 [notice] 1#0: openresty/1.15.8.1
2020/02/15 20:39:29 [notice] 1#0: built by gcc 8.3.0 (Alpine 8.3.0) 
2020/02/15 20:39:29 [notice] 1#0: OS: Linux 3.10.0-957.12.1.el7.x86_64
2020/02/15 20:39:29 [notice] 1#0: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2020/02/15 20:39:29 [notice] 1#0: start worker processes
2020/02/15 20:39:29 [notice] 1#0: start worker process 71
2020/02/15 20:39:29 [notice] 1#0: start worker process 72
2020/02/15 20:39:29 [notice] 1#0: start worker process 73
2020/02/15 20:39:29 [notice] 1#0: start worker process 74
2020/02/15 20:39:29 [notice] 1#0: start worker process 75
2020/02/15 20:39:29 [notice] 1#0: start worker process 76
2020/02/15 20:39:29 [warn] 75#0: *6 [lua] globalpatches.lua:52: sleep(): executing a blocking 'sleep' (0.001 seconds), context: init_worker_by_lua*
2020/02/15 20:39:29 [warn] 71#0: *2 [lua] globalpatches.lua:52: sleep(): executing a blocking 'sleep' (0.001 seconds), context: init_worker_by_lua*
2020/02/15 20:39:29 [warn] 73#0: *3 [lua] globalpatches.lua:52: sleep(): executing a blocking 'sleep' (0.001 seconds), context: init_worker_by_lua*
2020/02/15 20:39:29 [warn] 76#0: *5 [lua] globalpatches.lua:52: sleep(): executing a blocking 'sleep' (0.001 seconds), context: init_worker_by_lua*
2020/02/15 20:39:29 [warn] 74#0: *4 [lua] globalpatches.lua:52: sleep(): executing a blocking 'sleep' (0.001 seconds), context: init_worker_by_lua*
2020/02/15 20:39:29 [warn] 71#0: *2 [lua] globalpatches.lua:52: sleep(): executing a blocking 'sleep' (0.002 seconds), context: init_worker_by_lua*
2020/02/15 20:39:29 [warn] 76#0: *5 [lua] globalpatches.lua:52: sleep(): executing a blocking 'sleep' (0.002 seconds), context: init_worker_by_lua*
2020/02/15 20:39:29 [warn] 75#0: *6 [lua] globalpatches.lua:52: sleep(): executing a blocking 'sleep' (0.002 seconds), context: init_worker_by_lua*
2020/02/15 20:39:29 [warn] 73#0: *3 [lua] globalpatches.lua:52: sleep(): executing a blocking 'sleep' (0.002 seconds), context: init_worker_by_lua*
2020/02/15 20:39:29 [warn] 76#0: *5 [lua] globalpatches.lua:52: sleep(): executing a blocking 'sleep' (0.004 seconds), context: init_worker_by_lua*
2020/02/15 20:39:29 [warn] 75#0: *6 [lua] globalpatches.lua:52: sleep(): executing a blocking 'sleep' (0.004 seconds), context: init_worker_by_lua*
2020/02/15 20:39:29 [warn] 73#0: *3 [lua] globalpatches.lua:52: sleep(): executing a blocking 'sleep' (0.004 seconds), context: init_worker_by_lua*
2020/02/15 20:39:29 [notice] 71#0: *2 [lua] cache_warmup.lua:42: cache_warmup_single_entity(): Preloading 'services' into the cache ..., context: init_worker_by_lua*
2020/02/15 20:39:29 [warn] 71#0: *2 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/15 20:39:29 [warn] 75#0: *6 [lua] globalpatches.lua:52: sleep(): executing a blocking 'sleep' (0.008 seconds), context: init_worker_by_lua*
2020/02/15 20:39:29 [warn] 73#0: *3 [lua] globalpatches.lua:52: sleep(): executing a blocking 'sleep' (0.008 seconds), context: init_worker_by_lua*
2020/02/15 20:39:29 [warn] 73#0: *3 [lua] globalpatches.lua:52: sleep(): executing a blocking 'sleep' (0.016 seconds), context: init_worker_by_lua*
2020/02/15 20:39:29 [notice] 71#0: *2 [lua] cache_warmup.lua:81: cache_warmup_single_entity(): finished preloading 'services' into the cache (in 828ms), context: init_worker_by_lua*
2020/02/15 20:39:29 [notice] 71#0: *2 [lua] cache_warmup.lua:42: cache_warmup_single_entity(): Preloading 'plugins' into the cache ..., context: init_worker_by_lua*
2020/02/15 20:39:29 [warn] 71#0: *2 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/15 20:39:30 [notice] 71#0: *2 [lua] cache_warmup.lua:81: cache_warmup_single_entity(): finished preloading 'plugins' into the cache (in 922ms), context: init_worker_by_lua*
2020/02/15 20:39:30 [notice] 71#0: *2 [lua] cache_warmup.lua:42: cache_warmup_single_entity(): Preloading 'consumers' into the cache ..., context: init_worker_by_lua*
2020/02/15 20:39:30 [warn] 71#0: *2 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/15 20:39:30 [warn] 71#0: *2 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/15 20:39:31 [notice] 71#0: *2 [lua] cache_warmup.lua:81: cache_warmup_single_entity(): finished preloading 'consumers' into the cache (in 289ms), context: init_worker_by_lua*
2020/02/15 20:39:31 [notice] 71#0: *43 [lua] cache_warmup.lua:24: warming up DNS entries ..., context: ngx.timer
2020/02/15 20:39:31 [notice] 71#0: *43 [lua] cache_warmup.lua:34: finished warming up DNS entries' into the cache (in 55ms), context: ngx.timer
2020/02/16 02:21:59 [error] 74#0: *419596 [client 10.86.144.101] ModSecurity: Access denied with code 403 (phase 2). Matched "Operator `Ge' with parameter `5' against variable `TX:ANOMALY_SCORE' (Value: `8' ) [file "/usr/local/owasp-modsecurity-crs-3.2.0/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "79"] [id "949110"] [rev ""] [msg "Inbound Anomaly Score Exceeded (Total Score: 8)"] [data ""] [severity "2"] [ver ""] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "10.131.28.91"] [uri "/"] [unique_id "158181971940.369126"] [ref ""], client: 10.86.144.101, server: kong, request: "PROPFIND / HTTP/1.1", host: "10.175.238.204"
2020/02/16 02:21:59 [notice] 1#0: signal 17 (SIGCHLD) received from 73
2020/02/16 02:21:59 [alert] 1#0: worker process 73 exited on signal 11
2020/02/16 02:21:59 [notice] 1#0: start worker process 73880
2020/02/16 02:21:59 [notice] 1#0: signal 29 (SIGIO) received
2020/02/16 02:21:59 [warn] 73880#0: *419601 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:00 [warn] 73880#0: *419601 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:00 [warn] 73880#0: *419601 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:01 [warn] 73880#0: *419601 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:01 [warn] 73880#0: *419601 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:01 [warn] 73880#0: *419601 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:01 [warn] 73880#0: *419601 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:02 [warn] 73880#0: *419601 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:02 [warn] 73880#0: *419601 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:02 [warn] 73880#0: *419601 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:03 [warn] 73880#0: *419601 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:03 [warn] 73880#0: *419601 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:06 [notice] 1#0: signal 17 (SIGCHLD) received from 71
2020/02/16 02:22:06 [alert] 1#0: worker process 71 exited on signal 11
2020/02/16 02:22:06 [notice] 1#0: start worker process 73918
2020/02/16 02:22:06 [notice] 1#0: signal 29 (SIGIO) received
2020/02/16 02:22:06 [notice] 1#0: signal 17 (SIGCHLD) received from 72
2020/02/16 02:22:06 [alert] 1#0: worker process 72 exited on signal 11
2020/02/16 02:22:06 [notice] 1#0: start worker process 73919
2020/02/16 02:22:06 [notice] 1#0: signal 17 (SIGCHLD) received from 75
2020/02/16 02:22:06 [alert] 1#0: worker process 75 exited on signal 11
2020/02/16 02:22:06 [notice] 1#0: start worker process 73920
2020/02/16 02:22:06 [notice] 73918#0: *419760 [lua] cache_warmup.lua:42: cache_warmup_single_entity(): Preloading 'services' into the cache ..., context: init_worker_by_lua*
2020/02/16 02:22:06 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:06 [warn] 73919#0: *419761 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:06 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:06 [warn] 73920#0: *419762 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:07 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:07 [warn] 73919#0: *419761 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:07 [warn] 73920#0: *419762 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:07 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:07 [warn] 73919#0: *419761 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:07 [notice] 73918#0: *419760 [lua] cache_warmup.lua:81: cache_warmup_single_entity(): finished preloading 'services' into the cache (in 1186ms), context: init_worker_by_lua*
2020/02/16 02:22:07 [notice] 73918#0: *419760 [lua] cache_warmup.lua:42: cache_warmup_single_entity(): Preloading 'plugins' into the cache ..., context: init_worker_by_lua*
2020/02/16 02:22:07 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:07 [warn] 73920#0: *419762 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:08 [warn] 73920#0: *419762 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:08 [warn] 73919#0: *419761 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:08 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:08 [warn] 73920#0: *419762 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:08 [warn] 73919#0: *419761 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:08 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:08 [warn] 73920#0: *419762 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:08 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:08 [warn] 73919#0: *419761 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:09 [warn] 73920#0: *419762 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:09 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:09 [warn] 73919#0: *419761 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:09 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:09 [warn] 73920#0: *419762 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:09 [warn] 73919#0: *419761 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:10 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:10 [warn] 73920#0: *419762 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:10 [warn] 73919#0: *419761 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:11 [warn] 73919#0: *419761 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:11 [warn] 73920#0: *419762 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:11 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:11 [warn] 73919#0: *419761 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:11 [warn] 73920#0: *419762 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:11 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:11 [warn] 73919#0: *419761 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:11 [warn] 73920#0: *419762 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:11 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:12 [notice] 1#0: signal 17 (SIGCHLD) received from 73919
2020/02/16 02:22:12 [alert] 1#0: worker process 73919 exited on signal 11
2020/02/16 02:22:12 [notice] 1#0: start worker process 73945
2020/02/16 02:22:12 [notice] 1#0: signal 29 (SIGIO) received
2020/02/16 02:22:12 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:12 [warn] 73945#0: *419831 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:12 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:12 [warn] 73945#0: *419831 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:13 [notice] 73918#0: *419760 [lua] cache_warmup.lua:81: cache_warmup_single_entity(): finished preloading 'plugins' into the cache (in 5365ms), context: init_worker_by_lua*
2020/02/16 02:22:13 [notice] 73918#0: *419760 [lua] cache_warmup.lua:42: cache_warmup_single_entity(): Preloading 'consumers' into the cache ..., context: init_worker_by_lua*
2020/02/16 02:22:13 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:13 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:13 [warn] 73945#0: *419831 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:13 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:13 [warn] 73945#0: *419831 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:13 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:13 [warn] 73945#0: *419831 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:13 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:13 [notice] 73918#0: *419760 [lua] cache_warmup.lua:81: cache_warmup_single_entity(): finished preloading 'consumers' into the cache (in 918ms), context: init_worker_by_lua*
2020/02/16 02:22:13 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:14 [warn] 73945#0: *419831 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:14 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:14 [warn] 73945#0: *419831 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:14 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:14 [warn] 73945#0: *419831 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:14 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:14 [warn] 73945#0: *419831 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:15 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:15 [warn] 73945#0: *419831 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:15 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:15 [warn] 73945#0: *419831 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:15 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:15 [warn] 73945#0: *419831 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:16 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:16 [notice] 1#0: signal 17 (SIGCHLD) received from 73945
2020/02/16 02:22:16 [alert] 1#0: worker process 73945 exited on signal 11
2020/02/16 02:22:16 [notice] 1#0: start worker process 73958
2020/02/16 02:22:16 [notice] 1#0: signal 29 (SIGIO) received
2020/02/16 02:22:16 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:16 [notice] 1#0: signal 17 (SIGCHLD) received from 73880
2020/02/16 02:22:16 [alert] 1#0: worker process 73880 exited on signal 11
2020/02/16 02:22:16 [notice] 1#0: start worker process 73959
2020/02/16 02:22:16 [warn] 73958#0: *419892 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:16 [warn] 73959#0: *419893 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:16 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:16 [warn] 73958#0: *419892 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:16 [warn] 73959#0: *419893 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:17 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:17 [warn] 73958#0: *419892 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:17 [warn] 73959#0: *419893 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:17 [warn] 73918#0: *419760 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:17 [warn] 73958#0: *419892 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:17 [notice] 1#0: signal 29 (SIGIO) received
2020/02/16 02:22:17 [notice] 73918#0: *419911 [lua] cache_warmup.lua:24: warming up DNS entries ..., context: ngx.timer
2020/02/16 02:22:17 [warn] 73959#0: *419893 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:17 [notice] 1#0: signal 17 (SIGCHLD) received from 73918
2020/02/16 02:22:17 [alert] 1#0: worker process 73918 exited on signal 11
2020/02/16 02:22:17 [notice] 1#0: start worker process 73960
2020/02/16 02:22:17 [warn] 73958#0: *419892 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:17 [notice] 73960#0: *419923 [lua] cache_warmup.lua:42: cache_warmup_single_entity(): Preloading 'services' into the cache ..., context: init_worker_by_lua*
2020/02/16 02:22:17 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:18 [warn] 73959#0: *419893 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:18 [warn] 73958#0: *419892 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:18 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:18 [warn] 73959#0: *419893 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:18 [warn] 73958#0: *419892 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:18 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:18 [warn] 73958#0: *419892 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:18 [warn] 73959#0: *419893 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:18 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:19 [notice] 73960#0: *419923 [lua] cache_warmup.lua:81: cache_warmup_single_entity(): finished preloading 'services' into the cache (in 1523ms), context: init_worker_by_lua*
2020/02/16 02:22:19 [notice] 73960#0: *419923 [lua] cache_warmup.lua:42: cache_warmup_single_entity(): Preloading 'plugins' into the cache ..., context: init_worker_by_lua*
2020/02/16 02:22:19 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:19 [warn] 73958#0: *419892 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:19 [warn] 73959#0: *419893 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:19 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:20 [warn] 73959#0: *419893 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:20 [warn] 73958#0: *419892 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:20 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:20 [warn] 73959#0: *419893 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:20 [warn] 73958#0: *419892 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:20 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:21 [warn] 73959#0: *419893 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:21 [warn] 73958#0: *419892 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:21 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:21 [notice] 1#0: signal 29 (SIGIO) received
2020/02/16 02:22:21 [warn] 73959#0: *419893 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:21 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:21 [notice] 1#0: signal 29 (SIGIO) received
2020/02/16 02:22:22 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:22 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:22 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:22 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:23 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:23 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:23 [notice] 73960#0: *419923 [lua] cache_warmup.lua:81: cache_warmup_single_entity(): finished preloading 'plugins' into the cache (in 4443ms), context: init_worker_by_lua*
2020/02/16 02:22:23 [notice] 73960#0: *419923 [lua] cache_warmup.lua:42: cache_warmup_single_entity(): Preloading 'consumers' into the cache ..., context: init_worker_by_lua*
2020/02/16 02:22:23 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:23 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:24 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:24 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:24 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:24 [notice] 73960#0: *419923 [lua] cache_warmup.lua:81: cache_warmup_single_entity(): finished preloading 'consumers' into the cache (in 853ms), context: init_worker_by_lua*
2020/02/16 02:22:24 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:24 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:25 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:25 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:25 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:26 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:26 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:26 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:26 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:27 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:27 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:27 [warn] 73960#0: *419923 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
2020/02/16 02:22:27 [notice] 73960#0: *420075 [lua] cache_warmup.lua:24: warming up DNS entries ..., context: ngx.timer
2020/02/16 02:22:28 [notice] 73960#0: *420075 [lua] cache_warmup.lua:34: finished warming up DNS entries' into the cache (in 99ms), context: ngx.timer
2020/02/16 02:22:29 [error] 76#0: *420182 [client 10.130.2.1] ModSecurity: Access denied with code 403 (phase 2). Matched "Operator `Ge' with parameter `5' against variable `TX:ANOMALY_SCORE' (Value: `15' ) [file "/usr/local/owasp-modsecurity-crs-3.2.0/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "79"] [id "949110"] [rev ""] [msg "Inbound Anomaly Score Exceeded (Total Score: 15)"] [data ""] [severity "2"] [ver ""] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "10.131.28.91"] [uri "/.jsp"] [unique_id "158181974936.342298"] [ref ""], client: 10.130.2.1, server: kong, request: "POST /.jsp HTTP/1.1", host: "10.175.238.204"
2020/02/16 05:45:26 [error] 73960#0: *664690 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
2020/02/16 05:45:26 [error] 73960#0: *664793 [lua] handler.lua:893: could not rebuild router via timer: could not create router: could not categorize route, context: ngx.timer
2020/02/16 05:45:27 [error] 74#0: *664712 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
2020/02/16 05:45:28 [error] 73959#0: *664725 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
2020/02/16 05:45:34 [error] 73959#0: *664951 [lua] handler.lua:893: could not rebuild router via timer: could not create router: could not categorize route, context: ngx.timer
2020/02/16 05:45:43 [error] 73958#0: *665042 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
2020/02/16 05:45:43 [error] 73920#0: *665051 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
2020/02/16 05:45:47 [error] 76#0: *665117 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
2020/02/16 05:45:47 [error] 74#0: *665129 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
2020/02/16 05:45:47 [error] 73920#0: *665238 [lua] handler.lua:893: could not rebuild router via timer: could not create router: could not categorize route, context: ngx.timer
2020/02/16 05:45:47 [error] 74#0: *665232 [lua] handler.lua:893: could not rebuild router via timer: could not create router: could not categorize route, context: ngx.timer
2020/02/16 05:45:55 [error] 73958#0: *665287 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
2020/02/16 05:46:00 [error] 74#0: *665400 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
2020/02/16 05:46:02 [error] 73958#0: *665442 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
2020/02/16 05:46:02 [error] 73960#0: *665450 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
2020/02/16 05:46:02 [error] 73920#0: *665453 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
2020/02/16 05:46:03 [error] 76#0: *665456 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer

Also the fact a few workers failed was a totally odd behavior I have never seen before. Just to confirm its not due to a 3rd party module(NGX Mod Security WAF) I plan on running yet again 1 more test Sunday with the module totally disabled. Can say we see no nil pointer C exceptions though so far so that paging change in the init.lua def did something. I will also say that during this window of tear down the timeouts may be due to the fact I think a full C repair runs overnight and may have overlapped with some of this testing. No idea on the failed worker processes though.

Some of the logs my unit output are here, you can see how the minute aligns up on a few of them with a failed to rebuild router/plugins error and a few other rebuild errors I had never seen before around timers [lua] handler.lua:893: could not rebuild router via timer: could not create router: could not categorize route, context: ngx.timer was a new one for me hah.

Another interesting bit was the other datacenter node I was running in sandbox that wasn't taking the admin-api calls directly but probably getting lots of cluster events and invalidations through the db:

2020/02/15 20:45:08 [error] 74#0: *6884 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
--
  | 2020/02/15 20:45:08 [error] 76#0: *6887 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
  | 2020/02/15 20:45:08 [error] 75#0: *6890 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
  | 2020/02/15 20:45:08 [error] 71#0: *6898 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
  | 2020/02/16 03:33:06 [error] 72#0: *583782 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
  | 2020/02/16 03:33:06 [error] 73#0: *583770 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
  | 2020/02/16 03:33:06 [error] 71#0: *583763 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
  | 2020/02/16 03:33:06 [error] 74#0: *583854 [lua] handler.lua:893: could not rebuild router via timer: could not load routes: [cassandra] could not execute page query: timeout, context: ngx.timer
  | 2020/02/16 03:33:06 [error] 74#0: *583855 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
  | 2020/02/16 03:33:06 [error] 76#0: *583851 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
  | 2020/02/16 03:33:07 [notice] 71#0: *583884 [lua] cluster.lua:839: execute(): [lua-cassandra] retrying request on host at 10.204.90.237 reason: timeout, context: ngx.timer
  | 2020/02/16 03:33:08 [error] 74#0: *583972 [lua] handler.lua:904: could not rebuild plugins iterator via timer: ...luarocks/share/lua/5.1/kong/runloop/plugins_iterator.lua:323: table index is nil, context: ngx.timer

Few timeouts in there but the last line was an interesting one:

2020/02/16 03:33:08 [error] 74#0: *583972 [lua] handler.lua:904: could not rebuild plugins iterator via timer: ...luarocks/share/lua/5.1/kong/runloop/plugins_iterator.lua:323: table index is nil, context: ngx.timer

Running these tests are brutal hah, doing it on my work laptop connected over vpn, getting like 3-5 TPS against the admin API for tear up and tear down logic, I generally go do other things while its building/deleting and look at logs later. Probably a combination of vpn latency paired with admin-api calls will generally just be slow when your db backing solution like C* needs strong consensus settings w full node replication going on. Takes literal hours for me to build it up to 4k and sometimes requires re-runs of the scripting because the load balancers and cloud ingress in this environment are shaky too, connection resets randomly at times, endrant.

[jeremyjpj0916]

Okay, Sunday evening follow up, positive news. With ModSecurity v3 Module disabled only Kong logic getting hit and no workers crashed or lua garbage collector error oddities/craziness.

Build up saw no issues, creation of all resources went very smoothly from Kong perspective. Tear down I only ran into some C* timeout against it seems which caused router/plugin mapper rebuild errors which caused 2 routes to fail to delete, and subsequently prevented my script from deleting the associated services too. Deletion of all the consumer resources went smoothly too. Overall for 4k route/service/consumers being generated to only have a fail case during delete of 2 resources is pretty awesome, and I am not having to run any extra patchwork to achieve it which is great, less custom snippets I run inn my own env, the better.

Kong logs were as follows:

2020/02/16 06:57:52 [warn] 1#0: [lua] cluster.lua:597: refresh(): [lua-cassandra] found contact point with '0.0.0.0' as rpc_address, using '10.86.168.39' to contact it instead. If this is incorrect you should avoid using '0.0.0.0' in rpc_address
--
  | nginx: [warn] [lua] cluster.lua:597: refresh(): [lua-cassandra] found contact point with '0.0.0.0' as rpc_address, using '10.86.168.39' to contact it instead. If this is incorrect you should avoid using '0.0.0.0' in rpc_address
  | 2020/02/16 06:57:55 [notice] 1#0: using the "epoll" event method
  | 2020/02/16 06:57:55 [notice] 1#0: openresty/1.15.8.1
  | 2020/02/16 06:57:55 [notice] 1#0: built by gcc 8.3.0 (Alpine 8.3.0)
  | 2020/02/16 06:57:55 [notice] 1#0: OS: Linux 3.10.0-957.12.1.el7.x86_64
  | 2020/02/16 06:57:55 [notice] 1#0: getrlimit(RLIMIT_NOFILE): 1048576:1048576
  | 2020/02/16 06:57:55 [notice] 1#0: start worker processes
  | 2020/02/16 06:57:55 [notice] 1#0: start worker process 71
  | 2020/02/16 06:57:55 [notice] 1#0: start worker process 72
  | 2020/02/16 06:57:55 [notice] 1#0: start worker process 73
  | 2020/02/16 06:57:55 [notice] 1#0: start worker process 74
  | 2020/02/16 06:57:55 [notice] 1#0: start worker process 75
  | 2020/02/16 06:57:55 [notice] 1#0: start worker process 76
  | 2020/02/16 06:57:55 [warn] 76#0: *3 [lua] globalpatches.lua:52: sleep(): executing a blocking 'sleep' (0.001 seconds), context: init_worker_by_lua*
  | 2020/02/16 06:57:55 [notice] 71#0: *2 [lua] cache_warmup.lua:42: cache_warmup_single_entity(): Preloading 'services' into the cache ..., context: init_worker_by_lua*
  | 2020/02/16 06:57:55 [warn] 71#0: *2 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
  | 2020/02/16 06:57:56 [notice] 71#0: *2 [lua] cache_warmup.lua:81: cache_warmup_single_entity(): finished preloading 'services' into the cache (in 677ms), context: init_worker_by_lua*
  | 2020/02/16 06:57:56 [notice] 71#0: *2 [lua] cache_warmup.lua:42: cache_warmup_single_entity(): Preloading 'plugins' into the cache ..., context: init_worker_by_lua*
  | 2020/02/16 06:57:56 [warn] 71#0: *2 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
  | 2020/02/16 06:57:57 [notice] 71#0: *2 [lua] cache_warmup.lua:81: cache_warmup_single_entity(): finished preloading 'plugins' into the cache (in 927ms), context: init_worker_by_lua*
  | 2020/02/16 06:57:57 [notice] 71#0: *2 [lua] cache_warmup.lua:42: cache_warmup_single_entity(): Preloading 'consumers' into the cache ..., context: init_worker_by_lua*
  | 2020/02/16 06:57:57 [warn] 71#0: *2 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
  | 2020/02/16 06:57:57 [warn] 71#0: *2 [lua] socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*
  | 2020/02/16 06:57:57 [notice] 71#0: *2 [lua] cache_warmup.lua:81: cache_warmup_single_entity(): finished preloading 'consumers' into the cache (in 330ms), context: init_worker_by_lua*
  | 2020/02/16 06:57:57 [notice] 71#0: *29 [lua] cache_warmup.lua:24: warming up DNS entries ..., context: ngx.timer
  | 2020/02/16 06:57:57 [notice] 71#0: *29 [lua] cache_warmup.lua:34: finished warming up DNS entries' into the cache (in 356ms), context: ngx.timer
  | 2020/02/17 03:13:34 [error] 72#0: *1442243 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
  | 2020/02/17 03:13:34 [error] 71#0: *1442264 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
  | 2020/02/17 03:13:34 [error] 73#0: *1442222 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
  | 2020/02/17 03:13:34 [error] 75#0: *1442267 [lua] handler.lua:893: could not rebuild router via timer: could not load routes: [cassandra] could not execute page query: timeout, context: ngx.timer
  | 2020/02/17 03:13:34 [error] 76#0: *1442271 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
  | 2020/02/17 03:13:35 [error] 74#0: *1442291 [lua] handler.lua:893: could not rebuild router via timer: could not load routes: [cassandra] could not execute page query: timeout, context: ngx.timer
  | 2020/02/17 03:13:35 [error] 74#0: *1442274 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
  | 2020/02/17 03:13:36 [error] 72#0: *1442164 [kong] api_helpers.lua:310 could not execute selection query: timeout, client: 127.0.0.1, server: kong_admin, request: "DELETE /routes/cf1380ac-c488-4a60-a9ba-d70d007b2c71 HTTP/1.1", host: "localhost:8001"
  | 2020/02/17 03:13:39 [error] 71#0: *1442440 [lua] handler.lua:893: could not rebuild router via timer: could not load routes: [cassandra] could not execute page query: [Read timeout] Operation timed out - received only 1 responses., context: ngx.timer
  | 2020/02/17 03:13:39 [error] 71#0: *1442362 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
  | 2020/02/17 03:13:39 [error] 76#0: *1442369 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
  | 2020/02/17 03:13:40 [error] 73#0: *1442375 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
  | 2020/02/17 03:13:41 [error] 74#0: *1442414 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
  | 2020/02/17 03:13:42 [error] 75#0: *1442427 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
  | 2020/02/17 03:13:42 [error] 76#0: *1442429 [lua] handler.lua:893: could not rebuild router via timer: could not load routes: [cassandra] could not execute page query: timeout, context: ngx.timer
  | 2020/02/17 03:13:43 [error] 72#0: *1442439 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: timeout, context: ngx.timer
  | 2020/02/17 03:13:43 [error] 71#0: *1442503 [lua] handler.lua:893: could not rebuild router via timer: could not load routes: [cassandra] could not execute page query: [Read timeout] Operation timed out - received only 1 responses., context: ngx.timer
  | 2020/02/17 03:13:44 [error] 76#0: *1442529 [lua] handler.lua:904: could not rebuild plugins iterator via timer: [cassandra] could not execute page query: [Read timeout] Operation timed out - received only 1 responses., context: ngx.timer

Which corresponds well to my deletion script running time of failure on route delete:

21:47:50.075 [main] INFO  c.o.s.kong.gateway.client.KongClient - StargateGateway: https://gateway.company.com/admin-api needed to refresh jwt token, unauthorized occured
22:02:50.650 [main] INFO  c.o.s.kong.gateway.client.KongClient - StargateGateway: https://ping-kong-gateway-dev.ocp-elr-core-nonprod.optum.com/admin-api needed to refresh jwt token, unauthorized occured
22:13:37.279 [main] INFO  c.o.s.f.KongTearDownTest - Route: /api/unittest/route1349 could not be deleted
22:17:34.318 [main] INFO  c.o.s.f.KongTearDownTest - Route: /api/unittest/route2338 could not be deleted
22:17:50.564 [main] INFO  c.o.s.kong.gateway.client.KongClient - StargateGateway: https://gateway.company.com/admin-api needed to refresh jwt token, unauthorized occured
22:25:43.409 [main] INFO  c.o.s.f.KongTearDownTest - Service: UnitTestServiceName2338 could not be deleted
22:27:03.901 [main] INFO  c.o.s.f.KongTearDownTest - Service: UnitTestServiceName1349 could not be deleted

So I would suspect some of the problems that arose prior were not Kong related but the ModSec module. I have also disabled ModSec(in the hopes it will avoid the worker crashes) in our Stage to give better insight to the ACL bug for the week with this Cassandra paging patch added in with the hopes we don't see the ACL bug occur over the next 2 weeks. In the meantime looks like I will need to do some CORE dumps and tracing to figure out why ModSec's plugin is rekkin NGINX/Kong(likely, hard to 100% confirm it after just 1 test with it on and 1 off but lets say preliminary analysis shows it to be the culprit to those earlier issues, although I don't think it could be the lua gc issue because its a NGX Module thats not involved in Lua so they still may be semi on Kong there).

Side note,

socket.lua:159: tcp(): no support for cosockets in this context, falling back to LuaSocket, context: init_worker_by_lua*

Coulda sworn all these print statements ^ were fixed awhile back in Kong with a PR to not happen. I think I am senile now though so might be wrong there 😀.

[Tieske]

that's positive indeed 👍 , keep us posted.

[jeremyjpj0916]

I may run some more tests with a higher timeout to see if that potentially helps too if I get some spare time to build up and tear down our sandbox again. I have always left it on a 5 second timeout. Wonder if timeout of 10 seconds might resolve the 2 or so failures or if C* is just fried in those few moments of all the DELETES coming through and that query is doomed for a timeout from client perspective regardless of timeout value.

[jeremyjpj0916]

Still so far so good on ACL bug reported here. Has not reproduced again in Stage/Prod since C* paging patch implemented. Will watch another 2 weeks before we mark off that that somehow fixed it.

[jeremyjpj0916]

Welp we been monitoring for this bug error and still not seen it in stage or prod, and by now it certainly would have happened in stage prior to this fix. So calling it officially resolved. Glad this was fixed by the C* paging issue!

[Tieske]

@jeremyjpj0916 thorough work on your behalf, as usual. 👍

[jeremyjpj0916]

Just doing my humble part of some open source QA 😀 . Always nice seeing a PR resolve bugs that were not necessarily understood to be related at the time. Have a good one!