High KV write unit usage on Deno Deploy

[jakewhelan]

Here's some stats on the current cost of using done

According to /v1/admin/stats I have pushed ~1652 messages through the service, there were some retries in there for sure but it's not communicated in the /v1/admin/stats payload so I don't know how many.

The resulting cost was ~150k Deno Deploy KV write units, or 136MB of write data - 80KB of writes per message processed on average. The done payload sent with each message is very small: a callback URL, headers, and sometimes a body of ~1KB.

For comparison, the rest of my platform (~3 small services) consumed 6k Deno Deploy kv write units, collectively. These services are handling the callbacks from the queue, and storing the actual data output of the jobs.

My gut feel here is that something is wrong about how much it costs to run done, what is it writing that consumes so many KV units?

Based on the current Deno Deploy pricing and my workload, it will cost ~$2.50 USD per 100k messages or $25 USD per 1m messages. I know it's not a perfect example, but that is 60x more expensive than AWS SQS ($0.4 USD per million).

[jakewhelan]

I'm looking a little closer using kview at the done kv store.

Here's some numbers

• messages
  • Count: ~1600
  • Payload: 1.3KB (largest)
  • Estimated single-write per entry worst case scenario total: 2.5MB
  • Child keys:
    • None
  • logging
    • Count: ~5100 (including secondaries)
    • Payload:
      • STORE_CREATE_EVENT: 1.5KB (seems to contain the payload)
      • STORE_UPDATE_EVENT / DELIVER, QUEUED, SENT: ~250B
      • STORE_UPDATE_EVENT / RETRY: ~450B
  • Estimated single-write per entry worst case scenario total: 7.5MB
  • Child keys:
    • logging/secondaries/BY_MESSAGE_ID
      • Count: ~900
      • Payload: ~400B

kview is a bit busted so I can't dig very effectively in logging/secondaries, but I can see there are more keys: BY_STATUS and BY_PUBLISH_DATE for sure.

It's hard to conclude much from this at all, other than there is room for a small optimisation by not including the message payload in the persisted logs - it still does not explain 136MB of kv write usage, the evidence here in the DB accounts for maybe 10MB.

Some open questions:

1. Is it possible the same logging or messages entries are being written multiple times each?
2. Does each update to an entry cost the full size of the entry in write units? (probably?) If yes, how often does done update an entry?

[fightbulc]

Hey there, I haven't looked into this but the nature of adding relations in a kv store is expensive I would say. Also, processes are event based. Each message write/update will write a system message to let the app know (https://github.com/dnl-fm/done/blob/main/src/utils/store.ts#L285). We listen for these and decide what to do (https://github.com/dnl-fm/done/blob/main/src/managers/message-state-manager.ts).

I would have to take a look and see how many units Done is requiring for a single message and from there we need to see if there is room for optimisations.

The idea of this first iteration of Done was to have a very simple onboarding process. You simple deploy Done to Deno Deploy and everything runs. However, while building Done I wasn't too happy with the writing speed of KV (due to location) and in general adding relations to a KV store..., well it's not made for this.

I would allow to add storage containers which allows users to choose their store of preference. I would start to add Turso since I did already speed tests with them before launching Done.

[fightbulc]

@jakewhelan howdy!

I finally found some time and enabled a logging flag for the env variables. You can now turn off logs to save at least 2/3 of your kv writes. In best case Done writes 3 logs for 1 message. If there are any issues coming up sending a message this number increases with the amount of fails.

Thats a quick fix. I would still aim for the storage container feature.