Improve back-end resilience by using queues

[franckc]

We should add a DB backed queue for listener tasks that are business critical. In particular for the following:

• creation of offers for credit card payments.
• acceptance of offers

The queueing framework should be relatively generic so that we can use it for any type of tasks in the future.

[DanielVF]

Orders are currently created in _makeOffer.js and immediately sent to the blockchain.

This is currently a problem, since two simultaneous orders will get the same tx ID going to the blockchain, and one will be lost.

The majority of the order work is done in handleLog.js. HandleLog() is called by both the ongoing listener.js, and by the backfill.js.

handleLog does:

• Upsert a new event in the database (could uncle blocks mean the same TX is recorded and processed twice)?
• Read from that db event
• Skip if not an offer
• Get the encrypted offer via IPFS
• Decrypt data via much magic
• Upsert the order in the DB
• Send an email
• Send a discord webhook

Orders are manually submitted to printful by a human.

[DanielVF]

[franckc]

Great diagram :) It helps a lot to visualize the data flow.

[DanielVF]

I've tested out a faked version of our system using both bull and a quick, 30 minute, hand-rolled queue.

My secret weapon, called one or two times from each action:

randomFailure = (str) => {
    if(Math.random() < 0.30){
        throw(Error(str))
    }
}

Looks like bull will do a great job, and we won't need anything beyond it for the current set of needs.

Unfortunately, ioredis-mock, does not support enough features to run bull. I've stubbed out the the one crashing missing calls, but something else is silently failing, or not being called, and no jobs go through. Looks like redis will have to be a dependency, but as dependencies go, that's not that bad.

I'm going to start with the Stripe webhook / OfferCreation queue.

[franckc]

Thanks for the update sir! Haha. Love the secret weapon of yours :)

I wish Node.js would have a good npm package that implements a simple database work queue... But I can't find any good one... :(

Bull seems to be the de facto standard for queues. We actually used it in the DApp for cron-job and it does a very good job. The drawbacks I see are:

• Additional dependency on Redis
• Using Redis makes it a bit harder to get visibility into the current and past states of the queue when troubleshooting issues or just when wanting to check on the health of the system. Now, Bull has a GUI called Arena so even though I hate adding yet another npm dependency it seems worth integrating with it?

CCing @mikeshultz in case he has any opinion and also since he needs to know about the Redis dependency for building the deployabe VM image as well as for the multi-tenant back-end migration from Heroku to GCP.

[nick]

@DanielVF did you try redis-mock instead of ioredis-mock?

[mikeshultz]

I've never been a fan of bull, but if y'all want to use it then so be it. That said, building a DB queue would be relatively trivial.

[franckc]

Yeah I hear you @mikeshultz... @DanielVF did you weigh the pros/cons of using Bull vs building ourselves a super basic work queue backed by a db table?

[DanielVF]

@nick According to the someone far down in a github issue, redis-mock can work with bit of shims thrown in. I tested their code, and it doesn't work - it silently just lets bull use actual redis behind the scenes.

@franckc Yes, I initially leaned heavily towards building our own DB backed queue. However, it looks like bull has good thought put into the error handling side, and an almost perfect API for what we are doing. I think this will save us bugs on the queue side vs writing our own.

[nick]

This has been done now.