Redis - Run background process for every `eventsCreated` webhook POST

[daisycrego]

Background Jobs in Node.js with Redis

• Worker Dynos, Background Jobs and Queueing - Another Heroku article explaining background processes/worker dynos/queueing in scaling apps.

  A good rule of thumb is to avoid web requests which run longer than 500ms. If you find that your app has requests that take one, two, or more seconds to complete, then you should consider using a background job instead. This can be even more important for Node.js servers where computationally expensive requests can block the Event Loop and prevent the server from responding to any new requests until the computation is complete. Separating this computation out into a different process keeps your web server light and responsive. This article demonstrates using worker queues to accomplish this goal with a sample Node.js application using Bull to manage the queue of background jobs.

• bull: Queue package for handling distributed jobs and messages in NodeJS.
• redis: in-memory data structure store, used as a database, cache, and message broker.

Redis

• Redis, which stands for Remote Dictionary Server, is a fast, open-source, in-memory key-value data store for use as a database, cache, message broker, and queue.
• The project started when Salvatore Sanfilippo, the original developer of Redis, was trying to improve the scalability of his Italian startup. Redis now delivers sub-millisecond response times enabling millions of requests per second for real-time applications in Gaming, Ad-Tech, Financial Services, Healthcare, and IoT.
• Redis is a popular choice for caching, session management, gaming, leaderboards, real-time analytics, geospatial, ride-hailing, chat/messaging, media streaming, and pub/sub apps.

How does Redis work?

• All Redis data resides in-memory, in contrast to databases that store data on disk or SSDs. By eliminating the need to access disks, in-memory data stores such as Redis avoid seek time delays and can access data in microseconds.
• Redis features versatile data structures, high availability, geospatial, Lua scripting, transactions, on-disk persistence, and cluster support making it simpler to build real-time internet scale apps.
  • All Redis data resides in the server’s main memory, in contrast to databases such as PostgreSQL, Cassandra, MongoDB and others that store most data on disk or on SSDs.
• In comparison to traditional disk based databases where most operations require a roundtrip to disk, in-memory data stores such as Redis don’t suffer the same penalty. They can therefore support an order of magnitude more operations and faster response times. The result is – fast performance with average read or write operations taking less than a millisecond and support for millions of operations per second.

Redis - Popular Use Cases

Caching

• Redis is a great choice for implementing a highly available in-memory cache to decrease data access latency, increase throughput, and ease the load off your relational or NoSQL database and application.
• Redis can serve frequently requested items at sub-millisecond response times, and enables you to easily scale for higher loads without growing the costlier backend.
• Database query results caching, persistent session caching, web page caching, and caching of frequently used objects such as images, files, and metadata are all popular examples of caching with Redis.

Chat, messaging, and queues

• Redis supports Pub/Sub with pattern matching and a variety of data structures such as lists, sorted sets, and hashes. This allows Redis to support high performance chat rooms, real-time comment streams, social media feeds and server intercommunication.
• The Redis List data structure makes it easy to implement a lightweight queue. Lists offer atomic operations as well as blocking capabilities, making them suitable for a variety of applications that require a reliable message broker or a circular list.

Gaming leaderboards

• Redis is a popular choice among game developers looking to build real-time leaderboards.
• Simply use the Redis Sorted Set data structure, which provides uniqueness of elements while maintaining the list sorted by users' scores. Creating a real-time ranked list is as easy as updating a user's score each time it changes. You can also use Sorted Sets to handle time series data by using timestamps as the score.

Session store

• Redis as an in-memory data store with high availability and persistence is a popular choice among application developers to store and manage session data for internet-scale applications.
• Redis provides the sub-millisecond latency, scale, and resiliency required to manage session data such as user profiles, credentials, session state, and user-specific personalization.

Install Redis

• Install redis locally:

  brew install redis
  brew services start redis

• Note that like Postgres, redis can be provided by various providers (e.g. AWS ElastiCache for Redis), but it is also available as open source.

[daisycrego]

Todo

• [ ] Review Heroku article for using redis, includes a sample project with a Node.js app and bull to manage the queued jobs.
• [ ] Use redis to run a background process in response to every POST to /events/api/fub/callback.

[daisycrego]

Reading the code to understand how this works:

[daisycrego]

Code here

[daisycrego]

This is getting too long, I created a wiki

[daisycrego]

Now I understand how the example app works. It uses Node Foreman to actually run 2 processes. 1 process is to handle the workers, and supports clustering at scale (although we'll still be using a low-level Heroku dyno in this implementation, it could support scaling).

In the worker process (worker.js) throng is a worker manager for clustered Node.js apps. We don't really need a clustered Node.js app, but having the infrastructure is useful, so we won't explicitly remove that. Also, throng is touted as a "dead-simple one-liner for clustered Node.js apps. It "forks N workers and creates new ones if they go down". We define astartmethod which we pass tothrong, andthrongwill create multiple clusters and callstarton each of them. The number of worker processes (the number of clusters) is going to be eitherprocess.env.WEB_CONCURRENCY, which is set by Heroku based on your dyno, or2if that isn't set. You can play around with the value ofworkers` if you're trying to take advantage of more CPU cores.

Within start we have the interesting part of worker.js. We define a process for the Queue, workQueue, to perform whatever the job is and return a value once the job is done.

That's it. Now we can initialize a Queue using the same REDIS_URL in the web process (server.js) so that we can add jobs to the Queue, now that process is defined so we have something to do with the job. And we can also check the state of the job from another route in the web process to send that data back to the UI when it called the updateJobs method.

Adding this to the existing codebase shouldn't be too crazy. But we are somewhat changing the hierarchy of this app, so we need to go back to understanding how the package.json works currently before we just expect we can add Node Foreman into the mix with no hassles.

[daisycrego]

Integrated the basics of the bull work queue by adding a new worker process and some client-side code (currently injected in template.js but I can move it to somewhere else when I do the refactoring). At least was able to work around the concerns I had of making Node Foreman work with webpack. It all seems to work ok. And the jobs seem to be successfully created and then completed.

[daisycrego]

Defining the background process, here is the skeleton:

workQueue.process(maxJobsPerWorker, async (job) => {
    console.log(`Initiating a job in the worker process`);
    console.log(`job.data:`);
    console.log(job.data);
    // This is an example job that just slowly reports on progress
    // while doing no work. Replace this with your own job logic.
    let progress = 0;
    job.progress(progress);

    // do the work here

    // check if the event already exists
    // define the status of the event -> newLead, existingLead
    // send an alert if the status of the event === existingLead && source == Zillow or Zillow Flex
    // create a new Event object
    /* 
    incoming job.data will look like this:
    {
        eventId: 'dc6d4d68-f27a-4940-bd10-e60b4fe3b5dc',
        eventCreated: '2021-05-30T21:46:02+00:00',
        event: 'eventsCreated',
        resourceIds: [ 90490 ],
        uri: 'https://api.followupboss.com/v1/events/90490'
    }
    */

    progress = 100;
    job.progress(progress);

    // A job can return values that will be stored in Redis as JSON
    // This return value is unused in this demo application.
    return { value: "This will be stored" };
  });
}

[daisycrego]

The job will expect the following data:

{
        eventId: 'dc6d4d68-f27a-4940-bd10-e60b4fe3b5dc',
        eventCreated: '2021-05-30T21:46:02+00:00',
        event: 'eventsCreated',
        resourceIds: [ 90490 ],
        uri: 'https://api.followupboss.com/v1/events/90490'
    }

• If the uri field is missing AND the resourceIds array is empty, log that the job has failed.
• Otherwise, make a fetch to the uri for the rest of the event data (should include personId)
• Use the personId to fetch the rest of the person data, to determine if this is a new lead or not.

[daisycrego]

Progress Update

• The background process, triggered by new event data coming into the /api/events/fub/callback route, successfully runs and retrieves (1) extra event data, (2) person data to determine if this is a new lead, and then determines if an email should be sent to alert the admins this is a possible Zillow exemption. Either way, the important (really arbitrarily decided by me, could be changed) parts of the event data are stored in our own database, so we can follow up on these events.
• Haven't actually started sending the emails yet, choosing an email solution. Thinking about nodemailer. This medium article was has some code inspiration as well

[daisycrego]

Use an App Password for the SMTP

• Created an App Password to access one of my Gmail accounts.
• Better than using the password to the account, you can revoke it and you can only grant access to a subset of the google account data (e.g. Gmail). Passing the credentials, host, and port to the contructor for a Nodemailer SMTP instance.

[daisycrego]

Modified the logic to only send emails for "Zillow Flex" leads. Now we can set the email alert system for a time and let it run, see if it works. The Logs can also be used to backtrack.