Review specs from Dan

[daisycrego]

Dan:

1. Alerting us when we get a Zillow inquiry on a contact that is already in the database.
   • Sending an email alert to notify this has happened (maybe use webhook to make this quicker)
   • Creating a separate UI view to show only these zillow exceptions with an ability to set their exception status (No action, Notified Zillow, Zillow Approves Exemption, Zillow Rejected Exemption)
2. Allow to sort Zillow Flex inquiries by # days since last update updating zillow is manual and the update date in this app will need to be set manually as well.
3. Provide UI to compare Zillow Flex Status in Zillow vs Status in FUB
   • Zillow status will be manually updated in this app, FUB status to be refreshed and pulled in.

[daisycrego]

These are the webbook events supported in the FUB API (https://docs.followupboss.com/reference#webhooks-2):

peopleCreated, peopleUpdated, peopleDeleted, peopleTagsCreated, peopleStageUpdated,peopleRelationshipCreated,peopleRelationshipUpdated,peopleRelationshipDeleted

Looking at peopleUpdated, which seems the closest:

One or more of the following fields are updated in a person:

Name name firstName lastName Emails emails Phone Numbers phones Address addresses Price price Background background Assigned Agent assignedTo assignedUserId Assigned Lender assignedLenderName assignedLenderId Contacted contacted Stage stage stageId Lead Source source sourceUrl Tags tags Custom Fields Relationship is created, updated, or deleted

It's really not clear to me what these events, "Zillow inquiry"s, look like, where they appear, in the FUB API.

[daisycrego]

I need to figure out where a Zillow inquiry is going to appear in the data. Is it going to appear as an event? It doesn't seem like the webhook is going to work, because we can't expect the peopleUpdated webhook to encapsulate the zillow inquiries, and that was the closest one. So , hopefully, a zillow inquiry is an event in the API, let's see.

[daisycrego]

There actually might be a good webhook: eventsCreated (under the People Events section here:

eventsCreated: People perform an action on your IDX website, e.g. view a property.

We can register a webhook for a peopleEvent, and then we can filter all of those events to get only the zillow events, and then send out the emails accordingly. This way we might not have to use a scheduled process approach, we can take advantage of the webhook. But we should probably have the ability to do a manual sync run as well. To make sure the data is up to date.

[daisycrego]

Webhook best practices

Setup your system to de-couple receiving webhook events from fetching the resource specified in the event. For example--the web service that receives events could record the event in a local database table. A separate backend process could be created to monitor this table for new rows and fetch the resource specified in the event. By separating these, the web service that handles receiving webhooks is doing less and is, therefore less likely to fail. The process that processes the events can contain all the complex logic to fetch the resource from Follow Up Boss, reconcile with your local system and mark the event as processed in the table. If this process fails or breaks it can be debugged independently of the web service. The table that stores events is the source of truth as to what webhook events have been received and processed.

Debugging webhooks

Since webhooks require a publicly accessible URL to function they can be hard to test from your local machine. It is recommended that you use a service like Request Bin when you are developing your webhook endpoints.

Common webhook mistakes

• Make sure you provide the correct URL when registering a webhook. You can list all your webhooks at the /webhooks endpoint.
• Make sure the webhook endpoint is returning a 2XX HTTP status code such as 200
• When registering a webhook make sure you are setting the X-System header. You can also include the system param in the query string or post data. For example: https://api.followupboss.com/v1/webhooks/32?system=AcmeCo
• If you are setting up webhooks for multiple Follow Up Boss accounts be sure to include a Follow Up Boss account identifier in the registered webhook URLs. Your webhook URLs should look like this: https://acmeLeadProvider.com/callbacks/fub/account12/peopleCreated or https://acmeLeadProvider.com/callbacks/fub/peopleCreated?fub_account=12

[daisycrego]

So from what I understand, the webhooks are going to alert the system that there is new events data, but the system still has to check whether the event is relevant (Zillow, related to an existing lead), and then decide whether or not to send out an email with the alert.

We will just create a webhook endpoint for eventsCreated, so that whenever an event is created, we will get a POST to that endpoint. The endpoint is responsible for filtering the event and, if the event is a Zillow inquiry, it will send an email and also create some kind of record in the system (we're not going to be storing all events, just the ones that we send emails about?)

[daisycrego]

More Details

• Zillow exceptions: When Zillow refers the group a lead which is a new lead that the group then closes, JBG owes Zillow a 35% GCI referral fee. However, if JBG can establish an existing relationship with the lead within 24 hours, we can avoid the fee. Currently, the ISA is looking out for new zillow leads, but having a real time alert system will allow us to start our 24 hour clock a bit earlier, be able to capitalize on all of that time.
• Source: There should be both a "Zillow" source and a "Zillow Flex" source. So maybe we want to look for all sources that contain "Zillow", and then create the subcategory "Flex"/"Zillow Flex".
• Collect ALL of the events that are coming in for all sources (Zillow, Zillow Flex, and others), but in the UI, the admin only really cares about the Zillow leads, and being able to distinguish between the Zillow and the Zillow Flex. However, they should have the secondary option in the UI to see all lead sources. But the primary view is focused on Zillow leads.

Inbound Lead Number Tracking is valued

• Dan would be like to be able to calculate how many of the new incoming leads are true new leads vs. past clients that already existed in our DB, to have a better picture of the lead numbers. To get these numbers, we need to make a determination for each incoming lead as to whether this is an existing person or a new person, irregardless of the source. Then, once we decide that this is an existing person, if it is also Zillow, we send that update email because we want to make this a Zillow exemption. But no matter what type of lead this is, we need to correctly categorize all of them and store them in the db.

[daisycrego]

Todo (5/26)

• The "Sync Events" feature will act as the manual jumping off point for collecting all events that don't already exist in the db, in case a WebHook failed to capture an event for some reason. This feature is an important fail safe option to have for Dan, and is incomplete because we don't yet record all of the required information about each new Event:
  • If the user was a new lead or an existing FUB lead

New Lead vs. Existing Lead - FUB API

How can we tell from the FUB API if an event is related to a new lead or an existing lead? We need to also be capturing all of the new lead created events, so we have our own DB to refer to for when a lead came into our system. This way, when a person comes in with an event, we can know whether this is a new person (from the POV of FUB) or an existing lead being updated.

So we need to set up a web hook to track all new leads as they come in AND we need to pull all of the existing leads (as of some 0 day) so that we have our own internal lead database which is also synced with the live database through (1) WebHook and (2) manual sync option. That way, we can compare all of the incoming leads to the new leads to understand if this is a new lead or not.

• [ ] Set up a Lead schema for the database so we can keep track of basic contact details, internal FUB ID for the user (important! we possibly don't need to store full details just the FUB IDs!), and the created date. This way, we can always know when leads were created and if they already existed in the database.
• [ ] Set up a POST endpoint for sending in new leads. Pass this to a WebHook for personCreated. Whenever a person is created, add a new Lead object to the database.
• [ ] Update the Event object to store a new status: newLead (boolean). If this is NOT a new lead AND this is a Zillow or Zillow Flex source event, then we send out an email alert to get this lead as a possible Zillow exemption. Otherwise we just store the data for the event as usual.
• [ ] Set up a POST endpoint for new events. Pass this to a WebHook for eventCreated/personEvent/whatever. Whenever an event is created, see if a person already exists in the database before the last 10 minutes, for example.
  • If the lead already existed, we will be labelling this as a possible "Zillow Exemption" if this is Zillow, and sending out the email to alert the team.
  • If the lead didn't already exist, then this event is for a new lead. If the lead already existed, then this event is for an existing lead. Add this status to the Event object created in the database

[daisycrego]

Feedback

It may not be necessary to store all of this information in another database. Could actually make everything worse. Keep looking at the API. The /person endpoint might actually be sufficient.

created: This is going to be good enough for us, actually. We can also do a simple check if created exists AND created===updated AND created != today (not sure if the last check is really necessary but I would have some concerns about never actually checking the day). We need to get a better understanding of the format of the date information - how much of an exact timestamp am I getting? We could go down an a "within the last hour" comparison even, but only if that's the kind of time data FUB is storing. I do really like the elegance of created exists AND created===updated, this is a new lead. Otherwise it's not a new lead. Simple as that. Never need to actually see what the time is. Don't need to worry about narrowing things down to the minute/hour/day, etc. We just have to rely on FUB correctly updating that updated field. I do like that final check though, created=== today. Probably a smart move just to add that final check.

[daisycrego]

Revised Todo (5/26)

• [x] Update the Event object to store a new status: isNewLead (boolean). If this is NOT a new lead AND this is a Zillow or Zillow Flex source event, then we send out an email alert to get this lead as a possible Zillow exemption. Otherwise we just store the data for the event as usual.
• [ ] Set up a POST endpoint for new events. Pass this to a WebHook for eventCreated/personEvent/whatever. Whenever an event is created, see if a person already exists in the database before the last 10 minutes, for example.
  • If the lead already existed, we will be labelling this as a possible "Zillow Exemption" if this is Zillow, and sending out the email to alert the team.
  • If the lead didn't already exist, then this event is for a new lead. If the lead already existed, then this event is for an existing lead. Add this status to the Event object created in the database
• [ ] Add a special function/method/route for checking if a user is a new lead or not. To do this, we will pull in the person data from the FUB api (https://docs.followupboss.com/reference#events-id-get) using the personId from the event. Parse the created time and compare it to updated as well as today's date to see if this is a new lead or not.