Lead status comparison feature

[daisycrego]

Allow admin to manually enter Zillow lead status, pull lead status from FUB, display these side by side in the UI so the admin can visually know whether or not they are in sync, and bring them to sync if needed.

[daisycrego]

Requires a table of all the leads and a new fetch method for the /person/:id endpoint.

[daisycrego]

Could re-purpose the old sync-events button and feature to instead sync-people. We can use this to sync the people in the last x period of time, to get all of that recent zillow data we need, including the statuses. This will also be used every time the admin uses to page to make sure all the data is in sync. We could alternatively add another webhook for whenever a person is created, so we only keep track of new leads and we also capture updates to lead status without the admin having to manually sync with the FUB api every time they use our site.

• [ ] Decide: persons webhook vs. manual sync-persons button to keep person status up to date

[daisycrego]

personCreated/personUpdated webhooks vs. manual Sync Person or Sync People button

• The way to perform the fewest requests to get the accurate information on a person's status is going to be with webhooks. However, if a webhook event doesn't fire for some reason, we will miss the update. This shouldn't happen but it could, so having a simple Sync Person button is going to be a more surefast way of making sure this person has the most up-to-date stage possible. I don't want to do overkill for this, so I'm thinking that I could have a Sync Person button for each row in the table, a Sync People button for all of the currently visible rows in the PersonTable (to be created), and then MAYBE, some webhooks which are updating person statuses in the background. By developing the absolutely required features, Sync People and Sync Person first, and the nice-to-have but somewhat incomplete webhooks feature later, the app will be operational asap.

Might be useful to discuss with Dan how they want to use the feature, what the admin's workflow is most likely to benefit from. I'm thinking that the ability to sync an individual row and the webhook for personCreated/personUpdated is enough, and maybe the update-all-people (Sync People button) is really overkill.

tentative

• [ ] Sync Person button in each row of the PersonTable --> Makes a GET request to the FUB api at person/:id route, updates the Person model in the database with the latest info, including the status/stage. Alerts the user once the update was performed with a snackbar notification.
• [ ] Sync People button to perform an initial syncing of the data. In case the app was down for a period of time and needs to be caught up to speed. Should find all the leads (or all the Zillow leads?) in the last day, something like that.
• [ ] Webhook for personCreated, personUpdated to keep our Person data as up-to-date as possible, with the Sync Person button as a fallback.

[daisycrego]

Plan - Technical

• [ ] Define a new Leads page to display the leads. Model it off of the Events page.
• [ ] Define a new LeadsTable component to display the lead data. Model it off the EventsTable component. The data to be displayed in each row is: basic lead details (personName, source, ?) & 2 columns for FUB stage and Zillow stage which can be updated (similar to the status column already in the EventsTable). Keep the Export CSV button we will use it again, but need to change some of that code.
• [ ] Add a Sync Leads button to the LeadsTable which will retrieve all the leads in the last x period of time and store them as documents in the new Lead model in the db. For now, we will store all of the leads, but through the UI we will only show the Zillow leads by default because the primary purpose of the Leads page is to keep Zillow up to speed with FUB. However it would be useful to have more of the Leads, if those are available. We could even run an initial sync event to make sure that we have all of the leads in the database to begin with. If we have all of the leads, and we set up the webhooks to update our own people table, we can keep our own leads in sync with the FUB API. But we always need to have a Sync Leads feature available to make sure all the leads are in sync.
• [ ] Register webhooks to keep track of newly created people in FUB and updates to existing people (personCreated and personUpdated), as well as any other related webhooks (check the FUB webhooks api). Create endpoint(s) to service the webhooks. Whenever a webhook event occurs, schedule a background process in worker.js to run an additional query to the FUB /person/:id API to get the rest of the data (we need the stage most importantly). This way we will have updates to the people data going on in the background of the app, similar to the events webhook tracking we already have in place.

[daisycrego]

Made a commit with most of the initial code for the leads feature. Need to make some fixes to the Events table which I will also have to carry over to the newly created LeadsTable, which I why I haven't refined the LeadsTable, as it may be changed when I make the fixes to the EventsTable: rowsPerPage can't be changed from 10, and the larger change, state of the EventsTable needs to be passed higher up in the component hierarchy so when the user goes to the More link, they can go back and see the state of their previous search.

[daisycrego]

Planning

Resuming this Leads stage comparison feature to compare the FUB vs. Zillow stages, now that the Events page and EventsTable have been implemented.

• [x] Copy over Events, EventsTable, and Event components for the basis of Leads, LeadsTable, and Lead.
• [x] Set up a webhook to retrieve leads in real time. Handle webhook requests in the worker process.
• [x] Set up a Sync Leads button to catch up the system if the webhook or worker are down for some reason. During a Sync, make sure to look for updates in the STAGE of the lead, but don't overwrite the Zillow Stage of the lead to be null! Just update any data coming in from FUB. When performing a sync, how far back do you go? It's hard to know which leads are the ones to look at because any of them, regardless of original order, could have a stage updated. This is why using a Sync Lead button explicitly before handling any lead, and even better yet, using webhooks to actually keep track of the ongoing changes, will make for a faster process for the admin user. This app is multipurpose, and a major secondary function is a database of the leads. There's no way to really get out of having both the webhooks and the Sync Lead/s features.
• [ ] Set up a Sync Lead button on each lead page / each row of the LeadsTable to make sure that you have the most up-to-date information from the API without syncing all the leads again.
• [x] Define a Lead model/schema in the database.
• [x] Define lead routes on the server to handle API requests for lead data:
  • [ ] /api/leads -> GET, POST
  • [ ] /api/leads/sync -> GET (to run the leads sync process)
  • [ ] /api/leads/:leadId -> GET, PUT, DELETE (change individual leads)
• [x] Define a lead controller to process the API requests coming into the Lead routes from the frontend (Leads and Lead will be using methods like list and read in api-lead.js to make requests to this lead controller).
  • [ ] create
  • [ ] leadById
  • [ ] read
  • [ ] list
  • [ ] remove
  • [ ] update
  • [ ] syncLeads
  • [ ] createLeadsWebhookCallback

[daisycrego]

Issue - Don't overwrite zillowStage during syncs

When the webhook events are being processed - if we make an update to the stage of a person, make sure not to update the zillowStage, this will surely just overwrite it to null! Make sure to only update some of the fields, not all of them.

Update

• This won't be an issue for the webhooks, I've already written them to only update the relevant fields, not overwrite zillowStage.
• May have to rewrite the Sync Leads feature because when updating, it shouldn't overwrite the stage. Waiting to save those changes until the initial sync happening right now finishes. 40,000 leads.

[daisycrego]

Issue - QueryExceededMemoryLimitNoDiskUseAllowed - MongoError

• Error thrown during db query if the user were to try and pull all the data. The default is only Zillow Flex leads, around 500 leads currently, so the initial page loads fine. But if the user were to select all lead sources, a MongoError is thrown, QueryExceededMemoryLimitNoDiskUseAllowed, and the error message: Executor error during find command :: caused by :: Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting. It might be possible to opt into external sorting to avoid this error.

Update

• Maybe something like this after all of the sorting calls in the lead api:

  , { "allowDiskUse" : true }

• That only applies to aggregation. Maybe this:

  
  db.collection.find(<match>).sort(<sort>).allowDiskUse()