Now that we have the GET /people/:letter route working, there are two big steps to complete before it's really complete. First, we will implement the cache on this route. Second, within the cache filter we will calculate the number of occurrences to display.
Cache Implementation
Adding support for the cache to this backend route will be very easy.
First, import the cacheMiddleware at the top of the api route file.
Next, add the cacheMiddleware as a router-level middleware. Make sure that you add it AFTER the already implemented permissionsRoutemiddleware that is on this route. That way it will check that the requesting user has permission to access this route before it checks for a cached value to send back. As you can see in other implementations of the cacheMiddleware, there are two parameters that you need to pass in. First there is a parameter called a "Typescript Generic Parameter". This is because it is not a parameter of a function, but rather an interface that is passed in to indicate what type should be processed by the function. In this case, because the cached value will be a PersonListItem[], you will pass that in as the Typescript Generic Parameter by placing it between <> that appear here: cacheMiddleware<PersonListItem[]>(). Next, you will need to provide a filter as the only parameter of the actual function by passing it inside the parentheses. Just to get this route working, import the noFilter filter and pass it in here. We'll change this later.
Now, we want to make sure that the generated value is actually stored in the cache the first time that this route is called. This will make use of the cache.insert function. To access this, first make sure to use the serviceLocator to get the cache just like how you called sl.get to get the various DAOs that you use in this file.
Next, right before you use res.json() to send the response at the end of the route, we will make use of this function.
Create a constant called response and assign it by calling await cache.insert. This function will also take a Typescript Generic Parameter between <>. Once again it will be PersonLIstItem[]. Then, there are 3 function parameters to pass in. First, { req }. This uses object destructuring to pass in the request object which is used to generate a cache key. Don't worry too much about what that means. Second, the value to be cached. In this case that is the personListItem variable. Finally, we provide the filter to use. Once again, we will temporarily pass in the noFilter filter to get things working.
Finally, change the res.json so that it doesn't send the personListItem as the response, but rather sends the newly created response object that was generated by the cache.insert function. That way, the filtered version is still send as the response despite the full list of persons being inserted into the cache.
At this point, this route should support the cache. To test this, send a request in Postman to this route. Postman will display a time in milliseconds near the bottom half of the window that displays how long it took to get a response from the route. Make a mental note of that number. Send the same request again. You should see that the millisecond count dropped drastically, indicating that it was able to retrieve a cached value instead of accessing the database each time.
Text Occurrences
Next, we want to calculate the number of occurrences for each person. We will do this within a cache filter which will allow us to to keep a cached source of truth, while getting a user-specific occurrences count that excludes texts that the user is not allowed to view.
To start, we will create a new function in the PersonDao. First though, go ahead and remove the getAllPeopleBaseQuery and getAllPeople functions from that file as they are no longer used.
Create the new asynchronous function, calling it something like getPersonTextOccurrences. It will take 3 parameters. First, a string called uuid that will be the uuid of the person. Second, userUuid which can be either string | null. This will be used to filter out texts hidden from that user. Third, the optional trx transaction variable as seen in other DAO functions. It should return a Promise<number>.
Within the function block, we will calculate the number of occurrences for an individual person. First, create the k constant exactly how you see it in all other read DAO functions: const k = trx || knexRead(); Next, before we use knex to get the occurrences count, we will get a user-specific list of texts to hide. Luckily there is already a function in CollectionTextUtils that will do this for us! Simply use the service locator by calling sl.get('CollectionTextUtils') to get access to that class object. Then create a new constant called textsToHide by calling await CollectionTextUtils.textsToHide(userUuid, trx). This will get us an array of text uuids that the logged-in user is not allowed to see. We will use this in the next piece to filter those occurrences out of the list.
Next, we will use knex to get the number of occurrences. Write the equivalent of this SQL query where values found between <> indicate replacement with the relevant function parameters:
SELECT COUNT(uuid) FROM item_properties
WHERE object_uuid = <uuid>
AND WHERE reference_uuid IN (
SELECT uuid FROM text_discourse
WHERE text_uuid NOT IN <textsToHide>
)
You will also need to add .first() to the knex query so that the returned value is one row, instead of an array. Note: Using the .count() function can be a bit tricky, but is the best way to do this. Look at other examples of its use in other files for reference.
Once you have the number of occurrences, return it at the end of this DAO function.
Now that the new DAO function is working, we will make use of it in a new filter. In the filters.ts, create a new personsFilter function that mimics the setup of the other filter functions. However, the first parameter for this function should be called something like persons and should be of type PersonListItem[] because that is the type of the response that we will be filtering (as indicated by the type of the response for the /people/:letter API route you wrote). The function should also return a Promise wrapped PesrsonListItem[].
In the function block, use the serviceLocator to get the PersonDao. Then, within an await Promise.all, use the .map() function to map over the parameter array. Within each map loop, you'll use the spread operator (...) to copy the contents of each person into a new object. Then, you will manually add the occurrences key to override the existing null value. To set the occurrences key, you will call your newly created PersonDao.getPersonTextOccurrences function, passing in the person's uuid and the Useruuid if available, defaulting to null if not available.
Return that mapped version of the data.
Finally, go to the /people/:letter API route and replace both of the noFilter references with the personsFilter. Once this is complete, you should be able to send a Postman request to the route and see that the occurrences field is populated with user-specific text occurrences.
NOTE: You may notice that this route sometimes takes a REALLY long time to complete despite the cache. This is because it has to calculate occurrences each time outside of the cache, which isn't particularly efficient. We will have to discuss options going forward, but it might make sense to paginate this response or come up with some other way of speeding this up. We will save that for a future issue after further discussion though.
hbludworth
commented
2 years ago
Now that we have the
GET /people/:letterroute working, there are two big steps to complete before it's really complete. First, we will implement the cache on this route. Second, within the cache filter we will calculate the number of occurrences to display.Cache Implementation
Adding support for the cache to this backend route will be very easy.
First, import the
cacheMiddlewareat the top of the api route file.Next, add the
cacheMiddlewareas a router-level middleware. Make sure that you add it AFTER the already implementedpermissionsRoutemiddleware that is on this route. That way it will check that the requesting user has permission to access this route before it checks for a cached value to send back. As you can see in other implementations of thecacheMiddleware, there are two parameters that you need to pass in. First there is a parameter called a "Typescript Generic Parameter". This is because it is not a parameter of a function, but rather aninterfacethat is passed in to indicate what type should be processed by the function. In this case, because the cached value will be aPersonListItem[], you will pass that in as the Typescript Generic Parameter by placing it between<>that appear here:cacheMiddleware<PersonListItem[]>(). Next, you will need to provide a filter as the only parameter of the actual function by passing it inside the parentheses. Just to get this route working, import thenoFilterfilter and pass it in here. We'll change this later.Now, we want to make sure that the generated value is actually stored in the cache the first time that this route is called. This will make use of the
cache.insertfunction. To access this, first make sure to use theserviceLocatortogetthecachejust like how you calledsl.getto get the various DAOs that you use in this file.Next, right before you use
res.json()to send the response at the end of the route, we will make use of this function.Create a constant called
responseand assign it by callingawait cache.insert. This function will also take a Typescript Generic Parameter between<>. Once again it will bePersonLIstItem[]. Then, there are 3 function parameters to pass in. First,{ req }. This uses object destructuring to pass in the request object which is used to generate a cache key. Don't worry too much about what that means. Second, the value to be cached. In this case that is thepersonListItemvariable. Finally, we provide the filter to use. Once again, we will temporarily pass in thenoFilterfilter to get things working.Finally, change the
res.jsonso that it doesn't send thepersonListItemas the response, but rather sends the newly createdresponseobject that was generated by thecache.insertfunction. That way, the filtered version is still send as the response despite the full list of persons being inserted into the cache.At this point, this route should support the cache. To test this, send a request in Postman to this route. Postman will display a time in milliseconds near the bottom half of the window that displays how long it took to get a response from the route. Make a mental note of that number. Send the same request again. You should see that the millisecond count dropped drastically, indicating that it was able to retrieve a cached value instead of accessing the database each time.
Text Occurrences
Next, we want to calculate the number of occurrences for each person. We will do this within a cache filter which will allow us to to keep a cached source of truth, while getting a user-specific occurrences count that excludes texts that the user is not allowed to view.
To start, we will create a new function in the
PersonDao. First though, go ahead and remove thegetAllPeopleBaseQueryandgetAllPeoplefunctions from that file as they are no longer used.Create the new asynchronous function, calling it something like
getPersonTextOccurrences. It will take 3 parameters. First, astringcalleduuidthat will be the uuid of the person. Second,userUuidwhich can be eitherstring | null. This will be used to filter out texts hidden from that user. Third, the optionaltrxtransaction variable as seen in other DAO functions. It should return aPromise<number>.Within the function block, we will calculate the number of occurrences for an individual person. First, create the
kconstant exactly how you see it in all other read DAO functions:const k = trx || knexRead(); Next, before we useknexto get the occurrences count, we will get a user-specific list of texts to hide. Luckily there is already a function inCollectionTextUtilsthat will do this for us! Simply use the service locator by callingsl.get('CollectionTextUtils')to get access to that class object. Then create a new constant calledtextsToHideby callingawait CollectionTextUtils.textsToHide(userUuid, trx). This will get us an array of text uuids that the logged-in user is not allowed to see. We will use this in the next piece to filter those occurrences out of the list.Next, we will use knex to get the number of occurrences. Write the equivalent of this SQL query where values found between
<>indicate replacement with the relevant function parameters:You will also need to add
.first()to the knex query so that the returned value is one row, instead of an array. Note: Using the.count()function can be a bit tricky, but is the best way to do this. Look at other examples of its use in other files for reference.Once you have the number of occurrences,
returnit at the end of this DAO function.Now that the new DAO function is working, we will make use of it in a new filter. In the
filters.ts, create a newpersonsFilterfunction that mimics the setup of the other filter functions. However, the first parameter for this function should be called something likepersonsand should be of typePersonListItem[]because that is the type of the response that we will be filtering (as indicated by the type of the response for the/people/:letterAPI route you wrote). The function should also return aPromisewrappedPesrsonListItem[].In the function block, use the
serviceLocatortogetthePersonDao. Then, within anawait Promise.all, use the.map()function to map over the parameter array. Within eachmaploop, you'll use the spread operator (...) to copy the contents of each person into a new object. Then, you will manually add theoccurrenceskey to override the existingnullvalue. To set theoccurrenceskey, you will call your newly createdPersonDao.getPersonTextOccurrencesfunction, passing in the person'suuidand theUseruuidif available, defaulting tonullif not available.Return that mapped version of the data.
Finally, go to the
/people/:letterAPI route and replace both of thenoFilterreferences with thepersonsFilter. Once this is complete, you should be able to send a Postman request to the route and see that theoccurrencesfield is populated with user-specific text occurrences.NOTE: You may notice that this route sometimes takes a REALLY long time to complete despite the cache. This is because it has to calculate occurrences each time outside of the cache, which isn't particularly efficient. We will have to discuss options going forward, but it might make sense to paginate this response or come up with some other way of speeding this up. We will save that for a future issue after further discussion though.