All of the case oriented queries and analysis in Coconut Surveillance currently occur in the client. This is because the database is not capable of aggregating separate results (notification, facility, household, member) into a single result (case) which could then be indexed. Instead for a given date range, all of the separate results are downloaded to the client, where they are then aggregated together (in memory), and then analysis is performed on the in-memory results. This is a complicated and slow process that only occurs because the database doesn't allow for tertiary (3rd order) indexing (1: document, 2: results by case id, 3: cases). I see this as one of the main reasons that @scloo has been struggling to develop queries - it's just too complicated as it currently works. It is not difficult to add an index that returns full case data, it just can't be done purely within the database. The solution is to add a process that runs on the server that listens to the databases _changes feed. This process automatically makes sure that there is a single document for each case in the system, with all of the case's data cached and updated on changes. I am already doing something very similar to generate the CSVs. This will generalize that process to be used not just for CSVs but for all queries. This will include all of the summary properties in models/Case.coffee. This includes the following:
MalariaCaseID
indexCaseDiagnosisDate
LostToFollowup
District (if no household district uses facility)
District of Facility
shehia
isShehiaValid
highRiskShehia
village:
Patient Name"
ageInYears
Sex
isUnder5
hasCompleteFacility
notCompleteFacilityAfter24Hours
notFollowedUpAfter48Hours
followedUpWithin48Hours
indexCaseHasTravelHistory
indexCaseHasNoTravelHistory
completeHouseholdVisit
numberPositiveCasesAtIndexHousehold
numberPositiveCasesAtIndexHouseholdAndNeighborHouseholds
numberHouseholdOrNeighborMembersTested
numberPositiveCasesIncludingIndex
.... (and many more)
All of the case oriented queries and analysis in Coconut Surveillance currently occur in the client. This is because the database is not capable of aggregating separate results (notification, facility, household, member) into a single result (case) which could then be indexed. Instead for a given date range, all of the separate results are downloaded to the client, where they are then aggregated together (in memory), and then analysis is performed on the in-memory results. This is a complicated and slow process that only occurs because the database doesn't allow for tertiary (3rd order) indexing (1: document, 2: results by case id, 3: cases). I see this as one of the main reasons that @scloo has been struggling to develop queries - it's just too complicated as it currently works. It is not difficult to add an index that returns full case data, it just can't be done purely within the database. The solution is to add a process that runs on the server that listens to the databases _changes feed. This process automatically makes sure that there is a single document for each case in the system, with all of the case's data cached and updated on changes. I am already doing something very similar to generate the CSVs. This will generalize that process to be used not just for CSVs but for all queries. This will include all of the summary properties in models/Case.coffee. This includes the following: