Closed shankari closed 2 years ago
shankari
commented
2 years ago As of https://github.com/e-mission/e-mission-phone/pull/826/commits/0636a90cb9142999860f51eb8eaee82d2786d009, screenshot https://github.com/e-mission/e-mission-phone/pull/826#issuecomment-1120253675 we have the basic survey functionality working again. But there is still some special casing of "SURVEY" in the code that reads the manual inputs. Let's fix that next.
shankari
commented
2 years ago It is very tempting to refactor the trip button code even further to make it beautiful and to generalize to a single input per trip, but we don't have time for that right now. Our focus should be on what the enketo libraries do.
We should just comment/no-op everything to see what the survey code actually uses, and generalize that to work for the onboarding survey as well.
shankari
commented
2 years ago Tried to remove the special casing of "SURVEY" by moving the code to processManualInputs into the individual directives.
Loading the individual directive factories via the injector, but still running into a circular dependency
Error: [$injector:cdep] Circular dependency found: Timeline <- PostTripManualMarker <- DiaryHelper <- EnketoTripButtonService <- Timeline
http://errors.angularjs.org/1.5.3/$injector/cdep?p0=Timeline[object Object]3C-%6PostTripManualMarker%6%3C-%6DiaryHelper%6%3C-%6EnketoTripButtonService%6%3C-%6Timeline
@ionic://localhost/_app_file_/Users/kshankar/Library/Developer/CoreSimulator/Devices/72287858-3107-46F0-890C-F34F78427F7C/data/Containers/Data/Application/B3D241D4-6172-48AF-801D-6BB9C0BBCE1B/Library/NoCloud/phonegapdevapp/www/lib/ionic/js/ionic.bundle.js:13438:32
One fix might be to remove the dependency between the directive service and the diary helper.
const unprocessedLabelEntry = DiaryHelper.getUserInputForTrip(trip, nextTrip, inputList);There doesn't appear to be a reason why that matching needs to happen in the directive. Why doesn't it just happen in the service, right after we read the input, or in the controller?
shankari
commented
2 years ago maybe because we need to iterate over the user inputs - e.g.
mls.populateInputsAndInferences = function(trip, manualResultMap) {
...
trip.userInput = {};
ConfirmHelper.INPUTS.forEach(function(item, index) {
mls.populateManualInputs(trip, trip.nextTrip, item,
manualResultMap[item]);
});
...
}We could get the keys (e.g. MODE, PURPOSE) in addition to manual/mode_confirm, manual/purpose_confirm from the directives, but it is not clear why getUserInputForTrip is even in DiaryHelper. The rest of the functions there are around formatting the diary, and we moved them to a service because we wanted to reuse them in the common screen.
Since this is only used for the trip label matching, it seems to make sense to move it to a separate service that is directly in the survey module.
shankari
commented
2 years ago Moved the matching code to the newly created InputMatching service. But a new problem is that the enketo service mapping code also returns a promise. So we can't just use
resultMap[etbs.SINGLE_KEY] = EnketoSurveyAnswer.filterByNameAndVersion('TripConfirmSurvey', surveyResult)because then the value stored in the map is a promise
[Log] MULTILABEL: After processing manual results (3) (index.html, line 153)
[[], []] (2)
", resultMap "
{MODE: [], PURPOSE: []}
[Log] ENKETO: After processing manual results (3) (index.html, line 153)
[[]] (1)
", resultMap "
{SURVEY: Promise}The obvious fix is to listen to the promise and then set the value
EnketoSurveyAnswer.filterByNameAndVersion('TripConfirmSurvey', surveyResult)
.then((answer) => { resultMap[etbs.SINGLE_KEY] = answer });However, in this case, the result map is filled in asynchronously, which means that it is not properly accounted for in $scope.$apply
[Log] ENKETO: After processing manual results (3) (index.html, line 153)
[[]] (1)
", resultMap "
{}
[Log] ENKETO: After processing manual results (3) (index.html, line 153)
[[]] (1)
", resultMap "
{SURVEY: []}wait, that can't be the reason because
[Log] ENKETO: After processing manual results (3) (index.html, line 153)
[[]] (1)
", resultMap "
{SURVEY: []}and then
ENKETO: About to populate inputs and inferences for (3)
{data: Object, style: function, onEachFeature: function, pointToLayer: function, start_place: Object, …}
", with resultMap "
{SURVEY: []}The multi-label works, and it also has the manual map as blank at this stage
[Log] MULTILABEL: About to populate inputs and inferences for (3) (index.html, line 153)
{data: Object, style: function, onEachFeature: function, pointToLayer: function, start_place: Object, …}
", with resultMap "
{MODE: [], PURPOSE: []}Ah the multilabel works because the user_input is already in the trip
user_input: {mode_confirm: "walk", purpose_confirm: "library", replaced_mode: "drove_alone"}
verifiability: "already-verified"After overriding with the labels, we do get some new inputs
[Log] MULTILABEL: About to populate inputs and inferences for (3) (index.html, line 153)
{data: Object, style: function, onEachFeature: function, pointToLayer: function, start_place: Object, …}
", with resultMap "
Object
MODE: [Object] (1)
PURPOSE: [Object] (1)Reverting changes and restoring them one-by-one with logs to ensure that everything works.
[Log] ENKETO: populating trip, (3) (index.html, line 153)
{data: Object, style: function, onEachFeature: function, pointToLayer: function, start_place: Object, …}
" with result map"
Object
SURVEY: Array (24)
0 Object
data: {end_ts: 1651363948.849, xmlResponse: "<data xmlns:jr=\"http://openrosa.org/javarosa\" xmln…3e</instanceID>↵ </meta>↵ </data>", label: "1 purpose, 1 mode", start_ts: 1651362581.1776686, timestamp: "2022-05-09T16:47:43.148Z", …}
metadata: {time_zone: "America/Los_Angeles", plugin: "none", write_ts: 1652114863.1496382, platform: "ios", read_ts: 0, …}
Object Prototype
1 {data: Object, metadata: Object}
2 {data: Object, metadata: Object}
3 {data: Object, metadata: Object}
4 {data: Object, metadata: Object}
5 {data: Object, metadata: Object}
6 {data: Object, metadata: Object}
7 {data: Object, metadata: Object}
8 {data: Object, metadata: Object}
9 {data: Object, metadata: Object}
10 {data: Object, metadata: Object}
11 {data: Object, metadata: Object}
12 {data: Object, metadata: Object}
13 {data: Object, metadata: Object}
14 {data: Object, metadata: Object}
15 {data: Object, metadata: Object}
16 {data: Object, metadata: Object}
17 {data: Object, metadata: Object}
18 {data: Object, metadata: Object}
19 {data: Object, metadata: Object}
20 {data: Object, metadata: Object}
21 {data: Object, metadata: Object}
22 {data: Object, metadata: Object}
23 {data: Object, metadata: Object}Next steps:
Let's now try to understand the enketo wrappers better and pull out the load and save functionality into the directive.
shankari
commented
2 years ago Couple of obvious things to change:
In the enketo service:
const DATA_KEY = 'manual/survey_response';
const ENKETO_SURVEY_CONFIG_PATH = 'json/enketoSurveyConfig.json';From a performance perspective, we also read all entries
function showModal() {
const tq = $window.cordova.plugins.BEMUserCache.getAllTimeQuery();
return UnifiedDataLoader.getUnifiedMessagesForInterval(DATA_KEY, tq)
.then(answers => _restoreAnswer(answers))
.then(instanceStr => _loadForm({ instanceStr }));
}I'm also not sure why we need to check against the UUID; we can only retrieve entries for the same UUID. Ah here's a response from @atton16 https://github.com/e-mission/e-mission-docs/issues/674#issuecomment-926414998
My concern on uuid and timestamp is that the metadata and the provided user_id field is not ready until it is pushed. That is why I also embed it in data object.
https://github.com/e-mission/e-mission-docs/issues/674#issuecomment-927578617
Thank you for the clarification. I might be able to get rid of it in our survey and using the provided fields.
It looks like it is already not passed in
$scope.openPopover = function ($event, trip, inputType) {
return EnketoSurveyLaunch
.launch($scope, 'TripConfirmSurvey', { trip: trip })To fix the issue with reading all the entries again, we need to be a bit careful. The trip.user_input will be filled in once the data gets to the server. But in the interim, the trip.userInput["SURVEY"] only has the label, for reasons that I don't fully understand.
Let's make sure to store the full datastructure there.
shankari
commented
2 years ago Note to self: we should ensure that we support languages going forward https://github.com/enketo/enketo-core/blob/master/tutorials/10-configuration.md#explicitly-set-the-default-form-language
shankari
commented
2 years ago Now, we start exploring displaying the demographic survey during onboarding.
First, let's see if we can at least launch the survey from the onboarding screen as a button, so we have a nice backup. Then, we can see whether we can make it work inline.
shankari
commented
2 years ago Launching from button works.
https://user-images.githubusercontent.com/2423263/167929285-2a30be66-e319-4cee-87e2-33f3bafbb642.mov
shankari
commented
2 years ago I identified my previous attempts at creating the wrapper to work with the enketo form. https://github.com/e-mission/e-mission-phone/pull/563/files
Comparing it to the current wrapper, potentially modified by the UNSW folks, the changes are minimal. After removing the wrapper (modal vs. ion-view) and unifying the whitespace, we get
--- enketo-survey.html 2022-05-11 12:43:15.000000000 -0700
+++ enketo-survey-new.html 2022-05-11 12:43:22.000000000 -0700
@@ -1,5 +1,5 @@
- <ion-content overflow-scroll="true">
- <div class="main">
+ <ion-content class="enketo-plugin overflow-scroll">
+ <div class="main" id="survey-paper">
<article class="paper" data-tap-disabled="true">
<!--
This section is to be removed in application
@@ -29,8 +29,8 @@
mother application. The HTML markup can be changed as well.
-->
<a href="#" class="previous-page disabled" style="position:absolute; left: 10px; bottom: 40px;">Back</a>
- <button id="validate-form" class="btn btn-primary" ng-click="validateForm()" style="width:200px; margin-left: calc(50% - 100px);">Validate</button>
- <a href="#" class="btn btn-primary next-page disabled" style="width: 200px; margin-left: calc(50% - 100px)">Next</a>
+ <button id="validate-form" class="btn btn-primary" ng-click="enketoSurvey.validateAndSave()" style="width:200px; margin-left: calc(50% - 100px);">Save</button>
+ <a href="#survey-paper" class="btn btn-primary next-page disabled" style="width: 200px; margin-left: calc(50% - 100px)" ng-click="enketoSurvey.onNext()">Next</a>
<div class="enketo-power" style="margin-bottom: 30px;">Powered by <a href="http://enketo.org" title="enketo.org website"><img src="templates/survey/enketo_bare_150x56.png" alt="enketo logo" /></a> </div>
<div class="form-footer__jump-nav" style="display: flex; flex-direction: row;">The main differences are:
survey-paper, which is referenced from the bottom #survey-paper instead of #ng-click functions are different. Both of those seem like reasonable changes, and nothing too complicated. Let's try to display inline now...shankari
commented
2 years ago Aha! I have it working inline in https://github.com/e-mission/e-mission-phone/pull/826/commits/c987e3fabbc21e6f72c9aa9e2801a2dcb7b9c9c4.
https://user-images.githubusercontent.com/2423263/167994269-38bfedb1-effd-4fb4-b0f4-9599cd2f0acd.mov
I had to do some fairly hacky stuff which I will clean up later.
Notably:
<ion-content section from the popupinitSurvey export to the launcherngdone is uglyLet's clean each of those up one by one.
shankari
commented
2 years ago Trying to convert the <ion-content to a <div to allow us to inline the form outside of a full-ion content, but failing.
content.html:
<div class="enketo-plugin overflow-scroll" height="100%">
<div class="main overflow-scroll" id="survey-paper">survey.html:
<enketo-demographics-inline ng-done="finish"></enketo-demographics-inline>works
Putting the <ion-content with scrolling around it, causes it to fail. The div ends up with height 0 and does not display.
shankari
commented
2 years ago This is probably some subtle artifact of the scrolling and the height. We don't really need any other inlining of the survey now, so let's stick to the directive starting with the <ion-content and explore this later. But let's avoid copy-pasting the directive by re-using this in the popup.
shankari
commented
2 years ago One issue that I've been encountering consistently is that after we go through the onboarding screen, when we launch the popup, it doesn't show the form. If we reload the screen (e.g. by editing a file in the devapp), it works. I bet this is something subtle related to initSurvey since in the first approach, we call initSurvey twice and in the second, we call it once.
shankari
commented
2 years ago Aha! Figured it out! The problem is that after we have gone through the intro, there are two form.or selectors, one for the inline intro and one for the modal.
const formSelector = 'form.or:eq(0)';Both forms are populated - the modal from the previous load and in the inline from the intro screen
The form is reloaded, but the first form.or is selected, so the intro screen is loaded, not the popup. So there is no form in the popup and we can't see anyting.
Two potential solutions:
Let's explore the first option with a time bound.
shankari
commented
2 years ago This is even more obvious when you go back to the intro after launching the popup.
So popup -> intro (two copies) -> popup (second form added to intro) -> intro shows two forms in the slide. One of which is stuck at the last screen of the survey and the other which moves.
| Page 1 | Page 2 | Page 3 |
|---|---|---|
 |
 |
 |
shankari
commented
2 years ago ionicSlideBoxDelegate doesn't have the option to delete a slide.
http://man.hubwiz.com/docset/Ionic.docset/Contents/Resources/Documents/ionicframework.com/docs/api/service/%24ionicSlideBoxDelegate/index.html
But there is an option to take an angular directive out of the DOM https://stackoverflow.com/questions/33889454/angularjs-remove-custom-directive-and-child-directives-from-dom
shankari
commented
2 years ago Adding that functionality works
var validateAndSave = function() {
...
// remove this directive
$element.empty();
$scope.$destroy();
}But then the directive is not re-initialized when we go back through the intro.
So intro (works) -> popup (works) -> intro (blank screen)
Can we try to remove the entire intro view when we are done with it? It looks like the views are created when we navigate to them.
shankari
commented
2 years ago Aha! This works!
$("[state='root.intro']").remove();
$scope.$destroy();| Before going back to the intro screen (no intro view) | After going back to the intro screen (intro view) |
|---|---|
 |
 |
shankari
commented
2 years ago Current status:
I will probably not change the initSurvey for now, since we may want to change/refactor the popup and the button directive. Working on the final screen "without header" option
shankari
commented
2 years ago 
Checked everything down from the ion-view above the ion-tabs. All the sizes and the offsets are accurate +/- a few pixels.
The main difference comes in the ion-content, where the width on the right is 722px and the left is 797px.
shankari
commented
2 years ago Note that the "broken" version is missing the "Profile" header.
Which is an ion-header-bar and right below the body and above the ion-nav-view
shankari
commented
2 years ago Ah, the ion-header-bar is hidden on the broken one
ion-nav-bar class="bar-stable nav-bar-container hide" style="background-color: #212121 !important;" nav-bar-transition="ios" nav-bar-direction="none" nav-swipe=""
Removing the hide re-inserts the bar at the top, but it doesn't have any text and the content is not moved downwards, so we get something that looks like this.
shankari
commented
2 years ago Ok, so it turned out that almost all the other views (e.g. metrics, or diary list) have a
<ion-nav-bar class="bar-stable">
</ion-nav-bar>But the control didn't. Adding it there gives us this corrected spacing, but no Profile text.
shankari
commented
2 years ago This is because the nav bar is hidden. Unhiding it works. Not quite sure why it is hidden. This is not really a huge issue since in the normal course of events, the diary will be showed after onboarding, not the profile. Still, it would be helpful to fix this if we can. Let's see when the "hide" is set.
In fact, all the ion-nav-bar entries are set
0 <ion-nav-bar class="bar-stable nav-bar-container hide" style="background-color: #212121 !important;" nav-bar-transition="ios" nav-bar-direction="none" nav-swipe>…</ion-nav-bar>
1 <ion-nav-bar class="bar-stable nav-bar-container hide" nav-bar-transition="ios" nav-bar-direction="none" nav-swipe>…</ion-nav-bar>
2 <ion-nav-bar class="bar-stable nav-bar-container hide" nav-bar-transition="ios" nav-bar-direction="swap" nav-swipe>…</ion-nav-bar>I do notice that at the end of the trip, we try to go to inf-scroll which fails. Is that why the header is not restored?
Changing the default screen to diary to see if that fixes it.
shankari
commented
2 years ago That seems to have fixed it. The nav bar in the diary was hidden when we launched the intro, but unhidden when we returned to it. It looks like the nav bar is unhidden when we successfully open a view. Concretely, automatically loading profile also shows the profile bar correctly even if we don't add a nav bar to the control view.
Last issue left is loading any existing value. Not sure what the best UI for this might be, but as a start, we could populate with existing values and give people the option to skip if it is the same.
shankari
commented
2 years ago Current approach of pulling out the <ion-content as the common functionality (https://github.com/e-mission/e-mission-phone/pull/826/commits/ff7863d02a4d72c84de3bf5a20ef4c6aa05dbe31) is a bit complicated since it pushes the additional controls (skip-button, etc) to the intro controller rather than keeping it isolated in the component. Let's refactor so that only the form is pulled out and ng-include it in both the modal and the inline directives.
shankari
commented
2 years ago After putting the buttons into the <ion-content, we get the following, which is still a bit ugly.
If we do choose to edit and hide the little buttons, it works quite well.
| Buttons still visible | Buttons hidden |
|---|---|
 |
 |
shankari
commented
2 years ago We have a couple of choices in the case of an existing response. We can: (1) show only the buttons (2) show the buttons and the first screen (3) show the buttons and some kind of "preview"
Note that in options 1 and 3, there is no form to initialize when the controller is created, so we need to initialize when the "Edit response" button is clicked. Let's start with (1) since it is easier and we are running out of time.
shankari
commented
2 years ago Edit or skip screen is still a bit clunky. Let's see if we can display the existing result without a lot of work.
shankari
commented
2 years ago Let's see if we can display the existing result without a lot of work.
We can display the existing result, and it is not a lot of work. However, the naive approach of simply traversing the XML response iteratively to find all non-blank entries results in a not-very-pretty result.
More importantly, this non-prettiness is because the result is displayed using the field and answer codes (e.g. hh_size or not_currently_) instead of strings such as "household size". This means that it will not work at all with translated text. So this doesn't seem to be a great option.
I'll leave the preview code in there but will not use it for now. Unfortunately, the enketo core library does not appear to have a HTML preview https://enketo.github.io/enketo-core/global.html
shankari
commented
2 years ago Removed preview code in https://github.com/e-mission/e-mission-phone/pull/826/commits/4d5efb41a13eda6280e7a7f787fc72f4ff666704 Added border in https://github.com/e-mission/e-mission-phone/pull/826/commits/2308f0e0a793b4a396c67603986dc31998a86f24
Last two planned changes:
shankari
commented
2 years ago Seems like we would want to create the JSON copy on the client. We would need to do this anyway for data that has not yet been pushed to the server. So for simplicity, it is probably best to convert in the client; the save method seems like a good option.
shankari
commented
2 years ago Created copy on client. Made server changes, but have errors while saving the demographic survey. Need to use a different formatter that doesn't read the start and end timestamps.
Got error 'AttrDict' instance has no attribute 'start_ts' while saving entry AttrDict({'_id': ObjectId('628c6aee8f39cb68214a17d1'), 'metadata': {'time_zone': 'America/Los_Angeles', 'plugin': 'none', 'write_ts': 1652400058.8529959, 'platform': 'ios', 'read_ts': 0, 'key': 'manual/demographic_survey', 'type': 'message'}, 'user_id': UUID('a0a1b7f2-e4cd-463d-b29e-eee11d7c0f03'), }) -> NoneRemoved the start and end timestamps. Now the save fails with
raise WriteError(error.get("errmsg"), error.get("code"), error)
pymongo.errors.WriteError: The dollar ($) prefixed field '$' in 'data.jsonDocResponse.data.group_my6jo52.$' is not valid for storage., full error: {'index': 0, 'code': 52, 'errmsg': "The dollar ($) prefixed field '$' in 'data.jsonDocResponse.data.group_my6jo52.$' is not valid for storage."}
/Users/kshankar/e-mission/gis_branch_tests/emission/net/api/bottle.py:4082: DeprecationWarning: Flags not at the start of the expression '\\{\\{((?:(?mx)( ' (truncated)
patterns = [re.compile(p % pattern_vars) for p in patterns]Let's remove the JSON doc for now; we can easily recreated it on demand.
shankari
commented
2 years ago manual/trip_user_input seems like a fine choice and is consistent with our goals to combine the labels into one object in the future as wellmanual/demographic_survey, or do we have manual/one_time_survey with data.name = 'UserProfileSurvey'?
manual/one_time_survey is that we then support all kinds of surveys at one swoopmanual/demographic_survey is it spotlights the demographic survey, which we will always have, and makes it easier to retrieve using our standard timeseries interfaces.Concretely, if we used manual/one_time_survey, we would need to retrieve all surveys and then match against the name to find the most recent demographic survey; with manual/demographic_survey, the retrieval will be much easier.
I am tempted to just go with manual/demographic_survey for now and defer the decisions for the one time surveys until we have the time to engineer them properly.
Quick check: how easy will it be to migrate?
We can't just change all the existing keys on the server because some clients might not have migrated over yet.
manual/demographic_survey and manual/one_time_survey) for all server entriesmanual/one_time_survey entry comes in, we can delete the manual/demographic_survey since the client has migrated overmanual/demographic_survey entry comes in, we can store it and a copy since the client has not yet upgraded. Eventually, all clients would have upgraded, and the older versions will be deleted.Doesn't look too bad; let's go with demographic_survey for now. We can still call the wrapper onetimesurvey
shankari
commented
2 years ago The JSON error is due to https://stackoverflow.com/questions/70617689/mongodb-the-dollar-prefixed-field-is-not-valid-for-storage
we could choose to upgrade to mongo 5.0, but not sure how that will play with DocumentDB.
Let's just strip out the $ entries from the JSON before sending it over
Actually, the xml2json library has some config options https://github.com/sergeyt/jQuery-xml2json/blob/master/src/xml2json.js
var defaultOptions = {
attrkey: '$',
charkey: '_',
normalize: false
};so we can set it to something else. How about setting it to "attr"? That is not a special key and should be pretty safe. The only problem would potentially be with an actual attribute called attr, but I don't think that the survey results typically have that.
shankari
commented
2 years ago shankari
commented
2 years ago Just confirming that the survey edits also work...
One entry in the usercache
>>> edb.get_usercache_db().count_documents({"metadata.key": "manual/demographic_survey"})
1Run the pipeline; it moves to the timeseries, which now has two entries
>>> edb.get_usercache_db().count_documents({"metadata.key": "manual/demographic_survey"})
0
>>> edb.get_timeseries_db().count_documents({"metadata.key": "manual/demographic_survey"})
1Now, edit the survey from the profile
fmt_time: Tue May 24 2022 19:13:52 GMT-0700 (Pacific Daylight Time) {}
jsonDocResponse: {data: {…}}
label: "Answered"
name: "UserProfileSurvey"
ts: 1653444832.682
version: 1
__proto__: ObjectNow, end trip and force sync after clearing out the logs. Ah, the updated survey does show up.
2022-05-24 19:16:09,189:DEBUG:123145400045568:Updated result for user = e595f863-5c27-495e-9a57-8e55f9c3a12c,
key = config/consent, write_ts = 1653440310.839 = {'n': 1, 'nModified': 0, 'upserted': ObjectId('628d91698f39c
b68214d7d02'), 'ok': 1.0, 'updatedExisting': False}
2022-05-24 19:16:09,193:DEBUG:123145400045568:Updated result for user = e595f863-5c27-495e-9a57-8e55f9c3a12c,
key = stats/client_time, write_ts = 1653440372.654 = {'n': 1, 'nModified': 0, 'upserted': ObjectId('628d91698f
39cb68214d7d04'), 'ok': 1.0, 'updatedExisting': False}
2022-05-24 19:16:09,214:DEBUG:123145400045568:Updated result for user = e595f863-5c27-495e-9a57-8e55f9c3a12c,
key = stats/client_time, write_ts = 1653440372.74 = {'n': 1, 'nModified': 0, 'upserted': ObjectId('628d91698f3
9cb68214d7d06'), 'ok': 1.0, 'updatedExisting': False}
2022-05-24 19:16:09,216:DEBUG:123145400045568:Updated result for user = e595f863-5c27-495e-9a57-8e55f9c3a12c,
key = stats/client_time, write_ts = 1653440372.757 = {'n': 1, 'nModified': 0, 'upserted': ObjectId('628d91698f
39cb68214d7d08'), 'ok': 1.0, 'updatedExisting': False}
2022-05-24 19:16:09,218:DEBUG:123145400045568:Updated result for user = e595f863-5c27-495e-9a57-8e55f9c3a12c, key = manual/demographic_survey, write_ts = 1653444832.684 = {'n': 1, 'nModified': 0, 'upserted': ObjectId('628d91698f39cb68214d7d0a'), 'ok': 1.0, 'updatedExisting': False}
2022-05-24 19:16:09,220:DEBUG:123145400045568:Updated result for user = e595f863-5c27-495e-9a57-8e55f9c3a12c, key = statemachine/transition, write_ts = 1653444967.84 = {'n': 1, 'nModified': 0, 'upserted': ObjectId('628d91698f39cb68214d7d0c'), 'ok': 1.0, 'updatedExisting': False}
2022-05-24 19:16:09,222:DEBUG:123145400045568:Updated result for user = e595f863-5c27-495e-9a57-8e55f9c3a12c, key = statemachine/transition, write_ts = 1653444967.861 = {'n': 1, 'nModified': 0, 'upserted': ObjectId('628d91698f39cb68214d7d0e'), 'ok': 1.0, 'updatedExisting': False}And we now have entries in both the usercache and the timeseries.
>>> edb.get_usercache_db().count_documents({"metadata.key": "manual/demographic_survey"})
1
>>> edb.get_timeseries_db().count_documents({"metadata.key": "manual/demographic_survey"})
1Let's now try doing that again to ensure that the first time was not a fluke and that edits do in fact show up.
The consent entry overlaps again, but the rest of it is fine, and we do see the survey again
2022-05-24 19:22:51,100:DEBUG:123145431576576:Updated result for user = e595f863-5c27-495e-9a57-8e55f9c3a12c,
key = config/consent, write_ts = 1653440310.839 = {'n': 1, 'nModified': 0, 'ok': 1.0, 'updatedExisting': True}
2022-05-24 19:22:51,103:DEBUG:123145431576576:Updated result for user = e595f863-5c27-495e-9a57-8e55f9c3a12c,
key = background/battery, write_ts = 1653444968.051 = {'n': 1, 'nModified': 0, 'upserted': ObjectId('628d92fb8
f39cb68214d7dde'), 'ok': 1.0, 'updatedExisting': False}
2022-05-24 19:22:51,105:DEBUG:123145431576576:Updated result for user = e595f863-5c27-495e-9a57-8e55f9c3a12c,
key = stats/client_time, write_ts = 1653444968.299 = {'n': 1, 'nModified': 0, 'upserted': ObjectId('628d92fb8f
39cb68214d7de0'), 'ok': 1.0, 'updatedExisting': False}
2022-05-24 19:22:51,106:DEBUG:123145431576576:Updated result for user = e595f863-5c27-495e-9a57-8e55f9c3a12c,
key = stats/client_time, write_ts = 1653444968.423 = {'n': 1, 'nModified': 0, 'upserted': ObjectId('628d92fb8f
39cb68214d7de2'), 'ok': 1.0, 'updatedExisting': False}
2022-05-24 19:22:51,108:DEBUG:123145431576576:Updated result for user = e595f863-5c27-495e-9a57-8e55f9c3a12c,
key = stats/client_time, write_ts = 1653444968.444 = {'n': 1, 'nModified': 0, 'upserted': ObjectId('628d92fb8f
39cb68214d7de4'), 'ok': 1.0, 'updatedExisting': False}
2022-05-24 19:22:51,110:DEBUG:123145431576576:Updated result for user = e595f863-5c27-495e-9a57-8e55f9c3a12c, key = manual/demographic_survey, write_ts = 1653445366.183 = {'n': 1, 'nModified': 0, 'upserted': ObjectId('628d92fb8f39cb68214d7de6'), 'ok': 1.0, 'updatedExisting': False}
2022-05-24 19:22:51,111:DEBUG:123145431576576:Updated result for user = e595f863-5c27-495e-9a57-8e55f9c3a12c, key = statemachine/transition, write_ts = 1653445369.875 = {'n': 1, 'nModified': 0, 'upserted': ObjectId('628d92fb8f39cb68214d7de8'), 'ok': 1.0, 'updatedExisting': False}
2022-05-24 19:22:51,113:DEBUG:123145431576576:Updated result for user = e595f863-5c27-495e-9a57-8e55f9c3a12c, key = statemachine/transition, write_ts = 1653445369.894 = {'n': 1, 'nModified': 0, 'upserted': ObjectId('628d92fb8f39cb68214d7dea'), 'ok': 1.0, 'updatedExisting': False}And now we have two entries in the usercache and one in the timeseries.
>>> edb.get_usercache_db().count_documents({"metadata.key": "manual/demographic_survey"})
2
>>> edb.get_timeseries_db().count_documents({"metadata.key": "manual/demographic_survey"})
1The phone and server changes are now done. Phone: https://github.com/e-mission/e-mission-phone/pull/826 Server: https://github.com/e-mission/e-mission-server/pull/853
We can just merge the server code to master.
However, for the phone code, we only want to copy/merge the www/*/survey/enketo directory so that we don't introduce any inadvertent changes to the diary code.
Since we just merged from master, the chances of that happening are low, but don't want to introduce regressions at this point.
We probably want to do something like: https://stackoverflow.com/questions/1214906/how-do-i-merge-a-sub-directory-in-git
shankari
commented
2 years ago Looking at the actual set of changes in the PR, most of them are in the survey/enketo directory but some are outside - e.g. the refactoring of www/js/diary/service.js
https://github.com/e-mission/e-mission-phone/pull/826
| JS files | template files |
|---|---|
 |
 |
So maybe we should see what happens if we try to merge everything to master.
shankari
commented
2 years ago Comparing to master, we have: https://github.com/e-mission/e-mission-phone/compare/master...refactor_enketo
| Previous commits | Previous commits |
|---|---|
 |
 |
 |
 |
 |
 |
Everything from May of this year is code that I want to keep
shankari
commented
2 years ago Those initial changes https://github.com/e-mission/e-mission-phone/compare/15cb98ea80bbd855c104c53b98db10804ab9ef5f...425d47900ca392c7246dcfca1ca11fe0be50c672
are primarily to
most of which (intro, consent, etc) I will need to change back to non-RCITI values Can I merge everything other than those?
Although those are a good template of what to modify while merging to master.
shankari
commented
2 years ago Looks like there are basically two options:
cherry-pickcherry-pick is not considered a great option because the commit IDs will all be different and we could end up with merge conflicts
But once we have merged this to master, we can abandon the refactor_enketo branch so we don't have issues with merge conflicts later.
wrt "cherry picking a commit from one branch to another basically involves generating a patch, then applying it, thus losing history that way as well."
Most of the code is in new files, so not sure we are losing a lot of history. Let's try it on a new branch anyway and see what it looks like. This is the range of new commits this month: https://github.com/e-mission/e-mission-phone/compare/26a015faeb101b14fb2b63231d79274bc003422a...19f97a420a3388cb47dc34a651b7c88727622a4d
shankari
commented
2 years ago This doesn't sound very promising. The very second commit fails.
kshankar-35069s:phone-rciti-branch kshankar$ git cherry-pick 26a015faeb101b14fb2b63231d79274bc003422a...19f97a420a3388cb47dc34a651b7c88727622a4d
Auto-merging www/templates/main.html
[enketo_directives_only b2aea247] Comment out the dashboard screen
Date: Fri May 6 07:31:02 2022 -0700
1 file changed, 2 insertions(+), 2 deletions(-)
CONFLICT (rename/delete): www/templates/survey/enketo_bare_150x56.png deleted in HEAD and renamed to www/templates/survey/enketo/enketo_bare_150x56.png in 08d03f71 (Move all the enketo code into a separate directory). Version 08d03f71 (Move all the enketo code into a separate directory) of www/templates/survey/enketo/enketo_bare_150x56.png left in tree.
CONFLICT (rename/delete): www/templates/survey/enketo-survey-modal.html deleted in HEAD and renamed to www/templates/survey/enketo/enketo-survey-modal.html in 08d03f71 (Move all the enketo code into a separate directory). Version 08d03f71 (Move all the enketo code into a separate directory) of www/templates/survey/enketo/enketo-survey-modal.html left in tree.
Auto-merging www/js/survey/multilabel/multi-label-ui.js
CONFLICT (content): Merge conflict in www/js/survey/multilabel/multi-label-ui.js
CONFLICT (rename/delete): www/js/survey/enketo-survey-service.js deleted in HEAD and renamed to www/js/survey/enketo/enketo-survey-service.js in 08d03f71 (Move all the enketo code into a separate directory). Version 08d03f71 (Move all the enketo code into a separate directory) of www/js/survey/enketo/enketo-survey-service.js left in tree.
CONFLICT (rename/delete): www/js/survey/enketo-survey-launch.js deleted in HEAD and renamed to www/js/survey/enketo/enketo-survey-launch.js in 08d03f71 (Move all the enketo code into a separate directory). Version 08d03f71 (Move all the enketo code into a separate directory) of www/js/survey/enketo/enketo-survey-launch.js left in tree.
CONFLICT (rename/delete): www/js/survey/enketo-survey-answer.js deleted in HEAD and renamed to www/js/survey/enketo/enketo-survey-answer.js in 08d03f71 (Move all the enketo code into a separate directory). Version 08d03f71 (Move all the enketo code into a separate directory) of www/js/survey/enketo/enketo-survey-answer.js left in tree.
Auto-merging www/js/diary/services.js
CONFLICT (content): Merge conflict in www/js/diary/services.js
Auto-merging www/js/diary/list.js
CONFLICT (content): Merge conflict in www/js/diary/list.js
Auto-merging www/index.html
CONFLICT (content): Merge conflict in www/index.html
error: could not apply 08d03f71... Move all the enketo code into a separate directory
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit'
To use the behavioral data, we need to have a demographic survey for each user. We have typically used external surveys (Qualtrics/Google Forms) before, but those have the following limitations:
A solution that would address all those issues is to store the survey information in mongodb, just like everything else. But we don't want to create a survey builder. Instead, we will use kobotoolbox to create a survey which we will display using the enketo library.
The UNSW group has already integrated with enketo core in the https://github.com/e-mission/e-mission-phone/tree/rciti branch, so let's start by exploring that approach.
If we can generalize this sufficiently, we can also have a config option to create surveys for each trip instead of using the labeling approach.