Migrating trip segmentation feature to native Android project

xubowenhaoren commented 5 years ago

Hi, this is a request/ suggestion for the modularization of trip segmentation feature. E-mission is an excellent trip data collection solution by itself. However, researchers already working with other general data collection frameworks (AWARE, for instance) cannot easily integrate the e-mission functionalities. Therefore, we're requesting the trip segmentation feature to be modularized. Here we are listing some goals, where each goal is a "phase" that provides increasingly more modularization that we'd like to see.

Make the trip plugin available as a Maven repo. Core functionalities like recording movement, generating survey notifications, and local database storage should ideally work "out of the box".
- Relevant modules: e-mission-data-collection, e-mission-transition-notify
Provide a guide for creating a simple UI that lists previous trips and provides the "take survey" button for the participant to take the surveys.
- Relevant modules: e-mission/e-mission-phone: www/templates/diary, js/diary/
Handle database syncing:
- Expose the local database so e-mission data can be synced by the parent data collection framework
  - More work, but a more elegant solution. We understand that currently some final pipelines must be run on the remote server for the trip segmentation to work. How to address this is work TBD.
- Or create a lightweight e-mission-server that supports database syncing with a simple identifier like device_id. Remove other components like connectionConfig, auth, etc.
  - Final analysis pipelines can still be run on this lightweight e-mission-server. Minimal modification required for e-mission-data-collection.
Expose emission's location and activity recognition providers to the parent data collection framework to save battery life.
- Currently both emission and AWARE are making location requests at the same time but separately, and this has been shown to impact some users' battery life.
- Relevant modules: e-mission-data-collection

shankari commented 5 years ago

Responding to point 3

Or create a lightweight e-mission-server that supports database syncing with a simple identifier like device_id. Remove other components like connectionConfig, auth, etc.

The auth mechanisms support both "skip" and "tokenlist", in which you send a string from the phone and the server largely accepts it. The rest of the server can then be run. The challenge with "skip", (which is the closest to what you want) is that it is not really secure. If somebody else runs an app on the phone and determines your deviceid, they can retrieve your data through the REST API calls.

I am not sure what authentication mechanism AWARE uses, maybe you can write a version that reuses their auth method?

xubowenhaoren commented 5 years ago

AWARE used a simple syncing method, i.e. using device_id as the identifier and the actual data as payload because we don't support data downloading/ retrieving. That's why we didn't have this security concern.

shankari commented 5 years ago

One workaround to get you pretty close to what you want is to generate a unique uuid in the app and send it to the server. Other hackers or other apps cannot get access to this because it is not tied to the phone in anyway.

The disadvantage is that if the user uninstalls + reinstalls the app or switches to another phone, you cannot maintain the continuity of the data.

shankari commented 5 years ago

to generate a unique uuid in the app and send it to the server

Instead of having the app generate a uuid, you can also just use skip and have the user type in a unique to them secret which will work out of the box, and has the potential to be consistent across reinstalls. But if they lose the secret, you are SOL.

shankari commented 5 years ago

The auth is currently implemented in native code in https://github.com/e-mission/cordova-jwt-auth There are multiple supported auth types that are selected by a factory. Should be pretty easy to add a new one.

shankari commented 5 years ago

In particular, on android, if you wanted to use the native code directly, something like the following in someplace that has a context, typically the activity.

tokenCreator = AuthTokenCreationFactory.getInstance(this);
AuthPendingResult result = tokenCreator.uiSignIn(this);
result.setResultCallback(new ResultCallback<AuthResult>() {
    // successful login
}

should work

You will have override onActivityResult and onNewIntent on the main activity similar to the implementation in the plugin.

shankari commented 5 years ago

@xubowenhaoren instead of integrating the e-mission code into AWARE, have you considered launching the AWARE stuff from e-mission? It is pretty trivial to wrap native code in a cordova plugin (as you can see from https://github.com/e-mission/cordova-jwt-auth/blob/master/src/android/JWTAuthPlugin.java) and you can make a single "AWARE plugin" that encompasses all AWARE functionality.

Since AWARE does not appear to have a complex UI (since they do not retrieve data from the server), you would not have to make significant UX changes to the e-mission-phone repo.

This is actually not the best approach from the perspective of improvements to the e-mission roadmap (making libraries available as jar files would actually be a much more useful long-term contribution) but it just seems like it would be much easier for this particular use case.

shankari commented 5 years ago

wrt https://github.com/e-mission/e-mission-docs/issues/410#issuecomment-498386725 for the skip and tokenlist authentications only, we do expect to be running in a cordova app with a webview around here https://github.com/e-mission/cordova-jwt-auth/blob/master/src/android/PromptedAuth.java#L42

So if you are creating a native-only app with no webview, you probably want to write a different version of the promptedAuth auth method that uses a native prompt and returns the value directly.

xubowenhaoren commented 5 years ago

Some additional thoughts:

The trip segmentation plugin ultimately becomes a maven plugin, which anyone can easily import.
There could be an option to use the plugin without setting up the entire remote e-mission server.
There could be a guide to customize the local database used by the trip segmentation for seamless integration into other general-purpose data collection frameworks.

shankari commented 5 years ago

@xubowenhaoren can you add in the requirements for the resulting app? I really think we should approach this from the requirements and not from what can be done. Many things can be done, but we should figure out which of them make the most sense to do.

xubowenhaoren commented 5 years ago

The trip-segmentation migration should behave with the same accuracy as the Cordova original.
No UI is required, but the local DB used should have some easy access to the developer can implement a basic, but informative UI so that participants/ users can understand enough context for the trip.
AWARE already collects Google-fused-location and Google-activity-recognition once every minute. There could be a relatively setup to change the sensor provider from the plugin itself to AWARE (or other data collection framework).

shankari commented 5 years ago

@xubowenhaoren great! so you can certainly identify the broad modules that you would need to work with for these requirements

e-mission-data-collection
cordova-usercache (or if aware has a database with a similar interface, you can choose to load/store to it instead of to the e-mission usercache)
again, if AWARE is already logging data to a local database (I am not sure if they are), then you can read from that database directly for the duty cycling

xubowenhaoren commented 4 years ago

After discussing with my supervisor, we decided that:

We would utilize a 7-day local cache/DB for the local UI. The participants can answer the survey within these 7 days.
To simplify our survey requirements, we would use web-based Qualtrics or SurveyMonkey surveys. We would like to implement that, in the upcoming native UI, when a user clicks on the "take survey" button, the app will launch a WebView to show the web survey.

shankari commented 4 years ago

To simplify our survey requirements, we would use web-based Qualtrics or SurveyMonkey surveys. We would like to implement that, in the upcoming native UI, when a user clicks on the "take survey" button, the app will launch a WebView to show the web survey.

e-mission already supports this. It fills a field in the surveys with the user's UUID and/or trip information. You can check out the interscity channel for a demo https://e-mission.eecs.berkeley.edu/#/client_setup?new_client=interscity&clear_usercache=true&clear_local_storage=true

https://github.com/e-mission/e-mission-docs/blob/c02d87bb6005ce099eeba7b92a37b5d5d2460bad/docs/dev/front/how_to_embed_an_external_survey_in_the_app.md

The launch code is in javascript, but you should be able to port it to native code if that is what you want.

xubowenhaoren commented 4 years ago

In UserCachePlugin.java, TransitionNotifier, and DataCollectionPlugin, we observe that cordova.CallbackContext and cordova.CordovaPlugin are heavily used. However, in our discussion, you mentioned that they are yet again calling some native code instead of the JS interface. Can you please clarify?

shankari commented 4 years ago

The files that you have indicated are all *Plugin* which are the interface to the javascript modules in cordova. However, the plugins just call native implementations, and the plugins internally call the native implementations as well.

In general, background tracking plugins cannot rely on javascript for background processing because, at least on android, when the app goes to the background, the related activity is suspended/killed, which means that the javascript disappears (poof!)

shankari commented 4 years ago

As a concrete example, consider the user cache. The usercache interface is defined in src/android/UserCache.java The primary implementation is in src/android/BuiltinUserCache.java. The interface to the javascript (which has the cordova callbacks) is in src/android/UserCachePlugin.java

If we look at the data collection plugin, which makes heavy use of the usercache, we see that it uses the BuiltinUserCache exclusively. The plugin is only used by the UI when it needs to read the database directly.

C02KT61MFFT0:e-mission-data-collection shankari$ grep -r "UserCache" src/android/ | wc -l
      37
C02KT61MFFT0:e-mission-data-collection shankari$ grep -r "UserCache" src/android/ | grep Plugin
src/android//DataCollectionPlugin.java:import edu.berkeley.eecs.emission.cordova.usercache.BuiltinUserCache;
src/android//DataCollectionPlugin.java:        BuiltinUserCache.getDatabase(myActivity).putMessage(R.string.key_usercache_client_nav_event,

The only reference to Plugin is in src/android//DataCollectionPlugin.java which is the interface from javascript to the data collection plugin. All communication between the native modules is directly through the native interface.

shankari commented 4 years ago

If you prefer GitHub, a similar query is https://github.com/e-mission/e-mission-data-collection/search?q=UserCache&unscoped_q=UserCache

xubowenhaoren commented 4 years ago

In plugins including e-mission-data-collection and cordova-usercache, import edu.berkeley.eecs.emission.R; is frequently used. However, this can only be generated during cordova build android. Should I copy your xml files in your res folder(s) to my res folder of the native project?

shankari commented 4 years ago

Yup! the res folders can have multiple xml files, and that is in fact what cordova does when it builds the native app from the plugins (see plugin.xml has <resource-file src="res/android/statemachine.xml" target="res/values/statemachine.xml" />

shankari commented 4 years ago

wrt more control over native code, you can explore a new framework called Capacitor (https://capacitor.ionicframework.com/) from Ionic; the same organization made the plugin that I use for UX customization in ionic deploy. I haven't looked into it at all, but I believe their goal was to make it easier to see the native internals, and because of their history, they are likely to offer a reasonable migration path from cordova.

You don't have to use this obviously, but if you like playing around and learning new stuff on your own, this might be a reasonable time-bounded (1-2 day) exploration that may make your life simpler going forward.

xubowenhaoren commented 4 years ago

In UserCachePlugin.java, the execute method, it currently is making a lot of calls to a JsonArray and CallbackContext. We believe these are relevant to the e-mission UI. How does E-mission interact with the module based on the CallbackContext? Since our migration does not need the UI at the moment, would it be safe for us to remove all these calls?

shankari commented 4 years ago

the e-mission javascript UI makes calls to the UserCachePlugin.java. As you can see from https://github.com/e-mission/e-mission-docs/issues/410#issuecomment-509700428 the data collection plugin does not. You do not need to integrate any of the Plugin calls into your app if it is native-only

shankari commented 4 years ago

The files that you have indicated are all Plugin which are the interface to the javascript modules in cordova. However, the plugins just call native implementations, and the plugins internally call the native implementations as well.

The only exception to this rule is for callbacks to notification/activities. cordova takes care of dispatching these, but you need to figure out whether you are going to put them directly into your Activity or do some kind of plugin-based registration or ???

This does not affect the usercache, but the data collection plugin responds to notifications to re-enable location services when they are turned off etc https://github.com/e-mission/e-mission-data-collection/blob/master/src/android/DataCollectionPlugin.java#L214

shankari commented 4 years ago

The plugins are basically a thin layer over non-plugin code. Everything that you do through a plugin, you can do directly through native code without involving the plugin in any way. The only exception is the onActivityResult (https://github.com/e-mission/e-mission-docs/issues/410#issuecomment-511866186), which is only in a few plugins.

xubowenhaoren commented 4 years ago

Hello,

We've run into issues pushing trip-end survey notification in the migration of the trip segmentation plugin for a native Android project. We've identified the source of the issue at line 144 in TransitionNotificationNotifier.java:

            JSONObject notifyConfigWrapper = UserCacheFactory.getUserCache(context).getLocalStorage(eventName, false);
            if (notifyConfigWrapper == null) {
                Log.d(context, TAG, "no configuration found for event "+eventName+", skipping notification");
                return;
        }

According to our previous communication, notifyConfigWrapper fetches the user-custom format in Javascript to customize the trip-end survey page. However, this feature is not required in our study project as of yet. How should we change the code so it opens up a custom Activity and launch a survey?

xubowenhaoren commented 4 years ago

Another relevant question: in our migration, where is the UserCache stored and in which database format? (SQLite perhaps?) Is there documentation specifying the table structure?

shankari commented 4 years ago

@xubowenhaoren according to the requirments listed earlier https://github.com/e-mission/e-mission-docs/issues/410#issuecomment-507733985 you were not planning to have any significant UI components, so this is essentially a new kind of integration.

Although all of the remote sensing is completely native, all the UI is currently written in javascript and uses the plugin interfaces for communication. So you should expect to have much larger structural changes to support UI work.

Concretely, in this case, if you look at the notification plugin interface, you will see that it is designed to store and retrieve configurations for the notification to be displayed when a particular state machine transition occurs. Further, the configuration is in the format expected by the cordova local notification plugin.

This is critical for an extensible, configurable platform, but may not be as critical for a one-off app. Have you considered just listening to the broadcasts from the FSM directly in a broadcast receiver that you implement and displaying your one hardcoded notification? I am not sure you can/want to do this in an activity directly since the notification generation needs to run in the background. It needs to be a broadcast receiver

shankari commented 4 years ago

Also, you have to be careful about communication to the server. My understanding from https://github.com/e-mission/e-mission-docs/issues/410#issuecomment-498381261 is that AWARE does not support bi-directional communication. By default, the data for a trip is uploaded when a trip is complete. You will also generate a notification when the trip is complete, but the user may only respond to it hours later (maybe at the end of the day). By that time, the data will already be on the server. I am not sure how you plan to customize the survey with the trip details when they are no longer available locally and they are not retrievable from the server.

shankari commented 4 years ago

Another relevant question: in our migration, where is the UserCache stored and in which database format? (SQLite perhaps?) Is there documentation specifying the table structure?

There is a usercache plugin, and yes, it uses SQLite under the hood. The table structure is in the plugin. You can also email the database to yourself from the app and open it to see what the stored data looks like.

shankari commented 4 years ago

The plugin is at https://github.com/e-mission/cordova-usercache It is used directly by the data collection plugin (without going through the plugin interface). The UI uses the plugin interface (in UserCachePlugin.java) to read/write data to the usercache. Native code would use the direct native interface (UserCache.java)

This should Just Work since the plugin interface is essentially a thin wrapper over the native interface

shankari commented 4 years ago

If you want a pointer to the data structure, it is at https://github.com/e-mission/cordova-usercache/blob/master/src/android/BuiltinUserCache.java#L87

shankari commented 4 years ago

It is also pretty trivial to determine where the usercache is called directly from the data collection https://github.com/e-mission/e-mission-data-collection/search?q=UserCache&unscoped_q=UserCache

shankari commented 4 years ago

@xubowenhaoren can you please confirm that the basic background tracking is working in the native app without the trip end prompt code? Also, can you give us an idea of the changes you had to make to adapt the plugin to native work - I remember you said something about renaming the files.

@PatGendre's team is interested in hearing the results :)

xubowenhaoren commented 4 years ago

Yes, I can confirm that the basic background tracking is working, although the trip end prompt notification is currently missing.

We would share eventually the whole repo for the Android migration later, but right now, a rough summary for the migration goes as follows:

I forked data-collection, transition-notify, unified-logger, and usercache from the emission repos.
I copied over the respective Android/src folders to my native Android project.
Where necessary, I also copied over the relevant res files to the res folder in the native project.
I had to manually (as of yet) to rename all emission dependencies to the current, current paths.

xubowenhaoren commented 4 years ago

For the Cordova Plugin dependency, e.g. in DataCollectionPlugin.java file, I currently removed all of the dependencies as we do not have a Javascript UI and there's no need to interact with that. Keeping only the pluginInitialize method was sufficient for a minimal operation of the trip segmentation algorithm.

Finally, to start the algorithm, call

    UserCachePlugin.pluginInitialize(this);
    DataCollectionPlugin.pluginInitialize(this);

in onCreate of your Main Acitivity.

xubowenhaoren commented 4 years ago

We currently face a problem: we don't have explicit access to /data/ via non-root access. It seems like only the application itself has access to that location. However, during our conversation, emission had this feature to email both the debug DB and the logger DB to myself. Where is this relevant code so that we can use so that the native Android app can view the SQLite database?

xubowenhaoren commented 4 years ago

For the Cordova Plugin dependency, e.g. in DataCollectionPlugin.java file, I currently removed all of the dependencies as we do not have a Javascript UI and there's no need to interact with that. Keeping only the pluginInitialize method was sufficient for a minimal operation of the trip segmentation algorithm.

While it took us some time to reach this (amazing) result, the actual manual work should take less than 3 hours.

shankari commented 4 years ago

The emailing code in javascript is handled by the EmailHelper https://github.com/e-mission/e-mission-phone/search?q=EmailHelper&unscoped_q=EmailHelper

The implementation is in emailService.js, and the invocations to email the database (for example) is in recent.js

    $scope.emailCache = function () {
        EmailHelper.sendEmail("userCacheDB");
    }

shankari commented 4 years ago

As you can see from the implementation, the android code specifies the path as "app://databases" https://github.com/e-mission/e-mission-phone/blob/master/www/js/control/emailService.js#L40

To see what this maps to in native code, you need to check the cordova email composer plugin ($cordovaEmailComposer -> https://github.com/katzer/cordova-plugin-email-composer). More detail are in the associated PR https://github.com/katzer/cordova-plugin-email-composer/pull/158

PatGendre commented 4 years ago

Hi @xubowenhaoren thanks for the first explanations. We may start a native e-mission app project in a few months... @shankari thanks again for the complementary infos given :-)

xubowenhaoren commented 4 years ago

Hello, we now can read the logger DB in our native app. However, what pieces of logs are necessary to reconstruct the previous trips?

Plus, in our previous communication, you said that fetching previous trips works regardless of whether the user has an Internet connection. If we can guarantee that the local logger DB stays on the device for 7 days, then can we assure that reading the logger DB alone will fully reconstruct the trip data?

shankari commented 4 years ago

Hello, we now can read the logger DB in our native app. However, what pieces of logs are necessary to reconstruct the previous trips?

@xubowenhaoren the loggerDB is completely extraneous to reconstructing trips. It is essentially the same as the log files that you see in an reasonably complex system. You wouldn't expect to store your data in a log file.

The local data is stored in the usercacheDB.

If we can guarantee that the local logger DB stays on the device for 7 days, then can we assure that reading the logger DB alone will fully reconstruct the trip data?

Yes, if you replace loggerDB with usercacheDB in that statement.

xubowenhaoren commented 4 years ago

Last week we managed to get access to the userCacheDB and saw records relevant to the trip data, like "accuracy", "altitude", etc. We have a few questions:

How to read the streams of the local DB and detect the start and end of a recorded trip from the logs? Are there significant headers or contents from the data?
For each record of data, now separated from the local DB, how do we reconstruct the "trip info card" as displayed on the emission-original UI? For example, the start location, the end location, and the respective timing?
(Implementation detail) How do we enable a cache-like way to store previously parsed trip data, so we don't have to read the entire DB every time we wish to see the previous trips? Or should we keep only data for 7 days and leave this as-is?

shankari commented 4 years ago

I think that this answers both steps (1) and (2).

The current implementation of the trip UI, which is only in javascript, is at https://github.com/e-mission/e-mission-phone/tree/master/www/js/diary

The code to read the data and construct a timeline which is essentially formatted as GeoJSON (https://en.wikipedia.org/wiki/GeoJSON) is largely in readUnprocessedTrips https://github.com/e-mission/e-mission-phone/blob/master/www/js/diary/services.js#L883

Before we call readUnprocessedTrips, we first check to see how far the pipeline on the server has run, but you will not need to do that.

As an example of how to find the trip start/ends, we use the statemachine transition data that is stored in the usercache - in particular, we use UnifiedDataLoader.getUnifiedMessagesForInterval("statemachine/transition", tq) to read the transitions and then convert them into trips here var tripsList = transition2Trip(transitionList);

and so on.

xubowenhaoren commented 4 years ago

For readUnprocessedTrips, is it reading the trip data from only the local DB, or is it trying to also fetch data from the server?

shankari commented 4 years ago

It is merging both local and remote unprocessed data but it is doing so under the hood by using UnifiedDataLoader.getUnifiedMessagesForInterval e.g. in https://github.com/e-mission/e-mission-phone/blob/9e2f9d65f8e86443c6ba5ce4be2df25b74671673/www/js/services.js#L193

If you replace that with a call to only the local usercache, which essentially has the same arguments (e.g. BEMUserCache.getMessagesForInterval) it will only use local data.

shankari commented 4 years ago

I should also point out that after the data is read and converted into trips, we process it further in processOrDisplayNone as seen https://github.com/e-mission/e-mission-phone/blob/00a3f8e8345705c41a1fb47721280d7dc7686e6c/www/js/diary/services.js#L1032

xubowenhaoren commented 4 years ago

Hello,

In our communication today, we discussed the need to test the performance/accuracy of the emission-Aware against emission-original. However, in earlier tests, we discovered that install 2 apps with the same BroadCastReceiver names is not possible. Is there a patch that can solve this without dramatically changing the existing codebase?

shankari commented 4 years ago

@xubowenhaoren this was filed earlier as https://github.com/e-mission/e-mission-docs/issues/406

The French team contributed a patch in https://github.com/fabmob/e-mission-phone-fabmob/commit/df421d9d7524c692963933be20f4fa62fd445baf

but there were some issues with merging it to master (https://github.com/e-mission/e-mission-docs/issues/402#issuecomment-501429914), which they did not fix.

However, my comments were mainly to the hooks/before_build/android/android_set_openid_oauth_uri.js. You need hooks/before_build/android/android_set_provider.js, so you can just manually copy it.

Once you confirm that it works, it would be great if you would submit it as a PR 😄

e-mission / e-mission-docs

Migrating trip segmentation feature to native Android project #410