Build a scalable, accurate, easily matchable tour model

[shankari]

The tour model will be constructed on the server. The server code is in this repo https://github.com/e-mission/e-mission-server.git

The instructions to install and run the server and the pipeline are in the README and, given that we have had multiple undergrads install it over time, should be fairly robust. Everything should be installable on your laptop, but you can use virtualenv or even a separate VM if you prefer.

Getting some test data

You will need some test data to play with. If you send me your PGP key, I can send you encrypted zip files of users who have agreed to share their data, both on android and on iOS. Alternatively, we can use sneakernet, and I can give you a USB drive with all the zip files on Tuesday. (wouldn't it be nice to have delegation working? :)

If you want to get started even more quickly, you can also pull data from the public phones using the bin/request_public_data/request_public_data.py script written by @yw374cornell. https://github.com/e-mission/e-mission-server/blob/master/bin/request_public_data/request_public_data.py

I would recommend pulling data for Oct 15th - Nov 15th because the rest of the pipeline might not work well on the experiments with non-duty cycled data. This should contain data for iphone1, iphone4, android2 and android4 - the corresponding UUIDs are in the request_public_data.py script.

If you pull public data, the script automatically loads the data into your local database. If you use the real data, you will need to load it.

Instructions to load the data and run the intake pipeline are in the README.

Data model

The data model is represented by the set of python wrapper classes at emission/core/wrapper. I expect that you will want to start with the cleaned trip and place objects. The easiest way to access them is by using the timeseries abstraction layer. You might also find the timeline abstraction useful. https://github.com/e-mission/e-mission-server/blob/master/emission/core/wrapper/cleanedtrip.py https://github.com/e-mission/e-mission-server/blob/master/emission/core/wrapper/cleanedplace.py

I've attached Timeseries_Sample.ipynb file to this issue to showcase some of the operations that might be useful.

Prior work

The previous tour model was at: emission/analysis/modelling/tour_model https://github.com/e-mission/e-mission-server/tree/master/emission/analysis/modelling/tour_model

The entry point was https://github.com/e-mission/e-mission-server/blob/master/emission/storage/decorations/tour_model_queries.py#L48

Note that the existing tour model code:

• used sections instead of trips
• worked across all data since the install date instead of a time range, so completely unscalable
• had a kludgy, unscalable data model in which we stored the IDs of all trips that matched to a common trip/place as part of the trip/place. The goal was to match the trip/place on the client so that we could compare against norms - e.g. was a particular cleaned trip faster/slower or earlier/later than usual. We need a better data representation.

[shankari]

Uploaded sample ipython notebook. Timeseries_Sample.ipynb.gz

[shankari]

I just checked this into the main directory as well https://github.com/e-mission/e-mission-server/pull/447/commits/65ed7a8cca9c9340a0413f95b47e23eb556c9a69