e-mission / e-mission-docs

Repository for docs and issues. If you need help, please file an issue here. Public conversations are better for open source projects than private email.
https://e-mission.readthedocs.io/en/latest
BSD 3-Clause "New" or "Revised" License
15 stars 34 forks source link

incorrect mode detection #322

Closed PatGendre closed 5 years ago

PatGendre commented 5 years ago

Hi Shankari logs-29-31-jan.tar.gz , please find attached the logs for the 2 days where we have on a long distance trip (100km+), on january 29 : section from 4:06 to 6:01 pm detected as bus instead of car on jan 31 : air/hsr detected on section starting at 06:39 and ending at 7:25 am, instead of car Thanks for your help!

shankari commented 5 years ago

Loaded the two data from the two timeseries locally

(emission) shankari$ ./e-mission-py.bash bin/debug/load_timeline_for_day_and_user.py /tmp/2019-01-31.2019-01-31.timeline broken_modes@lfdm.fr
Connecting to database URL localhost
/tmp/2019-01-31.2019-01-31.timeline
Loading file /tmp/2019-01-31.2019-01-31.timeline
After registration, broken_modes@lfdm.fr -> 1c8256b3-0ec4-4733-b9f4-736a73578d2b
Finished loading 0 entries into the usercache and 1705 entries into the timeseries

(emission) shankari$ ./e-mission-py.bash bin/debug/load_timeline_for_day_and_user.py /tmp/2019-01-29.2019-01-29.timeline broken_modes@lfdm.fr
Connecting to database URL localhost
/tmp/2019-01-29.2019-01-29.timeline
Loading file /tmp/2019-01-29.2019-01-29.timeline
After registration, broken_modes@lfdm.fr -> 1c8256b3-0ec4-4733-b9f4-736a73578d2b
Finished loading 0 entries into the usercache and 9037 entries into the timeseries

Confirmed that they are in the database and in the correct order

$ ./e-mission-ipy.bash
Python 3.6.1 | packaged by conda-forge | (default, May 11 2017, 18:00:28)
Type "copyright", "credits" or "license" for more information.

IPython 5.3.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import emission.storage.timeseries.abstract_timeseries as esta

In [2]: from uuid import UUID

In [3]: ldfm_uuid = UUID("1c8256b3-0ec4-4733-b9f4-736a73578d2b")

In [4]: ldfm_ts = esta.TimeSeries.get_time_series(ldfm_uuid)
Connecting to database URL localhost

In [5]: ldfm_all_df = ldfm_ts.get_data_df(key="background/battery")

In [6]: ldfm_all_df.iloc[0].ts
Out[6]: 1548728744.688

In [7]: ldfm_all_df.iloc[0].fmt_time
Out[7]: '2019-01-29T03:25:44.688000+01:00'

In [8]: ldfm_all_df.iloc[-1].fmt_time
Out[8]: '2019-01-31T18:31:20.752000+01:00'

In [9]: len(ldfm_all_df)
Out[9]: 130
shankari commented 5 years ago

Delete the old data

(emission) shankari$ ./e-mission-py.bash bin/debug/purge_user.py -e broken_modes@lfdm.fr
Connecting to database URL localhost
INFO:root:For uuid = 1c8256b3-0ec4-4733-b9f4-736a73578d2b, deleting entries from the timeseries
DEBUG:root:db_array not passed in, looking up databases
INFO:root:result = {'n': 10751, 'ok': 1.0}
INFO:root:For uuid = 1c8256b3-0ec4-4733-b9f4-736a73578d2b, deleting entries from the analysis_timeseries
INFO:root:result = {'n': 428, 'ok': 1.0}
INFO:root:For uuid 1c8256b3-0ec4-4733-b9f4-736a73578d2b, deleting entries from the user_db
INFO:root:result = {'n': 1, 'ok': 1.0}

Reload the raw data

(emission) shankari$ ./e-mission-py.bash bin/debug/load_timeline_for_day_and_user.py /tmp/2019-01-29.2019-01-29.timeline broken_modes@lfdm.fr
Connecting to database URL localhost
/tmp/2019-01-29.2019-01-29.timeline
Loading file /tmp/2019-01-29.2019-01-29.timeline
After registration, broken_modes@lfdm.fr -> a4959cef-485c-4414-bc9a-a8c45dfb3294
Finished loading 0 entries into the usercache and 9037 entries into the timeseries

(emission) C02KT61MFFT0:e-mission-server shankari$ ./e-mission-py.bash bin/debug/load_timeline_for_day_and_user.py /tmp/2019-01-31.2019-01-31.timeline broken_modes@lfdm.fr
Connecting to database URL localhost
/tmp/2019-01-31.2019-01-31.timeline
Loading file /tmp/2019-01-31.2019-01-31.timeline
After registration, broken_modes@lfdm.fr -> a4959cef-485c-4414-bc9a-a8c45dfb3294
Finished loading 0 entries into the usercache and 1705 entries into the timeseries

Verify that it is still valid

In [1]: import emission.storage.timeseries.abstract_timeseries as esta

In [2]: from uuid import UUID

In [3]: ldfm_uuid = UUID("a4959cef-485c-4414-bc9a-a8c45dfb3294")

In [4]: ldfm_ts = esta.TimeSeries.get_time_series(ldfm_uuid)
Connecting to database URL localhost

In [5]: ldfm_all_df = ldfm_ts.get_data_df(key="background/battery")

In [6]: ldfm_all_df.iloc[0].fmt_time
Out[6]: '2019-01-29T03:25:44.688000+01:00'

In [7]: ldfm_all_df.iloc[-1].fmt_time
Out[7]: '2019-01-31T18:31:20.752000+01:00'

In [8]: len(ldfm_all_df)
Out[8]: 130
shankari commented 5 years ago

Re-delete the logs and re-run the pipeline

$ rm /var/tmp/intake_single.log

$ ./e-mission-py.bash bin/debug/intake_single_user.py -e broken_modes@lfdm.fr
Connecting to database URL localhost
analysis.debug.conf.json not configured, falling back to sample, default configuration
google maps key not configured, falling back to nominatim
nominatim not configured either, place decoding must happen on the client
transit stops query not configured, falling back to default
2019-02-26T18:40:20.608920-08:00**********UUID a4959cef-485c-4414-bc9a-a8c45dfb3294: moving to long term**********
2019-02-26T18:40:20.720354-08:00**********UUID a4959cef-485c-4414-bc9a-a8c45dfb3294: filter accuracy if needed**********
2019-02-26T18:40:20.739947-08:00**********UUID a4959cef-485c-4414-bc9a-a8c45dfb3294: segmenting into trips**********
2019-02-26T18:56:40.792797-08:00**********UUID a4959cef-485c-4414-bc9a-a8c45dfb3294: segmenting into sections**********
/Users/shankari/e-mission/e-mission-server/emission/analysis/intake/segmentation/section_segmentation_methods/flip_flop_detection.py:57: RuntimeWarning: invalid value encountered in double_scalars
  sm.update({"trip_pct": (curr_section_time * 100)/total_trip_time})
2019-02-26T18:56:45.287489-08:00**********UUID a4959cef-485c-4414-bc9a-a8c45dfb3294: smoothing sections**********
/Users/shankari/OSS/anaconda/envs/emission/lib/python3.6/site-packages/pandas/core/frame.py:891: UserWarning: DataFrame columns are not unique, some columns will be omitted.
  "columns will be omitted.", UserWarning)
2019-02-26T18:56:47.304117-08:00**********UUID a4959cef-485c-4414-bc9a-a8c45dfb3294: cleaning and resampling timeline**********
2019-02-26T18:56:52.173119-08:00**********UUID a4959cef-485c-4414-bc9a-a8c45dfb3294: inferring transportation mode**********
2019-02-26T18:56:56.251270-08:00**********UUID a4959cef-485c-4414-bc9a-a8c45dfb3294: checking active mode trips to autocheck habits**********
2019-02-26T18:56:56.255769-08:00**********UUID a4959cef-485c-4414-bc9a-a8c45dfb3294: storing views to cache**********
shankari commented 5 years ago

bus mode inference

Here's the section whose mode was misclassified.

2019-02-26 18:56:52,458:DEBUG:140735666316160:~~~~~~~~~~About to get predicted value for section 5c75fc73f6858f2a1381c254 (2019-01-31T16:06:48+01:00 -> 2019-01-31T18:01:03+01:00)~~~~~~~~~~

Corresponding to the code in https://github.com/shankari/e-mission-server/blob/ground_truth_matching/emission/analysis/classification/inference/mode/rule_engine.py#L169

There were 4 bus stops near the start The related map for each id is at https://www.openstreetmap.org/node/ e.g. the first stop is at https://www.openstreetmap.org/node/3667405161

2019-02-26 18:56:52,458:DEBUG:140735666316160:bbox_string = 43.54969796532126,7.01712066532126,43.55239743467874,7.01982013467874
2019-02-26 18:56:54,798:DEBUG:140735666316160:STOP 0: AttrDict({'type': 'node', 'id': 3667405161, 'lat': 43.5514811, 'lon': 7.0175961, 'tags': {'highway': 'bus_stop', 'name': 'Square Mérimée', 'operator': 'Palm Bus', 'public_transport': 'platform'},...)
2019-02-26 18:56:54,798:DEBUG:140735666316160:STOP 1: AttrDict({'type': 'node', 'id': 5873553955, 'lat': 43.5514365, 'lon': 7.0175812, 'tags': {'bus': 'yes', 'name': 'Square Mérimée', 'network': 'Palm Bus', 'public_transport': 'stop_position'},)
2019-02-26 18:56:54,798:DEBUG:140735666316160:STOP 2: AttrDict({'type': 'node', 'id': 5918551401, 'lat': 43.5509497, 'lon': 7.0194447, 'tags': {'bench': 'no', 'bus': 'yes', 'departures_board': 'yes', 'highway': 'bus_stop', 'lit': 'no', 'name': 'Palais des Congrès', 'passenger_information_display': 'no', 'public_transport': 'platform', 'shelter': 'no'},)
2019-02-26 18:56:54,798:DEBUG:140735666316160:STOP 3: AttrDict({'type': 'node', 'id': 5929127520, 'lat': 43.5509099, 'lon': 7.0194178, 'tags': {'bus': 'yes', 'name': 'Palais des Congrès', 'public_transport': 'stop_position'},)

and one bus stop near the end

2019-02-26 18:56:54,799:DEBUG:140735666316160:bbox_string = 43.47847996532126,5.3718971653212595,43.481179434678744,5.37459663467874
2019-02-26 18:56:56,013:DEBUG:140735666316160:STOP 0: AttrDict({'type': 'node', 'id': 256366216, 'lat': 43.4787813, 'lon': 5.3719407, 'tags': {'bench': 'yes', 'bus': 'yes', 'highway': 'bus_stop', 'lit': 'yes', 'name': 'Les Grottes', 'public_transport': 'platform', 'shelter': 'yes', 'wheelchair': 'yes'}, })

We then find the routes for each of the bus stops and try to match them. There are no matching routes.

2019-02-26 18:56:56,018:DEBUG:140735666316160:About to find matches in lists: [5940895, 5951943, 8827639, 5940895, 5951943, 8827639, 8827639, 8827639] 
 [6947211, 6979960, 7019221, 7106136, 7124028, 7155399, 7157283]
2019-02-26 18:56:56,019:DEBUG:140735666316160:matching routes = []

But areas with irregularly updated OSM data, such as the US, frequently have highway/bus_stop entries with no routes, so we try to match two highway/bus_stops if one of them is sparse

https://github.com/shankari/e-mission-server/blob/f1475af75618abbc992925dc80b9f32e18bd5b9b/emission/net/ext_service/transit_matching/match_stops.py#L130

2019-02-26 18:56:56,020:DEBUG:140735666316160:len(start_bus) = 4, len(end_bus) = 1
2019-02-26 18:56:56,020:DEBUG:140735666316160:Both start and end are bus stops, validating...
2019-02-26 18:56:56,021:DEBUG:140735666316160:One side is sparse, valid bus stop
2019-02-26 18:56:56,021:DEBUG:140735666316160:Got predicted transit mode ['BUS']
2019-02-26 18:56:56,021:DEBUG:140735666316160:train_mapped_modes = ['BUS']

And in this case, the destination was sparse, so we classified it as BUS.

shankari commented 5 years ago

AIR_OR_HSR mode inference

We set the inferred mode to AIR_OR_HSR if the cleaned mode was AIR_OR_HSR https://github.com/shankari/e-mission-server/blob/ground_truth_matching/emission/analysis/classification/inference/mode/rule_engine.py#L68

So let's see why the sensed mode was set to AIR_OR_HSR.

The related section is

2019-02-26 18:56:56,043:DEBUG:140735666316160:Updating sensed mode for section = 5c75fc71f6858f2a1381c1c1 to PredictedModeTypes.AIR_OR_HSR

Searching backwards for that section, we get

2019-02-26 18:56:49,709:DEBUG:140735666316160:air check: end_to_end_distance = 84405.04925970073, end_to_end_time = 2768.9519350528717, so end_to_end_speed = 30.48267042529543
2019-02-26 18:56:49,709:DEBUG:140735666316160:air check: end_to_end_speed 30.48267042529543 > ONE_FIFTY_KMPH 27.77777777777778, returning True 

So basically, the speed of the trip was detected to be more than 150 kmph, so we assumed that it was not a car.

The ONE_FIFTY_KMPH rule is a heuristic.

Wait, the code actually shows 100 kmph instead of 150...

    ONE_FIFTY_KMPH = old_div(float(100 * 1000), (60 * 60)) # m/s

Let's look at the history a bit.

The original support was added in https://github.com/shankari/e-mission-server/commit/93434ab12bda9441becf3cc54f8ed9b0040348d5

and it has always been float(100 * 1000) / (60 * 60). That is so bizarre. If it were float(150 * 1000) / (60 * 60) as the variable name suggests, it would have been 41.666666666666664 instead of 27.77777777777778 and this check would have failed.

Let me try to change that and see if it fixes it.

Before I do that, let's verify the actual trip.

The trip looks fine; I guess the person really did drive at over 100 kmph.

screen shot 2019-02-26 at 10 23 27 pm
shankari commented 5 years ago

Fixes

One of these fixes is related to differences in OSM labels between US and France. The other is just stupid; I guess there aren't a lot of people driving super fast in the reference dataset.

AIR_OR_HSR

I can fix the ONE_FIFTY_KMPH variable, but maybe the air speed rule needs to be different in Europe. The current rule was based on data from my trips to Hawaii and Austin (https://github.com/e-mission/e-mission-docs/issues/205#issuecomment-242983175) and is documented in https://github.com/shankari/e-mission-server/commit/93434ab12bda9441becf3cc54f8ed9b0040348d5

BUS

The bus fix is trickier because the quality of OSM data varies wildly and I don't want to break the mode detection in the Bay Area. One option is to only do the backup bus stops iff they don't have route information. Problem is that will still break if there is incomplete route information. I haven't seen any incomplete route information in the Bay Area though; if there is route information, it is complete. Otherwise, all route information is missing. So at least with Bay Area + France, it should be good enough to only do the fall back if both origin and destination bus stops are simple bus stops with no route information.

shankari commented 5 years ago

@PatGendre I can probably commit the fixes to the ground_truth_matching branch around this weekend. If you then reset the pipeline and re-run it, you should get the correct modes. I might try to make some of the parameters configurable so that you can tweak them based on the behavior that you see in your geographic location.

After this is fixed, do you mind if I add this timeline to the list of unit tests?

shankari commented 5 years ago

I fixed the ONE_FIFTY_KMPH variable because that was such an egregious bug, and now that trip is CAR

=== Trip: {"coordinates": [5.4426193, 43.5137525], "type": "Point"} -> {"coordinates": [5.4422358, 43.5235052], "type": "Point"}
  --- Section: {"coordinates": [5.4426193, 43.5137525], "type": "Point"} -> {"coordinates": [5.4422358, 43.5235052], "type": "Point"}  on  PredictedModeTypes.WALKING
=== Trip: {"coordinates": [5.4422358, 43.5235052], "type": "Point"} -> {"coordinates": [6.4150469, 43.4074562], "type": "Point"}
  --- Section: {"coordinates": [5.4422358, 43.5235052], "type": "Point"} -> {"coordinates": [6.4150469, 43.4074562], "type": "Point"}  on  PredictedModeTypes.CAR
shankari commented 5 years ago

I also attempted to fix the bus stop issue by saying that a bus stop is simple only if it has no routes.

 def validate_simple_bus_stops(p_start_stops, p_end_stops):
-    is_bus_stop = lambda s: "highway" in s.tags and \
-                            s.tags.highway == "bus_stop"
-
-    start_bus_stops = [s for s in p_start_stops if is_bus_stop(s)]
-    end_bus_stops = [s for s in p_end_stops if is_bus_stop(s)]
+    is_no_route_bus_stop = lambda s: "highway" in s.tags and \
+                                     s.tags.highway == "bus_stop" and \
+                                     "route_ref" not in s.tags and \
+                                     "routes" not in s
+
+    start_bus_stops = [s for s in p_start_stops if is_no_route_bus_stop(s)]
+    end_bus_stops = [s for s in p_end_stops if is_no_route_bus_stop(s)]

That ensured that the trip was not categorized as BUS, but it was categorized as UNKNOWN. Looking at the raw data, we basically got no points until the end, and the only reason it even looks like a trip is because we extrapolated to the previous end point. We have no motion activity in this range

2019-02-26 23:40:45,543:INFO:140735666316160:++++++++++++++++++++Processing trip 5c763ef9f6858f3854d8de8a for user b47ba409-7c4b-480f-b9e6-8e9dd01a225a++++++++++++++++++++
2019-02-26 23:40:45,545:DEBUG:140735666316160:Distance from raw_place 5c763ef8f6858f3854d8de89 to the end of raw_trip_entry 5c763ef9f6858f3854d8de8a = 132899.47851349282

Some points right at the end.

2019-02-26 23:40:45,661:DEBUG:140735666316160:Found 39 results
2019-02-26 23:40:45,663:DEBUG:140735666316160:After de-duping, converted 39 points to 39
2019-02-26 23:40:45,666:DEBUG:140735666316160:Resampling entry list                            fmt_time            ts  longitude   latitude
0  2019-01-31T18:00:20.069000+01:00  1.548954e+09   5.373176  43.479825
1  2019-01-31T18:00:33.630000+01:00  1.548954e+09   5.373213  43.479801
2  2019-01-31T18:00:43.095000+01:00  1.548954e+09   5.373239  43.479824
3  2019-01-31T18:00:47.938000+01:00  1.548954e+09   5.373245  43.479830
4  2019-01-31T18:00:48.936000+01:00  1.548954e+09   5.373240  43.479829 of size 39

and basically one medium confidence activity point

2019-02-26 23:40:45,740:DEBUG:140735666316160:curr activity = Motionactivity({'_id': ObjectId('5c763b17f6858f384a31132a'), 'confidence': 86, 'fmt_time': '2019-01-31T18:00:33.528000+01:00', 'type': 2)}), returning True

2019-02-26 23:40:45,741:DEBUG:140735666316160:curr activity = Motionactivity({'_id': ObjectId('5c763b17f6858f384a311343'), 'confidence': 40, 'fmt_time': '2019-01-31T18:00:56.073000+01:00', 'type': 4)}), returning False

2019-02-26 23:40:45,741:DEBUG:140735666316160:curr activity = Motionactivity({'_id': ObjectId('5c763b17f6858f384a311364'), 'confidence': 93, 'fmt_time': '2019-01-31T18:01:11.054000+01:00', 'type': 3, 'user_id': UUID('b47ba409-7c4b-480f-b9e6-8e9dd01a225a')}), returning False

2019-02-26 23:40:45,741:DEBUG:140735666316160:curr activity = Motionactivity({'_id': ObjectId('5c763b17f6858f384a311373'), 'confidence': 55, 'fmt_time': '2019-01-31T18:01:21.090000+01:00', 'type': 3, 'user_id': UUID('b47ba409-7c4b-480f-b9e6-8e9dd01a225a')}), returning False

These are the sensed points and the points after cleaning and reconstruction.

screen shot 2019-02-26 at 11 59 16 pm screen shot 2019-02-27 at 12 06 43 am

@PatGendre do you know why you have no points along that trip? Was is because of the location services toggling?

shankari commented 5 years ago

I looked at the statemachine transitions to see if I could find more clues, and it looks like there was some issue with the sensing.

The raw sensing detected a trip end at around 4pm and then a small set of locations around 6pm. The pipeline e.g. https://github.com/e-mission/e-mission-docs/issues/205#issuecomment-242957231 then detected that the location at 4pm was different from the location at 6pm and joined them together, which is why the trajectory is a straight line, as opposed to the trip on the 29th, which clearly tracks the trajectory of a road.

@PatGendre Is there anything special/weird about this phone? For example, does it have a cellular plan (otherwise, you may be running into https://github.com/e-mission/e-mission-data-collection/issues/166), also reported by the folks from Switzerland when they were on vacation in Italy.

If it does have a cellular plan, can you send me the phone logs so that I can take a look?

Analysis details (as exemplar)

Bunch of setup stuff

In [1]: import emission.core.get_database as edb
   ...: import pandas as pd
   ...:
Connecting to database URL localhost

In [2]: from uuid import UUID

In [3]: lfdm_uuid = UUID("b47ba409-7c4b-480f-b9e6-8e9dd01a225a")

In [4]: import emission.storage.timeseries.abstract_timeseries as esta
   ...: import emission.storage.decorations.analysis_timeseries_queries as esda
   ...: import emission.core.wrapper.entry as ecwe
   ...: import emission.storage.decorations.trip_queries as esdt
   ...: import emission.storage.timeseries.timequery as estt
   ...: import emission.core.wrapper.transition as ecwt
   ...:

In [5]: ts = esta.TimeSeries.get_time_series(lfdm_uuid)

Transitions during the time period. Note the STOPPED_MOVING at 16:06 followed by the EXITED_GEOFENCE at 18:00

In [6]: transitions = ts.get_data_df("statemachine/transition")

In [7]: transitions["transition_name"] = transitions.transition.apply(lambda t: ecwt.Tra
   ...: nsitionType(t))

In [9]: transitions[['fmt_time', "transition_name"]]
Out[9]:
                            fmt_time                 transition_name
0   2019-01-29T06:27:38.806000+01:00  TransitionType.EXITED_GEOFENCE
1   2019-01-29T06:34:10.904000+01:00   TransitionType.STOPPED_MOVING
2   2019-01-29T06:42:03.420000+01:00  TransitionType.EXITED_GEOFENCE
3   2019-01-29T09:26:26.392000+01:00           TransitionType.BOOTED
4   2019-01-29T09:26:44.496000+01:00       TransitionType.INITIALIZE
5   2019-01-29T10:06:23.414000+01:00       TransitionType.INITIALIZE
6   2019-01-29T18:04:46.546000+01:00  TransitionType.EXITED_GEOFENCE
7   2019-01-29T18:27:54.323000+01:00   TransitionType.STOPPED_MOVING
8   2019-01-31T15:59:06.914000+01:00  TransitionType.EXITED_GEOFENCE
9   2019-01-31T16:06:28.153000+01:00   TransitionType.STOPPED_MOVING
10  2019-01-31T18:00:27.229000+01:00  TransitionType.EXITED_GEOFENCE
11  2019-01-31T18:41:23.840000+01:00   TransitionType.STOPPED_MOVING

Get the list of trips, and then the raw locations associated with the fourth trip. They basically span ~ 1 minute.

In [10]: ct_df = ts.get_data_df("analysis/cleaned_trip", time_query=None)

In [11]: ct_df[["start_fmt_time", "end_fmt_time"]]
Out[11]:
                     start_fmt_time               end_fmt_time
0  2019-01-29T06:27:44.085000+01:00  2019-01-29T06:31:31+01:00
1  2019-01-29T06:39:08.048065+01:00  2019-01-29T07:25:17+01:00
2  2019-01-29T18:04:51.440000+01:00  2019-01-29T18:10:56+01:00
3  2019-01-31T15:49:27.739263+01:00  2019-01-31T16:03:48+01:00
4         2019-01-31T16:06:48+01:00  2019-01-31T18:01:03+01:00

In [12]: filtered_locs = ts.get_data_df("background/filtered_location",
    ...:                                time_query = esda.get_time_query_for_trip_like(
    ...:                                    "analysis/cleaned_trip", ct_df.iloc[4]._id))
    ...:

In [13]: len(filtered_locs)
Out[13]: 20

In [15]: filtered_locs.head(3).fmt_time
Out[15]:
0    2019-01-31T18:00:20.069000+01:00
1    2019-01-31T18:00:33.630000+01:00
2    2019-01-31T18:00:43.095000+01:00
Name: fmt_time, dtype: object

In [16]: filtered_locs.tail(3).fmt_time
Out[16]:
17    2019-01-31T18:01:01+01:00
18    2019-01-31T18:01:02+01:00
19    2019-01-31T18:01:03+01:00
Name: fmt_time, dtype: object
PatGendre commented 5 years ago

Hi Shankari, thank you so much for having worked on the 3 issues I opened, and the detailed is a very tutorial for us! Thank you in advance if you can fix the ground_truth_matching branch then, and of course no problem if you add the timeline data in your tests. Concerning the absence of points during a large portion of the return trip, I don't know, the data come from Yann's phone, and I don't think he's toggling location (and mobile data) like I do, I will ask him (I am in Paris currently for 2 days), but I am sure he has (like I do) what you call a data plan (i.e. as I understand it, his mobile subscription includes a 3G/4G mobile data service).

shankari commented 5 years ago

@PatGendre the two fixes are in https://github.com/shankari/e-mission-server/commit/c75528003d20b37b2ea7581bb227d1e9e96d8dc5 and https://github.com/shankari/e-mission-server/commit/784f197da8e018c6b0fb13941c13b6e5fe87cb9a

However, after the BUS fix, the trip is classified as UNKNOWN due to the sensing issues outlined in https://github.com/e-mission/e-mission-docs/issues/322#issuecomment-467763935 and https://github.com/e-mission/e-mission-docs/issues/322#issuecomment-467937371

If you are not sure why the sensing issues occur, or if they persist even with the fixes to location services toggling, please open a new issue for them and email me the phone logs.

shankari commented 5 years ago

closing this for now