Diary does not handle timezones correctly

shankari commented 8 years ago

For trips that were in a different time zone, we want to use the local time in that timezone in the trip diary so that it can be used to provide an accurate diary that matches up with the users recollection.

It looks like we are currently using the timestamp directly without applying the local timezone.

We should use the local_dt.day and local_dt.hour or use the correct timezone with the timestamp.

shankari commented 8 years ago

@krisma please fix this!

shankari commented 8 years ago

I noticed this from the trip to Hawaii

PatGendre commented 5 years ago

Hi, I don't know if our problem matches this issue, but we seem to have a problem wrt timezones, or at least a question : in the Json data, the timezone is set to Los Angeles, how do we configure this to change it to Paris Europe CET timezone? We seem to have "lost" data, so data when the app was active and tracking, where I have seen the data first on the diary, e.g. on Feb 27, and then the data disappear on the diary. I wonder if this could be to the fact the app would be on LA time and the server on Paris time for instance, and that when running the pipeline, when the daytime is <0 or >24 it would be deleted?

Another question, what happens if the trip lasts after midnight, is it processed smoothly?

PatGendre commented 5 years ago

I suppose though that I there were issues with timezones, people using e-mission in Australia or Germany would have noticed and solved it? Maybe it just because we did not configure properly the app / the server?

shankari commented 5 years ago

I think @krisma already fixed this, at least for the list view. So let me address the two questions separately.

in the Json data, the timezone is set to Los Angeles, how do we configure this to change it to Paris Europe CET timezone?

Is the timezone set to Los Angeles? It wasn't in the timeline data that you posted as part of https://github.com/e-mission/e-mission-docs/issues/322

The very first entry is

  {
    "_id": {
      "$oid": "5c50179cca83182f9b8482e8"
    },
    "metadata": {
      "key": "stats/client_nav_event",
      "platform": "android",
      "read_ts": 0,
      "time_zone": "Europe/Paris",
      "type": "message",
      "write_ts": 1548728744.634,
      "write_local_dt": {
        "year": 2019,
        "month": 1,
        "day": 29,
        "hour": 3,
        "minute": 25,
        "second": 44,
        "weekday": 1,
        "timezone": "Europe/Paris"
      },
      "write_fmt_time": "2019-01-29T03:25:44.634000+01:00"
    },
    "user_id": {
      "$uuid": "d0cbf849f9b14ba183be1a148ec56a1e"
    },
    "data": {
      "client_app_version": "2.7.1",
      "client_os_version": "8.1.0",
      "name": "sync_launched",
      "reading": -1,
      "ts": 1548728744.619
    }
  },

and as you can see, the write_fmt_time is "2019-01-29T03:25:44.634000+01:00"

We do set the timezone to America/Los_angeles if it doesn't come from the phone. https://github.com/e-mission/e-mission-server/blob/3a5e2c921f41ea1bbeaec0d49f4fc722d418794d/emission/net/usercache/formatters/android/incident.py#L21 which is a decision that made sense at the time, but maybe should be revisited now.

But we haven't really had encountered the use case in which people don't have the timezone set on their phone because then the time that shows on their phone would be wrong.

The other place where we set the timezone to America/Los_angeles is when creating server side entries such as the server stats for the timeline stages and API calls. Maybe we should make that also be configurable, but those should not affect diary entries. https://github.com/e-mission/e-mission-server/blob/master/emission/core/wrapper/metadata.py#L42

shankari commented 5 years ago

We seem to have "lost" data, so data when the app was active and tracking, where I have seen the data first on the diary, e.g. on Feb 27, and then the data disappear on the diary.

This should not have happened. When you first saw the trips, were they in "draft" mode? If so, there might be an issue with sending them to the server or with the processing in the pipeline.

Please try to follow the troubleshooting steps https://github.com/e-mission/e-mission-docs/blob/master/docs/e-mission-server/troubleshooting_tips_FAQ.md and file an issue if you find an issue that is not handled there.

PatGendre commented 5 years ago

Thanks a lot, I will check this page and tell you.

PatGendre commented 5 years ago

Hi, actually we understood yesterday why we lost data, simply because I reinstalled the app for tests and the data was not transmitted yet to the server. I also checked : the LA timezone in the timelines appears only for one datatype (stats/server_api_time), the other data types all have a Europe/Paris time zone. Thanks again, you can close the issue if the problem described for the Hawai trip is solved.

shankari commented 5 years ago

I am actually not sure that the problem described for the Hawaii trip is resolved. Let me leave it open for now. I am also cloning an issue for making the default timezone (e.g. for stats/server_api_time) be configurable.

shankari commented 5 years ago

New issues is https://github.com/e-mission/e-mission-docs/issues/329

PatGendre commented 5 years ago

Ok, thanks

shankari commented 5 years ago

Fixed in https://github.com/e-mission/e-mission-phone/pull/553 Please see https://github.com/e-mission/e-mission-phone/pull/553#issuecomment-489770767 for testing results

e-mission / e-mission-docs

Diary does not handle timezones correctly #145