feature request: add or clarify GPS fix timestamp and timezone

[jata1]

@gcobb321 - big thanks for this work/integration. I have it working well overall but I am having some challenges getting reasonably accurate and consistent speed (for driving) using GPS data - I know this is challenging.

I have been chatting away with the guy who has created an integration that combines device trackers into one composite and uses this data to estimate speed. However, it all really depends on having the GPS fix time and timezone for all sensors pulling gps data - see link below.

https://github.com/pnbruckner/ha-composite-tracker/issues/66

So what we need to move forward is:

1. clarify which entity / attribute in icloud3 is the GPS fix time and timezone
2. see if there is a way of getting the GPS fix time and timezone from the HA companion app for iOS

I'm hoping you can help me take this forward.

[gcobb321]

The iCloud3 device_tracker entity contains the info you want. You can click on the device tracker entity on the dashboard screen, then select attributes or go to Developer Tools > States and select it. Mine are:

source_type: gps
battery_level: 83
latitude: ##.726838473559194
longitude: -##.39054471527213
gps_accuracy: 5
gps: (##.726838473559194, -##.39054471527213)
------------------------: -------------------------
integration: iCloud3
name: Gary/gary_iphone
entity_picture: /local/images/gary.png
picture_file: /local/images/gary.png
track_from_zones: warehouse, home
primary_home_zone: Home
away_time_zone_offset: HomeZone
-------------------------: -------------------------
data_source: FamShr
device_status: Online/200
trigger: FamShr@, 10:37:
zone: home
last_zone: quail
from_zone: home
home_distance: 0
zone_distance: 0
max_distance: 0
calc_distance: 0.01860382
waze_distance: 0
distance_to: ⌘Lillian/iPhone-4.4ft (10:37a), ✓Lillian-Watch-4.4ft (10:37a), ✓Gary-iPad-3.7ft (10:31a), ×Gary-AirPods-0.0mi (7:00p)
zone_changed: Thu, Mar 14, 9:07:01a
last_located: Thu, Mar 14, 10:37:19a
last_update: Thu, Mar 14, 10:37:21a
next_update: Thu, Mar 14, 11:07:20a
--------------------------: -------------------------
icloud3_devices: gary_iphone, lillian_iphone, lillian_watch, gary_ipad, gary_airpods
icloud3_version: v3.0.1b3
event_log_version: v3.0.17
icloud3_directory: /config/custom_components/icloud3
icon: mdi:cellphone
friendly_name: Gary

It has the last located time as a date based value but I can easily add it as a full timestamp/secs without any problems if that will help.

You do not want the HA device tracker info since iCloud3 already incorporates this. It monitors the MobileApp location on a 5-sec interval, reacts to MobileApp triggers and monitors the iCloud acccount location data. It will select the latest location timestamp and report that data.

The update interval is normally determined by the distance from home to minimize the number of locate requests to the iCloud servers. This can be tailored on the Tracking Parameters config screen. You can also set a Fixed Interval on the Update Devices screen for each device being tracked.

You might want to review the Mobile App docs for the triggers it generates. Pay close attentio to the Significant Location Change and Periodic triggers.

Hope that helps,

[jata1]

It really helps. Thanks.

I am getting some very strange speed calculations using the iCloud data so I will pass this info onto the composite device tracker dev who is also helping me.

How do you handle the fix/update timestamp when you use the data from the HA app as the integration does not have this attribute?

and just checking that for either iCloud or HA app data, the last located attribute holds the fix time?

thanks again and great work!

[gcobb321]

I am finishing up v3.0.1 when I saw your note yesterday and have added some timestamp attributes to the device_tracker entity. They are:

source_type: gps 
battery_level: 52 
latitude: 27.726837158203125 
longitude: -80.3905442990145 
gps_accuracy: 5 
gps: (27.726837158203125, -80.3905442990145) 
located_local: 2024-03-14 15:47:57
located_utc: 2024-03-14 19:47:57

I have added and removed the actual timestamp seconds (local & utc) which can be derived from the located_local & located_utc.

I'll send you v3.0.1b4 later today or tomorrow for you to work with. I'll also contact the other developer to see if there is anything specific that would make everything work together easier.

The Mobile App location time comes from a last_update_time attribute from the device_tracker and the sensor.last_update_trigger entities. The code for getting the mobapp times is at the top of the /support/mobapp_data_handler.py and /helpers/entity_io.py modules.

[pnbruckner]

Any way you want to present the timestamp that corresponds to when the location values (latitude, etc.) were read/obtained on the device is fine. I just need to know how to interpret it, and that it won't be changing.

I started implementing using the last_located attribute that already existed. Based on my reading of your code, it seemed that was the correct attribute to use, and that the time was always "local" in HA's configured time zone.

If you want to change it to something else, that is fine, too. For me, the easiest would be to use a time zone aware format instead of a naive one. E.g., one of these:

>>> from datetime import UTC, datetime
>>> utc_time_str = "2024-03-14 19:47:57"
>>> print(datetime.fromisoformat(utc_time_str).replace(tzinfo=UTC))
2024-03-14 19:47:57+00:00
>>> print(datetime.fromisoformat(utc_time_str).replace(tzinfo=UTC).astimezone())
2024-03-14 14:47:57-05:00

Note that I'm in the US Central time zone, which is why the last one comes out that way. If the code ran in HA, it would be in HA's configured time zone (well, at least, the underlying OS's time zone, which should be the same.) Or, you could use something like:

>>> from homeassistant.util import dt as dt_util
>>> # Simulate HA's time zone configuration.
>>> dt_util.set_default_time_zone(dt_util.get_time_zone("US/Eastern"))
>>> print(dt_util.as_local(datetime.fromisoformat(utc_time_str).replace(tzinfo=UTC)))
2024-03-14 15:47:57-04:00

[jata1]

So good you two guys are discussing this together. I am an idiot in the middle. haha

Thanks so much to you both!

[jata1]

Here is some data from this morning. I went for a short run - 5km. You can see the icloud GPS data is doing something weird.

Here is the same chart without icloud so we can compare the other data sources. By the way - this data below looks correct.

[pnbruckner]

Let me get a beta out for the composite integration that uses the existing last_located attribute. I can always change it if the iCloud3 attribute changes.

[pnbruckner]

Ok, 3.4.0b0 ready for a try.

[jata1]

Thanks. Will try that now and report back.

[jata1]

Just to update this thread that we are making progress - latest chart below shows no more crazy speed spikes and overall data not too far away from what traccar app is reporting (based on a 2min update frequency)

[gcobb321]

I've changed the format of the located attribute to utc time.

source_type: gps
battery_level: 0
latitude: 27.72683080476227
longitude: -80.39054137033257
gps_accuracy: 35
gps: (27.72683080476227, -80.39054137033257)
located: (2024-03-15 05:37:58-04:00)

I've had to wrap the located attribute in parentheses (2024-03-15 05:37:58-04:00) so HA does not reformat it on the device_tracker > Attributes screen to March 15, 2024 at 7:11:17 AM where it reports the time and drops the time zone info.

Here is the latest v3.0.1b4

Unzip into the icloud3 directory and restart HA. icloud3_v3.0.1b4.zip

Change Log : v3.0.1 - 3/15/2024

1. UPDATE SENSOR & DEVICE TRACKER ENTITIES (Improvement) - Changed the method of updating these items at the request of @balloob (Paulus Schousten, HA Founder) to prevent a potential Home Assistant lockup when doing an update.
2. LOG FILE (Improvement) - Changed the method of updating the iCloud3 log files. Also changed some of the record formatting to improve readability.
3. WAZE & CALC DISTANCE SENSORS (Fixed) - They were not being created.
4. UPDATE DEVICES CONFIGURE SCREEN (Fixed) - The Picture parameter was listing every image file in the all of the www/ sub-directories. A filter was added to limit the search to specific directories. The filter is defined on the Tracking and Other Parameters screen. Only the first 25 image files in a directory will be displayed to prevent overloading the HA dropdown list handler.
5. FIXED INTERVAL (Improvement) - The lowest interval was reduced from 5-minutes to 3-minutes.
6. DIRECTION OF TRAVEL (Improvement) - Changed the way this is being updated. If the distance from Home is more than 400m/250mi, it will be set to FarAway.
7. STATIONARY ZONE (Fixed) - The second device to enter an existing Stationary Zone was not saving the Stationary Zone info correctly.

[pnbruckner]

I've changed the format of the located attribute to utc time.

located: (2024-03-15 05:37:58-04:00)

That is now a time zone aware time (as opposed to a naive time, which it was before), but it's not UTC. UTC would be "+00:00".

I've had to wrap the located attribute in parentheses (2024-03-15 05:37:58-04:00) so HA does not reformat it on the device_tracker > Attributes screen to March 15, 2024 at 7:11:17 AM where it reports the time and drops the time zone info.

That is not necessary. The attribute will still have the time zone suffix. It shouldn't matter if the frontend chooses to display the time as local (no matter what time zone offset the attribute has.) That is normal.

[gcobb321]

Understand, but was trying to avoid confusion by having the front end display the located time with the time zone instead of the utc time as a normal time which would confuse a lot of people.

I have changed the local time to located and it is displayed under the gps location. The utc time you want is now named located (utc) and at the bottom.

gps: (27.726837158203125, -80.3905447785501)
located: Fri, Mar 15, 12:08:45p
...
...
located (utc): 2024-03-15 08:08:45-04:00

A new iCloud3 v3.0.1b4 with this change. Let me know if other adjustments will help. icloud3_v3.0.1b4.zip

[pnbruckner]

I think you're confusing the terms "UTC" and "time zone aware". "located (utc): 2024-03-15 08:08:45-04:00" is time zone aware (meaning it has a UTC offset suffix), but it is not UTC. The equivalent UTC would be "located (utc): 2024-03-15 12:08:45+00:00". (Kind of by definition, a UTC time has a UTC offset of zero.)

The utc time you want is now named located (utc) and at the bottom.

As I stated previously, I don't care how the time is formatted, I just need to know how to interpret it, and that it doesn't change once decided on. A naive time (i.e., with no UTC offset suffix) is fine, as long as it is always in HA's local time. A time zone aware format (w/ a UTC offset suffix) is better, since it's not open to interpretation, but isn't absolutely necessary. The time zone aware format can be in any time zone, that doesn't matter, which is the whole point to a time zone aware format. 😄 Having said that, the HA core team generally prefers UTC.

So, at this point, the "located (utc)" attribute is, at least, misleading. I'd recommend changing its value to UTC, or changing the name of the attribute. Also, attribute names in HA are typically snake case, so located_utc is better than located (utc).

Putting that all together, instead of:

located (utc): 2024-03-15 08:08:45-04:00

it should probably be:

located_utc: 2024-03-15 12:08:45+00:00

[gcobb321]

Here you go.

located_utc: 2024-03-15 14:14:01+00:00

icloud3_v3.0.1b4.zip

[pnbruckner]

I'll update my integration accordingly, then @jata1 can test and let us know.

[pnbruckner]

Wait a second. 2024-03-15 14:14:01+00:00 is equivalent to 2024-03-15 10:14:01-04:00. But you posted that a little after 2:00 PM EDT, which would be 2024-03-15 14:14:01-04:00. Are you just changing the suffix, or actually converting to UTC? It still doesn't seem right.

[pnbruckner]

If you are starting with a "POSIX timestamp" (i.e., seconds since the "Epoch", which is 1/1/1970, and which is what time.time() returns), then this is very simple.

If you want a time zone aware time in UTC:

>>> import time
>>> from datetime import datetime, UTC
>>> current_time = round(time.time())
>>> print(datetime.fromtimestamp(current_time, UTC))
2024-03-15 20:02:57+00:00

If, however, you want a time zone aware time in the local time zone:

>>> import time
>>> from datetime import datetime, UTC
>>> current_time = round(time.time())
>>> print(datetime.fromtimestamp(current_time).astimezone())
2024-03-15 15:02:57-05:00

I've also shown above how to use an ISO formatted time string to get the same results.

If you could point me to your code, I could give specific suggestions. I've given generic solutions so far for how you might do this.

And, BTW, if you don't want to use a Python datetime object as the attribute value (which is another possibility), you can easily convert to a str, e.g.:

>>> attribute_value = str(datetime.fromtimestamp(current_time, UTC))
>>> print(attribute_value)
2024-03-15 20:02:57+00:00

[jata1]

Thanks to you both I have my driver speeding tracking system working so much appreciated.

@gcobb321 - I will try out the 3.0.1b4 and report back. The only other thing that would help me is if the integration requests gps data a few times (over a few minutes) when first entering a zone (especially home zone) - to confirm device not moving / speed zero - before switching to the defined inzone interval. Do you think this is possible?

@pnbruckner - will me switching to icloud 3.0.1b4 break the version of your composite tracker integration?

[gcobb321]

@jata1 It may be possible. When you enter the zone, set up an automation that calls the icloud3.update service call with a locate xxx seconds/minutes action. You may be able to set it to run several times. See docs Service Calls chapter for more info.

@pnbruckner - I'm back home now so ping me if you have any questions about my timer handling. More info -- when iCloud3 starts, it determines the local-utc time zone offset secs and saves the tz offset string. That is in the _timeutil.py module in the _def calculate_time_zoneoffset(): function (line 542). The offset is used to convert timestamps from the iCloud account device locate data and the MobileApp data into local time for all reporting and display purposes.

The time displayed in an earlier note as located_utc: 2024-03-15 14:14:01+00:00 was for a 2:14:01p locate (EDT) that was based on local epoch secs for that time.

Correct me if I am wrong but I thought that 14:14:01+00:00 = 10:14:01-04:00. I have had it both ways and changed based on this comment:

located (utc): 2024-03-15 08:08:45-04:00
it should probably be:
located_utc: 2024-03-15 12:08:45+00:00

I do not understand why the HA Frontend displays March 15, 2024 at 1:21:34 PM (from the dev_trkr > Attributes) when the Dev Tools > States attribute value is 2024-03-15 17:21:34+00:00 . The iPhone was actually located at 5:21:34p.

Also, the attribute is already being posted as a string. secs is the epoch time in the local tz

def secs_to_ymd_hms(secs, tz_info=False):
    time_struct = time.localtime(secs)
    ymd_hms     = time.strftime('%Y-%m-%d %H:%M:%S', time_struct)
    if tz_info:
        return f"{ymd_hms}+00:00"

    return f"{ymd_hms}"

[jata1]

When you enter the zone, set up an automation that calls the icloud3.update service call with a locate xxx seconds/minutes action.

good idea - i will try that. Thanks

[pnbruckner]

when iCloud3 starts, it determines the local-utc time zone offset secs and saves the tz offset string. That is in the _timeutil.py module in the _def calculate_time_zoneoffset(): function (line 542). The offset is used to convert timestamps from the iCloud account device locate data and the MobileApp data into local time for all reporting and display purposes.

Yes, I believe I saw that. But you're doing work the standard Python libraries and/or homeassistant.util.dt will do for you.

How are the raw time values formatted? Are they the number of seconds since the epoch (i.e., a Unix/POSIX timestamp)? Or are they in some other format, say a string that represents time?

If they are the former, then it's very easy (as I've explained) to convert that to any time zone aware time (UTC, the local time zone, or any other time zone.) If it's the latter, and it includes a UTC offset suffix, then again, it's very easy to convert to any time zone. If it's the latter, but does not contain any offset suffix, then you need to know what time zone it corresponds to, and then it's still very easy to convert to a time zone aware time in any time zone.

The time displayed in an earlier note as located_utc: 2024-03-15 14:14:01+00:00 was for a 2:14:01p locate (EDT) that was based on local epoch secs for that time.

If it was for EDT, then the suffix is wrong. "+00:00" means UTC. "-04:00" would be correct for EDT. Also, there is no such thing as "local epoch secs". A Unix/POSIX timestamp is defined as the number of seconds since the epoch in UTC.

Correct me if I am wrong but I thought that 14:14:01+00:00 = 10:14:01-04:00.

Yes, that is correct. Just like 2024-03-15 12:08:45+00:00 == 2024-03-15 08:08:45-04:00. My comment was concerning the name of the attribute. If the time is "xxx-04:00", then it is not UTC, so the name of the attribute should not imply it is.

I do not understand why the HA Frontend displays March 15, 2024 at 1:21:34 PM (from the dev_trkr > Attributes) when the Dev Tools > States attribute value is 2024-03-15 17:21:34+00:00 . The iPhone was actually located at 5:21:34p.

Because 17:21:34+00:00 is the same as 13:21:34-04:00, assuming your local time zone is EDT. That is also equivalent to 1:21:34 PM. The frontend always displays times in local time, no matter what time zone they are reported in. The same goes for naive times (ones that don't have a UTC offset suffix), because naive times are assumed to be in local time.

You say the iPhone was actually located at 5:21:34p. That is a naive time, so in what time zone? If local, then the attribute should not have been 2024-03-15 17:21:34+00:00; it should have been either 2024-03-15 17:21:34-04:00 or 2024-03-15 21:21:34+00:00.

secs is the epoch time in the local tz

Again, there is no such thing. From the Python docs:

The epoch is the point where the time starts, the return value of time.gmtime(0). It is January 1, 1970, 00:00:00 (UTC) on all platforms.

[pnbruckner]

will me switching to icloud 3.0.1b4 break the version of your composite tracker integration?

Yes, it will. You'll have to wait until iCloud3 has an attribute with the correct data in an appropriate format. So far we haven't quite gotten there.

[gcobb321]

Everything in iCloud3 is in the local time of the HA server which is where the Home zone is located. Distances and times are based on that zone since that is where most automations the user wants to manage are. It also supports other zones at other locations and can track devices using the distance and travel times to the other location. There is a feature in iCloud3 v3 that will let someone display a time in the local time it’s they are in another time zone (not released for general use yet) but all device tracking is still based on the time zone of the HA server, not the devices location.

The epoch secs of the HA server Location is used for managing this. It does not matter that it is not a”true” epoch secs in a Python sense. I have never used that term before communicating with someone because I never had to. The way I have been handling the time has met my needs for a long time and not presented any problems. If I was starting over, I would probably do it differently but really have no need or desire to redesign it at this time since it works.

Your comments are interesting in an educational sense and you have a better grasp of the terminology between utc, time zones, how they should be displayed, what the attributes field should be called than I do. I’ve never really needed to use local or utc times in anything so have not dig into it.

I can make the field name anything you would like that I am not already using. I can make the field value in any format you want. It can be the:

• local hh:mm:ss+##:## (offset to utc) … 2:41:34p edt = 14:41:34-04:00 (I would think this one)
• utc hh:mm:ss+##:## (no offset to local) … 2:41:34p edt = 16:41:34+00:00
• Local hh:mm:ss+##:## (no offset to local) … 2:41:34p edt = 14:41:34+00:00
• the “local” secs
• the utc secs
• And if the device is in another time zone and displaying that time on the display screens, it can be in that time.
• something else

My understanding of your app is to determine the devices speed between 2 locations and you need the gps of both locations and the time it was at each one so you can calculate the time it took to get from A to B to calculate the v=d/t value.

It will take me only a few minutes to make the code changes to s set the field format to what you would like. Let me know its name and format.

Gary

[pnbruckner]

• Local hh:mm:ss+##:## (no offset to local) … 2:41:34p edt = 14:41:34+00:00
• the “local” secs

The first is wrong, and the second is misleading and unconventional, so let's avoid those.

FWIW, the composite integration already looks for an attribute with either of these names, in this order: last_seen, last_timestamp

And it accepts any of the following as values:

• Naive Python datetime object in HA's local time zone.
• Time zone aware Python datetime object in any time zone.
• A POSIX timestamp as a float, int or str representation of either.

Please note it currently does not support a str representation of time for the first two options, but that is easy enough to add.

Also, a POSIX timestamp is what I believe you refer to as "utc secs."

So, if you could create an attribute with one of the names composite already looks for, and whose value is any of those it already supports, then it would just work with the current release of the composite integration. But, it's no problem to add support for something different.

[gcobb321]

Would you verify this info. The device was located at : Sat, Mar 16, 12:55:03p EDT

last_seen: 2024-03-16 12:55:03
last_timestamp: 1710608103

Thanks

[pnbruckner]

According to https://www.unixtimestamp.com/, 1710608103 is "Sat Mar 16 2024 16:55:03 GMT+0000", which agrees with "Sat, Mar 16, 12:55:03p EDT", which also agrees with "2024-03-16 12:55:03" (assuming EDT, since it's naive.)

Between the two, last_timestamp is unambiguous, whereas last_seen requires the assumption of HA's configured time zone, so I would recommend just including last_timestamp and not including last_seen. There's no need for both, at least not for the composite integration. If you want to include "2024-03-16 12:55:03", then please name the attribute anything else (otherwise composite will use it and not last_timestamp.) Of course, if the assumption always holds, then both are fine.

Thanks!

[gcobb321]

Done...

last_timestamp: 1710612834

Here you go with the _lasttimestamp attribute in the device_tracker entity. icloud3_v3.0.1b4.zip

[jata1]

Thanks @gcobb321 - I will give this a try with the composite device_tracker and report back.

Can you explain/confirm what is used for last_timestamp when GPS coordinates are from HA app rather than icloud?

[gcobb321]

When the mobile app is used, the gps and time come from the mobile app device tracker and sensor entities. When the iCloud data is used, the gps and location time come from iCloud.

[jata1]

Great thanks Gary. I was asking as there does not seem to be this info in the HA mobile app in the device tracker attributes.

source_type: gps
battery_level: 100
latitude: -33.xxxx
longitude: 151.xxxx
gps_accuracy: 10.388014941340675
altitude: 37.83988952636719
vertical_accuracy: 19.732730865478516
friendly_name: Jago iPhone HA

[gcobb321]

@pnbruckner As I was looking at the timestamp data and providing into info you needed, I realized that all times were being kept in posix/unix timestamp time and converted to local time when they were to be displayed. Thanks for getting me back into code I wrote 5-years ago. I think I’ll change some variable names to prevent confusion 5-years from now.

I’ll mention that iCloud3 supports your app next time I get into the documentation

[gcobb321]

The time comes from the last_changed our last_updated attribute which is not displayed I in the Developer tools >states screen it also comes from the sensor.last_update_trigger entity. Whichever is the latest.

[jata1]

Hey Gary @gcobb321 - I wanted to thank you for your assistance with this. I think I have a workable solution now.

FYI - linking to my recent post on the composite device tracker thread https://github.com/pnbruckner/ha-composite-tracker/issues/66#issuecomment-2002320000