Crazy speed data - like 2,200km/h

[jata1]

Hey @pnbruckner - First of all thanks for this integration.

I have just installed this and I am using it to combine 2 GPS device trackers that both work well (HA companion app and icloud3 integration).

The main reason for using this is to have a speed sensor for my iphone. Generally it works ok but I am getting the occasional crazy speed - like over 2000 km/h :-)

Can you help me to trouble shoot this or is it something I will have to live with (or find a way to deal with the bogus data)?

I need accurate speed data as I am trying to create an open source and data privacy based system for monitoring my teenage son's driving. Basically life 360 but home grown.

[pnbruckner]

It does have some minimal filtering. E.g., if two data points are too close together in time (3 seconds for the same input, and 9 seconds for different inputs), it will not update the speed.

The problem may be the types of trackers you are using. Many do not have an attribute that specifies when the location data actually changed (e.g., last_seen.). In this case, the composite has to use when the tracker updated as a backup. Those two can be very different.

Can you share the attributes of the two types of trackers? Obviously, obfuscate any sensitive data.

[jata1]

Appreciate the help. I am getting so close to my overall project goal... Accurate speed is my last hurdle.

icloud3 tracker:

source_type: gps
battery_level: 45
latitude: -33.759368896484375
longitude: 151.29751418740335
gps_accuracy: 11
gps: (-33.759368896484375, 151.29751418740335)
"------------------------": "-------------------------"
integration: iCloud3
name: Jago iPhone/jago_iphone
entity_picture: /local/None
picture_file: /local/None
track_from_zones: home
primary_home_zone: Home
away_time_zone_offset: HomeZone
"-------------------------": "-------------------------"
data_source: MobApp
device_status: Online/200
trigger: launch@10:26:23p
zone: home
last_zone: home
from_zone: home
home_distance: 0
zone_distance: 0
max_distance: 3.753
calc_distance: 0.002433
waze_distance: 0
distance_to: ———
zone_changed: "2024-03-12 21:02:20"
last_located: "2024-03-12 22:26:23"
last_update: "2024-03-12 22:26:25"
next_update: "2024-03-12 22:36:25"
"--------------------------": "-------------------------"
icloud3_devices: jago_iphone
icloud3_version: v3.0.rc10.4
event_log_version: v3.0.17
icloud3_directory: /config/custom_components/icloud3
icon: mdi:cellphone
friendly_name: Jago iPhone

and the HA app device tracker

source_type: gps
battery_level: 45
latitude: -33.759368896484375
longitude: 151.29751418740335
gps_accuracy: 11.158759062906178
altitude: 38.62611389160156
vertical_accuracy: 24.223766326904297
friendly_name: Jago iPhone HA

[pnbruckner]

Are those the state attributes of the device_tracker entities? The formatting is a bit unexpected, at least for the icloud3 source.

The HA app tracker does not seem to have any sort of location timestamp. That is a shame, but not unexpected. I've tried to convince the powers-that-be how important such an attribute is, but my arguments have fallen on deaf ears.

For the icloud3 input, the opposite seems to be the case. There are several timestamps. Any idea which one, if any, indicates when the GPS fix was taken? If I had to guess, I'd say last_located. I wonder how much that differs from the device_tracker entity's last_updated state field (which is not to be confused with the tracker entity's last_update attribute.)

Maybe you could run the following Jinja template for each of the trackers and post the results here:

{% set state = states.device_tracker.XYZ %}
{{ '{:20}: '.format('last_updated') ~ state.last_updated }}
{% for attr, val in state.attributes.items() -%}
{{ '{:20}: '.format(attr) ~ val }}
{% endfor %}

[jata1]

Thanks for this. I will report back later today.

Maybe I could use another app/service with the nevessary gps attributes?

the other thing is that icloud3 uses the gps data from the HA app somehow. So this might explain the issue

[jata1]

I have run the Jinja code and it looks like both do have a last_updated timestamp.

Not sure why this attribute was not showing in Developer tools for the HA app entity. Must be a hidden attribute or something.

Here is the output for the HA tracker using the jinja above,

last_updated        : 2024-03-12 19:23:47.036138+00:00
source_type         : gps
battery_level       : 100
latitude            : -33.759388704333986
longitude           : 151.29751418740335
gps_accuracy        : 11.158759062906178
altitude            : 43.66053009033203
vertical_accuracy   : 27.649686813354492
friendly_name       : Jago iPhone HA

and for icloud3

last_updated        : 2024-03-12 20:21:15.875056+00:00
source_type         : gps
battery_level       : 100
latitude            : -33.759388704333986
longitude           : 151.29751418740335
gps_accuracy        : 11
gps                 : (-33.759388704333986, 151.29751418740335)
------------------------: -------------------------
integration         : iCloud3
name                : Jago iPhone/jago_iphone
entity_picture      : /local/None
picture_file        : /local/None
track_from_zones    : home
primary_home_zone   : Home
away_time_zone_offset: HomeZone
-------------------------: -------------------------
data_source         : FamShr
device_status       : Online/200
trigger             : FamShr@07:21:14
zone                : home
last_zone           : home
from_zone           : home
home_distance       : 0.0
zone_distance       : 0.0
max_distance        : 0.0
calc_distance       : 0.003511
waze_distance       : 0.0
distance_to         : ———
zone_changed        : 2024-03-13 02:00:26
last_located        : 2024-03-13 07:21:14
last_update         : 2024-03-13 07:21:15
next_update         : 2024-03-13 07:31:15
--------------------------: -------------------------
icloud3_devices     : jago_iphone
icloud3_version     : v3.0.rc10.4
event_log_version   : v3.0.17
icloud3_directory   : /config/custom_components/icloud3
icon                : mdi:cellphone
friendly_name       : Jago iPhone

[jata1]

I also have some new data points for us to look at to investigate. Here are the updates that were pushed from HA to my traccar server. Lots of data points look more or less correct but I have highlighted the rows that are rubbish.

I can compare to the native traccar client that I am also using for testing but is not a good long term solution as it is a battery hog.

Files uploaded are ready to assist!

report_trip_composite.xlsx report_trip_traccar_client.xlsx

[jata1]

I have found one error in my work. Traccar is expecting data to be passed in kts and I am using kmh. I think this will partially fix the issue but It is still not right where I have flagged the data in the xls.

Attaching the traccar server log - this is where / how I found the kn v. kmh issue

tracker-server_copy.log

[pnbruckner]

I have run the Jinja code ...

Thanks for doing that! I guess I was surprised by those icloud3 attributes, and thought you were originally showing some sort of report or other info and not simply the entity's attributes. Very unexpectedly, that entity has three very unusual and strange attributes whose names and values are a bunch of "-" characters. Weird.

...and it looks like both do have a last_updated timestamp. Not sure why this attribute was not showing in Developer tools for the HA app entity. Must be a hidden attribute or something.

The last_updated timestamp you see is not an attribute, it is a field of the entity state object. You can see it here:

It is not directly set by the integration, but rather by the system when the integration updates the entity. It records when the entity is updated. It is not a direct indication of when the GPS values were obtained at the source (i.e., in the cell phone.)

Those two times (when the GPS values are obtained in the phone, and when the HA entity is updated with those values) can be very different, depending on how those values are transferred to the HA entity. There could be seconds between them. I've even seen with some tracker entities where GPS values are received out of order due to network delays, etc. This is why it is very important (when trying to combine multiple GPS sources and/or trying to calculate speed) to have the data timestamped with when the data was actually obtained at the source.

When this timestamp information is not available, the composite integration does the only thing it can and will use that last_updated field instead, which almost certainly is wrong. Not only that, but it's also using HA's time source, which could be different from the phone's time source (which becomes much more important when combining multiple sources, where some have timestamps from the phone's time source, and others are using HA's time source, although, these days, most of these devices keep their time synced pretty well.)

So, again, the HA tracker has no timestamp for the GPS data. And I have no idea how much delay there may be between when the GPS values are obtained at the source and when the corresponding HA entity is updated, if it's consistent or if there's a lot of jitter, if data can be received out of order, etc. Here the only thing the composite integration can do is to use the last_updated field, with all the caveats mentioned above.

And looking at the icloud3 entity, there are multiple time values, where last_located is probably the correct one to use. But, unfortunately, it's a naive time (i.e., no time zone info.) Comparing it to last_updated (which is time zone aware in UTC), it looks like its offset from UTC is +11:00. That seems to jive with the GPS coordinates. The problem here, though, is, how can the composite integration know what the correct time zone is? It could assume it's "local time" where HA is, but what if the phone is actually in a different time zone? What time zone will the last_located field correspond to? HA? The phone? Yes, composite could use the GPS coordinates to look up the time zone, but I just took those features out of the composite integration.

If you could check with the icloud3 author as to which attribute, if any, records the actual time when the GPS fix was taken, and what time zone it corresponds to, then I could enhance composite to use that data.

[jata1]

Thanks for the detailed and very helpful explanation. I will follow up with the icloud3 author and also make a request to HA core team to add the fix time info to the iOS companion app.

Just thinking out loud but I wonder if the HA companion app for android has this attribute

[pnbruckner]

also make a request to HA core team to add the fix time info to the iOS companion app.

You can ask, but they'll likely not do it. Even if they add the data, they won't add it as an attribute. I tried to modify the gpslogger integration to do this and it was rejected. They wanted the data in a separate sensor, which is basically useless.

[jata1]

Well I will try and ask them.

What is the issue with a separate sensor? Is it related to having gps coordinates and fix time synchronous?

I can’t believe how hard it is to use gps to estimate speed. I thought there would be plenty of integrations. How is life360 doing it?

[jata1]

I think I have a possible workaround for my use case. I just need to identify rubbish speed data then not push this to my Traccar database/ststem.

Just need to work out how to identify bogus data on the fly. I think the issue is related to when the composite integration gets twitchy gps data very close together

[jata1]

This is data that i am pushing with an automation and REST API from HA to Traccar. While I am sleeping with phone on charge - so not moving. All I need to do is just identify and optionally not push these updates to traccar. Any ideas?

[pnbruckner]

Yes, in your case, it seems you need to work around the lack of data which is generating incorrect results. The better solution would be to obtain the data you need. For the HA app, I think that is a non-starter. For icloud3, I suspect it's possible. I started looking at the custom integration code, but it's pretty big and will take a bit to get through it enough to try and figure this out myself.

The alternative is to use better data sources. I myself use Google Maps (my own custom version of it), as well as gpslogger (yep, my own custom version of it, too.) You could use either or both of those if you like. The nice thing about Google Maps is, people often already have that on their phones, even Apple phones, and either already have location sharing set up, too, or can easily add it.

[jata1]

righto. I will see what I can find out about about icloud3 and I will report back.

I do have google maps on my phone so I will try that integration and also gpslogger too.

I just went to take my son to the dentist then to school and I was driving at 290km/h at one point. LOL

[pnbruckner]

I wonder if the outliers are due to the interaction of the two trackers, or if it's mostly from just one. Might be interesting to add two more composite entities, where each uses just one of the two "input" trackers. Then you'd get speed readings for each individually. If those are reasonable, then it's the interaction (which is probably due to lack of proper timestamps for the location data.)

[pnbruckner]

I read the icloud3 code some more, and it's quite, um, different. It seems to like to do things the hard way.

Anyway, as best as I can tell, the last_located (and other time attributes) are in the "local" time of the phone HA. If the phone is in a different time zone than HA, then the away_time_zone_offset will reflect that. E.g., when the phone is in the same time zone as HA, that attribute's value will be "HomeZone" (as it is above in your captured output.) But, if the phone is in a different time zone, then it will be "HomeZone X hours", where X is between -12 & 12. Then, when the phone returns to HA's time zone, it ... um ... wait, it should change back to "HomeZone", but I think there's a bug and it will still indicate the offset of the last time zone the phone was in that wasn't HA's time zone. Argh!

Do any of your phones that use the icloud3 integration ever go in a different time zone? It would be interesting to see if I'm reading the code right.

[pnbruckner]

Wait. Just realized, if the times are always in HA's time zone, then that should work fine. I'll add support for it. I'll try to check with the author to see if my interpretation is correct.

[jata1]

Thanks for such awesome help with this. Above and beyond for sure.

I will try out your google maps integration to see how that works for me. Although probably only for testing.

My project goal is to be able to keep an eye on my son's driving as he is getting to the age where he will be driving and in cars with friends. The snag is the traccar client app is a total battery hog but the location and speed data is OK.

Right now I want to use a combo of GPS device trackers that give me high location accuracy & frequency that do not drain the battery on my phone. Once I get everything working for me then I will setup my son in the same way.

[jata1]

Do any of your phones that use the icloud3 integration ever go in a different time zone? It would be interesting to see if I'm reading the code right.

Nope. All local in here in NSW Australia at the moment. Only on my phone currently and I installed it a few days ago so its all quite new.

[jata1]

I wonder if the outliers are due to the interaction of the two trackers, or if it's mostly from just one. Might be interesting to add two more composite entities, where each uses just one of the two "input" trackers. Then you'd get speed readings for each individually. If those are reasonable, then it's the interaction (which is probably due to lack of proper timestamps for the location data.)

I have been thinking something similar myself. One feature of the icloud3 tracker is that is already interacting with the HA tracker. I think it uses HA tracker together with icloud to handle location and maybe this is part of the issue.

I will setup a composite just with HA tracker and see how that works but we already know that the HA tracker is missing an important date/time attribute.

GPSlogger is android only so that's not an option.

I have google setup but can't see any location data in the entity so something went wrong with my setup/cookie stuff i think.

[pnbruckner]

I have google setup but can't see any location data in the entity so something went wrong with my setup/cookie stuff i think.

Are you using my custom version, and latest version? If so, you should enable debug for it. Then, on the log page, type "custom_components.google_maps" in the search box and click LOAD FULL LOGS. There should be some messages that can give some clues as to what's going on.

[jata1]

I am using your google integration and installation / cookie went fine I think.

below are the only logs I can find - with debug on. Looks OK to me. I was expecting to see GPS info in the attributes but there is nothing.

2024-03-14 14:45:10.387 DEBUG (SyncWorker_0) [custom_components.google_maps.gm_loc_sharing] jagoxxxxx@gmail.com: Cookies: 1P_JAR: 2024-04-13 11:45:50, AEC: 2024-09-10 11:42:58, SEARCH_SAMESITE: 2024-09-10 11:44:22, NID: 2024-09-13 11:43:33, SIDCC: 2025-03-14 14:43:56, __Secure-1PSIDCC: 2025-03-14 14:43:56, __Secure-3PSIDCC: 2025-03-14 14:43:56, APISID: 2025-04-18 11:45:41, HSID: 2025-04-18 11:45:41, SAPISID: 2025-04-18 11:45:41, SID: 2025-04-18 11:45:41, SSID: 2025-04-18 11:45:41, __Secure-1PAPISID: 2025-04-18 11:45:41, __Secure-1PSID: 2025-04-18 11:45:41, __Secure-3PAPISID: 2025-04-18 11:45:41, __Secure-3PSID: 2025-04-18 11:45:41
2024-03-14 14:45:12.000 DEBUG (MainThread) [custom_components.google_maps.coordinator] Finished fetching jagoxxxx@gmail.com data in 1.184 seconds (success: True)
2024-03-14 14:47:12.784 DEBUG (MainThread) [custom_components.google_maps.coordinator] Finished fetching jagoxxx@gmail.com data in 1.004 seconds (success: True)

[jata1]

So just working on the HA side of things. This is data from my latest test drive in the car. Circled in yellow is actual driving. The rest is nonsense. What this is telling me (for this trip/drive) is overall not bad.

no crazy spikes (might be lucky) HA app data is quite good but low update frequency icloud not bad but with a few drops (as I didn't stop) Traccar app was set to 60 second updates and is picking up lots of false movement/speed after the drive

So i'm hoping that with a bit of your help we can create a fairly good speed sensor with icloud and HA app.

I'm going to create a sensor that uses the average for HA and icloud and use this to push into traccar server - even better would be to do this and drop crap data as this will skew the data.

[jata1]

Just one more update. Went for another drive and it is the icloud3 tracker that keeps generating crazy speed values so I think this is where the root of the issue is that I need to find/fix - or just stop using the icloud3 integration. After all, I am only using it as I thought it would be better and/or combine well with the HA tracker.

It would be good to get the google tracker working then I have something else to compare with...

[pnbruckner]

I'm a bit confused about the Traccar data. You keep referring to pushing data to it, but you're also getting data from it??? Is it providing independent GPS data, or is it somehow data derived from the other trackers?

Note that the speed sensor only updates when it gets valid input that represents a new position. If the input(s) aren't changing, or the inputs are being rejected (for various reasons, typically they contain older data than the composite has already seen), then the speed sensor output won't change. That could be why the iCloud spike lasts a while. Also, it won't "drop" due to lack of new data.

Regarding the Google Maps integration, that's very strange. Which account did you add to the integration? Your personal account that is associated with your phone? Given the name of the tracker entity, it does appear you're using the "Account Holder" location. If you're using a different Google account (e.g., one you created just for use with this HA integration), then you should not use the "account holder" tracker, and you need to share your personal Google account's location with the account you add to the HA integration. Hopefully, that makes sense.

[jata1]

I'm a bit confused about the Traccar data. You keep referring to pushing data to it, but you're also getting data from it??? Is it providing independent GPS data, or is it somehow data derived from the other trackers?

There is the traccar server and also a client app for ios. In HA there are integrations for both the server and client.

To help with this testing, I have the traccar iOS client app (a GPS tracker) --> Traccar Client in HA -this gives me good test/baseline data for speed.

In addition, I am working with you on other options for GPS tracking for my iphone. In all cases (traccar client or HA/icloud/google) I need to push the data from HA to the traccar server. Pushing the data into the traccar server is all working fine. I have this part working great it can even test speed v. location based speed limit and notifications when speeding.

Guess what? Driving at 2,000 km/h triggers a speeding notification. :-)

[jata1]

I am just using my personal google account in 'account holder' mode. For testing this is all I need I think. Issue is I'm not getting any data through this integration... Yet... Not sure what I have done wrong. Your instructions were quite clear and everything seemed to work fine - just not getting data into HA.

[pnbruckner]

When you get no location data with the account holder tracker, that generally means it's not linked to an actual device. Do you have Google Maps installed on your iPhone and logged into the same Google account? If you open https://www.google.com/maps in a browser, logged into that same account, can you see your device location?

[jata1]

100% yes. Logged into my phone on google maps app.

Also I heard back from the iCloud3 dev. Link to the topic here

https://github.com/gcobb321/icloud3_v3/issues/286

hope it helps ?

[jata1]

So I deleted google integration and started from scratch using a windows pc rather than a mac and now it is working. So I guess it was third time lucky getting this setup. So now I have 3-4 GPS sources all available in HA.

HA app - working normally but does not have a fix time attribute icloud3 - working but with an update coming soon that might help google - working traccar app - working

I'm going for a run this morning so I will take my phone and see what the data looks like. Not driving speed (i'm a very slow runner) but should be useful data.

[jata1]

Not sure what went wrong with google maps and the composite integration. No data from my run. I have deleted and re-added it as a new composite sensor using a different - hopefully now getting data into the speed sensor for this now too.

[pnbruckner]

What do you have max gps accuracy set to? Again, check the log for debug messages. It will tell you if it's, say, ignoring updates due to inaccurate location data.

[jata1]

will put debug logging on and let you know if I find anything

[jata1]

Some more data - and we are making some good progress. I have created a composite tracker using icloud3 and HA and this is Working quite well now. Not one speed spike from icloud3 so I think the changes to composite tracker has been effective.

It would be good to get google working as a validation but the logs are clean just data not flowing. I will continue to work on google and let you know if I make progress.

It seems that the traccar GPS app (set to update every 2 mins) has quite a few data points with higher speed but that was not unexpected given frequency and driving type (city/start/stop/short)

Can you confirm that the 'last_seen' attribute of the composite tracker is the fix time for the GPS coordinates?

[jata1]

One more set of data - overall looking good now using the composite tracker or icloud3. Just need to find a way to get better detection of the end of driving to reset speed.

Plus data is being pushed through to my traccar server with successful speeding detection.

[pnbruckner]

I think further explanation of how all this works might be helpful. (I might add this to the docs in a Theory of Operation section. 😄 )

First and foremost, these are sampling systems. And like all sampling systems, sometimes hiccups can happen. E.g., a sample might be "bad", or a sample might be skipped, or the transmission of a sample from one system to another can get garbled, dropped, or received out of order.

With that in mind, the google maps integration periodically polls the google server for location data for the registered account, as well as any other accounts that share their location data with the registered account. When location data is received, for each account, there is a timestamp that indicates when that data was obtained, as well as an accuracy value (where bigger values are less accurate.) The HA integration then decides whether or not to use that data based on the timestamp, accuracy, configured max accuracy, and the last valid data received and used (if any.)

When the very first data is received, it is used, regardless of the accuracy.

After that, new location data will only be used if 1) it is not older than the previously used data (i.e., it rejects data received out of order, or "old" data, which does happen), and 2) it is at least as accurate as the previously used data (i.e., it rejects "less accurate" data.) This continues until the accuracy is less than or equal to the configured max accuracy.

After that, the second rule changes to using the data only if its accuracy is less than or equal to the configured max accuracy (i.e., it rejects "inaccurate" data), regardless of the accuracy value of the last used data.

So, at any given time, the state of the google maps tracker entity reflects the last known good location, its accuracy and its timestamp (which is in the last_seen attribute.) This even applies across HA restarts.

Now, moving on to the composite integration, its approach is based on "newer is always better." And it is "event driven", meaning it reacts to changes in the states of its inputs.

So, it does not poll its inputs (except for when it is first configured), but instead reacts only to "state changed" events of its inputs. When one of the inputs changes, like the google maps integration, it has to determine whether or not to use the new state (i.e., location.)

It starts by first checking the "last_seen" attribute to compare it to the value previously received for that input. If it is older, then the state change is rejected. (This handles out of order data for entities whose integrations don't do that.) I quote last_seen because, as we've discussed, not all entities have exactly this attribute, or even one similar to it. After trying several different attribute names (last_seen, last_timestamp & last_located), if it can't find one, it uses the state's last_updated field (which, again, is not an attribute but a field that indicates when the state & its attributes were last updated by its integration.)

Next it compares the last_seen value for this input to the last_seen value from the last used state change (from any input.) If it is older, then the update is rejected.

And, as I think we've already discussed, the speed sensor is only updated on new, accepted location data, and if the other requirements are met (mostly time between samples.)

So, as you've seen, it's possible for the speed to stay a certain value for a while, even after you know the speed has changed. This is due to the constraints above, basically that there isn't sufficient new data to override the current value. Although, even when the device isn't moving, there should be valid updates that indicate this, and the speed should change to zero (or there abouts.)

[pnbruckner]

Getting back to the google maps & composite entities, and your observations of their states, you should enable debug for both. They both write DEBUG messages when data is rejected with an explanation of why.

If that is not fully explaining the situation with your google maps entity, I can add some more debug output that dumps the raw data coming from the server. But, FWIW, some of that may already be in the logs. E.g., there might be messages that say, "Missing location or other data for person:", and are followed by the raw data for that person.

[jata1]

Thanks Phil - so good that you provide me with all this info so I can better understand this stuff. I had google & composite debug on yesterday and didn't see anything obvious wrong with the google integration.

We are an all apple household so the icloud3 integration is probably the best for us in any case. But I do want to understand why I can't get google to work and I do want to compare the data with icloud3 so I will keep trying.

Looking at the google integration I see that it is just not updating the GPS data - even though I went out for dinner. I don't have google maps location set to 'always on' on my iphone so maybe this is the issue. Will try changing that next (with debug on)

[jata1]

Going for a run. google maps app loaded. location set to always on but still the integration is reporting the last seen as yesterday afternoon. Will see if I get anything to report when back from my run.

[jata1]

New data (running not driving) does show that I am still getting very strange speed spikes - I think this is happening when icloud uses HA app GPS data.

Checkout how well HA app works comparing to the traccar client that is reporting GPS and speed every 2mins. Maybe this is all I need. Keep it simple!

[pnbruckner]

Yeah, as long as the HA app is transferring the GPS updates consistently, with little jitter, then maybe that's all you need. That is, as long as you're not trying to combine it with another GPS entity, because without accurate, low-skew timestamps, there's no way to know how much difference there is in timestamps between entities.

I'm still very curious why the google maps integration isn't working for you. I know you may not ultimately use it long term, but I'd like to get to the bottom of it in case there's a problem that may also affect others.

Are you absolutely sure you have debug enabled, and there are no useful debug, or other (warnings, errors) messages in the log? If so, then I'd like to add some more verbose logging if you're willing to give it a try.

[jata1]

Very happy to help try to work out what's happening with google maps with you. I do have debug logging on and the logs all look good - getting updates every minute - but it is not being pushed through to the entity? Here are the logs with debug enabled.

2024-03-16 12:36:54.132 DEBUG (SyncWorker_2) [custom_components.google_maps.gm_loc_sharing] jagoxxxx@gmail.com: Changed cookies since last saved: updated: SIDCC, __Secure-3PSIDCC, __Secure-1PSIDCC
2024-03-16 12:36:54.133 DEBUG (MainThread) [custom_components.google_maps.coordinator] Finished fetching jagoxxxx@gmail.com data in 0.470 seconds (success: True)
2024-03-16 12:37:54.237 DEBUG (MainThread) [custom_components.google_maps.coordinator] Finished fetching jagoxxxx@gmail.com data in 0.572 seconds (success: True)
2024-03-16 12:38:54.172 DEBUG (MainThread) [custom_components.google_maps.coordinator] Finished fetching jagoxxxx@gmail.com data in 0.507 seconds (success: True)
etc etc etc

but in the sensor in HA

[pnbruckner]

Do you know how to edit code in the custom_components folder, and if so, are you comfortable doing that? If you could, that would make adding the extra verbose debug output much easier.

Basically you'd have to add a single line of code in this block:

    def get_new_data(self) -> None:
        """Get new data from Google server."""
        try:
            resp = self._session.get(_URL, params=_PARAMS, verify=True)
            resp.raise_for_status()
        except (RequestException, MaxRetryError) as err:
            raise RequestFailed(f"{err.__class__.__name__}: {err}") from err
        raw_data = resp.text
        _LOGGER.debug("%s: Raw data: %s", self._account_email, raw_data) # <<< ADD THIS LINE
        try:
            self._data = json.loads(raw_data[5:])
        except (IndexError, json.JSONDecodeError) as err:
            raise InvalidData(f"Could not parse: {raw_data}") from err
        if not isinstance(self._data, Sequence):
            raise InvalidData(f"Expected a Sequence, got: {self._data}")
        try:
            if self._data[6] == "GgA=":
                self._dump_cookies()
                _LOGGER.debug("%s: Parsed data: %s", self._account_email, self._data)
                raise InvalidCookies("Invalid session indicated")
        except IndexError:
            raise InvalidData(f"Unexpected parsed data: {self._data}") from None

[pnbruckner]

Regarding the iCloud3 location timestamp attribute, I'm thinking of adding a feature to this integration that allows the user more control of how it gets this data.

First, I'd definitely add the ability to use an attribute that is a string representation of the time. It would accept anything that homeassistant.util.dt.parse_datetime accepts. (It's already using homeassistant.util.dt.utc_from_timestamp for POSIX timestamps.)

Next, for each input, if it doesn't already have a supported attribute, it would allow either specifying the name of another existing attribute (that would need to be formatted in one of the several supported ways), or a template, which would give the user full flexibility in getting the data from the right source (another attribute, another entity, etc.) and possibly reformatting it if necessary.

But, for the purposes of this issue, hopefully iCloud3 will be modified to included a supported (or easily supportable) attribute.

[jata1]

I’m happy to modify the code in the google maps integration. Just let me know the file name and approx line number so I can easily find that code block in my text editor.

[pnbruckner]

Sorry, I should have included that. It's gm_loc_sharing.py, and the line should be inserted after line 235.

[pnbruckner]

Also, you probably saw there's a new beta of iCloud3 that should work with composite (any version.)

[jata1]

Also, you probably saw there's a new beta of iCloud3 that should work with composite (any version.)

Not yet. Just woken up. No coffee even yet. haha

I will try the new beta and see how it goes. My concern is that we still are getting speed spikes with icloud3 and I think it is when icloud3 uses GPS coordinates from the HA app but does not have fix time so it is using something else from HA or holds the last fix time from icloud.

[jata1]

Have added logging code to google maps. I will restart the integration with debug enabled and see if we catch anything.