Cannot import data from Google

[jzitnik-dev]

Hi, I really like this webapp but I cannot import Google Timeline Data. I don't understand Google Timeline that much but as from I understood there is like a new version of Google Timeline and now all the location history is stored on the device. When I did Takeout there were no useful files. Just a Timeline Edits.json, Settings.json and one txt file called Encrypted Backups.txt that has this message inside:

You have encrypted Timeline backups stored on Google servers.

I found this thread on google support that says you can download the location history directly from my phone. I did it but I cannot import it. I think it is because the JSON file has different structure that the one from the Takeout, but I'm not sure, I don't have the backup from Takeout.

I give you a structure of the JSON file. But I'm not 100% sure what is actual location data and what is just some non private data. The original file is over a million lines long.

The file structure

{
  "semanticSegments": [
    {
      "startTime": "2019-04-03T08:00:00.000+02:00",
      "endTime": "2019-04-03T10:00:00.000+02:00",
      "timelinePath": [
        {
          "point": "(coordinates of my home)",
          "time": "2019-04-03T08:14:00.000+02:00"
        },
        {
          "point": "(some other coordinates from my town)",
          "time": "2019-04-03T08:46:00.000+02:00"
        }
      ]
    },
    {
      "startTime": "2019-04-03T08:13:57.000+02:00",
      "endTime": "2019-04-03T20:10:18.000+02:00",
      "startTimeTimezoneUtcOffsetMinutes": 120,
      "endTimeTimezoneUtcOffsetMinutes": 120,
      "visit": {
        "hierarchyLevel": 0,
        "probability": 0.8500000238418579,
        "topCandidate": {
          "placeId": "some random id",
          "semanticType": "UNKNOWN",
          "probability": 0.44970497488975525,
          "placeLocation": {
            "latLng": "(coordinates of a random house)"
          }
        }
      }
    }
  ],
  "rawSignals": [
    {
      "activityRecord": {
        "probableActivities": [
          {
            "type": "STILL",
            "confidence": 0.9599999785423279
          },
          {
            "type": "IN_VEHICLE",
            "confidence": 0.009999999776482582
          },
          {
            "type": "ON_FOOT",
            "confidence": 0.009999999776482582
          },
          {
            "type": "WALKING",
            "confidence": 0.009999999776482582
          },
          {
            "type": "UNKNOWN",
            "confidence": 0.009999999776482582
          },
          {
            "type": "IN_ROAD_VEHICLE",
            "confidence": 0.009999999776482582
          },
          {
            "type": "IN_RAIL_VEHICLE",
            "confidence": 0.009999999776482582
          },
          {
            "type": "IN_ROAD_VEHICLE",
            "confidence": 0.009999999776482582
          }
        ],
        "timestamp": "2024-04-26T20:54:38.000+02:00"
      }
    },
    {
      "activityRecord": {
        "probableActivities": [
          {
            "type": "STILL",
            "confidence": 0.9900000095367432
          },
          {
            "type": "UNKNOWN",
            "confidence": 0.009999999776482582
          }
        ],
        "timestamp": "2024-04-26T20:55:45.000+02:00"
      }
    }
  ],
  "userLocationProfile": {
    "frequentPlaces": [
      {
        "placeId": "some random id",
        "placeLocation": "coordinates of my work",
        "label": "WORK"
      },
      {
        "placeId": "some random id",
        "placeLocation": "my home coordinates",
        "label": "HOME"
      }
    ]
  }
}

[Freika]

Hi! I'd suggest you to try and export Google Takeout, and import its data instead. There are already 2 (two) different ways to import Google exported data in Dawarich and I would rather stop there and not implement a third way :)

Let me know if you succeed in your importing!

[jzitnik-dev]

Hi, thanks for your reply.

As I said

When I did Takeout there were no useful files. Just a Timeline Edits.json, Settings.json and one txt file called Encrypted Backups.txt.

I found this post by Google that says that nowadays they don't store the timeline history in Google Cloud. Instead it is stored directly on the users device. https://support.google.com/maps/answer/14169818?visit_id=638524670623014961-4265012978&p=maps_odlh&rd=1

Since the data shown on your Timeline comes directly from your device, Timeline won’t be available on Maps on your computer after your data is moved to your phone. To move your data to your phone, download the Google Maps app.

I can turn on Google Timeline backup that will store encrypted version of this backup in the Google Cloud but this backup will not show in the Google Takeout.

The Encrypted Backups.txt file in the Google Takeout is just saying that I have this backup in the Google Cloud but it is not showing it in the Takeout. Probably because it is encrypted :)

[Freika]

In this section, can you provide a realistically looking example of point coordinates value?

      "timelinePath": [
        {
          "point": "(coordinates of my home)",
          "time": "2019-04-03T08:14:00.000+02:00"
        },
        {
          "point": "(some other coordinates from my town)",
          "time": "2019-04-03T08:46:00.000+02:00"
        }

[jzitnik-dev]

Yeah, here it is 50.0506312°, 14.3439906°

[Freika]

@JZITNIK-github got it, thank you. I'll see what I can do for this kind of json format. I guess that will be the third format for Google Takeout 🤡

[jzitnik-dev]

Thanks, I would do the Google Takeout if it was possible :) But it looks like that Google is moving users to save their timeline on their device for some reason.

[Freika]

Hi @JZITNIK-github! If you update to 0.5.2, you'll be able to see a new option in "new import" page, "google_phone_takeout". I'd like to ask you to try selecting it with your file exported from your phone, and see if the import is successful or not. If it's not, I'd appreciate it if you could share a file of your export with me, replacing your real coordinates with dummy ones (but preserving the same structure of values, it's important).

Thanks!

[jzitnik-dev]

I'll look at it tomorrow.

[jzitnik-dev]

Sadly It didn't work. It uploaded the file but not imported the locations. I'm not able to send you the whole json file because it is really large and it would take ages to change all the coordinates to some dummy one. I was trying to search some documentation on this but sadly didn't find anything. But I don't need this feature that much. I thought that maybe I was doing something wrong, not that it would need to be complete new feature or something. But thanks for great app, definitely gonna use it.

[Freika]

@JZITNIK-github you could try something like Sublime Text's regex replacement to replace your values in bulk (https://stackoverflow.com/questions/11819886/regular-expression-search-replace-in-sublime-text-2), or even awk regex replace (https://unix.stackexchange.com/questions/25122/how-to-use-regex-with-awk-for-string-replacement), it should be much easier than doing it manually.

Anyway, it would be great to be able to handle more cases in terms of importing data from Google, but no pressure, at some point someone will probably encounter the same problem, and I'll fix it :) Cheers!

[jzitnik-dev]

Yeah, I wrote python script for doing that but it was running for ages. Also the JSON file doesn't seem very well structured and there are some weird values. I don't have much free time right now so I'm not gonna be able to help you much with this. Sorry.

[Freika]

No worries at all!

[etuck]

I'm having almost exactly the same issue. I've moved to the newest version of Google Timeline where all location data is stored on the device. When I export from the device (Settings > Location > Location Services > Timeline > "Export Timeline data") it creates a location-history.json file. I've tried the "google_phone_takeout" import and it does import the first 21 datapoints but that's all.

I'll include the first 279 lines of the file here. That's the first 21 and a few subsequent datapoints. It's a little bit of location data from 2013 so I'm not that concerned about it. { "semanticSegments": [ { "startTime": "2013-08-22T16:00:00.000-04:00", "endTime": "2013-08-22T18:00:00.000-04:00", "timelinePath": [ { "point": "34.0804288°, -83.8713145°", "time": "2013-08-22T17:22:00.000-04:00" }, { "point": "34.0832221°, -83.8620083°", "time": "2013-08-22T17:39:00.000-04:00" }, { "point": "34.0786764°, -83.8731066°", "time": "2013-08-22T17:43:00.000-04:00" }, { "point": "34.0804288°, -83.8713145°", "time": "2013-08-22T17:44:00.000-04:00" }, { "point": "34.0832221°, -83.8620083°", "time": "2013-08-22T17:54:00.000-04:00" }, { "point": "34.0804288°, -83.8713145°", "time": "2013-08-22T17:57:00.000-04:00" }, { "point": "34.0788556°, -83.8741344°", "time": "2013-08-22T17:58:00.000-04:00" } ] }, { "startTime": "2013-08-22T17:21:56.000-04:00", "endTime": "2013-08-22T18:30:45.000-04:00", "startTimeTimezoneUtcOffsetMinutes": -240, "endTimeTimezoneUtcOffsetMinutes": -240, "visit": { "hierarchyLevel": 0, "probability": 0.6399999856948853, "topCandidate": { "placeId": "ChIJPbskHZDr9YgRxZkdKXA-dpw", "semanticType": "UNKNOWN", "probability": 0.9720451235771179, "placeLocation": { "latLng": "34.078686°, -83.8783857°" } } } }, { "startTime": "2013-08-22T18:00:00.000-04:00", "endTime": "2013-08-22T20:00:00.000-04:00", "timelinePath": [ { "point": "34.0804288°, -83.8713145°", "time": "2013-08-22T18:08:00.000-04:00" }, { "point": "34.0785788°, -83.8733269°", "time": "2013-08-22T18:13:00.000-04:00" }, { "point": "34.0786122°, -83.8728445°", "time": "2013-08-22T18:31:00.000-04:00" }, { "point": "34.0800978°, -83.8753984°", "time": "2013-08-22T18:32:00.000-04:00" }, { "point": "34.0692154°, -83.906818°", "time": "2013-08-22T18:42:00.000-04:00" }, { "point": "34.0724091°, -83.9304494°", "time": "2013-08-22T18:47:00.000-04:00" }, { "point": "34.09416°, -83.9380305°", "time": "2013-08-22T18:49:00.000-04:00" }, { "point": "34.0942772°, -83.9447775°", "time": "2013-08-22T18:50:00.000-04:00" }, { "point": "34.114336°, -83.9949922°", "time": "2013-08-22T18:55:00.000-04:00" }, { "point": "34.0760879°, -84.047659°", "time": "2013-08-22T19:02:00.000-04:00" }, { "point": "34.0571969°, -84.0648377°", "time": "2013-08-22T19:05:00.000-04:00" }, { "point": "34.0633914°, -84.0734024°", "time": "2013-08-22T19:06:00.000-04:00" }, { "point": "34.0633914°, -84.0734024°", "time": "2013-08-22T19:06:00.000-04:00" }, { "point": "34.0645838°, -84.0491773°", "time": "2013-08-22T19:28:00.000-04:00" } ] }, { "startTime": "2013-08-22T18:30:45.000-04:00", "endTime": "2013-08-22T19:06:04.000-04:00", "startTimeTimezoneUtcOffsetMinutes": -240, "endTimeTimezoneUtcOffsetMinutes": -240, "activity": { "start": { "latLng": "34.0791232°, -83.8727417°" }, "end": { "latLng": "34.0636508°, -84.0725485°" }, "distanceMeters": 0.0, "topCandidate": { "type": "UNKNOWN_ACTIVITY_TYPE", "probability": 0.0 } } }, { "startTime": "2013-08-22T19:06:04.000-04:00", "endTime": "2013-08-22T19:18:03.000-04:00", "startTimeTimezoneUtcOffsetMinutes": -240, "endTimeTimezoneUtcOffsetMinutes": -240, "visit": { "hierarchyLevel": 0, "probability": 0.699999988079071, "topCandidate": { "placeId": "ChIJVYAAUw6X9YgRRGJWIqiNJd0", "semanticType": "UNKNOWN", "probability": 0.698940634727478, "placeLocation": { "latLng": "34.0641526°, -84.0721277°" } } } }, { "startTime": "2013-08-22T19:27:57.000-04:00", "endTime": "2013-08-24T07:20:08.000-04:00", "startTimeTimezoneUtcOffsetMinutes": -240, "endTimeTimezoneUtcOffsetMinutes": -240, "visit": { "hierarchyLevel": 0, "probability": 0.9900000095367432, "topCandidate": { "placeId": "ChIJ1T37rvqW9YgRlit5n2x75cU", "semanticType": "HOME", "probability": 0.9999732375144958, "placeLocation": { "latLng": "34.0645228°, -84.0490043°" } } } }, { "startTime": "2013-08-22T20:00:00.000-04:00", "endTime": "2013-08-22T22:00:00.000-04:00", "timelinePath": [ { "point": "34.0644334°, -84.0489579°", "time": "2013-08-22T20:00:00.000-04:00" } ] }, { "startTime": "2013-08-24T06:00:00.000-04:00", "endTime": "2013-08-24T08:00:00.000-04:00", "timelinePath": [ { "point": "34.0646311°, -84.0490813°", "time": "2013-08-24T07:20:00.000-04:00" }, { "point": "34.0667913°, -84.0517416°", "time": "2013-08-24T07:21:00.000-04:00" }, { "point": "34.0675353°, -84.0571015°", "time": "2013-08-24T07:22:00.000-04:00" }, { "point": "34.0580451°, -84.0654205°", "time": "2013-08-24T07:23:00.000-04:00" }, { "point": "34.0621942°, -84.0739047°", "time": "2013-08-24T07:24:00.000-04:00" }, { "point": "34.0680489°, -84.0778913°", "time": "2013-08-24T07:26:00.000-04:00" }, { "point": "34.0757092°, -84.0732632°", "time": "2013-08-24T07:27:00.000-04:00" }, { "point": "34.089429°, -84.0734914°", "time": "2013-08-24T07:29:00.000-04:00" }, { "point": "34.1019517°, -84.0828494°", "time": "2013-08-24T07:30:00.000-04:00" }, { "point": "34.1069565°, -84.0840503°", "time": "2013-08-24T07:31:00.000-04:00" }, { "point": "34.1141426°, -84.0794228°", "time": "2013-08-24T07:32:00.000-04:00" }, { "point": "34.1185254°, -84.077011°", "time": "2013-08-24T07:32:00.000-04:00" }, { "point": "34.1279388°, -84.095421°", "time": "2013-08-24T07:34:00.000-04:00" }, { "point": "34.1557704°, -84.1076551°", "time": "2013-08-24T07:36:00.000-04:00" }, { "point": "34.1640998°, -84.1176282°", "time": "2013-08-24T07:38:00.000-04:00" }, { "point": "34.1843092°, -84.1390445°", "time": "2013-08-24T07:44:00.000-04:00" }, { "point": "34.1613771°, -84.1678686°", "time": "2013-08-24T07:47:00.000-04:00" }, { "point": "34.1604478°, -84.1734737°", "time": "2013-08-24T07:48:00.000-04:00" }, { "point": "34.1600131°, -84.1879906°", "time": "2013-08-24T07:51:00.000-04:00" }, { "point": "34.1545192°, -84.2063043°", "time": "2013-08-24T07:52:00.000-04:00" }, { "point": "34.155703°, -84.2151482°", "time": "2013-08-24T07:53:00.000-04:00" }, { "point": "34.1514526°, -84.2099559°", "time": "2013-08-24T07:54:00.000-04:00" }, { "point": "34.1521487°, -84.2122473°", "time": "2013-08-24T07:55:00.000-04:00" } ] },

[Freika]

Hi @etuck! In release 0.8.2 I fixed support for files such as yours, proceed to the Imports page of your Dawarich instance, select "Google Phone Takeout" as the source, and select your location-history.json file and import it as usual. It should work :)

[etuck]

Yes, that what I did but it only imported the first 21 datapoints in a 44 MB file. So, it imported about half the ones in the sample json I posted above. But that json is just a small percentage of the full file. I'm happy to send you the full file if you want but I don't want to post it to a public message board.

Edit: Sorry, you did say version 0.8.2 and I'm on 0.8.1. I'll give that a shot.

[etuck]

I'm afraid 0.8.2 didn't even import the 21 datapoints. This is the error I'm seeing in the sidekiq log:

`2024-06-30T20:02:14.201Z pid=54 tid=h7u class=ImportJob jid=c86db3fd0305a8ee67a35ee2 elapsed=1.489 INFO: fail

2024-06-30T20:02:14.201Z pid=54 tid=h7u WARN: {"context":"Job raised exception","job":{"retry":true,"queue":"imports","wrapped":"ImportJob","args":[{"job_class":"ImportJob","job_id":"fa9081ef-0423-4fc2-b019-3bc7997aedcf","provider_job_id":null,"queue_name":"imports","priority":null,"arguments":[1,9],"executions":0,"exception_executions":{},"locale":"en","timezone":"Europe/London","enqueued_at":"2024-06-30T20:00:29.304886392Z","scheduled_at":null}],"class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","jid":"c86db3fd0305a8ee67a35ee2","created_at":1719777629.3052788,"enqueued_at":1719777732.7115953,"error_message":"undefined method `flat_map' for nil:NilClass","error_class":"NoMethodError","failed_at":1719777631.1152928,"retry_count":2,"retried_at":1719777693.08547}}

2024-06-30T20:02:14.201Z pid=54 tid=h7u WARN: NoMethodError: undefined method `flat_map' for nil:NilClass

2024-06-30T20:02:14.203Z pid=54 tid=h7u WARN: app/services/google_maps/semantic_history_parser.rb:47:in `parse_json'

app/services/google_maps/semantic_history_parser.rb:12:in 'call'

app/jobs/import_job.rb:10:in 'perform' `

[Freika]

@etuck I can see that you selected "google location history" as source, while you're trying to upload a file that should be imported with "google phone takeout" source :)

Yeah, dawarich has 3 different ways to import google data, unfortunately, so it's easy to mix. But only due to google having 3 different formats for takeout files :)

[etuck]

Not sure how I did that but you're right, re-importing as "google location history" worked. It imported 160K datapoints.