search

joshstrange / day-one-narrato-import

Importer for text Narrato posts into Day One
1 stars 1 forks source link

Getting location data? #2

Open mcdent opened 9 years ago

mcdent commented 9 years ago

Hi, I was just reminded about this from the recent post. I'd like to get my data out of narrato and in to Day One but I have a fair amount of location data which it would be a shame to lose. It's far beyond my skills to add this feature but wondered how much work it might be for Josh to add this support, especially as you have just been re-aquanted with the new json format from Narrato ;)

If its not feasible I guess I will have to pull it in without this data. Photos would be the icing on the cake but I guess that might be asking too much! Many thanks

Mike

joshstrange commented 9 years ago

If you give a mouse a cookie.... :) Just kidding. Ok, so DayOne doesn't "really" support just location points like Narrato did but you can create a blank note with just a location which I think I can do. Would this be acceptable or would you like the note text to but something like "Location only" or something like that?

joshstrange commented 9 years ago

Hmm, this is going be a bit more trouble than I thought. Here is the DayOne format:

<key>Location</key>
<dict>
    <key>Administrative Area</key>
    <string>KY</string>
    <key>Country</key>
    <string>United States</string>
    <key>Latitude</key>
    <real>38.076193551568387</real>
    <key>Locality</key>
    <string>Lexington</string>
    <key>Longitude</key>
    <real>-84.552054712627594</real>
    <key>Place Name</key>
    <string>2601 Coronado Rdg</string>
    <key>Region</key>
    <dict>
        <key>Center</key>
        <dict>
            <key>Latitude</key>
            <real>38.076188640000098</real>
            <key>Longitude</key>
            <real>-84.551940157499899</real>
        </dict>
        <key>Radius</key>
        <real>28.189055850031355</real>
    </dict>
</dict>

And Narrato's format is: 

{
    "created": "2013-07-20T22:33:14Z",
    "created_offset": -14400,
    "day_string": "2013-07-20",
    "is_deleted": false,
    "location": {
      "lat": 36.16086789749094,
      "lon": -86.78100696507192
    },
    "modified": "2013-08-31T15:06:24Z",
    "range_token": "20130720223314000117dc9d",
    "timezone": "America/New_York",
    "type": "co.narrato.journal.checkinpost",
    "uuid": "B2936B29-45A6-4781-80DE-D1198117DC9D",
    "version": "7344925"

}

As you can see DayOne has quite a bit more info. I can't test right now but I will check if I just drop in Lat/Lon into DayOne if it will automatically fill out the rest of the info. If not I'll have to use something like Google's Maps API (Which I am fairly familiar with) to get the needed data but that will take longer obviously.

mcdent commented 9 years ago

:) Interesting, looks like Day one records actual lat/long and some kind of matched lat/long too? Anything you can do would be very much appreciated!

Mike

joshstrange commented 9 years ago

Yes, that's what it looks like, the "Center" data is probably what the GPS kicks back and the Lat/Lon at the top level is the reverse geocoded point. The good news is I have experience with both Google Maps API and reverse geocoding. I've only ever done it from the browser though and not node so hopefully it's just as easy (And free, I use a paid version for work).

joshstrange commented 9 years ago

Ok, I've just committed location support. I used google maps but it looks like DayOne cleans that up if data is wrong/missing which is nice. The note text will be "Location Only". If you want that changed let me know.

mcdent commented 9 years ago

Awesome! Thanks very much, I will check this out when I get a moment.

mcdent commented 9 years ago

Just gave this a whirl now after installing node from https://nodejs.org/en/

I get:-

Mikes-MacBook-Air:day-one-narrato-import mikedent$ node ./index.js --help module.js:338 throw err; ^

Error: Cannot find module 'commander' at Function.Module._resolveFilename (module.js:336:15) at Function.Module._load (module.js:286:25) at Module.require (module.js:365:17) at require (module.js:384:17) at Object. (/Users/mikedent/Documents/Day one import/day-one-narrato-import/index.js:1:77) at Module._compile (module.js:434:26) at Object.Module._extensions..js (module.js:452:10) at Module.load (module.js:355:32) at Function.Module._load (module.js:310:12) at Function.Module.runMain (module.js:475:10)

I'm probably doing something stupid or missing something? Thanks! Mike

joshstrange commented 9 years ago

Did you run "npm install" in the folder after installing node? You need to run that to install the dependencies. Look in the README under usage.

mcdent commented 9 years ago

Thanks, got past that now and started the import. However it seems to be bombing out on some invalid characters.

node index.js import ./items.json /Users/mikedent/Documents/Day one import/day-one-narrato-import/node_modules/dayone/node_modules/plist/node_modules/xmlbuilder/lib/XMLStringifier.js:150 throw new Error("Invalid character (" + chr + ") in string: " + str + " at index " + chr.index); ^

Error: Invalid character (�) in string: Funny thing. Heard hair dryer coming from Cameron's bed room tonight, asked him what he was doing? "Warming the bed before I get in" 😃 at index 148

In the json file the character looks like:-

before I get in\" \ud83d\ude03", "timezone": "Europe/London"

I got past a couple of these by removing the spurious crap before the closing quote.

It then proceeded to import about 1000 entries and 3 locations. Not sure what happened with the locations? I checked on my iPhone in the old Journal app and the locations are Foursquare checkins, not sure if that has confused matters?

I could probably paste some of the json file with an appropriate location if you think it would help?

Thanks again!

Mike

joshstrange commented 9 years ago

If you can post some data from the locations that didn't get imported I bet I can get it working. That said they might not store the data alongside the rest so I can't make any promises.

mcdent commented 9 years ago

Thanks, whats the best way to get the data from a json file so it is more readable and I can paste it here? If i do it from the shell with 'less' its not very easy to read :)

joshstrange commented 9 years ago

I find pasting the contents of your items.json into a tool like this: http://www.jsoneditoronline.org/ (On the left hand side) then clicking this icon in the top left formats it nicely then you paste it in as a comment (feel free to strip identifiable info) indented with 4 spaces

{
   "so": "it shows up like this"
}

works best.

mcdent commented 9 years ago

Thanks! hopefully this looks ok?

"created": "2014-09-11T12:43:30Z",
"created_offset": 0,
"data": {
  "comments": {
    "count": 0
  },
  "createdAt": 1410439410,
  "id": "541198f2498eff8664be878c",
  "like": false,
  "likes": {
    "count": 0,
    "groups": []
  },
  "photos": {
    "count": 0,
    "items": []
  },
  "posts": {
    "count": 0,
    "textCount": 0
  },
  "source": {
    "name": "Swarm for iOS",
    "url": "https://www.swarmapp.com"
  },
  "timeZoneOffset": 60,
  "type": "checkin",
  "venue": {
    "categories": [
      {
        "icon": {
          "prefix": "https://ss1.4sqi.net/img/categories_v2/food/coffeeshop_",
          "suffix": ".png"
        },
        "id": "4bf58dd8d48988d1e0931735",
        "name": "Coffee Shop",
        "pluralName": "Coffee Shops",
        "primary": true,
        "shortName": "Coffee Shop"
      }
    ],
    "contact": {
      "formattedPhone": "01524 36139",
      "phone": "0152436139"
    },
    "id": "4afedd16f964a520023122e3",
    "like": false,
    "location": {
      "address": "23 Market Street",
      "cc": "GB",
      "city": "Lancaster",
      "country": "United Kingdom",
      "formattedAddress": [
        "23 Market Street",
        "Lancaster",
        "Lancashire",
        "LA1 1HZ"
      ],
      "lat": 54.04864968513448,
      "lng": -2.80102014541626,
      "postalCode": "LA1 1HZ",
      "state": "Lancashire"
    },
    "name": "Caffè Nero",
    "stats": {
      "checkinsCount": 601,
      "tipCount": 2,
      "usersCount": 132
    },
    "url": "http://www.caffenero.com",
    "verified": false
  }
},
"day_string": "2014-09-11",
"is_deleted": false,
"location": {
  "lat": 54.04864968513448,
  "lon": -2.80102014541626
},
"modified": "2014-09-13T13:38:13Z",
"range_token": "20140911124330000ec37683",
"timezone": "GMT+0100",
"type": "com.foursquare.checkin",
"uuid": "37597089-0D6A-4CBB-97FA-2xxxxxxxxx",
"vendor_id": "541198f2498exxxxx4be878c",
"version": "9632608"

},

joshstrange commented 9 years ago

Ok, that's exactly what I needed. I just added support for these as well. That said if you re-run the import it will re-import ALL of your data again leading to duplicates. I'm not sure of a way to handle this ATM... The best I think I could do is allow you to specify a type of post you want to import (so you could import just the foursquare posts).