Integrate Goose for database migrations

[brianfoshee]

Working on parts of #9 Closes #9 #20 #19 #18 #16

• [x] Add uuid-ossp extension
• [x] Add device table
• [x] Add reading table
• [x] Update readme
• [x] Integrate with Heroku
• [x] Update postgres manager and tests

CREATE EXTENSION "uuid-ossp";
CREATE TABLE device (
  -- id is for internal use as a primary key
  id uuid PRIMARY KEY DEFAULT uuid_generate_v4(),
  -- Device generated identifier
  identifier character varying NOT NULL,
  -- timestamps which set to current time in UTC on record creation
  updated_at timestamp without time zone NOT NULL DEFAULT (now() AT TIME ZONE 'UTC'),
  created_at timestamp without time zone NOT NULL DEFAULT (now() AT TIME ZONE 'UTC')
);

CREATE TABLE reading (
  device_id uuid NOT NULL REFERENCES device (id) ON DELETE CASCADE,
  readings json NOT NULL DEFAULT '{}'::json
);
CREATE INDEX reading_device_id ON reading (device_id);

-- SQL function to add a key to json field
CREATE OR REPLACE FUNCTION "json_object_set_key"(
  "json"          json,
  "key_to_set"    TEXT,
  "value_to_set"  anyelement
)
  RETURNS json
  LANGUAGE sql
  IMMUTABLE
  STRICT
AS $function$
SELECT COALESCE(
  (SELECT ('{' || string_agg(to_json("key") || ':' || "value", ',') || '}')
     FROM (SELECT *
             FROM json_each("json")
            WHERE "key" <> "key_to_set"
            UNION ALL
           SELECT "key_to_set", to_json("value_to_set")) AS "fields"),
  '{}'
)::json
$function$;

-- Insert a device into the db
INSERT INTO device (identifier) VALUES ('ABC123') RETURNING "id";

-- Option 1 for inserting a reading column:
-- insert an empty reading (default empty json object on create)
INSERT INTO reading (device_id) 
SELECT id FROM device WHERE device.identifier = 'ABC123';

-- Option 2 for inserting a reading column:
INSERT INTO reading (device_id, readings) 
SELECT id, '{"2014-11-01T12:30:00Z": {"ph":7,"tds":100}}'  
FROM device WHERE device.identifier = 'ABC123';

-- Insert additional JSON into the record:
UPDATE reading
SET readings = json_object_set_key(readings, '2014-11-05T21:00:00Z', '{"ph":4, "tds":121}'::json)
WHERE device_id = (
  SELECT id
  FROM device
  WHERE identifier = 'ABC123'
);

UPDATE reading
SET readings = json_object_set_key(readings, '2014-12-01T23:00:00Z', '{"ph":7, "tds":101}'::json)
WHERE device_id = (
  SELECT id
  FROM device
  WHERE identifier = 'ABC123'
);

[brianfoshee]

@letsgitgrowing are we safe to drop everything on the Heroku database? I'm going to test this out.

[brianfoshee]

@letsgitgrowing I was invited to beta test a feature on Heroku which lets us setup automatic deploys to Heroku when code is pushed to the master branch. I set it up on this app and repository to test it out.

https://devcenter.heroku.com/articles/github-sync

Also, the database is full on Heroku so we'll have to delete all the data and set up the new schema so that we can start using it again.

[brianfoshee]

@letsgitgrowing I had some fun with Postgres last night at the event I went to with Kirsten (it was a design-related talk so I got away with not paying attention). I got the database schema squared away and figured out the function that'll be required to allow us to UPDATE a JSON field (and it supports nested JSON on Postgres 9.3 already).

See the original post at the top of this pull request for the raw SQL schema, function, and examples of adding a device then adding individual readings. These are all updated in the database migrations in the goose branch (this PR).

[brianfoshee]

This is what the database looks like afterwards:

aquaponics-data-dev=# \d
                  List of relations
 Schema |          Name           |   Type   | Owner
--------+-------------------------+----------+-------
 public | device                  | table    | brian
 public | goose_db_version        | table    | brian
 public | goose_db_version_id_seq | sequence | brian
 public | reading                 | table    | brian
(4 rows)

aquaponics-data-dev=# \d device
                                  Table "public.device"
   Column   |            Type             |                   Modifiers
------------+-----------------------------+-----------------------------------------------
 id         | uuid                        | not null default uuid_generate_v4()
 identifier | character varying           | not null
 updated_at | timestamp without time zone | not null default timezone('UTC'::text, now())
 created_at | timestamp without time zone | not null default timezone('UTC'::text, now())
Indexes:
    "device_pkey" PRIMARY KEY, btree (id)
Referenced by:
    TABLE "reading" CONSTRAINT "reading_device_id_fkey" FOREIGN KEY (device_id) REFERENCES device(id) ON DELETE CASCADE

aquaponics-data-dev=# \d reading
             Table "public.reading"
  Column   | Type |          Modifiers
-----------+------+-----------------------------
 device_id | uuid | not null
 readings  | json | not null default '{}'::json
Indexes:
    "reading_device_id" btree (device_id)
Foreign-key constraints:
    "reading_device_id_fkey" FOREIGN KEY (device_id) REFERENCES device(id) ON DELETE CASCADE

aquaponics-data-dev=# select * from device;
                  id                  | identifier |         updated_at         |         created_at
--------------------------------------+------------+----------------------------+----------------------------
 6c7ae8aa-3853-4c27-a1cc-3d991b56ff29 | ABC123     | 2014-12-04 21:16:05.593885 | 2014-12-04 21:16:05.593885
(1 row)

Time: 0.395 ms

aquaponics-data-dev=# select * from reading;
              device_id               |                                        readings
--------------------------------------+-----------------------------------------------------------------------------------------
 6c7ae8aa-3853-4c27-a1cc-3d991b56ff29 | {"2014-11-05T21:00:00Z":{"ph":4, "tds":121},"2014-12-01T23:00:00Z":{"ph":7, "tds":101}}
(1 row)

Time: 0.316 ms

[brianfoshee]

Required changes to the schema for converting to jsonb:

-- We have to remove the default value of '{}'::json because it's incompatible with jsonb
ALTER TABLE reading ALTER COLUMN readings DROP DEFAULT;
-- Change the column type, and cast existing data to jsonb
ALTER TABLE reading ALTER COLUMN readings TYPE jsonb USING readings::jsonb;
-- Add the default value back, of course using jsonb this time
ALTER TABLE reading ALTER COLUMN readings SET DEFAULT '{}'::jsonb;

I'll make a migration for this once we have the refactoring-goose branch finished up. So far I'm seeing query times cut in half for selecting on a particular json key. It'll be interesting to see what an index will do when we're dealing with a ton of data.

Also, need to modify the json_object_set_key function types.