api.ly

api.ly.g0v.tw endpoint source and utility scripts

Development

Linux

We recommend using docker for developing. Docker will install all the packages, configurations, database needed in a image (a file), not in your computer.

Workflow
1. Install docker
  
  If you're using ubuntu / mint, please follow these steps: Remove sudo, Remove docker and local DNS server warnings
2. Build base image
```
$ ./docker/baseimage/scripts/build-image.sh
```
3. Run database container (store database only, please see below)
4. Run postgres container (please see below)
5. Now you can
  - Run web server container
  - Run worker containers
  - Dig into database
  - Backup the database
  - Restore the database
Web server container (API endpoint)

Build image
```
$ ./docker/app/scripts/build-image.sh
```
Open another terminal, and run:
```
$ ./docker/app/run.sh
```
Open you browser, see http://127.0.0.1:3000/collections/sittings

Woker containers

Build image

$ ./docker/workers/scripts/build-image.sh

Open another terminal, and run:

Calendar worker
```
$ ./docker/workers/calendar.sh
```
Sitting worker
```
$ ./docker/workers/sitting.sh
```
Motion & Bill worker
```
$ ./docker/workers/motion-and-bill.sh
```
Bill-details worker
```
$ ./docker/workers/bill-details.sh
```

Postgres container

Build image

$ ./docker/postgres/scripts/build-image.sh

Open another terminal, and run:

$ ./docker/postgres/run.sh

Dig into database

$ ./docker/postgres/scripts/psql.sh
psql (9.3.4)
Type "help" for help.

ly=> \d
            List of relations
 Schema |       Name        | Type  | Owner 
--------+-------------------+-------+-------
 public | amendments        | table | ly
 public | bills             | table | ly
 public | calendar          | table | ly
 public | ivod              | table | ly
 public | laws              | table | ly
 public | motions           | table | ly
 public | sittings          | table | ly
 public | ttsbills          | table | ly
 public | ttsinterpellation | table | ly
 public | ttsmotions        | table | ly
(10 rows)

Database container

Build image
```
$ ./docker/dbdata/scripts/build-image.sh
```
Open another terminal, and run:
```
$ ./docker/dbdata/run.sh
```
- Backup the database
```
$ ./docker/dbdata/scripts/backup.sh # save to backup/dbdata.tar
```
- Restore the database
```
$ ./docker/dbdata/scripts/restore.sh
```
  And restart postgres container
Please backup the database first before stopping database container running.

Delete the images

$ docker images
REPOSITORY  TAG     IMAGE ID      CREATED         VIRTUAL SIZE
lyapi-xxx   latest  569514b1aad4  42 minutes ago  1.439 GB
$ docker rmi 569514b1aad4
Deleted: 569514b1aad4...

or simply:

$ docker rmi $(doc images | grep lyapi- | awk '{ print $3 }')

Windows or Mac
1. Install Virtualbox 4.3.x
2. Install Vagrant 1.4.x
3. Install berkshelf:
```
$ sudo gem install berkshelf
```
4. Install vagrant plugins:
```
$ vagrant plugin install vagrant-berkshelf
$ vagrant plugin install vagrant-cachier
```
5. Run
```
% cd cookbooks/ly.g0v.tw/
% vagrant up
```
  You should now have localhost:6988 served by pgrest within the vagrant, try to access http://localhost:6988/v0/collections/sittings

Host

Besides Vagrant, of course you can run a api server in your host.

the server provides RESTFUL service by pgrest. pgrest rely on postgresql, so you should install postgresql and related components to your host.

For example, in Debian/Ubuntu

Create /etc/apt/sources.list.d/pgdg.list, add this line:

Replace the codename with the actual distribution you are using.

examples (distribution -> codename):

Debian 7.0 -> wheezy

Ubuntu 14.04 -> trusty
```
deb http://apt.postgresql.org/pub/repos/apt/ codename-pgdg main
```

Import the repository key

$ wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
$ sudo apt-get update

Install the packages.

$ sudo apt-get install postgresql-9.3
$ sudo apt-get install skytools3 skytools3-ticker postgresql-9.3-pgq3  # for pgq
$ sudo apt-get install postgresql-9.3-plv8  # for plv8 extension

Preparation
```
$ sudo su postgres
$ export PLV8XDB=ly
```

init

% npm i

Then refer to the cookbook to initialize your postgresql.

Bootstrap with the initial dump file:

% createdb ly
% psql ly -c 'create extension plv8'
% lsc app.ls --db ly --boot
% curl -SsL 'https://www.dropbox.com/s/6j0zlzlubq8it5u/api.ly.bz2?dl=1' | bzcat |  psql ly -f -

run pgrest

$ lsc app.ls tcp://ly:password@localhost/ly
or
$ lsc app.ls tcp://ly:password@localhost:5433/ly    # if your postgresql is running on port 5433

pgrest will bind a local port to serve

data flow

fulltext from gazettes

(not automated yet)

ivod clip metadata

populated with update-ivod.sh

TTS data (WIP)

National Parliament Library provides a database called TTS. to work with it you'll need to have lisproxy.psgi up and running, and install phatomjs and casperjs to use scripts/tts.coffee

% sudo apt-get install cpanminus
% cpanm Plack::App::Proxy
% npm i -g bower
% npm i -g casperjs
% npm i -g phantomjs
% bower install jquery

% plackup lisproxy.psgi &

bill and motion metadata

populated with populate-ttsmotions.ls and populated-ttsbills.ls

interpellation

populated with populate-ttsinter.ls

API listing for api.ly

http://docs.twly.apiary.io/

To regenerate api, just run docgen.s then commit apiary.apib, the document will hook to apiary automatically

cd ~/api.ly
lsc docgen.ls
git commit apiary.apib

License

MIT: http://g0v.mit-license.org

g0v / api.ly

readme