Javascript, Python and SQL instructions for using Data Package

User stories: As a Consumer I want to download a DataPackage’s data one coherent SQLite database so that I can get it easily in one form

As a Publisher I want to send a link to an SQL/SQLite Database user colleague about how to use my data so that they can grab it and start using it

As a Consumer [Javascript user] I want to load a Data Package from Javascript, so that I can immediately start playing with it.

As a Publisher I want to send a link to Javascript user colleague about how to use my data so that they can grab it and start using it.

Acceptance Criteria

[x] When I visit a showcase page there are instructions for Javascript users (that are easy to find and link to)
[x] When I visit a showcase page there are instructions for SQL users (that are easy to find and link to)

Tasks

[x] Install Python/SQLAlchemy/SQLite
- [x] Create template instructions (and test)
- [x] Embed in page
- [x] Make sure we can direct link to it!
[x] Install Node.js
- [x] Create template instructions (and test)
- [x] Embed in page
- [x] Make sure we can direct link to it!

Analysis

Instructions on using Data Package in JavaScript (Node)

To use this Data Package in JavaScript, please, follow instructions below:

Install datapackage using npm:

$ npm install datapackage@0.8.1

Once the package is installed, use code snippet below

const Datapackage = require('datapackage').Datapackage

async function fetchDataPackageAndData(dataPackageIdentifier) {
  const dp = await new Datapackage(dataPackageIdentifier)
  await Promise.all(dp.resources.map(async (resource) => {
    if (resource.descriptor.format === 'geojson') {
      const baseUrl = resource._basePath.replace('/datapackage.json', '')
      const resourceUrl = `${baseUrl}/${resource._descriptor.path}`
      const response = await fetch(resourceUrl)
      resource.descriptor._values = await response.json()
    } else {
      // we assume resource is tabular for now ...
      const table = await resource.table
      // rows are simple arrays -- we can convert to objects elsewhere as needed
      const rowsAsObjects = false
      resource.descriptor._values = await table.read(rowsAsObjects)
    }
  }))

  // see the data package object
  console.dir(dp)

  // data itself is stored in Resource object, e.g. to access first resource:
  console.log(dp.resources[0]._values)

  return dp
}

fetchDataPackageAndData('https://bits.datapackaged.com/metadata/core/finance-vix/_v/latest/datapackage.json');

Our JavaScript is written using ES6 features. We are using node.js v7.4.0 and passing --harmony option to enable ES6:

$ node --harmony index.js

Instruction on using Data Package in Python

In order to work with Data Packages in Python you need to install our packages:

$ pip install datapackage

To get Data Package into your Python environment, run following code:

import datapackage

dp = datapackage.DataPackage('http://bits.datapackaged.com/metadata/core/s-and-p-500-companies/_v/latest/datapackage.json')

# see metadata
print(dp.descriptor)

# get list of csv files
csvList = [dp.resources[x].descriptor['name'] for x in range(0,len(dp.resources))]
print(csvList) # ['constituents', 'constituents-financials']

# access csv file by the index starting 0
print(dp.resources[0].data)

Instructions on using SQL/SQLite in DataHub in Python

In order to work with Data Packages in SQL you need to install our packages:

$ pip install datapackage
$ pip install jsontableschema-sql
$ pip install sqlalchemy

To import Data Package to your SQLite Database, run following code:

import datapackage
from sqlalchemy import create_engine

data_url = 'https://bits.datapackaged.com/metadata/core/s-and-p-500/_v/latest/datapackage.json'
engine = create_engine('sqlite:///:memory:')
# to load Data Package into storage
storage = datapackage.push_datapackage(data_url, 'sql', engine=engine)
# to see datasets in this package
storage.buckets
# to execute sql command
storage._Storage__connection.execute('select * from data__data limit 1;').fetchall()
# desciption of the table columns
storage.describe('data__data')

openknowledge-archive / dpr-api