Indikatorenportal Statistisches Amt Basel-Stadt

Find and display statistical indicators from the canton of Basel-Stadt, Switzerland. See live version here.

Update charts from ftp server

• Run the following command in a terminal window:

  npm run deployNewCharts
  npm run build

• Build, commit and push as explained further down.

Create png/pdf files of each chart within an indikatorenset

• In Chrome, open print.html?Indikatorenset=indikatorensetname or print.html?Indikatorenset=indikatorensetname&type=pdf
  • use keyword view to decide if indicator-title is included
    • &view=print: no title
    • &view=portal: title
    • &view=indikatorenset: number and title
• Chrome will download a png/pdf file of each chart in the given Indikatorenset to the local downloads directory. You can then manually move them to their target folder.
• It uses a Highcharts Node.js Export Server deployed on the StatA-Server pdstatasvpapp05
• To preview single charts in print view, use chart.html with the url parameter "view=print", e.g. chart.html?view=print&id=5824.
• To download a single chart as pn/pdfg, use chart.html?thumbnailOfflineExporting=false&thumbnailType=png&view=print&exportServer=https://highcharts-export.stata.pd.intranet.bs.ch/&id=[chart-id]&thumbnailType=[pdf/png]

Manually create svg thumbnails

• For the portal view: In Chrome, open thumbnails.html
• For the indikatorenset view: In Chrome, open thumbnails.html?view=indikatorenset This will download all svg files to the local downloads directory. You can then manually move them to their respective directory below /images/.

Local Installation

Install node.js, then run the following command in the console:

npm install

Getting Started

Locally build, start http server, open browser:

npm start

Get Charts from "Umweltbericht beider Basel"

• In Indikatoren-App (Access) go to > "Spezialtabellen" > "Umweltbericht Indikatoren" and run:

  • Metadaten einlesen (gets Metadaten of all Indicators from UB-Webpage)
  • Metadaten abgleichen (Imports Metadata to local Indicators DB)

• Set all indicators to status "Bereit für Live" and click "publizieren" to export all JSONs with correct "zuletzt geändert" date

• Github: Create new issue and branch

• VS Studio: pull, checkout branch

• Copy metadata (JSON) and tsv files (only for incdicators where data is not copied from Umweltbericht) to branch as described above in section "Update charts from ftp server".

• In the command line: run

  npm run build:create_umwelt_charts
  npm run build:clean_umwelt_charts

  This uses casperJs / phantomJs to open all charts of Indikatorenset "Umwelt" and save their Highcharts configuration in charts/configs/portal. If a Umwelt chart metadata's datenInChartIntegriert is: 1) false, the chart's Highcharts config is cleaned and saved as js file in charts/templates/[id].js (This allows using a Umwelt chart Higcharts configuration with data from an externally provided tsv (manually uploaded, default for all other indicators)). 2) undefined or true: its tsv file is downloaded ftom UB-website and saved to data/[id].tsv (This allows retrieving the tsv file from the Highcharts hamburger menu.)

• Create svg files by running the command

  npm run build

• run local-server, check everything

• add, commit, push

• add branch/issue-nr to all indicators in the umwelt-set via StatApp

• set all indicators to status "bereit für Live" in Indikatoren-App

• (inform git-admin about update)

URL Parameters

• Most of the URL Paramaters outlined below can be used in combination.
• Use ? to separate server + document from the list of parameters, then & before 2nd, 3rd (and so on) oparameter in the URL. See examples for stufe below.
• Filter parameters can be used even if the respective filter element is hidden.
• Url encoding of filter parameter values is automatically performed by the browser, just type in the value in the browser's url bar and hit enter.

Development

Add or Update Data

Copy the data as tab-separated tsv file named id.tsv into the folder 'data'. Refresh browser.

Add or Update Metadata

• Copy the json file named id.json into the folder 'metadata/single'.
• On test system (https://github.com/statabs-test/indikatoren), the master branch is automatically built and deployed to github pages via travis-ci - thus no local build necessary.
• For a local build: Rebuild the project to regenerate the json databases for indikatorensets (in folder 'metadata/sets') and for all indikatoren (file metadata/portal/indikatoren.js):

  npm run build

Add or Update Chart Configurations

• Copy the chart configuration file named id.js into the folder 'charts/templates'. The file needs to contain a self-calling function that returns the Highchart configuration details that differ from the chart's template. At least the "series" node is required.
• On test system (https://github.com/statabs-test/indikatoren), the master branch is automatically built and deployed to github pages via travis-ci - thus no local build necessary.
• For a local build: Rebuild the project to regenerate metadata databases, final chart configurations, and svg thumbnail images for all indicators:

  npm run build

Create Kennzahlensets for Print: Copy Data and Chart Configurations

• If a new kennzahlenset is created for print, it can be initialized with data and chart configurations of their respective parent kennzahlenset.
• To do this, first upload the metadata files to metadata/single.
• Then run

  npm run build:init_print_charts

• This will check each metadata file for matching data/[id].tsv and charts/templates/[id].js. If tsv or js file is missing, it will be copied from the parent chart's file.
• This is an excellent starting point to refine charts for printing.

Build Application Locally

To build the application, create the json config files and the svg images of the charts that have changed since the last build:

npm run build

To build the application and rebuild all json config files and all chart's svg images:

npm run rebuild

Manually create svg thumbnails:

• For the portal view: In Chrome, open thumbnails.html.
• For the indikatorenset view: In Chrome, open thumbnails.html?indikatorensetView=true. This will download all svg files to the local downloads directory. You can then manually move them to their respective directory below /images/.

What does the build script do?

The build script does the following things:

• npm run build:database: Creates metadata database files to be used in portal view:
  • Loads each chart's metadata file from metadata/single/[id].json and evaluates if it is visible (visible == true) and/or visible in portal view (visibleInPortal == true).
  • Metadata of each charts that has property visible == true gets copied into the file metadata/all/indikatoren.json
  • A chart gets the property visibleInPortal set to true if it:
    • has property visibleInPortal set to true or undefined, and
    • is not member of kennzahlenset "Test", "-print", and
    • has no parent chart of which the metadata file is present or has a parent chart and links to that chart because it is e.g. the same data but presented differently.
  • Metadata of each chart with metadata visibleInPortal == true gets copied into the file metadata/portal/indikatoren.json.
  • Files metadata/all/kuerzelById.json, idByKuerzel.json, templateById.json, all.md are created. Those can be used by humans to quickly look up id, kuerzel, kennzahlenset, visible, visibleInPortal and template of each chart.
• npm run build:find_changed_charts: Checks which charts must be rebuilt since last build:
  • Calculates hash code of each file in data/, metadata/single/, charts/templates/ and compares it to the hash code after the previous build (created by npm run build:save_checksums). If a chart's data, metadata, config file or template config have changed since last build, its id is added to file tmp/chartsToBuild.json.
• npm run build:partial_databases: Creates a json database for each kennzahlenset:
  • Runs through all files metadata/single/*.json and adds charts that have property visible == true or visible == undefined to a file metadata/sets/[kennzahlenset].json. This file is loaded by index.html when parameter ?indikatorenset is specified, instead of the larger file metadata/portal/indikatoren.json.
• npm run build:charts: Creates Highcharts config file for each chart:
  • Runs through each chart in tmp/chartsToBuild.json to create a highchart config file charts/configs/portal/[id].json by combining the chart's from data/[id].tsv, metadata from metadata/single/[id].json, template from charts/templates/[id].js and template's config from charts/templates/[template].js.
• npm run build:images: Creates the svg file for each chart to be used as a preview:
  • Runs through each chart in tmp/chartsToBuild.json to create a file images/portal/[id].svg from its Highcharts config file charts/config/[id].json.
• npm run build:images_viewbox: Adds viewbox to the generated svg files so that IE displays them nicely.
• npm run build:optimize_images: Optimizes svg files so that they are smaller.
• npm run build:save_checksums: Saves checksums of all files to metadata/all/hasesAfterBuild.json so that during next build, npm run build:find_changed_charts can find changes.
• npm run build:copy_modules: Copies npm modules that are required for the live website (defined in packages.json, key "dependencies") to assets/js/modules.
• npm run build:copy_data_per_set: Copies tsv files to a folder defined by the chart's kennzahlenset (data/sets/[kennzahlenset]/[id].tsv)
• npm run build:deployNewCharts: Downloads data, metadata and config for charts that are ready to be updated:
  • It downloads the file https://indikatoren.statabs.ch/verzeichnis.txt which contains all charts that are ready to be updated in csv format with columns "indikator", "branch".
  • For each chart that is ready to be updated, it downloads its respective tsv and metadata json file from https://indikatoren.statabs.ch/
  • If a branch name ist given for a chart, it retrieves that chart's js file through git from origin/[branch name].

    Update dependencies

    Update version numbers in package.json, then run the following command to do a clean reinstall:

    npm run reinstall

Develop in a private github repository

• Develop in a private github repositoriy as you would do in the public repo: Create an issue, create a branch called 'issue-XXX' with XXX being the id of the issue.
• When it's time to release the new functionality onto the public repository:
  • In the private repo, define the public repo as a remote repo named "upstream" (of course use the correct https url, not the following dummy url. Use e.g. the address from the browser and add '.git' at the end):

    git remote add upstream https://github.com/user/indikatoren.git

  • Add, commit and push your changes to the private repo as usual.
  • Create a branch with the same name on the public github repository ('upstream').
  • Pull the latest commits from the public branch to the private branch:

    git checkout issue-XXX
    git pull upstream issue-XXX

  • If there are conflicts (public and private branch have changes in common files), manually resolve the conflicts, then add, commit and push the resolved conflicts to the private branch.

    git add .
    git commit -m"Merge upstream"
    git push

  • Push your changes to the branch 'issue-XXX' on the public repo:

    git push upstream issue-XXX

Develop using cloud9

• Create new hosted workspace based on the node.js template and the correct github repo. Use the SSH repo path.
• Run the following command. This will set node.js version to 6, install true type fonts, and install the application.

  ./c9-setup.sh

• Close the bash terminal and open a new one: click on the + symbol, choose "New Terminal"
• Enter the following command in terminal and press enter:

  npm run reinstall

• Run application on c9: Click "Run", "New Run Configuration...", click "Runner", click "Apache httpd". Click into the field "Run Config Name" and type "Apache httpd" to give this configuration a name. Now click Run, then click the url displayed in the console log: https://<c9-vm-name>-<c9-username>.c9users.io
• To make this runner configuration the default, right-click the green "Run" button in the menu bar, click "Manage...", click "Set as Default". Now, this runner is always invoked when you click the green "Run" button.

How to create new charts

• Upload the new metadata json file to metadata/single/[id].json
• Upload the tsv file to data/[id].json
• Copy an existing chart definition file to charts/templates/[id].js, or create a new chart definition file that contains:
  • a self-invoking javascript function that returns a javascript object, which contains:
  • an array named "series" which contains one object for each column in the tsv that needs to be used in the chart (starting with the 2nd column in the data file, no object necessary for the first column),
  • all deviations from the chart template file that is defined in the chart's metadata file.
• Edit chart-dev.html to use the chart id and template file for your current chart
• Load chart-dev.html in the browser, and check if it matches your requirements. If not:
• Adapt your chart definition file [id].js until it matches your requirements by following the Highcharts API Documentation.
• To get ideas check e.g. this book.
• To see which charts are based on which templates: check metadata/all/templatesById.json
• To add the chart to portal and/or indikatorenset View: Build the application
• If you decide not to copy and adapt an existing chart, choose from the following existing templates (click onto image to open interactive chart):
  • area001: e.g. 6548:
  • befragungen001: Use for survey results if numbers are given as counts:
  • befragungenProzent001: Use for survey results if numbers are given in percentages:
  • bubble001:
  • dotplot:
  • line001:
  • map001: Wohnviertel Choropleth Map with Ranks read from data file displayed in tooltip
  • map002: Simple Wohnviertel Choropleth Map without ranks and with simple tooltip
  • mapcolumn002:
  • mappie001:
  • pie001:
  • populationPyramid001
  • stock001: Time axis, mini chart to filter, e.g. 4132:
  • template001: General Purpose template to create your own charts. Most bar, column or combination charts are based on this template.

Licensing

Highcharts is free for personal, school or non-profit projects under the Creative Commons Attribution - Non Commercial 3.0 License. For commercial and governmental websites and projects, you need to buy a license. See License and Pricing.