Server and Docker configuration for OB.eu components

[pwalsh]

Description

All the custom components for OpenBudgets.eu will run out of Fraunhofer servers, while the OpenSpending components are much more generic and run from servers run by Open Knowledge International.

In order to simplify to setup of the ob.eu-specific components, it is proposed to orchestrate them using Docker.

Tasks

• [ ] Containerise all required components (@pwalsh recommends using Alpine Linux as the base image, and OKI can give any advice on the use of Docker if required)
  • [x] LinkedPipes
  • [x] 2 x Triple Stores
  • [x] NGINX server
  • [x] url rewriting for https://github.com/openbudgets/platform/issues/2
  • [x] reverse proxy for Rudolf
  • [x] access control for triple stores, linked pipes
  • [x] Rudolf (API server in php)
  • [ ] Alignment UI
  • [ ] File server for static data mining outputs
  • [x] Custom UIs (Viewer running on RDF API is the only one right now)
• [x] Build orchestration config, using Docker Compose and or other tools link Ansible, Fabric or Salt (as required)
• [ ] Work out the required integration points that will require environment variables and/or other configuration for communication across components
• [x] Work out the server requirements and get those servers running and accessible
• [x] Make an initial deployment of the OB.eu components required for WP4

[schmaluk]

Hi @pwalsh , have added https://github.com/openbudgets/integration Are there any CI-Tools which you can recommend for this project? Ainsible (over/or in combination with) Jenkins or alternatively gitlab-ci. If we have selected a CI-tool I can then use it for integrating all components. It is not an urgent matter: But if Jenkins is ok, Jenkins can be made accessible via ssh Tunneling or alternatively via https & protected with Basic Auth by nginx. Also the other tools such as LinkedPipes can be then made accessible via https & Basic Auth. That would simplify the work for some of my colleagues. Then they dont have to use ssh-tunneling anymore in order to access Linked Pipes and possibly other Tools on the platform. In this case probably a certificate would be needed.

[pwalsh]

@mlukasch I do not see any point in setting up our own Jenkins server, when we can use Travis for free, and have less headaches. At @okfn we use it extensively.

[pwalsh]

@mlukasch also: why do we need to use GitLab?

[schmaluk]

@pwalsh if all of our components will be open-source, then github & travis will be fine. gitlab has a CI-tool: gitlab-ci which works only in conjunction with gitlab. Gitlab can be Setup in way that it mirrors the github-repo. gitlab-ci would be an alternative to Jenkins.

[pwalsh]

I'm very familiar with GitLab and GitLab-CI.

What is the value you are providing by dealing with them now? We have so many actual moving components we need to setup in the OB.eu stack, all of which will work with existing, free, web-based tools for code hosting and CI.

Of course, you are free to choose to work on that, but I'm not convinced it is the right effort at this stage: see the list above with the other components we actually need to work on and orchestrate.

[schmaluk]

@pwalsh, by "access control for triple stores, linked pipes" do you mean mainly restricting the access to specific ip-addresses? LinkedPipes has also a frontend which is used by @fathoni .Currently he uses ssh for accessing the frontend which could be replaced by https and Basic Auth. Would it be okay & make sense to spend some time on enabling https and serving the Frontend of LinkedPipes via https and Basic Auth. (Maybe not a real problem, if we dont do this.)

[schmaluk]

Currently deployed & containerized:

• Rudolf (not yet connected to the triplestore on the server)
• Production Triplestore
• Staging Triplestore
• nginx as reverse proxy
• linkedpipes.

Everything on the public server. We have 2 servers. 1 publicly accessible and 1 internal one. Currently I have installed all components (especially both of the triplestores) just on the public server. Can leave it like this for the moment, if that is okay. And later split everything up and distribute on 2 servers. If that is not recommended, which components should be installed on the private one? Was it just the staging triplestore? Still missing at this point:

• Fdp2Rdf transformer
• Alignment UI
• OS-Viewer using Rudolf
• Minor: Access control of the Triplestores, LinkedPipes is done just via ssh-tunneling (Can be done via ssl and password protection)

Overview of the Servers:

• Public Server: 16GB Ram, 1 TB storage, 4 cores, Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz
• Internal server: 12GB Ram, 1TB storage, 4 cores, Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz

[pwalsh]

@mlukasch I suggest you connect directly with the various people involved with the components you are missing, to built a timeline.deployment plan for integrating those.

[schmaluk]

@pwalsh @akariv Im trying to connect the OS-Viewer now to a working Rudolf on our OBEU-server. Rudolf is exposed on http://eis-openbudgets.iais.fraunhofer.de/api/3/cubes Im using this image: openspending/os-viewer And this environment variables in docker-compose.yml-file:

• WORKDIR=app
• PORT=8000

  For dev:

• OS_EXTERNAL_ADDRESS=http://localhost
• OS_VIEWER_API_HOST=http://localhost

  For prod:

      #- OS_EXTERNAL_ADDRESS=http://eis-openbudgets.iais.fraunhofer.de
      #- OS_VIEWER_API_HOST=http://nginx:80

• OS_VIEWER_AUTH_HOST=http://next.openspending.org
• OS_VIEWER_SEARCH_HOST=http://next.openspending.org That means that everything except the API (Rudolf on OBEU-side) will be used from OpenSpending. Is that correct? How do I build the URL for example to a treemap in order to test the Viewer with Rudolf in the backend? The OS-viewer is deployed: http://eis-openbudgets.iais.fraunhofer.de/viewer/ And we have a dataset with the name: budget-thessaloniki-expenditure-2011__803aa according to: http://eis-openbudgets.iais.fraunhofer.de/api/3/cubes

[larjohn]

In OS Viewer's app/config/index.js, you can set

const DEFAULT_HOST = 'http://eis-openbudgets.iais.fraunhofer.de';

This should suffice.

[schmaluk]

Thanks @larjohn for the help. It is deployed now with that change: http://eis-openbudgets.iais.fraunhofer.de/viewer/ Though Im not sure, how to access visual components backed by Rudolf in order to verify.

[pwalsh]

@mlukasch @larjohn looks like the datapackage is pulled from openspending, not from the triple store api ( http://eis-openbudgets.iais.fraunhofer.de/viewer/ )

[schmaluk]

@pwalsh @larjohn my fault. Had still the os-viewer-image in the compose.yml... Now Im getting a 404 for GET http://eis-openbudgets.iais.fraunhofer.de/permit/lib on http://eis-openbudgets.iais.fraunhofer.de/viewer/ This /permit/lib is the os-conductor-app which is not hosted by our server.

[pwalsh]

@mlukasch

• https://github.com/openspending/os-viewer/blob/master/app/config/index.js#L14
• https://github.com/openspending/os-viewer/blob/master/app/config/index.js#L34

( right @akariv ? )

[akariv]

Yes, /permit/lib is the authentication library which allows logging-in to the server. You could use the one hosted on next.openspending.org (so that it loads successfully) using the pointers that @pwalsh gave, although it may not be enough to allow logging in in your setup.

On Thu, 7 Jul 2016 at 15:51 Paul Walsh notifications@github.com wrote:

@mlukasch https://github.com/mlukasch

- https://github.com/openspending/os-viewer/blob/master/app/config/index.js#L14

https://github.com/openspending/os-viewer/blob/master/app/config/index.js#L34

( right @akariv https://github.com/akariv ? )

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openbudgets/platform/issues/1#issuecomment-231068105, or mute the thread https://github.com/notifications/unsubscribe/AAQMdT1t5cB3haMtliguOBKCyirvREl-ks5qTPY4gaJpZM4IumrG .

[schmaluk]

@akariv @pwalsh Bit better now. So I have changed:

• OS_VIEWER_AUTH_HOST=http://next.openspending.org
• OS_VIEWER_SEARCH_HOST=http://next.openspending.org and the Default_host is: http://eis-openbudgets.iais.fraunhofer.de with this result: http://eis-openbudgets.iais.fraunhofer.de/viewer/ So the search for datasets is done on OS-side by the conductor, since there is no search app on our side afaik.

[larjohn]

There is a route in rudolf for searching. I am not saying it currently does anything that useful, still it should populate the dropdown correctly:

Route::get("search/package", "SearchController@index" );

This is what I see in my own config for the search endpoint:

  search: {
    url: searchHost + '/search/package'
  },

where searchHost is:

var searchHost = process.env.OS_VIEWER_SEARCH_HOST || DEFAULT_HOST; which in turn should lead back to rudolf.

[schmaluk]

Hello, there are some overdue issues listed above, which have not been resolved yet.

1) Access control for triple stores, linked pipes: Currently this is done just by SSH Tunneling in order to access the Admin-Frontend of Fuseki, LinkedPipes (Although this has some disadvantages due to the security policy @FhG, I would consider this as done.)

2) Alignment UI: Postponed due to minor priority.

3) FileServer for static data mining outputs: Postponed. Data Mining is not in the stage where it generates any output. But can be added, if it is needed.

4) Work out the required integration points that will require environment variables and/or other configuration for communication across components. Some Integration endpoints have been already defined and connected to each other, for example:

• OS-packager to FhG server for triggering the FDP-to-RDF-Pipeline: endpoint defined & provided
• FDP-to-RDF-Pipeline connected to the staging triplestore
• staging triplestore connected to the production triplestore for the time being by copying the underlying database-folders (This should be ideally replaced by a Pipeline in LinkedPipes later on defined by the datasets experts.)
• Rudolf connected to the production triplestore
• OS-Viewer connected to Rudolf

Some other integration endpoints between OBEU components are atm more of a continous task to be defined & still in progress and can be maybe resolved better by the experts working on those Topics. As I recall from our meeting today for example in datamining:

• Required CSV format output for datamining input
• Common Datamining-backend-to-Frontend Interface

Will provide the Integration as soon as a Task here has become clear & the component is in the stage to be integrated.

[pwalsh]

@badmotor I'm closing this. It must be out of date, and any specific issues can be raised with distinct issues.