search

Eaternity / eaternity-database-public

Source code of Eaternity's DBMS solution based on GitHub, Jekyll, Heroku and Prose
12 stars 1 forks source link

readme

Eaternity Database User Manual

Access to the current running instance of the Application can be found on http://my-edb.herokuapp.com. There are three tools available: Products DB, Nutrition DB and EDB-Prose. How to work with the Eaternity database:

  1. Make sure you have i) a GitHub account and ii) sufficient rights to access the data stored in in https://github.com/Eaternity/eaternity-database-public -> Master Branch -> _data/prods or _data/nutrs -> repository
  2. Open the Jekyll-DB overview on all products/nutritions. You can search any text displayed on the page and sort by columns. Click on the product/nutrition you want to edit.
  3. If you are already logged in with your GitHub account, the file is displayed as a form. If you are not yet logged in, a page will appear telling you to please AUTHORIZE ON GITHUB. You will be redirected to a GitHub page where you will be asked for your GitHub username and password. If you do not have sufficient rights to access the linked data, you will receive an error page telling you that there is no such resource/web page available.
  4. Edit the data in the input form and save your data using the "Save" icon on the right. You will have to provide a short text explaining your changes before saving.

Adding new products or nutrition files

  1. Open Prose.io. If you are not yet logged in to your Github page, do it now.
  2. Navigate to the repository (e.g. eaternity-database-public) where you want to add a file to. (There is a dropdown menu on the right upper side of the page where you can switch branches.)
  3. Go to the _data/prod directory. You will see a list of all existing products.
  4. Before you continue, decide which id you want to use for this product. The id should be a positive number not yet given to any other product. This is is an important step: Setting an invalid id can lead to a confusion between products.
  5. Click on the New File button.
  6. Set the filename according to this pattern: <id>-<name>-prod.json.
  7. Delete all existing text from the file. Add this basic json code: { "id": <id>, "name": "<name>". Make sure you put the name in double quotes ("") but not the id.
  8. On the right of the page, click the "Save" button. You will be asked to provide a short text explaining what you did. Confirm your changes and save.

Eaternity Database Technical Documentation

The Eaternity Database is a collection of files and tools.

  1. File Structure: Product and nutrition data is stored in SON data format in a non-public GitHub repository in the master branch. There is one .json file per product/nutrition.
  2. Jekyll-DB: Jekyll-DB is used to display an overview of all ingredients to the user. The user can search for keywords and select the file to edit.
  3. Prose: Data is edited in an adapted version of Prose. Thus, the full advantage of GitHub's versioning system is offered to the users.

This is a schematic overview on the whole EDB solution: EDB Overview There are three different physical locations involved: Heroku.com, GitHub.com und the local computer. Whereas the website's content is stored in a repository on GitHub.com at https://github.com/Eaternity/eaternity-database-public, the website itself is hosted on and served from an app running at Heroku.com under the URL https://my-edb.herokuapp.com. When new changes are pushed to the repository on GitHub.com, the Jekyll server running at Heroku.com is automatically notified, the website is rebuilt and after a few minutes the changes are reflected to the user. The GitHub repository is private. GitHub's gh-pages branch is never used, because everything published under a gh-pages branch is accessible to the whole world. The repository belongs to a GitHub organization Eaternity. This organization has a team edb-website-users. All members of this team have admin access to the eaternity-database-public repository, and only members of this team will be allowed to access the website and read or write data through it.

Workflow

The website including all files are stored in a GitHub repository available in the master branch. The website consists of three different sets of files - all more or less mingled together:

  1. Data: Data is kept inside the _data/prod, _data/nutr etc. directories.
  2. EDB Prose: EDB Prose is an adapted version of prose relying on JSONForm to display user forms. For more information, see further below. Although all the code is inside the repo's prose directory, the file prose/index.html expects a few other Javascript files to exist one level above it, i.e. in prose/../...
  3. Jekyll-Auth: Jekyll-Auth is used to control access to the website hosted on Heroku.

When a user makes any changes to the local clone of the GitHub repository, he then adds git add..., commits git commit -m "blah" and pushes git push these changes to the remote GitHub repository. A webhook is available for the eaternity-database-public repository, and it is through this webhook that the Jekyll server running on Heroku is informed about pending updates. The Heroku server process grabs the changed files from the GitHub repository and rebuilds the whole website. (The whole process usually takes several minutes.) This automatic deployment from GitHub's repo to Heroku's repo is set up using a GitHub integration feature for Heroku apps. To see the current settings, log in to the Heroku Dashboard as <owner of my-edb Heroku app> and click on the Deploy menu item. (The GitHub integration feature is currently in beta status. An older, but still working alternative would be to use Heroku deployer, but this is more difficult to set up).

The Heroku server process has its own Git repository available under code>git@heroku.com:my-edb.git</code, but this repository should not be pushed to with git push heroku master in most cases. The reason is that the corresponding "sibling" repository on GitHub eaternity-database-public will of course not be notified of any changes made to Heroku.

Authorization and Authentication

When a user tries to access the website available under https://my-edb.herokuapp.com several things happen in sequence.

  1. First, Jekyll-Auth running on Heroku tries to access a local session cookie on the user's computer. If there is no such session cookie, the user is forwarded to a GitHub website and asked to provide his GitHub username and password. Next, if never done before he must explicitly authorize the Heroku website process to access this repository on his behalf, i.e. using his username and password.
  2. GitHub checks to see whether this user is a registered member of the team edb-website-users or not and returns this information to the Heroku server process. If ok the requested web content is displayed to the user. If not ok then the user sees a 403.html "access denied" page (which may look different in different browsers). If someone forgot to add this user to edb-website-users, the user must first delete his cookies and close the browser down before attempting again to open the website - otherwise the authorization/authentication will fail again!
  3. The user now browses through the website's content. At one point, he clicks on a product file link, e.g. "1234-blah-prod.json". This link effectively resolves to an EDB Prose page, itself requesting access to the file Eaternity/eaternity-database-public/_data/prod/1234-bla-prod.json in the GitHub repository. A second authorization/authentication is triggered at this step, which is independent of the first one performed earlier. Whereas through the first authorization/authentication procedure website access is controlled, the second controls access of the user to any data file on the GitHub repository through EDB Prose. Therefore, first the edb-prose-gatekeeper app requests the user's credentials, which are by now probably stored locally in a session cookie. These credentials are sent to GitHub, and GitHub returns ok/nok for this user.
  4. If ok, EDB Prose tries to load the requested data file in a JSONForm form. The user makes changes through the form and finally submits the data to the GitHub repository through EDB Prose. This again notifies the my-edb Jekyll server running on Heroku of changes, which ultimately leads to a website rebuild.

1. Repository File Structure

Every product is stored in the JSON data format in a file with a filename of <id>-<product-name>-prod.json. For nutrition files the filename is <id>-<nutrition-name>-nutr.json. All files are placed in a _data/prods/ resp. _data/nutr/ subdirectory in the gh-pages branch of the project. All YAML- or JSON-files placed in such a subdirectory can be accessed and looped over via Liquid templates:

{% for prod in site.data.prods %} ...do something... {% endfor %}.

The variable prod[0] will contain the filename (without leading path and without file ending), whereas prod[1] will contain the key/value-pairs specified in YAML or JSON.

It is now easy to build a HTML page containing a list of all ingredients with selected attributes: 

...  
<ul>
{% for prod in site.data.prods %}  
  <li>Filename: {{ prod[0] }}.json; name: {{ prod[1].name }}</li>  
{% endfor %}  
</ul>  
...

2. Jekyll-DB

Jekyll-DB is an open source and easy-to-use tool based on List.js to index, filter, search, sort and display html tables and lists. Basically, Jekyll-DB always processes a file's frontmatter, which might be prepended in either YAML or JSON format. The file itself must be in the _posts (or _data) directory and be recognized by Jekyll as a valid post, i.e. it must have a valid file ending for markdown (.md, .markdown) or html (.html, .htm) posts. From a file's frontmatter, Jekyll-DB builds an adaptable display table that can be searched and filtered by keywords and sorted by columns. It is also possible to place links in the display table e.g. to a open a given post item in a GitHub or Prose editor. Data columns can be specified, as can the number of displayed items. (This feature requires Jekyll-DB > v2.1.0.)

Although the official documentation does not state it explicitly, Jekyll-DB not only works with YAML/JSON-frontmatter of files placed inside a _posts directory, but also with YAML/JSON-files in a _data/prods/ (or _data/nutrs/) subdirectory. If a _posts/ directory is used, in Liquid templates frontmatter data can be accessed directly, e.g. {% for post in site.posts %} ...{{ post.myFrontmatterKey }}... {% endfor %}. For _data/prods/ subdirectories, additionally indices must be used to differentiate between filename at index 0 and frontmatter content at index 1: {% for prod in site.data.prods %} ...{{ edb[1].myFrontmatterKey }}... {% endfor %}.

If JSON files inside a _data subdirectory should be indexed, a Jekyll server instance >= v.2.1.0 is required. Indexing frontmatter only in a _posts directory also works with older Jekyll server instances. Re-indexing all -prod.json files on Github can take 30 seconds to a few minutes.

Adding or removing columns

To add new columns or remove existing ones, follow the steps described in Jekyll-DB's README file.

3. EDB Prose

Whereas Jekyll-DB allows to display data, Prose can be used to actually edit the underlying data. Prose is an open-source editor built on top of GitHub pages. There is a free-to-use online instance running at the given link, but it is also possible to run one's own Prose in any YAML-aware webserver such as Jekyll or a Node.js web server. We use an adapted version of prose which additionally supports files with file names ending on -prod.json or -nutr.json. Such files are expected to be in the EDB JSON data format. Opening such a file in the adapted version of Prose will automatically open a JSONForm editor window.

Installation manual for Prose

A user must be able to authorise access of Prose (or any descendant) to his/her Github repositories. For this purpose, besides installing and running Prose, also a gatekeeper server must be installed and run on Heroku. The basic steps are:

Step 1: Register GitHub application

GitHub applications such as Prose need to follow the OAuth procedure for authentication. Thus, we need to register a new OAuth application on GitHub. As a result, we will receive a clientId (e.g. da429541964e3302c62f) and a client secret (another string consisting of random numbers and letters). This is the information which will be required by the gatekeeper instance.

Step 2: Create Heroku account

In the case of Prose, a dedicated gatekeeper server is needed for the OAuth procedure with GitHub to work. This server is neither provided by GitHub nor can it be run on GitHub and therefore must be set up manually externally. For this purpose, we need a (free) repository on Heroku, there we will run our own gatekeeper server. We register a new account at Heroku. At this time, there is no need to set up a sample application.

Step 3: Install and run Prose gatekeeper server on Heroku

Next, we need to install our own gatekeeper server on Heroku. The installation steps are described in Prose's documentation on Gatekeeper. Here they are:

  1. Clone the gatekeeper server repository:
    git clone git@github.com:prose/gatekeeper.git
  2. Install the dependencies: cd gatekeeper && npm install
  3. (Hint: We will not use cake, therefore this step can be skipped. Nevertheless, here's an up-to-date version of how the config.json file should look like.) {
    "oauth_client_id": "da429541964e3302c62f",
    "oauth_client_secret": "XXX",
    "oauth_host": "github.com",
    "oauth_port": 443,
    "oauth_path": "/login/oauth/access_token",
    "oauth_method": "POST"
    }
     We do not want to run the gatekeeper server locally and directly continue to the deployment on Heroku.
  4. Because we are not going to use coffescript's cake command command, the next steps slightly deviate from Prose's instructions. Make sure you are in your local gatekeeper directory: 
    gatekeeper> heroku create  
gatekeeper> heroku apps:rename my-edb-prose-gatekeeper  
gatekeeper> heroku config:set OAUTH_CLIENT_ID=da429541964e3302c62f OAUTH_CLIENT_SECRET=XXX  
gatekeeper> git push heroku master

    In our case, we chose to use the name "my-edb-prose-gatekeeper" for our application, which will result in an url of http://my-edb-prose-gatekeeper.herokuapp.com. The client secret will not bo provided here but must be taken from GitHub's application registration procedure. The gatekeeper server should now be up and running. This can be checked in our Heroku account.

Step 5: Prepare and push Prose to the GitHub repository, project and branch

We are now ready to rebuild our adapted version of Prose, so that authorisation through gatekeeper runs smoothly.

  1. For this purpose, we need to adapt the file oauth.json: 
    {  
"api": "https://api.github.com",  
"site": "https://github.com",  
"clientId": "da429541964e3302c62f",  
"gatekeeperUrl": "https://my-edb-prose-gatekeeper.herokuapp.com"  
}

    Once we've made our changes, we rebuild prose.js/prose.min.js with the commands make clean and make. Note that the file oauth.json is added to .gitignore and will not be uploaded to the server. Instead, during the build process the file's contents are read out and appended to prose.js/prose.min.js. (Attention: Using the make command in Windows is cumbersome, as this presumes an installation of a make-tool, e.g. the one provided by cygwin. After installation, the make commands must be run manually. Still this somehow results in problems, because after these steps the modified Prose cannot be started with jekyll serve anymore, but only with npm's serve command. If using Jekyll, an error message is thrown: _jekyll 2.1.0 | Error: undefined method 'defaultproc=' for "this --- is correct":String.)

  2. Finally, we commit & push our adapted version of Prose to the gh-pages branch of our project repository (http://eaternity-agent.github.io/Eaternity-Datenbank/). We need to wait a few minutes until GitHub's Jekyll server is ready to serve our version of Prose under the url http://my-edb.herokuapp.com/.

4. Adding a new data type to Prose and JSONForm

Adding a new data type (e.g. -proc.json) to Prose and JSONForm requires the manipulation of several files. After the manipulation, Prose must be rebuilt using a make command.

Step 1: Extending Prose's functionality

Manipulation of Prose requires the adaptation of 4 files:

In case you would like to create dropdown menus in JSONForm, you will probably have to set up additional Javascript files similar to id-prod.js and id-nutr.js. Do not forget to import these Javascript files in Eaternity-Datenbank/prose/index.html. You can then refer to the list of product ids, nutrition ids or process ids inside the jsonformformat.js file as ID_PROD, ID_NUTR or ID_PROC.

Step 2: Rebuilding Prose

We have made changes to Prose, thus we need to rebuild it. For this purpose, the make tool is required. Unfortunately, installing make on a Windows machine is quite a pain, and I don't recommend it. Hint: There might exist an alternative to the make command now using gulp.

  1. Make sure you are located inside the eaternity-database-public/prose directory and that make is in your path. Call make clean to clean up first.
  2. Call make to rebuild Prose. There might be warnings, but there should not be an error.

Step 3: Adding data

Inside the _data directory, create a new subdirectory, e.g. data/procs. Therein create a first test file, e.g. -1-test-proc.json_. Add, commit and push the data files to the remote GitHub repository.

Step 4: Testing the input form locally

Before we push our changes to the server, we would like to try out the new input forms locally. Once we pushed our changes to the server, the new file will be available under the link http://github.com/prose/#Eaternity/eaternity-database-public/edit/gh-pages/_data/proc/-1-test-proc.json. Until then accessing the newly created data files will run through a local Jekyll server instance, thus the link is http://localhost:4000/prose/#Eaternity/eaternity-database-public/edit/gh-pages/_data/proc/-1-test-proc.json.

Step 5: Setting up Jekyll-DB

Next, we want to set up everything in order to present a Jekyll-DB overview page to the user. The set up is fairly simple.

  1. cd into your _includes directory. Create a copy of one of the existing files interactive-table-prod.html or interactive-table-nutr.html and name it interactive-tale-proc.html. Edit the HTML table inside this file to contain all the columns for Jekyll-DB to display, including the file link inside the HTML anchor tag. Also don't forget to adjust the valueNames variable inside the option Javascript object at the bottom of the file. Save and exit.
  2. cd back into eaternity-database-public's root. Create a copy of one of the existing files index-prod.html or index-nutr.html. In the file, replace all occurrences of prod or nutr with proc. Save and exit.
  3. Still in the root directory, create a copy of either output-prod.json or output-nutr.json and name it output-proc.json. The file must match the changes made in interactive-table-proc.html. Save and exit.
  4. If you want, you can also adjust the root directory's index.html file.

Now, if you restart the local Jekyll server with jekyll serve --host localhost the whole site is rebuilt. This might take a while. The new website is available at http://localhost:4000/index-proc.html resp. will be available at http://my-edb.herokuapp.com/index-proc.html respectively. Add, commit and push your changes to the remote GitHub repository and wait for the site to be rebuilt.

5. Installation of Jekyll-Auth

A complete installation guide to Jekyll-Auth can be found here: http://fabian-kostadinov.github.io/2014/11/13/installation-of-jekyll-auth/.