search

guardian / giant

Platform for journalists to search, analyse, categorise and share unstructured data
Apache License 2.0
53 stars 3 forks source link
production

readme

Giant

Giant makes it easier for journalists to search, analyse, categorise and share unstructured data. It takes many file formats, indexes them (including converting images to text using OCR) and provides a UI for search. Users can upload their own files but it also scales up to terabytes of data.

Screenshot of Giant search

Giant is part of the Guardian's "Platform for Investigations" suite, you will see references to pfi in the code. Under development since 2017, it's written in Scala and Typescript and is maintained by the Investigations & Reporting team.

If Giant doesn't fit your needs, check out Aleph from the OCCRP and Datashare from the ICIJ.

(Users) Getting started

(Developers) Getting started

Giant has the following pre-requisites for local development:

Giant uses three databases, run locally in Docker through docker-compose.yaml:

There are two optional dependencies:

Elasticsearch requires Docker to have at least 4GB of memory from the preferences menu otherwise it will exit with no log output and error 137.

For Guardian developers:

Select the correct version of node:

nvm use

Then run the setup script:

./scripts/setup.sh

Seed the configuration:

./scripts/cluster-setup.sh

Run the Scala backend:

./scripts/start-backend.sh

This will also automatically launch the databases in the background by running docker-compose up -d.

In a separate terminal, run the Create React App frontend:

./scripts/start-frontend.sh

The frontend script will wait for the backend to start before launching Giant at http://localhost:3000.

Once Giant has started, follow the admin quickstart guide.

dev-nginx proxy

You can use dev-nginx to more easily access Giant and the backing databases whilst running locally.

dev-nginx setup-app util/nginx-mapping.yml

Running Tests

To run all unit tests:

sbt test

To run all integration tests:

sbt int:test

To run a specific integration test:

sbt 'int:testOnly controllers.api.WorkspacesITest'

Stopping databases

To terminate the databases without losing data:

docker-compose down

To terminate and delete data:

docker-compose down -v

Contributing

The Guardian welcomes contributions to Giant. We do not yet have a publicly accessible CI server but please ensure all tests pass by running the build script locally:

./scripts/teamcity.sh

We do not yet publish deployment templates for Giant in either cloud hosts or locally. If you are interested in deploying Giant please get in touch by raising a GitHub issue on this repository.

Architecture

architecture diagram for uploading files

https://docs.google.com/drawings/d/1wcTY9KLhkYqxmwzsyZ3DsWcc0v-ax5kMKWtYb4HZgF0

Licensing

Giant uses the Apache 2.0 licence. Some libraries used are licensed separately:

Supported file formats

Experimental features

Experimental features are enabled through feature flags in the Settings page:

Credits

In addition to any contributors named in this repository, the following contributed to Giant whilst it was closed source at the Guardian: