Crowd-sourced visual interpretation of on-demand satellite imagery
** Installation Requirements
*** Required
** Application Configuration
On startup, the CEO application reads a file called ~config.edn~ in the top level directory of this repository, which contains various user-configurable system parameters. A sample file called ~config.example.edn~ is provided with this repository to get you started. To begin, simply copy it to ~config.edn~. Then edit it according to the following sections.
Please note that you must replace all values surrounded with angle brackets (e.g., "
** PostgreSQL Database Setup
PostgreSQL needs to be installed on the machine that will be hosting this website. This installation task is system specific and is beyond the scope of this README, so please follow the instructions for your operating system and PostgreSQL version. However, please ensure that the database server's superuser account is named ~postgres~ and that you know its database connection password before proceeding.
*** PostgreSQL Server
To confirm that PostgreSQL is running as a server on your machine, you'll need to check its ~systmed~ or ~sysvinit~ process. To figure out which you're using on your system, run ~ps -p 1 -o comm=~. Once you know if you're using ~systemd~ or ~sysvinit~, you can run the following command:
sudo systemctl status postgresql.service
sudo service postgresql status
If it's not running, you'll need to start it by running:
sudo systemctl start postgresql.service
sudo service postgresql start
Finally, to ensure that your PostgreSQL server always restarts when your system reboots, you can run:
sudo systemctl enable postgresql.service
sudo update-rc.d postgresql enable
*** CEO Database
Once the PostgreSQL database server is running on your machine, you should navigate to the top level directory (i.e., the directory containing this README) and run the database build command as follows:
clojure -M:build-db build-all --dev-data
This will begin by creating a new database and role called ~ceo~ and then add the ~postgis~ and ~pgcrypto~ extensions to it. Next, the script will populate the database with the schemas, tables, and functions that are necessary for storing and processing ceo's data. Finally, it will load some default data into these tables that is necessary for the website to function properly. You can optionally load dev data with ~--dev-data~. This will initialize the DB with 3 users, an imagery source, and a project.
If you wish to use a live copy of the CEO database instead of the dev data and you have a ~.dump~ file, run the following command:
clojure -M:build-db restore -f /path/to/ceo/database/ceo-db-
*** Performance Settings
If you want to improve the performance of your Postgresql server, one way is to visit the [[https://pgtune.leopard.in.ua/][pgtune]] website and input your system settings. This will generate a set of configuration options that you can add to the postgresql.conf file in your system-specific Postgresql data directory.
If you would like to keep these settings separate for your own reference, you can add them to a file called "performance.conf" that you should place in your Postgresql data directory. You can then import these settings into postgresql.conf by adding this line to the end of that file:
include = 'performance.conf'
Note that the Postgresql server will need to be restarted for these changes to take effect.
** Environment Variable Setup
In order for your command line shell to find the programs installed above, you will need to update your environment variables. Under GNU/Linux, BSD, or MacOS X, you can set these through your shell's .rc or .*profile configuration files in your $HOME directory. In a Windows environment, you will need to set either the USER (for the current user only) or SYSTEM (for any user) environment variables under your system settings menu.
*** Step 1: Add JAVA_HOME
On a Unix machine using bash (e.g., GNU/Linux or MacOS X):
export JAVA_HOME=
On a Unix machine using csh or tcsh (e.g., *BSD):
setenv JAVA_HOME
On a Windows machine:
JAVA_HOME = C:\Program Files\Java\jdk-X.X.X
Replace X.X.X with the JDK version installed on your computer. In order to run CEO, your JDK version has to be either 11 or 17.
*** Step 2: Update PATH
On a Unix machine using bash (e.g., GNU/Linux or MacOS X):
export PATH=$PATH:$JAVA_HOME/bin
On a Unix machine using csh or tcsh (e.g., *BSD):
setenv PATH $PATH\:$JAVA_HOME/bin
On a Windows machine, add these entries to the PATH environment variable:
%JAVA_HOME%\bin C:\Program Files\PostgreSQL\X\lib C:\Program Files\PostgreSQL\X\bin C:\Program Files\7-Zip
Replace X with the PostgreSQL version installed on your computer.
** Configuration
Copy ~config.default.edn~ to ~config.edn~ and update any settings that you wish. The base set of defaults should be sufficient to run CEO in a development setting.
** Python dependencies
Use ~pip~ to install dependencies. The python / clj interop does not support virtual environments.
pip install -r requirements.txt pip install earthengine-api --upgrade
** Development Web Server
*** Google Earth Engine authentication
You will need a Google Earth Engine account to use the GeoDash. To apply for a GEE account, visit https://signup.earthengine.google.com/#!/. Once you receive an account, you can store local authentication information with:
earthengine authenticate
This command will open a window in your browser containing a key that you should copy and paste back into your terminal. You do not need to update the ~:gee~ section of ~config.edn~ if you use this method. If the above method is giving you issues, try following [[the official step-by-step instructions provided by Google][https://developers.google.com/earth-engine/guides/python_install#expandable-2]].
*** Modifying hosts file
Add the following line to your hosts file. This local URL should be used for local development in place of ~localhost~ due to CORS settings on CEO's Mapbox account.
127.0.0.1 local.collect.earth
*** Create accounts
To be able to crate accounts without having to send email for confirmation, set ~:auto-validate?~ of ~:mail~ to ~true~ in ~config.edn~.
There can only be one superuser account with ~user_uid~ of ~1~ ~adminstrator~ flag set to ~true~. Activating this flag for other users can cause their queries to sometimes not return the full result, as some of those queries assume only one root superuser to be existing.
*** Compile and run
To compile and run the web application, navigate to the top level project directory and run:
npm install npm run vite-dev
Now, in a separate terminal window (because webpack needs to be running in the background), run the following commands:
npm run server-dev
The website will then be available at http://localhost:8080 unless a port is specified. These can also be configured using the ~:server~ section in your ~config.edn~ file. When using the ~clojure -M:server start~ command (which is what ~npm run server-dev~ is an alias for) an http port can be specified with -p and an https port can be specified with -P. In dev mode, server-side exceptions will be displayed in the browser and JavaScript source files will be reloaded whenever you refresh the page.
Going forward you will usually only need to run ~npm run webpack-dev~ and ~npm run server-dev~ (still in separate terminals) to get your dev environment set up. Watch for updates to SQL files or ~package.json~ and run ~npm run build-db-functions~ or ~npm install~ respectively.
*** Checking for Reflection Warnings
From the top level project directory run:
clojure -M:check-reflection
This will emit warnings when reflection is needed to resolve Java method calls or field accesses. To resolve any that appear, add [[https://clojure.org/reference/java_interop#typehints][type hints]] to your code. Resolving all reflection warnings can improve system performance.
** Production Web Server
*** Sessions
It is very important to change the default ~:session-key~ in ~config.edn~. This key is used to encrypt user session data and should be unique to each deployment. The key must be exactly 16 characters long.
*** Email Server
To set up the email server for system emails, open the "config.edn" file in the root directory of the application. Edit the default EDN object containing server details to the file, replacing the values with your own.
*** Enabling HTTPS (optional)
To enable HTTPS from within the server, view the [[https://github.com/sig-gis/triangulum#triangulumhttps][Triangulum HTTPS]] page for further instructions on enabling HTTPS.
*** Google Earth Engine service account
For production it is recommended that you use a service account with a key file. You can obtain your key file by logging into your service account, navigating to the account menu, and clicking "Create key > JSON". Then, download that JSON key file and place it in the root directory of CEO. Set the email for your service account and key path in the ~:gee~ section of config.edn.
:gee {:ee-account "example@gmail.com" :ee-key-path "ceo-gee-key.json"}
*** Launching the Web Server
To compile and run the web application, navigate to the top level project directory and run:
npm install npm run webpack-prod clojure -M:build-db functions -d ceo clojure -M:server start -m [dev|prod] [-p 8080] [-P 8443] [-r]
The website will then be available at http://localhost:8080 unless a port is specified. These can also be configured using the ~:server~ section in your ~config.edn~ file. An http port can be specified with -p and an https port can be specified with -P. In dev mode, server-side exceptions will be displayed in the browser and Clojure source files will be reloaded whenever you refresh the page. These features are disabled in prod mode. If -m is unspecified, it will default to prod mode.
*** Running the Web Server as a System Service
View the [[https://github.com/sig-gis/triangulum#triangulumsystemd][Triangulum Systemd]] page for further instructions on enabling the app as a system service.
*** Maintaining Daily Logs
By default the server will log to standard out. If you would like to have the system log to YYYY-DD-MM.log, use the "-o path" option to specify an output path. You can either specify a path relative to the top level directory of this repository or an absolute path on your filesystem. The logger will keep the 10 most recent logs.
*** Using the Announcement Banner
On each page load clojure will read the value of ~announcement.txt~. If text is found, the value will be inserted into a HTML element that displays as a red banner at the top of the page. To add a new announcement, edit ~announcement.txt~ and add a new message. To remove the announcement, edit ~announcement.txt~ and remove all text.
** Contact
Authors:
** License and Distribution
Copyright © 2016-2022 FAO.
Collect Earth Online is distributed by FAO under the terms of the MIT License. See LICENSE in this directory for more information.