search

clstoulouse / motu

Motu is a high efficient and robust Web Server which fills the gap between heterogeneous Data Providers to End Users. Motu handles, extracts and transforms oceanographic huge volumes of data without performance collapse.
GNU Lesser General Public License v3.0
19 stars 5 forks source link

readme

Motu Project

@author tjolibois@cls.fr: Project manager & Product owner
@author smarty@cls.fr: Scrum master, Software architect, Quality assurance, Continuous Integration manager

How to read this file? Use a markdown reader: plugins chrome exists (Once installed in Chrome, open URL chrome://extensions/, and check "Markdown Preview"/Authorise access to file URL.), or for firefox (anchor tags do not work) and also plugin for notepadd++.

Be careful: Markdown format has issue while rendering underscore "_" character which can lead to bad variable name or path.

Summary

Overview

Motu is a robust web server allowing the distribution of met/ocean gridded data files through the web. Subsetter allows user to extract the data of a dataset, with geospatial, temporal and variable criterias. Thus, user download only the data of interest.
A graphic web interface and machine to machine interfaces allow to access data and information on data (metadata). The machine-to-machine interface can be used through a client written in python, freely available here https://github.com/clstoulouse/motu-client-python. Output data files format can be netCDF3 or netCDF4.
An important characteristic of Motu is its robustness: in order to be able to answer many users without crashing, Motu manages its incoming requests in a queue server.
The aim is to obtain complete control over the requests processing by balancing the processing load according to criteria (volume of data to extract, number of requests to fulfill for a user at a given time, number of requests to process simultaneously).
Moreover, Motu implements a request size threshold. Motu masters the amount of data to extract per request by computing, without any data processing, the result data size of the request.
Beyond the allowed threshold, every request is rejected. The threshold is set in the configuration file. Motu can be secured behind an authentication server and thus implements authorization. A CAS server can implement the authentication. Motu receives with authentication process user information, including a user profile associated with the account. Motu is configured to authorize or not the user to access the dataset or group of datasets which user is trying to access.
For administrators, Motu allows to monitor the usage of the server: the logs produced by Motu allow to know who (login) requests what (dataset) and when, with extraction criterias.

Architecture

Motu is a Java Web Application running inside the Apache Tomcat application server. Motu can be run as a single application or can be scaled over several instances.

Architecture overall

Architecture single instance

The clients "motu-client-python" or an HTTP client like a web browser are used to connect to Motu services.
A frontal web, Apache HTTPd for example, is used as a reverse proxy to redirect request to Motu server and also to serve the downloaded data from Motu download folder.
Motu server, runs on a Apache Tomcat server and can serve files either directly "DGF" or by delegating extraction to Thredds server with NCSS or OpenDap protocols.
A NFS server is used to share the netcdf files between Thredds and Motu DGF when they are not deployed on the same host.
An (SSO CAS server)[#ConfigurationSystemCASSSO] is used for the authentication of users but Motu can also be deployed without any authentication system.
The Apache HTTPd, on the top right corner is used to serve the graphic chart when several Motu Web server are deployed.

The schema below shows an example of Motu scalability architecture. The "i1, i2" are the Motu server deployed. They have to share the same business configuration file and the download folder.

Software architecture

Architecture scalability

To run Motu over several instances, a Redis server has to be deployed in order to share to request id and status. The download folder of Motu has also to be shared between the different Motu instances.
It can be on a NFS server or a GLusterFS server.
The frontal web server "Apache HTTPd" must serve the downloaded files and implement the load balencer between all Motu instances.
All other servers, CAS, NFS remains as on the single instance architecture.
The same source code is used to run Motu with a single architecture or with several instances. It is just done by configuration.
When Motu is scalable, one Motu server instance can run a download request, another distinct Motu server instance can respond to a "get status" request and a last one can respond the URL of the result file.

Software architecture

Interfaces

Server interfaces

All ports are defined in motu.properties configuration file.

External interfaces with other systems or tools

Motu has interfaces with other systems:

Design

The Motu application has been designed by implementing the Three-Layered Services Application design. It takes many advantages in maintenance cost efficiency and in the ease of its future evolutivity.
Three layers are set in the core "motu-web" project:

Each layer is an entry point of the application designed with a singleton. These three singletons gives access to high level managers which provides services by implementing a Java interface. High level managers handle for example the configuration, the request, the catalog, the users.

A common package is also defined to provide utilities: log4j custom format, XML, String ...

Design details

The main project is "motu-web". This project is divided in three main layers detailed below:

Motu-web project

Layers

USL

BLL

DAL

Daemon threads

"motu-web" project starts daemon threads for:

Other projects

Algorithm details

Downloading 1 point

Schema below displays a subset of a dataset variable as an array of 2 longitudes and 2 latitudes. At each intersection, we have got 1 real value (10, 11, 12, 13) as defined in the gridded data.
But which result value is returned by Motu when the request target a location between those longitudes and latitudes ?
As we can see below, 4 areas are displayed and the nearest value from the requested location is returned.

Downloading 1 point

Development

Source code

Source code can be downloaded directly from Github. 

mkdir motugithub  
cd motugithub  
git clone https://github.com/clstoulouse/motu.git  
    #In order to work on a specific version  
git tag -l  
git checkout motu-X.Y.Z  
git status  
cd motu  
cd motu-parent

Development environment

Configure Eclipse development environment

Run/Debug Motu

Click Debug configurations...> Under Apache Tomcat, debug "Motu Tomcat v7.0 Server at localhost"

Open a web browser and test:
http://localhost:8080/motu-web/Motu?action=ping

it displays "OK - response action=ping"

For more details about Eclipse launchers, refers to /motu-parent/README-eclipseLaunchers.md.

Compilation

Maven is used in order to compile Motu.
You have to set maven settings in order to compile.
Copy/paste content below in a new file settings.xml and adapt it to your information system by reading comments inside.

    <settings>  
    <!-- localRepository: Path to the maven local repository used to store artifacts. (Default: ~/.m2/repository) -->  
    <localRepository>J:/dev/cmems-cis-motu/testGitHub/m2/repository&lt;/localRepository>  
    <!-- proxies: Optional. Set it if you need to connect to a proxy to access to Internet -->  
    <!--   
    <proxies>  
       <proxy>  
          <id>cls-proxy</id>  
          <active>true</active>  
          <protocol>http</protocol>  
          <host></host>  
          <port></port>  
          <username></username>  
          <password></password>  
          <nonProxyHosts></nonProxyHosts>  
        </proxy>  
      </proxies>  
    -->   
    <!-- Repositories used to download Maven artifacts in addition to https://repo.maven.apache.org   
         cls-to-ext-thirdparty : contains patched libraries and non public maven packaged libraries  
         geotoolkit: contains geographical tools libraries  
    -->  
    <profiles>    
       <profile>  
         <id>profile-cls-cmems-motu</id>  
         <repositories>  
            <repository>  
              <id>cls-to-ext-thirdparty</id>  
              <name>CLS maven central repository, used for CMEMS Motu project</name>  
              <url>http://mvnrepo.cls.fr:8081/nexus/content/repositories/cls-to-ext-thirdparty</url>  
            </repository>  
            <repository>  
              <id>geotoolkit</id>  
              <name>geotoolkit</name>  
              <url>http://maven.geotoolkit.org/</url>  
            </repository>  
        </repositories>  
       </profile>  
     </profiles>  
     </settings>

This step is used to generate JAR (Java ARchives) and WAR (Web application ARchive).

In the GitLab Continuous Integration context, the access to the Maven repository are configured with maven password encryption using the ci/settings-security.xml master key, according to https://maven.apache.org/guides/mini/guide-encryption.html. It is recommended to use maven encryption, and for changing a password, ensure that the encryption is done on a maven repository using the same master key than the one used on the continuous integration server, or update with your local master key of your own settings-security.xml. 

mkdir motu  
cd motu   
  #Copy paste the content above inside settings.xml  
vi settings.xml  
  #Get source code of the last Motu version  
git clone https://github.com/clstoulouse/motu.git  
cd motu/motu-parent  
  #  This remove the maven parent artifact from pom.xml, or remove lines below manually:  
  #  <parent>  
  #        <artifactId>cls-project-config&lt;/artifactId>  
  #        <groupId>cls.commons&lt;/groupId>  
  #        <version>1.2.00&lt;/version>  
  # </parent>  
sed -i '6,10d' pom.xml  
  #Compile the source code  
mvn -s ../../settings.xml -gs ../../settings.xml -Pprofile-cls-cmems-motu -Dmaven.test.skip=true clean install  
...  
[INFO] BUILD SUCCESS  
...

All projects are built under target folder.
The Motu war is built under "/motu-web/target/motu-web-X.Y.Z-classifier.war".
It embeds all necessary jar libraries.

Packaging

This packaging process can be run only on CLS development environment. This is an helpful script used to packaged as tar.gz the different projects (products, distribution (server and client), configuration, themes).
So if you try to run it outside of CLS development environment, you will have to tune and remove many things to run it successfully (in particular all which is related to motu-config and motu-products). This step includes the compilation step. Once all projects are compiled, it groups all archives in a same folder in order to easy the final delivery.
You have to set ANT script inputs parameter before running it.
See /motu-distribution/build.xml header to get more details about inputs. 

cd /motu-distribution  
ant  
cd target-ant/delivery/YYYYMMDDhhmmssSSS

4 folders are built containing archives:

Installation

Prerequisites

In this chapter some paths are set. For example "/opt/cmems-cis" is often written to talk about the installation path. You are free to install Motu in any other folder, so in the case, replace "/opt/cmems-cis" by your installation folder.
This installation is used to install Motu on a single instance. To scale Motu on several instances, refers to Install a scalable Motu over several instances.

Motu host, hardware settings

OS target: Linux 64bits (Tested on centos-7.2.1511)

Minimal configuration for an operational usage:

Once started, you can check performance.

For test usage we recommend:

Motu host, software settings

Motu embeds all its dependencies (Java , Tomcat, CDO). All versions of these dependencies will be visible in the folder name once the Motu product archive is extracted.
For example: 

ls -1 /opt/cmems-cis/motu/products  
apache-tomcat-7.0.69  
cdo-group  
jdk1.7.0_79  
README  
version-products.txt

So bash shell is only required on the Linux host machine.

External interfaces

Motu is able to communicate with different external servers:

The installation of these two servers is not detailed in this document. Refer to their official web site to know how to install them.

Several Motu instances on a same host

If you need to instance several instances of Motu server on a same host, you have to:

Upgrade from Motu v2.x

Check this section only if you have installed Motu v2.x and you want to install Motu 3.x. In this section we consider that your Motu installation folder of version 2.x is "/opt/atoll/misgw/".

Upgrade the software

First stop Motu v2.x: /opt/atoll/misgw/stop-motu.
Then install the version 3.x of Motu from scratch. Before starting the new Motu version, upgrade its configuration by ready section below.
Once the version 3.x of Motu runs well, you can fully remove the folder of version 2.x is "/opt/atoll/misgw/".
To avoid any issue, perhaps backup the folder of Motu v2.x before removing it definitively. 

motu2xInstallFolder=/opt/atoll/misgw  
rm -rf $motu2xInstallFolder/deliveries  
rm -rf $motu2xInstallFolder/motu-configuration-common-2.1.16  
rm -rf $motu2xInstallFolder/motu-configuration-sample-misgw-1.0.5  
rm -rf $motu2xInstallFolder/motu-web  
rm -rf $motu2xInstallFolder/start-motu  
rm -rf $motu2xInstallFolder/stop-motu  
rm -rf $motu2xInstallFolder/tomcat7-motu  
rm -rf $motu2xInstallFolder/tomcat-motu-cas

Upgrade the configuration

Business configuration, Product & dataset: motuConfiguration.xml

The new version of Motu is compatible with the motuConfiguration.xml file configured in Motu v3.x.
So you can use exactly the same file, but it is important to update some fields to improve performance and future compatibility. Copy your old motuConfiguration.xml file to the folder /opt/cmems-cis/motu/config, for example: 

cp  /opt/atoll/misgw/motu-configuration-sample-misgw/resources/motuConfiguration.xml /opt/cmems-cis/motu/config

Then update attributes below:

Upgrade DGF

Resource URN attributes are not handled in the same way in Motu v3.
In Motu v2.x, URN attributes had values set with an ontology URL: 

<resource urn="http://purl.org/myocean/ontology/product/database#dataset-bal-analysis-forecast-phys-V2-dailymeans">

In Motu v3.x, you have to upgrade URN attributes with only the value found after the # character: 

<resource urn="dataset-bal-analysis-forecast-phys-V2-dailymeans">

Log files

In CMEMS-CIS context the log file motuQSlog.xml is stored in a specific folder in order to be shared.
You have so to check that with the new version this log file is well written in the shared folder. Here is where this log files were written in Motu v2.x: 

grep -i "motuQSlog.xml" /opt/atoll/misgw/motu-configuration-sample-misgw-1.0.5/resources/log4j.xml
<param name="file" value="/opt/atoll/misgw/tomcat-motu-cas/logs/motuQSlog.xml" />

The folder set in the value attribute shall be the same as the one defined in new the Motu configuration file. Replace $path below by the folder path: 

grep -i "motuQSlog.xml" /opt/cmems-cis/motu/config/log4j.xml
fileName="$path/motuQSlog.xml"
filePattern="$path/motuQSlog.xml.%d{MM-yyyy}"

Install Motu from scratch

Motu installation needs two main step: the software installation and optionally the theme installation.
The software installation brings by default the CLS theme. The theme installation is there to customize or change this default theme.

Install Motu software, for example on a Dissemination Unit

Copy the installation archives and extract them. 

cd /opt/cmems-cis  
cp motu-products-A.B.tar.gz .  
cp motu-distribution-X.Y.Z.tar.gz .  
cp motu-config-X.Y.Z-$BUILDID-$TARGET-$PLATFORM.tar.gz .  
tar xzf motu-products-A.B.tar.gz  
tar xzf motu-distribution-X.Y.Z.tar.gz  
tar xzf motu-config-X.Y.Z-$BUILDID-$TARGET-$PLATFORM.tar.gz  
cd motu

At this step, Motu is able to start. But static files used for customizing the web theme can be installed.
In the CMEMS context, the installation on a dissemination unit is ended, static files are installed on a central server.

Now you can configure the server:

Refer to configuration in order to check your configuration settings.

Motu is installed and configured. You can start Motu server.
Then you can check installation.

Install Motu theme (public static files)

As a dissemination unit administrator, in CMEMS context, this section is not applicable.

Public static files are used to customized Motu theme. When several Motu are installed, a central server eases the installation and the update by referencing static files only once on a unique machine. This is the case in the CMEMS context, where each dissemination unit host a Motu server, and a central server hosts static files.
If you runs only one install of Motu, you can install static files directly on Motu Apache tomcat server.

On a central server

Extract this archive on a server.

tar xvzf motu-web-static-files-X.Y.Z-classifier-$timestamp-$target.tar.gz

Then use a server to make these extracted folders and files accessible from an HTTP address.

Example: The archive contains a motu folder at its root. Then a particular file is "motu/css/motu/motu.css" and this file is served by the URL http://resources.myocean.eu/motu/css/motu/motu.css in the CMEMS CIS context.

Directly on Motu Apache tomcat server

If you do not use a central entity to serve public static files, you can optionally extract the archive and serve files directly by configuring Motu.
First extract the archive: 

tar xzf motu-web-static-files-X.Y.Z-classifier-$timestamp-$target.tar.gz -C /opt/cmems-cis/motu/data/public/static-files

Then edit "motu/tomcat-motu/conf/server.xml" in order to serve files from Motu.
Add then "Context" node as shown below. Note that severals "Context" nodes can be declared under the Host node. 

[...]  
<Host appBase="webapps" [...]  
        <!-- Used to serve public static files -->  
        <Context docBase="/opt/cmems-cis/motu/data/public/static-files/motu" path="/motu"/>    
 [...]

Finally in motuConfiguration.xml, remove all occurrences of the attribute named: httpBaseRef in motuConfig and configService nodes. (Do not set it empty, remove it).

If you want to set another path instead of "/motu", you have to set also the business configuration parameter named httpBaseRef.

Check installation

Start motu

./motu start

Check messages on the server console

When you start Motu, the only message shall be: 

tomcat-motu - start

Optionaly, when this is your first installation or when a software update is done, an INFO message is displayed: 

INFO: War updated: tomcat-motu/webapps/motu-web.war [$version]

If any other messages appear, you have to treat them.

As Motu relies on binary software like CDO, error could be raised meaning that CDO does not runs well. 

ERROR: cdo tool does not run well: $cdo --version  
[...]

In this case, you have to install CDO manually.

Check Motu web site available

Open a Web browser, and enter: http://$motuUrl/motu-web/Motu?action=ping
Where $motuUrl is: ip adress of the server:tomcat port Refer to configuration regarding the tomcat port

Response has to be: 

OK - response action=ping

Open a Web browser, and enter: http://$motuUrl/motu-web/Motu
If nothing appears, it is because you have to add dataset.

Check Motu logs

Check that no error appears in Motu errors log files.

CDO manual installation

This section has to be read only if Motu does not start successfully.
Select one option below to install "cdo". If you have no idea about cdo installation, choose the Default option.

Install cdo

"cdo" (Climate Data operators) are commands which has to be available in the PATH when Motu starts.
By default, Motu provides a built of CDO and add the "cdo" command to the PATH, but with some Linux distribution it is necessary to install it.
Motu provides some help in order to install CDO.

First check your GLibC version: 

ldd --version  
ldd (GNU libc) 2.12  
[...]

If your GlibC lower than 2.14, you have to install GLIBC 2.14, but to highly recommend to upgrade your Linux operating system to get an up to date GLIBC version:
INSTALL GLIBC 2.14 

export MotuInstallDir=/opt/cmems-cis  
cd $MotuInstallDir/motu/products/cdo-group  
wget http://ftp.gnu.org/gnu/glibc/glibc-2.14.tar.gz  
tar zxvf glibc-2.14.tar.gz  
cd glibc-2.14  
mkdir build  
cd build  
mkdir $MotuInstallDir/motu/products/cdo-group/glibc-2.14-home  
../configure --prefix=$MotuInstallDir/motu/products/cdo-group/glibc-2.14-home  
make -j4  
make install  
cd $MotuInstallDir/motu

Now check if "cdo" runs well: 

export MotuInstallDir=/opt/cmems-cis  
$MotuInstallDir/motu/products/cdo-group/cdo.sh --version  
Climate Data Operators version x.y.z (http://mpimet.mpg.de/cdo)  
[...]

If error appear like ones below, it certainly means that GLIC is not in the LD_LIBRARY_PATH.

$MotuInstallDir/motu/products/cdo-group/cdo-x.z.z-home/bin/cdo: error while loading shared libraries: libhdf5.so.10: cannot open shared object file: No such file or directory

or 

cdo: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by cdo)
cdo: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /opt/cmems-cis-validation/motu/products/cdo-group/hdf5-1.8.17-home/lib/libhdf5.so.10)

In this case, edit $MotuInstallDir/products/cdo-group/cdo.sh and add "$GLIBC-home/lib" to LD_LIBRARY_PATH.

Now check again if "cdo" runs well.

If it runs well, you can now start Motu.

cdo is already installed on this machine

If "cdo" is installed in another folder on the machine, you can add its path in "$MotuInstallDir/motu/motu" script: 

__setPathWithCdoTools() {  
  PATH=$MOTU_PRODUCTS_CDO_HOME_DIR/bin:$PATH  
}

Optionnaly set LD_LIBRAY_PATH in $MotuInstallDir/products/cdo-group/setLDLIBRARYPATH.sh

Try MOTU without cdo installation

Note that without CDO, some functionalities on depth requests or on download product won't work successfully. If any case, you can disable the CDO check by commented the check call:

How cdo is built?

CDO is automaticcly build from the script $MotuInstallDir/motu/products/cdo-group/install-cdo.sh Also in order to get full details about CDO installation, you can get details in /opt/cmems-cis/motu/products/README and search for 'Download CDO tools'.

Installation folder structure

Once archives have been extracted, a "motu" folder is created and contains several sub-folders and files:
motu/

Setup a frontal Apache HTTPd server

Apache HTTPd is used as a frontal HTTP server in front of Motu Http server. It has several aims:

See sample of the Apache HTTPd configuration in the Motu installation folder: config/apache-httpd-conf-sample
The configuration is described for Apache2 contains files:

When an SSO cas server is used, you have to st the property cas-auth-serverName to http://$serverHostName

Apache HTTPd can be used at different levels. The Apache HTTPd above is the one installed on the same machine as Motu. An Apache HTTPd can be used as a frontal to manage Apache HTTPd load balancing. In the case, you can set up with the following example: 

 # Use to authenticate users which want to download transaction files  
< Location /datastore-gateway/transactions/* >  
|--AuthType Basic  
|--AuthName "XXX"  
|--AuthUserFile /XXX/password.conf  
|--Require valid-user  
< / Location>   

 # Used to serve URL requested after a CAS authentication  
 # Because Motu SSO client set a redirection URL directly to its webapp name  
 # so we have to take into account the webapp name in Apache HTTPd  
ProxyPass /motu-web http://$motuTomcatIp:$motuTomcatPort/motu-web  
ProxyPassReverse /motu-web http://$motuTomcatIp:$motuTomcatPort/motu-web  

 # Used to serve Motu requests   
    # /mis-gateway-servlet These rules are used for retro compatibility between Motu v2.x and Motu v3.x  
ProxyPass /mis-gateway-servlet http://$motuTomcatIp:$motuTomcatPort/motu-web  
ProxyPassReverse /mis-gateway-servlet http://$motuTomcatIp:$motuTomcatPort/motu-web  
ProxyPreserveHost On  
    # /motu-web-servlet This URL is sometimes used.  
    # It can be customized depending of your current installation. If you have any doubt, keep this rule.  
ProxyPass /motu-web-servlet http://$motuTomcatIp:$motuTomcatPort/motu-web  
ProxyPassReverse /motu-web-servlet http://$motuTomcatIp:$motuTomcatPort/motu-web  

ProxyPass /datastore-gateway/transactions http://$apacheHTTPdOnMotuHost/datastore-gateway/transactions  
ProxyPassReverse /datastore-gateway/transactions http://$apacheHTTPdOnMotuHost/datastore-gateway/transactions  

ProxyPass /datastore-gateway/deliveries http://$apacheHTTPdOnMotuHost/datastore-gateway/deliveries  
ProxyPassReverse /datastore-gateway/deliveries http://$apacheHTTPdOnMotuHost/datastore-gateway/deliveries  

< Location /motu-web-servlet/supervision>  
|--Order allow,deny  
|--Allow from All  
< /Location>

Security

Run Motu as an HTTPS Web server

Motu is a web server based on Apache Tomcat.
In order to secure HTTP connections with a client, HTTPs protocol can be used.
You have two choices:

Motu and Single Sign On

In order to manage SSO (Single Sign On) connections to Motu web server, Motu uses an HTTPs client.
All documentation about how to setup is written in chapter CAS SSO server.

Install a scalable Motu over several instances

Scalability is provided with two main components, the Redis database, for sharing the status of the requests between Motu instances, and Tomcat with the session replication mechanism to share login and authentication data.

Scalability prerequisites:

Scalability configuration:

Motu configuration:

Motu apache Tomcat configuration:

Frontal load balancer configuration (httpd or other)

Deactivate sticky session mechanisms:

Configuration

This chapter describes the Motu configuration settings.
All the configuration files are set in the $installDir/motu/config folder.

Configuration directory structure

cd $installDir/motu/config

Business settings

motuConfiguration.xml: Motu business settings

This file is watched and updated automatically. This means that when Motu is running, this file has to be written in a atomic way.

You can configure 3 main categories:

If you have not this file, you can extract it (set the good version motu-web-X.Y.Z.jar): 

/opt/cmems-cis/motu/products/jdk1.7.0_79/bin/jar xf /opt/cmems-cis/motu/tomcat-motu/webapps/motu-web/WEB-INF/lib/motu-web-X.Y.Z.jar motuConfiguration.xml

If you have this file from a version anterior to Motu v3.x, you can reuse it. In order to improve global performance, you have to upgrade some fields:

Attributes defined in motuConfig node

defaultService

A string representing the default action in the URL /Motu?action=$defaultService
The default one is "listservices".
All values can be found in the method USLRequestManager#onNewRequest with the different ACTION_NAME.

dataBlockSize

Amount of data in Ko that can be requested in a single query from Motu to TDS. Default is 2048Kb.
If this amount is lower than the maxSizePerFile (in MegaBytes), Motu will launch several sub-requests to TDS to gather all the data.
Higher value (up to the maxSizePerFile) leads to less requests, but with higher data volume to transfer by request from TDS to Motu. And the tds.http.sotimeout has to be long enough for letting TDS the time to read and transfer the whole data to Motu.
Lower values will imply more requests, but shorter. It consumes more CPU on both Motu and TDS with the advantages to allow TDS to answer to other parallel request it would receive, and to reduce communication times for each request.
Concerning performance, we tried 200Mb and 1024Mb for a 1024Mb request, and durations were similar, but note that the shapes of the sub-requests may not match the shapes of the netcdf files, and that could imply a supplementary delay for hard drive data storage.

If the tds.http.sotimeout can be set to a high value (such as 900s for a 1Gb max size request), the safest is to use the maxSizePerFile value for the dataBlockSize parameter (mind the units Kb/Mb).
Else from the max timeout the environment can support (external constraints or "004-27" error, see tds.http.sotimeout), extrapolate a volume of data that can be transfered during this delay, lower it to keep a margin, and use it as the dataBlockSize.

maxSizePerFile

This parameter is only used with a catalog type set to "FILE" meaning a DGF access.
It allows download requests to be executed only if data extraction is lower than this parameter value.
Unit of this value is MegaBytes.
Default is 1024 MegaBytes.
Example: maxSizePerFile="2048" to limit a request result file size to 2GB.

maxSizePerFileSub

This parameter is only used with a catalog type used with Opendap or Ncss.
It allows download requests to be executed only if data extraction is lower that this parameter value.
Unit of this value is MegaBytes.
Default is 1024 MegaBytes.
Example: maxSizePerFileSub="2048" to limit request result file size to 2GB.

maxSizePerFileTDS

@Deprecated from v3 This parameter is not used and has been replaced by maxSizePerFile and maxSizePerFileSub.
Number of data in Megabytes that can be written and download for a Netcdf file. Default is 1024Mb.

extractionPath

The absolute path where files downloaded from TDS are stored.
For example: /opt/cmems-cis/motu/data/public/download It is recommended to set this folder on an hard drive with very good performances in write mode. It is recommended to have a dedicated partition disk to avoid freezing Motu if the hard drive is full. By default value is $MOTU_HOME/data/public/download, this folder can be a symbolic link to another folder.
String with format ${var} will be substituted with Java property variables. @See System.getProperty(var)

downloadHttpUrl

Http URL used to download files stored in the "extractionPath" described above. It is used to allow users to download the result data files.
This URL is concatenated to the result data file name found in the folder "extractionPath".
When a frontal HTTPd server is used, it is this URL that shall be configured to access to the folder "extractionPath".
String with format ${var} will be substituted with Java property variables. @See System.getProperty(var)

httpBaseRef

Http URL used to serve files from to the path where archive motu-web-static-files-X.Y.Z-classifier-buildId.tar.gz has been extracted.
For example:

IMPORTANT: When Motu URL starts with "HTTPS", if you set an URL in httpBaseRef, this URL has also to start with "HTTPS". On the contrary, when Motu URL starts with "HTTP", if you set an URL in httpBaseRef, this URL can start with "HTTP" or "HTTPS".

cleanExtractionFileInterval

In minutes, oldest result files from extraction request which are stored in the folder set by extractionpath are deleted. This check is done each "runCleanInterval" minutes.
Default = 60min

cleanRequestInterval

In minutes, oldest status (visible in debug view) than this time are removed from Motu. This check is done each "runCleanInterval" minutes.
Default = 60min

runCleanInterval

In minutes, the waiting time between each clean process. The first clean work is triggered when Motu starts.
A clean process does:

Default = 1min

extractionFilePatterns

Patterns (as regular expression) that match extraction file name to delete in folders:

Default is "..nc$|..zip$|..tar$|..gz$|.*.extract$"

extractionFileCacheSize

Size in Mbytes.
A clean job runs each "runCleanInterval". All files with a size higher than this value are deleted by this job. If value is zero, files are not deleted.
Default value = 0.

describeProductCacheRefreshInMilliSec

Provide the delay to wait to refresh the meta-data of products cache after the last refresh.
Motu has a cache which is refreshed asynchronously. Cache is first refreshed as soon as Motu starts.
Then Motu waits for this delay before refreshing again the cache.
This delay is provided in millisecond.
The default value is 60000 meaning 1 minute.

Logbook file (motu/log/logbook.log) gives details about time taken to refresh cache, for example: 

INFO  CatalogAndProductCacheRefreshThread.runProcess Product and catalog caches refreshed in 2min 19sec 75msec

Logbook file gives details per config service ($configServiceId) about dedicated time taken to refresh cache, for example: 

INFO  CatalogAndProductCacheRefreshThread.runProcess Refreshed statistics: $configServiceId@Index=0min 34sec 180msec, $configServiceId@Index=0min 31sec 46msec, ...

They are sorted by config service which has taken the most time first.
@Index All config services are refreshed sequentially. This index is the sequence number for which this cached has been refreshed.

Example of archived data with several TB of data. Cache is refreshed daily: describeProductCacheRefreshInMilliSec=86400000
Example of real time data with several GB of data. Cache is refreshed each minute: describeProductCacheRefreshInMilliSec=60000

runGCInterval

@Deprecated from v3 This parameter is not used.

httpDocumentRoot

@Deprecated from v3 This parameter is not used. Document root of the servlet server.

wcsDcpUrl

Optional attribute. Used to set the tag value "DCP" in the response of the WCS GetCapabilities request with a full URL. The WCS DCP URL value is define using the following priority order:

useAuthentication

@Deprecated from v3 This parameter is not used. It is redundant with parameter config/motu.properties#cas-activated.

defaultActionIsListServices

@Deprecated from v3 This parameter is not used.

Configure the Proxy settings

@Deprecated from v3 This parameter is not used. To use a proxy in order to access to a Threads, use the JVM properties, for example: 

tomcat-motu-jvm-javaOpts=-server -Xmx4096M  ... -Dhttp.proxyHost=monProxy.host.fr -Dhttp.proxyPort=XXXX -Dhttp.nonProxyHosts='localhost|127.0.0.1'
refreshCacheToken

This token is a key value which is checked to authorize the execution of the cache refresh when it is request by the administrator . If the token value provided by the administrator doesn't match the configured token value, the refresh is not executed and an error is returned. A default value "a7de6d69afa2111e7fa7038a0e89f7e2" is configured but it's hardly recommended to change this value. If this token is not changed, it is a security breach and a log ERROR will be written while the configuration will be loaded. The value can contains the characters [A-Za-z] and specials listed here ( -_@$*!:;.,?()[] ) It's recommended to configure a token with a length of 29 characters minimum.

downloadFileNameFormat

Format of the file name result of a download request.
2 dynamic parameters can be used to configure this attribute:

If this attribute is not present, default value is: "@@productId@@_@@requestId@@.nc"

motuConfigReload

Configure how motu configuration is reloaded.
Arguments are only 'inotify' or an 'integer in seconds'. 'inotify' is the default value.

Attributes defined in configService node

name

String to set the config service name If the value of this attribute contains some special caracters, those caracters have not to be encoded. For example, if the value is an URL, the caracters ":" and "/" have not to be encoded like "%2E" or "%3A".

group

String which describes the group

description

String which describes the service

profiles

Optional string containing one value, several values separated by a comma or empty (meaning everybody can access).
Used to manage access right from a SSO cas server.
In the frame of CMEMS, three profiles exist:

Otherwise, it's possible to configure as many profiles as needed.
Profiles are configured in LDAP within the attribute "memberUid" of each user. This attribute is read by CAS and is sent to Motu once a user is logged in, in order to check if it matches profiles configured in Motu to allow a user accessing the data.
In LDAP, "memberUid" attribute can be empty, contains one value or several values separated by a comma.

veloTemplatePrefix

Optional, string used to target the default velocity template. It is used to set a specific theme.
Value is the velocity template file name without the extension.
Default value is "index".

refreshCacheAutomaticallyEnabled

Optional, boolean used to determine if the current config service have its cache updated automatically by Motu or not. Default value is "true". "true" means that the config service cache update is executed automatically by Motu.

httpBaseRef

Optional, used to override motuConfig httpBaseRef attribute for this specific service.

defaultLanguage

@Deprecated from v3 This parameter is not used.

Attributes defined in catalog node

name

This catalog name refers a TDS catalog name available from the URL: http://$ip:$port/thredds/m_HR_MOD.xml Example: m_HR_OBS.xml

type

Example: tds

ncss

Optional parameter used to enable or disable the use of NetCDF Subset Service (NCSS) in order to request the TDS server. ncss must be enabled only with regular grid. The datasets using curvilinear coordinates (like ORCA grid) can not be published with ncss. Thus, ncss option must be set to disable or empty. Without this attribute or when empty, Motu connects to TDS with Opendap protocol. If this attribute is set to "enabled" Motu connects to TDS with NCSS protocol in order to improve performance.
We recommend to use "enabled" for regular grid datasets. Values are: "enabled", "disabled" or empty.

urlSite

Attributes defined in queueServerConfig node

maxPoolAnonymous

Maximum number of request that an anonymous user can send to Motu before throwing an error message.
Value of -1 means no check is done so an unlimited number of user can request the server.
Default value is 10
In case where an SSO server is used for authentication, this parameter is not used. In this you you will be able to fix a limit by setting "maxPoolAuth" parameter value.

maxPoolAuth

Maximum number of request that an authenticated user can send to Motu before throwing an error message.
Value of -1 means no check is done so an unlimited number of user can request the server.
Default value is 1 In case where no SSO server is used for authentication, this parameter is not used. In this you you will be able to fix a limit by setting "maxPoolAnonymous" parameter value.

defaultPriority

@Deprecated from v3 This parameter is not used.

Attributes defined in queues

id

An id to identify the queue.

description

Description of the queue.

batch

@Deprecated from v3 This parameter is not used.

Child node: maxThreads

Use to build a java.util.concurrent.ThreadPoolExecutor an to set "corePoolSize" and "maximumPoolSize" values.
Default value is 1
The total number of threads should not be up to the total number of core of the processor on which Motu is running.

Child node: maxPoolSize

Request are put in a queue before being executed by the ThreadPoolExecutor. Before being put in the queue, the queue size is checked. If it is upper than this value maxPoolSize, an error message is returned. Value of -1 means no check is done.
Default value is -1

Child node: dataThreshold

Size in Megabyte. A request has a size. The queue in which this request will be processed is defined by the request size. All queues are sorted by size ascending. A request is put in the last queue which has a size lower than the request size. If the request size if higher than the bigger queue dataThreshold, request is not treated and an error message is returned.
This parameter is really useful when a Motu is used to server several kind of file size and you want to be sure that file with a specific size does no slow down request of small data size.
In this case you can configure two queues and set a number of threads for each in order to match the number of processors. The JVM, even if requests for high volume are running, will be able to process smallest requests by running the thread on the other processor core. Sp processing high volume requests will not block the smallest requests.

Child node: lowPriorityWaiting

@Deprecated from v3 This parameter is not used.

Attributes defined in redisConfig node

This optional node is used to run Motu in a scalable architecture. Do not add this node when you just run one single Motu instance.
Once this node is added, Motu stores all its request ids and status in Redis.

host

Define the host (ip or server name) where is deployed the Redis server od Redis cluster used by Motu to share the RequestId and RequestStatus data. Default value is localhost

port

Define the port used by the Redis server or Redis cluster used by Motu to share the requestId and RequestStatus data. Default value is 6379

prefix

Define the prefix used to build the RequestId value of the shared RequestStatus data. Default value is requestStatus

isRedisCluster

Define if the redis server in in cluster mode. This is a boolean value. By default is set to false and the cluster mode is not activate. To activate the cluster, the value have to be set on true.

System settings

motu.properties: Motu system settings

System settings are configured in file config/motu.properties
All parameters can be updated in the file.

Java options

The three parameters below are used to tune the Java Virtual Machine, and the tomcat-motu-jvm-javaOpts parameter can include any Java property in the form "-D\<java property name>=\<value>":
# -server: tells the Hostspot compiler to run the JVM in "server" mode (for performance)
tomcat-motu-jvm-javaOpts=-server -Xmx4096M -Xms512M -XX:MetaspaceSize=128M -XX:MaxMetaspaceSize=512M
tomcat-motu-jvm-port-jmx=9010
tomcat-motu-jvm-address-debug=9090
tomcat-motu-jvm-umask=tomcat|umask|0000 (More details...)

Tomcat umask

By default, if tomcat-motu-jvm-umask is not set, motu sets the umask with result of the command umask
tomcat-motu-jvm-umask=umask|tomcat|0000

Java property tds.http.sotimeout

By default this parameter is at "300". It represents the maximum delay in seconds for TDS to answer a MOTU request (reading timeout of the socket).
For queries involving lots of files, TDS might need more than the default 5 minutes to answer, and to avoid the error "004-27 : Error in NetcdfWriter finish", this parameter can be set to a higher value in:
tomcat-motu-jvm-javaOpts=-server [...] -XX:MaxPermSize=512M -Dtds.http.sotimeout=4000

Java property tds.http.conntimeout

By default this parameter is at "60". It represents the maximum delay in seconds for TDS to accept a MOTU request (connection timeout on the socket).
The paramater can be customized and added in:
tomcat-motu-jvm-javaOpts=-server [...] -XX:MaxPermSize=512M -Dtds.http.conntimeout=100

Tomcat network ports

The parameters below are used to set the different network ports used by Apache Tomcat.
At startup, these ports are set in the file "$installdir/motu/tomcat-motu/conf/server.xml".
But if this file already exist, it won't be replaced. So in order to apply these parameters, remove the file "$installdir/motu/tomcat-motu/conf/server.xml".

tomcat-motu-port-http=9080
# HTTPs is in a common way managed from a frontal Apache HTTPd server. If you really need to use it from Tomcat, you have to tune the SSL certificates and the protocols directly in the file "$installdir/motu/tomcat-motu/conf/server.xml".
tomcat-motu-port-https=9443
tomcat-motu-port-ajp=9009
tomcat-motu-port-shutdown=9005

CAS SSO server

# true or false to enable the SSO connection to a CAS server
cas-activated=false

# Cas server configuration to allow Motu to access it
# @see https://wiki.jasig.org/display/casc/configuring+the+jasig+cas+client+for+java+in+the+web.xml

# The start of the CAS server URL, i.e. https://cas-cis.cls.fr/cas
cas-server-url=https://cas-cis.cls.fr/cas

# The Motu HTTP server URL, for example: http://misgw-ddo-qt.cls.fr:9080 or http://motu.cls.fr
# If you use a frontal HTTPd server, you have to known if its URL will be called once the user will be login on CAS server.
# In this case, set the Apache HTTPd server. The value will be http://$apacheHTTPdServer/motu-web/Motu So, in Apache HTTPd, you have to redirect this URL to the Motu Web server
cas-auth-serverName=http://$motuServerIp:$motuServerPort

# The proxy callback HTTPs URL of the Motu server ($motuServerIp is either the Motu host or the frontal Apache HTTPs host ip or name. $motuServerHttpsPort is optional if default HTTPs port 443 is used, otherwise it is the same value as defined above with the key "tomcat-motu-port-https", or it is the port defined for the HTTPs server on the frontal Apache HTTPd)
cas-validationFilter-proxyCallbackUrl=https://$motuServerIp:$motuServerHttpsPort/motu-web/proxyCallback

IMPORTANT: Motu uses a Java HTTPs client to communicate with the CAS server. When the CAS server has an untrusted SSL certificate, you have to add it to Java default certificates or to add the Java property named "javax.net.ssl.trustStore" to target a CA keystore which contains the CAS Server SSL CA public key. For example, add this property by setting Java option tomcat-motu-jvm-javaOpts: 

tomcat-motu-jvm-javaOpts=-server -Xmx4096M -Xms512M -XX:MetaspaceSize=128M -XX:MaxMetaspaceSize=512M -Djavax.net.ssl.trustStore=/opt/cmems-cis/motu/config/security/cacerts-with-cas-qt-ca.jks

The following part is not relevant in the CMEMS context as the SSO CAS server has been signed by a known certification authority.
If you need to run tests with your own SSO CAS server without any certificate signed by a known certification authority, you have to follow the following steps.

How to build the file cacerts-with-cas-qt-ca.jks on Motu server?

NetCdf standard names

When NetCdf variables are read in data files, either by Threads or directly by Motu, Motu wait for a standard name metadata sttribute to be found for each variable as requiered by the CF convention. Due to any production constraints, some netcdf files does not have any standard_name attribute.
In the case, you can add directly in the configuration folder, a file named standardNames.xml in order to map a standard_name to a netcdf variable name.
You can find an example in Motu source: /motu-web/src/main/resources/standardNames.xml

Supervision

To enable the status supervision, set the parameter below:
tomcat-motu-urlrewrite-statusEnabledOnHosts=localhost,*.cls.fr

This parameter is used to set the property below in the WEB.XML file: 

        <!-- Documentation from http://tuckey.org/urlrewrite/manual/3.0/
        you may want to allow more hosts to look at the status page
        statusEnabledOnHosts is a comma delimited list of hosts, * can
        be used as a wildcard (defaults to "localhost, local, 127.0.0.1") -->
        <init-param>  
            <param-name>statusEnabledOnHosts</param-name>  
            <param-value>${tomcat-motu-urlrewrite-statusEnabledOnHosts}</param-value>  
        </init-param>

For more detail read:
org.tuckey UrlRewriteFilter FILTERS : see http://tuckey.org/urlrewrite/manual/3.0/

Log settings

Log are configured by using log4j2 in file config/log4j.xml

Motu queue server logs: motuQSlog.xml, motuQSlog.csv

This log files are used to compute statistics about Motu server usage.
Two format are managed by this log, either XML or CSV.
To configure it, edit config/log4j.xml

Log format: XML or CSV

Update the fileFormat attribute of the node "MotuCustomLayout": A string either "xml" or "csv" to select the format in which log message are written.
Also update the log file name extension of the attributes "fileName" and "filePattern" in order to get a coherent content in relationship with value set for MotuCustomLayout file format.
If this attribute is not set, the default format is "xml". 

        <RollingFile name="log-file-infos.queue"   
            fileName="${sys:motu-log-dir}/motuQSlog.xml"   
            filePattern="${sys:motu-log-dir}/motuQSlog.xml.%d{MM-yyyy}"    
            append="true">   
            <!-- fileFormat=xml or csv -->  
            <MotuCustomLayout fileFormat="xml" />  
            <Policies>  
                <TimeBasedTriggeringPolicy interval="1" modulate="true"/>  
            </Policies>  
        </RollingFile>
Log path

In the dissemination unit, Motu shares its log files with a central server.
Log files have to be save on a public access folder.
Set absolute path in "fileName" and "filePattern" attributes. This path shall be serve by the frontal Apache HTTPd or Apache Tomcat.

For example, if you want to share account transaction log files, you edit config/log4j.xml. Update content below: 

<RollingFile name="log-file-infos.queue" fileName="${sys:motu-log-dir}/motuQSlog.xml"
            filePattern="${sys:motu-log-dir}/motuQSlog.xml.%d{MM-yyyy}"

with: 

<RollingFile name="log-file-infos.queue" fileName="/opt/cmems-cis/motu/data/public/transaction/motuQSlog.xml"
            filePattern="/opt/cmems-cis/motu/data/public/transaction/motuQSlog.xml.%d{MM-yyyy}"

Note that both attributes fileName and filePattern have been updated.
Then the frontal Apache HTTPd server has to serve this folder.

Theme and Style

In Motu you can update the theme of the website. There is 2 mains things in order to understand how it works?

By default, the template and style are integrated in the "war". But the Motu design enable to customize it easily.

The main HTML web page structure is defined by the index.vm velocity template. For example, in you create a file motu/config/velocityTemplates/index.vm containing an empty html page, website will render empty web pages.
"index.vm" is the default theme. The name can be updated for each motuConfig#configService by setting veloTemplatePrefix="". By default veloTemplatePrefix="index".

Operation

Start, Stop and other Motu commands

All operations are done from the Motu installation folder.
For example: 

cd /opt/cmems-cis/motu

Start Motu

Start the Motu process. 

./motu start

Stop Motu

At the shutdown of Motu, the server waits for none of the pending or in progress request to be in execution.
If it's the case, the server waits the end of the request before shutdown.
Note that after waiting 10 minutes, server will automatically shutdown without waiting any running requests.
So command below can respond quickly if no requests are in the queue server or takes time to process them. 

./motu stop

If you needs to understand what Motu is waiting for, you can check the logbook: 

tail -f log/logbook.log  
Stop in progress...  
Stop: Pending=0; InProgress=2  
Stop: Pending=0; InProgress=2  
...  
Stop: Pending=0; InProgress=1  
Stop: Pending=0; InProgress=0  
...  
Stop done

During the stop step, from a web browser, the user will be able to ends its download request if a front web server (Apache HTTPd) serves the statics files and the downloaded product. In case where Motu is installed as a standalone web server, user will get a 500 HTTP error. For example in development or qualification environment, this could lead to block the download of the files if Motu is used to serve both static and requested product files.

Advanced commands

Restart Motu

./motu restart

Status of the Motu process

./motu status

Status are the following:

Help about Motu parameters

./motu ?

Monitor performance

Once started, you can use the Linux command "top" to check performance:

Example of top command: 

top - 11:07:01 up 19:46,  3 users,  ***load average: 0,05, 0,09, 0,25***  
Tasks: 395 total,   2 running, 393 sleeping,   0 stopped,   0 zombie  
%Cpu0  :  1,0 us,  1,0 sy,  0,0 ni, 98,1 id,   ***0,0 wa***,  0,0 hi,  0,0 si,  0,0 st  
%Cpu1  :  1,0 us,  0,0 sy,  0,0 ni, 99,0 id,  ***0,0 wa***,  0,0 hi,  0,0 si,  0,0 st  
KiB Mem : 10224968 total,  ***4034876 free***,  3334576 used,  2855516 buff/cache    
...

Logbooks

Log messages are generated by Apache Log4j 2. The configuration file is "config/log4j.xml".
By default, log files are created in the folder $MOTU_HOME/log. This folder contains Motu log messages.
Tomcat log messages are generated in the tomcat-motu/logs folder.

Add a dataset

In order to add a new Dataset, you have to add a new configService node in the Motu business configuration.
When Motu read data through TDS (Opendap or NCSS service) url, the data shall be configured in TDS before this configuration is saved in Motu. The TDS configuration is not explained here.

Within CMEMS, the datasets are organized in a tree structure, where the product granularity appears above the dataset granularity.
To be noticed: 

<  CMEMS DU xxx Thredds Catalog >  
| ------ < GLOBAL_ANALYSIS_PHYS_001_016  >   
|------- < dataset-armor-3d-v5-myocean >  
|----------------- < GLOBAL_REP_PHYS_001_013  >   
|------- < dataset-armor-3d-rep-monthly-v3-1-myocean >                                                                     
|------- < dataset-armor-3d-rep-weekly-v3-1-myocean>

The Motu configuration (motuConfiguration.xml) should reference the node corresponding to one XML file in the TDS configuration.

Examples: 

<configService description="Free text to describe your dataSet" group="HR-Sample" httpBaseRef="" name="HR_MOD-TDS" veloTemplatePrefix="" profiles="external">  
        <catalog name="m_HR_MOD.xml" type="tds" ncss="enabled" urlSite="http://$tdsUrl/thredds/"/>  
</configService>  
<configService description="Free text to describe your dataSet" group="HR-Sample" httpBaseRef="" name="HR_MOD-TDS" veloTemplatePrefix="" profiles="external">  
        <catalog name="m_HR_MOD.xml" type="tds" ncss="" urlSite="http://$tdsUrl/thredds/"/>  
</configService>  
<configService description="Free text to describe your dataSet" group="HR-Sample" profiles="internal, external, major" httpBaseRef="" name="HR_MOD-TDS" veloTemplatePrefix="">  
           <catalog name="catalogFILE_GLOBAL_ANALYSIS_PHYS_001_016.xml" type="file" urlSite="file:///opt/cmems-cis-data/data/public/inventories"/>  
</configService>

An an example, the file __catalogFILE_GLOBAL_ANALYSIS_PHYS_001_016.xml__ contains: 

< ?xml version="1.0" encoding="UTF-8"?>  
<!DOCTYPE rdf:RDF [  
<!ENTITY atoll "http://purl.org/cls/atoll/ontology/individual/atoll#">  
]>  
<catalogOLA xmlns="http://purl.org/cls/atoll" name="catalog GLOBAL-ANALYSIS-PHYS-001-016">  
        <resourcesOLA>  
                <resourceOLA urn="dataset-armor-3d-v5-myocean" inventoryUrl="file:///opt/cmems-cis-data/data/public/inventories/dataset-armor-3d-v5-myocean-cls-toulouse-fr-armor-motu-rest-file.xml"/>  
        </resourcesOLA>  
</catalogOLA>

File dataset-armor-3d-v5-myocean-cls-toulouse-fr-armor-motu-rest-file.xml: 

< ?xml version="1.0" encoding="UTF-8"?>  
<!DOCTYPE rdf:RDF [  
<!ENTITY atoll "http://purl.org/cls/atoll/ontology/individual/atoll#">  
<!ENTITY cf "http://purl.org/myocean/ontology/vocabulary/cf-standard-name#">  
<!ENTITY cu "http://purl.org/myocean/ontology/vocabulary/cf-unofficial-standard-name#">  
<!ENTITY ct "http://purl.org/myocean/ontology/vocabulary/forecasting#">  
<!ENTITY cp "http://purl.org/myocean/ontology/vocabulary/grid-projection#">  
]>  
<inventory lastModificationDate="2016-01-27T00:10:10+00:00" xmlns="http://purl.org/cls/atoll" updateFrequency="P1D">  
  <service urn="cls-toulouse-fr-armor-motu-rest-file"/>  
  <resource urn="dataset-armor-3d-v5-myocean">  
    <access urlPath="file:///data/atoll/armor/armor-3d-v3/"/>  
    <geospatialCoverage south="-82" north="90" west="0" east="359.75"/>  
    <depthCoverage min="0" max="5500" units="m"/>  
    <timePeriod start="2014-10-01T00:00:00+00:00" end="2016-01-26T23:59:59+00:00"/>  
    <theoricalTimePeriod start="2014-10-01T00:00:00+00:00" end="2016-01-26T23:59:59+00:00"/>  
    <variables>  
      <variable name="zvelocity" vocabularyName="http://mmisw.org/ont/cf/parameter/eastward_sea_water_velocity" units="m/s"/>  
      <variable name="height" vocabularyName="http://purl.org/myocean/ontology/vocabulary/cf-standard-name#height_above_geoid" units="m"/>  
      <variable name="mvelocity" vocabularyName="http://mmisw.org/ont/cf/parameter/northward_sea_water_velocity" units="m/s"/>  
      <variable name="salinity" vocabularyName="http://mmisw.org/ont/cf/parameter/sea_water_salinity" units="1e-3"/>  
      <variable name="temperature" vocabularyName="http://mmisw.org/ont/cf/parameter/sea_water_temperature" units="degC"/>  
    </variables>  
  </resource>  
  <files>  
    <file name="ARMOR3D_TSHUV_20141001.nc" weight="327424008" modelPrediction="http://www.myocean.eu.org/2009/resource/vocabulary/forecasting#" startCoverageDate="2014-10-01T00:00:00+00:00" endCoverageDate="2014-10-07T23:59:59+00:00" creationDate="2015-03-17T00:00:00+00:00" availabilitySIDate="2016-01-27T00:10:10+00:00" availabilityServiceDate="2016-01-27T00:10:10+00:00" theoreticalAvailabilityDate="2015-03-17T00:00:00+00:00"/>  
    <file name="ARMOR3D_TSHUV_20141008.nc" weight="327424008" modelPrediction="http://www.myocean.eu.org/2009/resource/vocabulary/forecasting#" startCoverageDate="2014-10-08T00:00:00+00:00" endCoverageDate="2014-10-14T23:59:59+00:00" creationDate="2015-03-17T00:00:00+00:00" availabilitySIDate="2016-01-27T00:10:10+00:00" availabilityServiceDate="2016-01-27T00:10:10+00:00" theoreticalAvailabilityDate="2015-03-17T00:00:00+00:00"/>  
    ...  
    <file name="ARMOR3D_TSHUV_20160120.nc" weight="327424008" modelPrediction="http://www.myocean.eu.org/2009/resource/vocabulary/forecasting#" startCoverageDate="2016-01-20T00:00:00+00:00" endCoverageDate="2016-01-26T23:59:59+00:00" creationDate="2016-01-26T11:11:00+00:00" availabilitySIDate="2016-01-27T00:10:10+00:00" availabilityServiceDate="2016-01-27T00:10:10+00:00" theoreticalAvailabilityDate="2016-01-26T11:11:00+00:00"/>  
  </files>  
</inventory>

Tune the dataset metadata cache

In order to improve response time, Motu uses an in-memory cache which stores datasets metadata. This cache is indexed by config service.
You can tune the cache behaviour in order to manage both real time and archived datasets effectively.
At startup, Motu loads datasets metadata of each configService by turn. Once done, cache is refreshed either periodically in an automatic manner or either when asked by triggering a specific action.
The cache is kept in memory and all Motu requests are based on it. When a cache refresh is asked, a second cache loads new metadata and when fully loaded, Motu main cache is replaced. So until the full loading, old cache is used in Motu responses.

For config services which manages real time datasets, meaning datasets which are daily updated, you can set the following configuration in motuConfiguration.xml:

<?xml version="1.0"?>
<motuConfig  ...
<configService ... refreshCacheAutomaticallyEnabled="true"
...

For config services which manages archived datasets, meaning dataset which not updated frequently for example only once a week, you can set the following configuration in motuConfiguration.xml:

<?xml version="1.0"?>
<motuConfig  ...
<configService ... refreshCacheAutomaticallyEnabled="false"
...

In this case, when you want to refresh metadata cache of these datasets, you can use this dedicated action.

Debug view

From a web browser access to the Motu web site with the URL: 

/Motu?action=debug

You can see the different requests and their status.
You change the status order by entering 4 parameters in the URL: 

/Motu?action=debug&order=DONE,ERROR,PENDING,INPROGRESS

Clean files

Logbook files

Logbook files are written by Apache Tomcat server and Motu application.

Apache Tomcat Logbook files

Tomcat writes log files in folder tomcat-motu/logs.
You can customize this default configuration by editing tomcat-motu/conf/logging.properties
This file is the default file provided by Apache Tomcat. There is a daily rotation so you can clean those files to fullfill the harddrive.
crontab -e
0 find /opt/cmems-cis/motu/tomcat-motu/logs/.log -type f -mmin +14400 -delete >/dev/null 2>&1
0 find /opt/cmems-cis/motu/tomcat-motu/logs/.txt -type f -mmin +14400 -delete >/dev/null 2>&1

Motu Logbook files

Logbook files are written in the folder(s) configured in the log4j.xml configuration file.
All logs are generated daily except for motuQSLog (xml or csv) which are generated monthly.
You can clean those files to avoid to fullfill the harddrive.
crontab -e
0 find /opt/cmems-cis/motu/log/.log -type f -mmin +14400 -delete >/dev/null 2>&1
0 find /opt/cmems-cis/motu/log/.out -type f -mmin +14400 -delete >/dev/null 2>&1
0 find /opt/cmems-cis/motu/log/.xml -type f -mmin +144000 -delete >/dev/null 2>&1
0 find /opt/cmems-cis/motu/log/.csv -type f -mmin +144000 -delete >/dev/null 2>&1

Note that Motu is often tuned to write the motuQSLog in a dedicated folder. So you have to clean log files in this folder too. For example:
0 find /opt/cmems-cis/motu/data/public/transaction/.xml -type f -mmin +144000 -delete >/dev/null 2>&1
0 find /opt/cmems-cis/motu/data/public/transaction/.csv -type f -mmin +144000 -delete >/dev/null 2>&1

Log Errors

The code pattern

The error codes of Motu as the following format "XXXX-Y":

For example, the web browser can display:
011-1 : A system error happened. Please contact the administrator of the site.

Here, we have the error code in order to understand better what happens. But the end user has a generic message and no detail is given to him. These end user messages are described in the file "/motu-web/src/main/resources/MessagesError.properties". The file provided with the project is a default one and can be customized for specific purposes. Just put this file in the "config" folder, edit it and restart Motu to take it into account. So when a user has an error, it just have to tell you the error code and you can check the two numbers with the descriptions below.

Action codes

The Action Code => A number matching the HTTP request with the action parameter.

001 => UNDETERMINED_ACTION
002 => PING_ACTION
003 => DEBUG_ACTION
004 => GET_REQUEST_STATUS_ACTION
005 => GET_SIZE_ACTION
006 => DESCRIBE_PRODUCT_ACTION
007 => TIME_COVERAGE_ACTION
008 => LOGOUT_ACTION
010 => DOWNLOAD_PRODUCT_ACTION
011 => LIST_CATALOG_ACTION
012 => PRODUCT_METADATA_ACTION
013 => PRODUCT_DOWNLOAD_HOME_ACTION
014 => LIST_SERVICES_ACTION
015 => DESCRIBE_COVERAGE_ACTION
016 => ABOUT_ACTION
018 => WELCOME_ACTION
019 => REFRESH_CACHE_ACTION
020 => HEALTHZ_ACTION
021 => CACHE_STATUS_ACTION

Error types

The Error Type Code => A number defining a specific error on the server.

0 => No error.
1 => There is a system error. Please contact the Administrator.
2 => There is an error with the parameters. There are inconsistent.
3 => The date provided into the parameters is invalid.
4 => The latitude provided into the parameters is invalid.
5 => The longitude provided into the parameters is invalid.
6 => The range defined by the provided dates is invalid.
7 => The memory capacity of the motu server is exceeded.
8 => The range defined by the provided latitude/longitude parameters is invalid.
9 => The range defined by the provided depth parameters is invalid.
10 => The functionality is not yet implemented.
11 => There is an error with the provided NetCDF variables.
12 => There is not variables into the variable parameter.
13 => NetCDF parameter error. Example: Invalid date range, invalid depth range, ...
14 => There is an error with the provided NetCDF variable. Have a look at the log file to have more information.
15 => The number of maximum request in the queue server pool is reached. it's necessary to wait that some requests are finished.
16 => The number of maximum request for the user is reached. It's necessary to wait that some requests are finished for the user.
18 => The priority of the request is invalid in the queue server manager. Have a look at the log file to have more information.
19 => The id of the request is not know by the server. Have a look at the log file to have more information.
20 => The size of the request is greater than the maximum data managed by the available queue. It's impossible to select a queue for this request. It's necessary to narrow the request.
21 => The application is shutting down. it's necessary to wait a while before the application is again available.
22 => There is a problem with the loading of the motu configuration file. Have a look at the log file to have more information.
23 => There is a problem with the loading of the catalog configuration file. Have a look at the log file to have more information.
24 => There is a problem with the loading of the error message configuration file. Have a look at the log file to have more information.
25 => There is a problem with the loading of the netcdf file. Have a look at the log file to have more information.
26 => There is a problem with the provided parameters. Have a look at the log file to have more information.
27 => There is a problem with the NetCDF generation engine. Have a look at the log file to have more information.
28 => The required action is unknown. Have a look at the log file to have more information.
29 => The product is unknown.
30 => The service is unknown.
31 => The request cut the ante meridian. In this case, it's not possible to request more than one depth. It's necessary to change the depth selection and to select in the "from" and the "to" the values that have the same index into the depth list.
32 => Due to a known bug in Thredds Data Server, a request cannot be satisfied wit netCDF4. User has to request a netCDF3 output file.
101 => WCS specific error code : A WCS mandatory parameter is missing
102 => WCS specific error code : A WCS parameter doesn't match the mandatory format
103 => WCS specific error code : The WCS version parameter is not compatible with the Motu WCS server
104 => WCS specific error code : A system error append.
105 => WCS specific error code : The coverage ident doesn't exist
106 => WCS specific error code : The list of coverage id is empty
107 => WCS specific error code : The provided parameter used to define a subset is invalid
108 => WCS specific error code : The provided axis label doesn't match any available label

Motu clients & REST API

You can connect to Motu by using a web browser or a client.

Python client

Motu offers an easy to use Python client. Very useful in machine to machine context, it enables to download data by running a python script.
Project and all its documentation is available at https://github.com/clstoulouse/motu-client-python.

OGC WCS API

Motu offers a Web Service interface which implements the OGC WCS standard described, in particular, by the two following documents on the OGC web site:

Available Web Services are:

Parameters can be added to each request and they are described with their cardinality [x,y].

WCS: Get Capabilities

The GetCapabilities request retrieves all available products defined on Motu server.

URL: http://localhost:8080/motu-web/wcs?service=WCS&version=2.0.1&request=GetCapabilities

Parameters:

Return: A XML document as shown below:

  
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ns3:Capabilities version="2.0.1" xmlns:ns6="http://www.opengis.net/swe/2.0" xmlns:ns5="http://www.opengis.net/gmlcov/1.0" xmlns:ns2="http://www.w3.org/1999/xlink" xmlns:ns1="http://www.opengis.net/ows/2.0" xmlns:ns4="http://www.opengis.net/gml/3.2" xmlns:ns3="http://www.opengis.net/wcs/2.0">
    <ns1:ServiceIdentification>
        <ns1:Title>Motu</ns1:Title>
        <ns1:Abstract>Motu WCS service</ns1:Abstract>
        <ns1:ServiceType>OGC WCS</ns1:ServiceType>
        <ns1:ServiceTypeVersion>2.0.1</ns1:ServiceTypeVersion>
        <ns1:Profile>http://www.opengis.net/spec/WCS/2.0/conf/core</ns1:Profile>
        <ns1:Profile>http://www.opengis.net/spec/WCS_protocol-binding_get-kvp/1.0/conf/get-kvp</ns1:Profile>
    </ns1:ServiceIdentification>
    <ns1:OperationsMetadata>
        <ns1:Operation name="GetCapabilities">
            <ns1:DCP>
                <ns1:HTTP>
                    <ns1:Get ns2:href="http://localhost:8080/motu-web/wcs"/>
                </ns1:HTTP>
            </ns1:DCP>
        </ns1:Operation>
        <ns1:Operation name="DescribeCoverage">
            <ns1:DCP>
                <ns1:HTTP>
                    <ns1:Get ns2:href="http://localhost:8080/motu-web/wcs"/>
                </ns1:HTTP>
            </ns1:DCP>
        </ns1:Operation>
        <ns1:Operation name="GetCoverage">
            <ns1:DCP>
                <ns1:HTTP>
                    <ns1:Get ns2:href="http://localhost:8080/motu-web/wcs"/>
                </ns1:HTTP>
            </ns1:DCP>
        </ns1:Operation>
    </ns1:OperationsMetadata>
    <ns3:ServiceMetadata>
        <ns3:formatSupported>application/netcdf</ns3:formatSupported>
    </ns3:ServiceMetadata>
    <ns3:Contents>
        <ns3:CoverageSummary>
            <ns3:CoverageId>HR_MOD_NCSS-TDS@HR_MOD</ns3:CoverageId>
            <ns3:CoverageSubtype>ns3:GridCoverage</ns3:CoverageSubtype>
        </ns3:CoverageSummary>

            ...

        </ns3:Contents>
</ns3:Capabilities>

WCS: Describe Coverage

The DescribeCoverage request retrieves the parameters description and the list of available variables. For the parameters description, low and high values are provided.

URL: http://localhost:8080/motu-web/wcs?service=WCS&version=2.0.1&request=DescribeCoverage&coverageId=$coverageId

Parameters:

Return: A XML document as shown below:

  
  <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ns4:CoverageDescriptions xmlns:ns6="http://www.opengis.net/ows/2.0"
    xmlns:ns5="http://www.opengis.net/swe/2.0" xmlns:ns2="http://www.w3.org/1999/xlink"
    xmlns:ns1="http://www.opengis.net/gml/3.2" xmlns:ns4="http://www.opengis.net/wcs/2.0"
    xmlns:ns3="http://www.opengis.net/gmlcov/1.0">
    <ns4:CoverageDescription ns1:id="$covergaeId">
        <ns1:boundedBy>
            <ns1:Envelope
                uomLabels="latitude longitude depth date in seconds since 1970, 1 jan"
                axisLabels="Lat Lon Height Time">
                <ns1:lowerCorner>-80.0 -180.0 0.0 1.3565232E9</ns1:lowerCorner>
                <ns1:upperCorner>90.0 180.0 5728.0 1.4657328E9</ns1:upperCorner>
            </ns1:Envelope>
        </ns1:boundedBy>
        <ns4:CoverageId>$covergaeId</ns4:CoverageId>
        <ns1:domainSet>
            <ns1:Grid dimension="4"
                uomLabels="latitude longitude depth date in seconds since 1970, 1 jan"
                axisLabels="Lat Lon Height Time" ns1:id="Grid000">
                <ns1:limits>
                    <ns1:GridEnvelope>
                        <ns1:low>-80 -180 0 1356523200</ns1:low>
                        <ns1:high>90 180 5728 1465732800</ns1:high>
                    </ns1:GridEnvelope>
                </ns1:limits>
            </ns1:Grid>
        </ns1:domainSet>
        <ns3:rangeType>
            <ns5:DataRecord>
                <ns5:field name="uice">
                    <ns5:Quantity>
                        <ns5:uom code="m s-1" />
                    </ns5:Quantity>
                </ns5:field>
                <ns5:field name="salinity">
                    <ns5:Quantity>
                        <ns5:uom code="1e-3" />
                    </ns5:Quantity>
                </ns5:field>
                <ns5:field name="vice">
                    <ns5:Quantity>
                        <ns5:uom code="m s-1" />
                    </ns5:Quantity>
                </ns5:field>
                <ns5:field name="hice">
                    <ns5:Quantity>
                        <ns5:uom code="m" />
                    </ns5:Quantity>
                </ns5:field>
                <ns5:field name="u">
                    <ns5:Quantity>
                        <ns5:uom code="m s-1" />
                    </ns5:Quantity>
                </ns5:field>
                <ns5:field name="v">
                    <ns5:Quantity>
                        <ns5:uom code="m s-1" />
                    </ns5:Quantity>
                </ns5:field>
                <ns5:field name="temperature">
                    <ns5:Quantity>
                        <ns5:uom code="K" />
                    </ns5:Quantity>
                </ns5:field>
                <ns5:field name="ssh">
                    <ns5:Quantity>
                        <ns5:uom code="m" />
                    </ns5:Quantity>
                </ns5:field>
                <ns5:field name="fice">
                    <ns5:Quantity>
                        <ns5:uom code="1" />
                    </ns5:Quantity>
                </ns5:field>
            </ns5:DataRecord>
        </ns3:rangeType>
    </ns4:CoverageDescription>
</ns4:CoverageDescriptions>

WCS: Get Coverage

The GetCoverage request is used to run an extraction on a dataset using some filtering parameters and a list of required variables.

URL:

Parameters:

Return: A Netcdf file. When you request for one point, a specific algorithm is used, see Downloading 1 point.

MOTU REST API

MOTU REST API defines a set of services accessible from an HTTP URLs.
All URLs have always the same pattern: http://motuServer/${context}/Motu?action=$actionName
Other parameters can be added and they are described with their cardinality [x,y].

$actionName is an action, they are all listed below:

About

Display version of the archives installed on Motu server
URL: http://localhost:8080/motu-web/Motu?action=about

Parameters: No parameter.

Return: An HTML page. Motu-static-files (Graphic chart) is refreshed thanks to Ajax because its version file can be installed on a distinct server.
Example: 

Motu-products: 3.0  
Motu-distribution: 2.6.00-SNAPSHOT  
Motu-configuration: 2.6.00-SNAPSHOT-20160623173246403  
Motu-static-files (Graphic chart): 3.0.00-RC1-20160914162955422

Debug

Display all requests status managed by Motu server in the last cleanRequestInterval minutes. Tables are sorted by time ascending.
4 status are defined:

URL: http://localhost:8080/motu-web/Motu?action=debug

Parameters:

Return: An HTML page

Describe coverage

Get coverage data in relationship with a dataset.
URL: http://localhost:8080/motu-web/Motu?action=describecoverage&service=HR_MOD-TDS&datasetID=HR_MOD

Parameters:

Return: A XML document 

<dataset xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  xsi:noNamespaceSchemaLocation="describeDataset.xsd" name="HR_MOD" id="HR_MOD">  
<boundingBox>  
  <lon min="-180.0" max="179.91668701171875" units="degrees_east"/>
  <lat min="-80.0" max="-80.0" units="degrees_north"/>
</boundingBox>
<dimension name="time" start="2012-12-26T12:00:00.000+00:00" end="2016-06-12T12:00:00.000+00:00" units="ISO8601"/>  
<dimension name="z" start="" end="" units="m"/>  
<variables>  
<variable id="northward_sea_water_velocity" name="v" description="Northward velocity" standardName="northward_sea_water_velocity" units="m s-1">  
<dimensions></dimensions>  
</variable>  
...  
</variables>  
</dataset>

Describe product

Display the product meaning dataset description. Result contains notably: the datasetid, the time coverage, the geospatial coverage, the variable(s) (with the standard_name and unit), eventually the vertical coverage.
There is 2 ways to call describe product, both returning a same response.

Way 1

URL: http://localhost:8080/motu-web/Motu?action=describeproduct&service=HR_MOD-TDS&product=HR_MOD

Parameters:

Way 2 (Deprecated)

URL: http://localhost:8080/motu-web/Motu?action=describeproduct&data=http://$tdsServer/thredds/dodsC/path_HR_MOD&xmlfile=http://$tdsServer/thredds/m_HR_MOD.xml

Parameters:

Return: An XML document 

<productMetadataInfo code="OK" msg="OK" lastUpdate="Not Available" title="HR_MOD" id="HR_MOD">  
<timeCoverage code="OK" msg="OK"/>  
<availableTimes code="OK" msg="OK">  
1993-01-15T12:00:00Z/2001-01-01T00:00:00Z/P2D,2001-01-01T00:00:00Z/2012-03-01T00:00:00Z/PT6H
</availableTimes>  
<availableDepths code="OK" msg="OK">  
0.49402;1.54138;2.64567;...  
</availableDepths>  
<geospatialCoverage code="OK" msg="OK"/>  
<variablesVocabulary code="OK" msg="OK"/>  
<variables code="OK" msg="OK">  
<variable description="Northward velocity" units="m s-1" longName="Northward velocity" standardName="northward_sea_water_velocity" name="v" code="OK" msg="OK"/>  
<variable description="Eastward velocity" units="m s-1" longName="Eastward velocity" standardName="eastward_sea_water_velocity" name="u" code="OK" msg="OK"/>  
...  
</variables>  
<dataGeospatialCoverage code="OK" msg="OK">  
<axis code="OK" msg="OK" description="Time (hours since 1950-01-01)" units="hours since 1950-01-01 00:00:00" name="time_counter" upper="582468" lower="552132" axisType="Time"/>  
<axis code="OK" msg="OK" description="Longitude" units="degrees_east" name="longitude" upper="179.91668701171875" lower="-180" axisType="Lon"/>  
<axis code="OK" msg="OK" description="Latitude" units="degrees_north" name="latitude" upper="90" lower="-80" axisType="Lat"/>  
<axis code="OK" msg="OK" description="Depth" units="m" name="depth" upper="5727.9169921875" lower="0.4940249919891357421875" axisType="Height"/>  
</dataGeospatialCoverage>  
</productMetadataInfo>

availableTimes XML tag

In the XML result file the tag "availableTimes" provides the list of date where data are available for the requested product. The format of the date follows the convention ISO_8601 used to represent the dates and times. (https://en.wikipedia.org/wiki/ISO_8601) Foreach available time period, the period definition format is "StartDatePeriod/EndDatePeriod/DurationBetweenEachAvailableData".
The "availableTimes" contains a list of time period separated by a ",".

For DGF datasets, the list of available times is built from the start and end date of each file of the dataset, ignoring the other time values if any. As a consequence, there might be more available times than those listed in this attribute for DGF datasets with there are more than 2 time values per file.

StartDate and EndDate format

The format of the StartDate and EndDate is YYYY-MM-DDThh:mm:ssZ where:

Examples:

DurationBetweenEachAvailableData

The formation of the duration is PnbyersYnbmonthsMnbdaysDTnbhoursHnbminutesMnbsecondsS.nbmillisec where:

By convention, P1M defines a duration of 1 month and PT1M defines a duration of 1 minutes.

Examples:

"lastUpdate" XML attribute

Note about "lastUpdate" attribute of the "productMetadataInfo" field: it has the same value than "Last update" field of the list catalog page.

Download product

Request used to download a product

URL: http://localhost:8080/motu-web/Motu?action=productdownload
example:
http://localhost:8080/motu-web/Motu?action=productdownload&service=HR_MOD-TDS&product=HR_MOD&x_lo=-2&x_hi=2&y_lo=-2&y_hi=2&output=netcdf&t_lo=2016-06-12+12%3A00%3A00&t_hi=2016-06-12+12%3A00%3A00&z_lo=0.49&z_hi=5727.92

Parameters:

Return: Several ways depending of the selected http parameter mode. When you request for one point, a specific algorithm is used, see Downloading 1 point.

Request status

Get a request status to get more details about a download state.

URL: http://localhost:8080/motu-web/Motu?action=getreqstatus&requestid=123456789

Parameters:

Return: An XML document or an HTML page if requestId does not exists.
Validated by the schema /motu-api-message/src/main/schema/XmlMessageModel.xsd#StatusModeResponse
Example: 

<statusModeResponse code="004-0" msg="" scriptVersion="" userHost="" userId="" dateSubmit="2016-09-19T16:56:22.184Z" localUri="/$pathTo/HR_MOD_1474304182183.nc" remoteUri="http://localhost:8080/motu/deliveries/HR_MOD_1474304182183.nc" size="1152.0"dateProc="2016-09-19T16:56:22.566Z" requestId="1474304182183" status="1"/>

Size is in MegaBytes or at NaN while still not estimated.

Get size

Get the size of a download request. Result contains the size of the potential result file, with a unit, and the maximum allowed size for this service.

URL: http://localhost:8080/motu-web/Motu?action=getsize

Parameters:

Parameters below are exactly the same as for Download product

Return: An XML document.
The unit is "KB" means Kilobyte. Validated by the schema /motu-api-message/src/main/schema/XmlMessageModel.xsd#RequestSize
Example: 

<requestSize code="005-0" msg="OK" unit="kb" size="1.5104933E8" maxAllowedSize="9.961472E8"/>

List catalog

Display information about a catalog (last update timestamp) and display link to access to download page and dataset metadata.

Note about field "Last update" : On MOTU start-up, this date is set to the most recent date of the dataset if it is before of the current day, or to the "last update date" returned by the TDS server also if it is before current day, or else to "Not available".
On MOTU cache update, if the most recent available date of the dataset changes, the "last update date" is set with the current date.

URL: http://localhost:8080/motu-web/Motu?action=listcatalog&service=HR_MOD-TDS

Parameters:

Return: An HTML page

List services

Display the service web page URL: http://localhost:8080/motu-web/Motu?action=listcatalog&service=HR_MOD-TDS

Parameters:

Return: An HTML page

Ping

Used to be sure that server is up. You can also use the supervision URL.

URL: http://localhost:8080/motu-web/Motu?action=ping

Parameters: No parameter.

Return: An plain text 

OK - response action=ping

Refresh config services metadata cache

Force the refresh of the cache of config service metadata instead of waiting the automatic refresh.
This action is secured and is only triggered when a valid token is given.
Moreover a list of config services needed to be refreshed is shared with the automatic update process.
This add robustness because a job refreshes only cache of the config services which are is the list. So when this action is called several times, if a config service in already in this waiting list, it is not added a second time. A soon as a cache for a config service is refreshed, config service is removed from this waiting list.

URL: http://localhost:8080/motu-web/Motu?action=refreshcache&token=tokenValid&configServiceNames=all

Parameters:

Return: A plain text which specify if the refresh is launched or if an error occurred, e.g. "OK: config service AAA cache refresh in progress" or "ERROR: Unknwon config service UnknownConfigService"

OK cache refresh in progress

Healthz

Gives healthz information about Motu server health.

URL: http://localhost:8080/motu-web/Motu?action=healthz

Parameters: No parameter

Return: An http status and a short message:

Welcome

HTML page which gives access to several web pages, in particular the Motu listservices web page.

URL:

Parameters: No parameter

Return: An HTML web page

Product download home

Display an HTML page in order to set the download parameters.

URL: http://localhost:8080/motu-web/Motu?action=productdownloadhome&service=HR_OBS-TDS&product=HR_OBS

Parameters:

Return: An HTML page

Product metadata Home

Display an HTML page with the geographical and temporal coverage, the last dataset update and the variables metadata.

Note about field "Date" of "Last dataset update" section: this date has the same value than the "Last update" field.

URL: http://localhost:8080/motu-web/Motu?action=listproductmetadata&service=HR_OBS-TDS&product=HR_OBS

Parameters:

Return: An HTML page

Time coverage

Display an HTML page with the geographical and temporal coverage, the last dataset update and the variables metadata.

URL: http://localhost:8080/motu-web/Motu?action=gettimecov&service=HR_MOD-TDS&product=HR_MOD

Parameters:

Return: A XML document 

<timeCoverage code="007-0" msg="OK" end="2016-09-17T00:00:00.000Z" start="2007-05-13T00:00:00.000Z"/>

Supervision

Gives information about Motu server.
For more details, see [https://jolokia.org/reference/html/agents.html].

URL: http://localhost:8080/motu-web/supervision

Parameters: No parameter

Return: A JSON document 

{"timestamp":1474638852,"status":200,"request":{"type":"version"},"value":{"protocol":"7.2","config":{"agentId":"10.1.20.198-18043-2df3a4-servlet","agentType":"servlet"},"agent":"1.3.3","info":{"product":"tomcat","vendor":"Apache","version":"7.0.69"}}}

CacheStatus

Gives the status of the dataset cache of Motu.
On start-up, Motu reads its configuration file, and gets a list of "configService" nodes referencing a dataset catalog with an URL, and starts caching them.

URL: http://localhost:8080/motu-web/Motu?action=cachestatus

Parameters: No parameter

Return: A JSON document 

{"cachestatus":
    {"state":
        {"nbTotal":8,"nbSuccess":6,"nbFailure":2,"lastUpdate":"2019-11-19T10:56:37.357Z","lastUpdateDuration":"PT1M17.561S"},
     "configServices": [
        {"Sea_Surface_Temperature_Global-TDS":
            {"state":
                {"status":"FAILURE","lastUpdate":"2019-11-19T10:57:56.398Z","lastUpdateDuration":""},
             "conf":
                {"refreshCacheAutomaticallyEnabled":true,"type":"tds","ncss":"enabled"}
            }
        },
        {"HR_MOD_NCSS-TDS":
            {"state":
                {"status":"SUCCESS","lastUpdate":"2019-11-19T10:57:01.436Z","lastUpdateDuration":"PT1.688S"},
             "conf":
                {"refreshCacheAutomaticallyEnabled":true,"type":"tds","ncss":"enabled"}
            }
        },
        .....]
     },
 "version":
    {"motu-products":"Unknow version","motu-distribution":"Unknow version","motu-configuration":"3.11.04-20190716151835979"}
 }

The nbTotal is the total number of ConfigServices.
The nbSuccess and the nbFailure are the number of currently loaded ConfigServices in success/failure.
Note that lastUpdate and lastUpdateDuration fields can be empty if the system hasn't still refreshed the cache or if the access failed.

Welcome

HTML page which gives access to several web pages, in particular the Motu listservices web page.

URL:

Parameters: No parameter

Return: An HTML web page

Motu Docker distribution

Motu comes in a Docker release based on CentOS.

Docker image content

The Motu specific elements are located under /opt/motu.
The required libraries are installed on the Docker image.
The Tomcat is located under /opt/motu/tomcat-motu.
The CDO scripts (merge.sh and cdo.sh) can be found in the folder /opt/motu/products/cdo-group.

Docker mounted directories

To configure the Motu Docker image, some of the folders have to be mounted from the execution environment:

Other folders for logs and Apache configuration are optionnals:

Configuration files are similar to standard Motu configuration files with some exceptions to take into account:

Run Motu with Docker

The following command:

docker run -d -v /data/shared/motu/result:/opt/motu/data/download/public -v /home/motu/motu/config_qt/motu_config:/opt/motu/config -v /home/motu/motu/config_qt/apache_config:/opt/motu/tomcat-motu/conf -v /home/motu/motu/config_qt/log_motu:/opt/motu/log -v /home/motu/motu/config_qt/log_apache:/opt/motu/tomcat-motu/logs -p 8080:8080 -p 8443:8443 -p 8009:8009 -p 8005:8005 registry-ext.cls.fr:443/motu/motu/motu-distribution

Will run the Motu server. Logs (from logbook, warning and error MOTU files) can be accessed using:

tail /var/log/syslog

If a reverse proxy for downloading the results is to be added in the platform where MOTU server docker image is deployed, use:

docker run -d -v /data/shared/motu/result:/var/www -v /home/motu/motu/config_qt/motu_config:/opt/motu/config -p $NGINX_PORT:8070 -e NGINX_PORT=$NGINX_PORT -e MOTU_DOWNLOAD_PATH=/motu-web -e MOTU_URL=http://$(hostname):18080 -d registry-ext.cls.fr:443/motu/motu/motu-nginx