search

clstoulouse / motu-client-python

Motu is a high efficient and robust Web Server which fills the gap between heterogeneous data providers to end users. Motu handles, extracts and transforms oceanographic huge volumes of data without performance collapse. This client enables to extract and download data through a python command line.
GNU Lesser General Public License v3.0
49 stars 22 forks source link

readme

Motu Client Python Project

@author Product owner tjolibois@cls.fr
@author Scrum master, software architect smarty@cls.fr
@author Quality assurance, continuous integration manager smarty@cls.fr

How to read this file? Use a markdown reader: plugins chrome exists (Once installed in Chrome, open URL chrome://extensions/, and check "Markdown Preview"/Authorise access to file URL.), or for firefox (anchor tags do not work) and also plugin for notepadd++.

Be careful: Markdown format has issue while rendering underscore "_" character which can lead to bad variable name or path.

Summary

Overview

Motu client "motuclient-python" is a python script used to connect to Motu HTTP server in order to:

This program can be integrated into a processing chain in order to automate the downloading of products via the Motu.

Build

From the root folder runs the command: 

./patchPOMtoBuild.sh  
mvn clean install -Dmaven.test.skip=true
[...]
[INFO] BUILD SUCCESS
[...]

This creates two archives in the target folder:

Installation

Prerequisites

Since motuclient release version 3.X.Y, you must use python version 3.7.10 or later.
/!\ motuclient does not work with the OpenSSL library release 1.1.1.e. Either use an older version such as the 1.1.1.d or jump to the 1.1.1.f release.
There are two methods to install the client, by using PIP or from a tar.gz file.
setuptools python package has be installed in order to display the motuclient version successfully.

Using PIP

Python Package Index is used to ease installation.
If your host needs a PROXY set it, for example: 

export HTTPS_PROXY=http://myCompanyProxy:8080

Then run: 

pip install motuclient --upgrade

Now "motuclient" is installed, you can configured it and use it.

From tar.gz file

Deploy the archive (file motuclient-python-$version-bin.tar.gz available from GitHub release) in the directory of your choice. 

tar xvzf motuclient-python-$version-$buildTimestamp-bin.tar.gz

Create a configuration file and set the user and password to use to connect to the CAS server.

Install setuptools python package

"Setuptools" python package has to be installed in order to display the version with option --version, here is how to install it:

If your host needs a PROXY set it, for example: 

export HTTPS_PROXY=http://myCompanyProxy:8080

Then run: 

sudo apt install python-pip  
pip install --upgrade setuptools

Configuration file

All parameters can be defined as command line options or can be written in a configuration file.
The configuration file is a .ini file, encoded in UTF-8 without BOM. This file is located in the following directory:

The expected structure of file is: 

[Main]  
# Motu credentials  
user=john  
pwd=secret  

motu=http://motu-ip-server:port/motu-web/Motu  
service_id=GLOBAL_ANALYSIS_FORECAST_PHY_001_024-TDS   
product_id=global-analysis-forecast-phy-001-024-hourly-t-u-v-ssh  
date_min=2019-03-27  
date_max=2019-03-27  
latitude_min=-30  
latitude_max=40.0  
longitude_min=-10  
longitude_max=179.9166717529297    
depth_min=0.493    
depth_max=0.4942  
# Empty or non set means all variables  
# 1 or more variables separated by a coma and identified by their standard name  
variable=sea_water_potential_temperature,sea_surface_height_above_geoid 
# Accept relative or absolute path. The dot character "." is the current folder  
out_dir=./out_dir  
out_name=test.nc  

# Logging
# https://docs.python.org/3/library/logging.html#logging-levels  
# log_level=X {CRITICAL:50, ERROR:40, WARNING:30, INFO:20, DEBUG:10, TRACE:0}   
log_level=0   

# block_size block used to download file (integer expressing bytes) default=65535
# block_size=65535  
socket_timeout=120000  

# Http proxy to connect to Motu server
# proxy_server=proxy.domain.net:8080  
# proxy_user=john  
# proxy_pwd=secret

A configuration file in another location can be specified by the --config-file option. It is even possible to split the configuration into two or more files. This is useful, for example, to keep server configuration in one file and dataset configuration in another:

./motuclient.py --config-file ~/server.ini --config-file ~/mercator.ini

If by chance there is a parameter listed in both configuration files, the value in the last file (e.g. mercator.ini) is the one actually used.

Note that the password must be encoded in UTF-8.
If it contains UTF-8 special characters, on Windows host only, you only have to double the "percent" character. If password is CMS2017@%! then enter 

pwd = CMS2017@%%!

Example of server.ini on Windows host only, with user password is
Password: 

loginForTesting2 &~#"'{([-|`_\^@)]=}¨^£$ µ*§!/:.;?,%<>

server.ini: 

[Main]
user = loginForTesting2@groupcls.com
pwd = loginForTesting2 &~#"'{([-|`_\^@)]=}¨^£$ µ*§!/:.;?,%%<>
auth-mode = cas
motu = http://motuURL:80/motu-web/Motu
out_dir = J:/dev/CMEMS-CIS-MOTU/git/motu-validkit/output/04-python-client/MOTU-208

Example of server.ini on Linux host only, with user password is
Password:

loginForTesting2 &~#"'{([-|`_\^@)]=}¨^£$ µ*§!/:.;?,%<>

server.ini:

[Main]
user = loginForTesting2@groupcls.com
pwd = loginForTesting2 &~#"'{([-|`_\^@)]=}¨^£$ µ*§!/:.;?,%<>
auth-mode = cas
motu = http://motuURL:80/motu-web/Motu
out_dir = J:/dev/CMEMS-CIS-MOTU/git/motu-validkit/output/04-python-client/MOTU-208

Usage

Starts the motu python client.

Usage from PIP installation

Since version 1.8.0: 

motuclient -h  
motuclient [options]

Before version 1.8.0: 

python -m motu-client -h  
python -m motu-client [options]

Options are listed below.
Method to used when it has been installed with PIP method.

Usage from tar.gz installation

./motuclient.py  -h  
motuclient.py [options]

Method to used when it has been installed with tar.gz method.
Usefull if host is offline and has no Internet access.

Options:

Usage examples

In the following examples, variable ${MOTU_USER} and ${MOTU_PASSWORD} are user name and user password used to connect to the CAS server for single sign on.
${MOTU_SERVER_URL} is the URL on the MOTU HTTP(s) server, for example http://localhost:8080/motu-web/Motu.
Commands "./motuclient.py" has to be replaced by "python -m motuclient" if it has been installed with PIP method.

Download

Download and save extracted file on the local machine

This command writes the extraction result data in file: /data/test.nc 

./motuclient.py --verbose --auth-mode=none -m ${MOTU_SERVER_URL} -s HR_MOD_NCSS-TDS -d HR_MOD -z 0.49 -Z 0.50 -x -70 -X 25 -y -75 -Y 10 -t "2016-06-10" -T "2016-06-11" -v salinity -o /data -f test.nc

Display on stdout the HTTP(s) URL of the NC file available on the Motu server

The HTTP(s) URL is displayed on stdout. This URL is a direct link to the file which is available to be downloaded. 

./motuclient.py --quiet --auth-mode=cas -u ${MOTU_USER} -p ${MOTU_PASSWORD}  -m ${MOTU_SERVER_URL} -s HR_MOD_NCSS-TDS -d HR_MOD -z 0.49 -Z 0.50 -x -70 -X 25 -y -75 -Y 10 -t "2016-06-10" -T "2016-06-11" -v salinity -o console

GetSize

See https://github.com/clstoulouse/motu#ClientAPI_GetSize for more details about XML result.

Get the XML file which contains the extraction size on the local machine

./motuclient.py --size --auth-mode=cas -u ${MOTU_USER} -p ${MOTU_PASSWORD}  -m ${MOTU_SERVER_URL} -s HR_MOD_NCSS-TDS -d HR_MOD -z 0.49 -Z 0.50 -x -70 -X 25 -y -75 -Y 10 -t "2016-06-10" -T "2016-06-11" -v salinity -o /data -f getSizeResult.xml

Display the extraction size as XML on stdout

./motuclient.py --quiet --size --auth-mode=cas -u ${MOTU_USER} -p ${MOTU_PASSWORD}  -m ${MOTU_SERVER_URL} -s HR_MOD_NCSS-TDS -d HR_MOD -z 0.49 -Z 0.50 -x -70 -X 25 -y -75 -Y 10 -t "2016-06-10" -T "2016-06-11" -v salinity -o console

DescribeProduct

See https://github.com/clstoulouse/motu#describe-product for more details about XML result.

Get the XML file which contains the dataset description on the local machine

./motuclient.py -D --auth-mode=cas -u ${MOTU_USER} -p ${MOTU_PASSWORD}  -m ${MOTU_SERVER_URL} -s HR_MOD_NCSS-TDS -d HR_MOD -o /data -f describeProductResult.xml

Display the dataset description XML result on stdout

./motuclient.py --quiet -D --auth-mode=cas -u ${MOTU_USER} -p ${MOTU_PASSWORD}  -m ${MOTU_SERVER_URL} -s HR_MOD_NCSS-TDS -d HR_MOD -o console

Licence

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.

Troubleshooting

Unable to download the latest version watched on GitHub from PIP

Example: 

pip install motuclient  
Collecting motuclient  
  Using cached https://test-files.pythonhosted.org/packages/4a/7d/41c3bdd973baf119371493c193248349c9b7107477ebf343f3889cabbf48/motuclient-X.Y.Z.zip  
Installing collected packages: motuclient  
  Running setup.py install for motuclient ... done  
Successfully installed motuclient-X.Y.Z

Clear your PIP cache: On Windows, delete the folder %HOMEPATH%/pip. On archlinux pip cache is located at ~/.cache/pip. After re run the command: 

pip install motuclient  
Collecting motuclient  
  Using https://test-files.pythonhosted.org/packages/4a/7d/41c3bdd973baf119371493c193248349c9b7107477ebf343f3889cabbf48/motuclient-X.Y.Z.zip  
Installing collected packages: motuclient  
  Running setup.py install for motuclient ... done  
Successfully installed motuclient-X.Y.Z

From Windows, Parameter error

From Windows, the command "motuclient.py --version" returns an error.
10:44:24 [ERROR] Execution failed: [Excp 13] User (option 'user') is mandatory when 'cas' authentication is set. Please provide it.

Analyse:
This issue comes from the fact that Windows command line does not pass parameters to python command.

Solution: 

Edit the Windows Registry Key "HKEY_CLASSES_ROOT\py_auto_file\shell\open\command" and append at the end of the value %*  
Exemple: "C:\dvltSoftware\python\Python27\python.exe" "%1" %*

Error on all motuclient commands

For example the command "motuclient.py --version" returns this kind of error: 

Traceback (most recent call last):
  File "C:\dvlt\python\python2.7.18\Scripts\motuclient-script.py", line 11, in <module>
    load_entry_point('motuclient==3.0.0.post1', 'console_scripts', 'motuclient')()
  File "c:\dvlt\python\python2.7.18\lib\site-packages\motuclient\motuclient.py", line 352, in main
    initLogger()
  File "c:\dvlt\python\python2.7.18\lib\site-packages\motuclient\motuclient.py", line 336, in initLogger
    logging.addLevelName(utils_log.TRACE_LEVEL, 'TRACE')
AttributeError: 'module' object has no attribute 'TRACE_LEVEL'

Analyse:
This issue comes from a too old python installation version. You must use Python 3.7.10 or higher.

Solution:
Find and install the Python 3 distribution for your operating system.