We are making use of the V3 API of openBIS (https://wiki-bsse.ethz.ch/display/openBISDoc1605/openBIS+V3+API) in order to interact with the data management system from the command line, in order to provide a quick data retrieval on the server or cluster resources, where the download via the qPortal is impractical.
If you want to build from source, check out the commit of your choice and execute mvn clean package. We only recommend this if you are familiar with Java build systems though, as we cannot give you support here. In the normal case, the binary of a stable release is sufficient.
Requirements
You need to have Java JRE or JDK installed (openJDK is fine), at least version 17. And the client's host must have allowance to connect to the server, which is determined by our firewall settings. If you are unsure, if your client is allowed to connect, contact us at support@qbic.zendesk.com.
Configuration
Postman uses picocli file arguments. Therefore, a config file has to be passed with the '@' prefix to its path:
Example:
java -jar postman.jar list -u <user> <sample> @path/to/config.txt
The structure of the configuration file is:
[-cliOption] [value]
For example: To set the ApplicationServerURL to another URL we have to use: -as [URL]
To use our openBIS URL we write the following lines in the config file:
(Anything beginning with '#' is a comment)
# Config file for postman-cli
# Replace the values defined after the respective CLI parameters (e.g. -u)
# with your value of choice (e.g. qbcab01 as value for -u)
--suffix .txt,.fastq,.fastq.gz
-u qbc001
--password:env MY_PASSWORD
Just execute the following command to get an overview of the available options.
java -jar postman-cli.jar -h
Usage: postman-cli [-hV] COMMAND
Description:
A software client for downloading data from QBiC.
Options:
-h, --help Show this help message and exit.
-V, --version Print version information and exit.
Commands:
download Download data from QBiC.
list lists all the datasets found for the given identifiers
Optional: specify a config file by running postman with '@/path/to/config.txt'.
A detailed documentation can be found at
https://github.com/qbicsoftware/postman-cli#readme.
Printed file sizes
File sizes printed by postman Kilobyte, Megabyte, Gigabyte, Terabyte and Petabyte use base 2.
How to specify your log directory
By default, postman will create log files in a directory ./logs/ in your working directory.
You can specify where logs are written by setting the system property log.path to the desired directory.
How to provide your credentials
After gaining access by applying through support@qbic.zendesk.com, you can log in using your credentials.
The username is provided by us or if you have a uni-tuebingen account by the university of Tuebingen.
Provide your username:
Use the option -u / --user to provide us with your username.
Provide your password:
Please never send your password over email. We will never ask you for it!
When using the application, you can either:
enter your password interactively --password
enter the name of a system property containing your password --password:prop my.awesome.property
To specify which data you want to list or download, you need to provide us with QBiC identifiers.
A QBiC project identifier begins with Q followed by four characters (QTEST). QBiC sample identifiers contain their project identifier.
You can provide identifiers either using the command line directly:
Please note: When you use the * character to search for all files in a project, escape your identifier using quotation marks.
Provide identifiers using a file:
You can provide identifiers using a file containing the identifiers. Lines starting with # are ignored.
# all files for the project
QTEST*
# all files associated with these samples
QSTTS001AB
QSTTS002BC
java -jar postman.jar -f myids.txt
How to filter files by suffix
Both the download and the list command allow you to filter files by their suffix. The suffixes provided are not case-sensitive.
.TXT and .txt will have the same effect.
Multiple suffixes can be provided separated by a comma. A suffix does not have to be the file ending but can be any substring from the end of a file's name.
If you only want to download fastq and fastq.gz files you can run postman with
java -jar postman.jar -s .fastq,.fastq.gz
list
Usage: postman-cli list [-hV] [--exact-filesize] [--with-checksum]
[--without-header] [--format=<outputFormat>] -u=<user>
[-s=<suffix>[,<suffix>...]]... (--password:
env=<environment-variable> | --password:
prop=<system-property> | --password) (-f=<filePath> |
SAMPLE_IDENTIFIER...)
Description:
lists all the datasets found for the given identifiers
Parameters:
SAMPLE_IDENTIFIER... one or more QBiC sample identifiers
Options:
--password:env=<environment-variable>
provide the name of an environment variable to
read in your password from
--password:prop=<system-property>
provide the name of a system property to read in
your password from
--password please provide your password
-u, --user=<user> openBIS user name
-f, --file=<filePath> a file with line-separated list of QBiC sample ids
-s, --suffix=<suffix>[,<suffix>...]
only include files ending with one of these
(case-insensitive) suffixes
--with-checksum list the crc32 checksum for each file
--exact-filesize use exact byte count instead of unit suffixes:
Byte, Kilobyte, Megabyte, Gigabyte, Terabyte and
Petabyte
--format=<format> The format to list files in. Case-insensitive.
Possible values: LEGACY, TSV
Default: LEGACY
--without-header remove the header line from the output. Only takes
effect for tabular output formats.
-h, --help Show this help message and exit.
-V, --version Print version information and exit.
Optional: specify a config file by running postman with '@/path/to/config.txt'.
A detailed documentation can be found at
https://github.com/qbicsoftware/postman-cli#readme.
The list command comes with some special options. You can change how your output looks by
listing the exact number of bytes for each file --exact-filesize
removing the header from the tabular output --without-header
listing the crc32 checksum for every file --with-checksum
# Dataset NGSQSTTS015A0 (20211026111452695-847006)
# Source NGSQSTTS015A0
# Registration 2021-10-26T09:14:53.143812Z
# Size 1,00 B
1,00 B my_genome.fastq
# Dataset NGSQSTTS015A0 (20211026111452695-847006)
# Source NGSQSTTS015A0
# Registration 2021-10-26T09:14:53.143812Z
# Size 1,00 B
1,00 B fa0a3f23 my_genome.fastq
download
Usage: postman-cli download [-hV] [-o=<outputPath>] -u=<user> [-s=<suffix>[,
<suffix>...]]... (--password:
env=<environment-variable> | --password:
prop=<system-property> | --password) (-f=<filePath>
| SAMPLE_IDENTIFIER...)
Description:
Download data from QBiC.
Parameters:
SAMPLE_IDENTIFIER... one or more QBiC sample identifiers
Options:
--password:env=<environment-variable>
provide the name of an environment variable to
read in your password from
--password:prop=<system-property>
provide the name of a system property to read in
your password from
--password please provide your password
-u, --user=<user> openBIS user name
-f, --file=<filePath> a file with line-separated list of QBiC sample ids
-s, --suffix=<suffix>[,<suffix>...]
only include files ending with one of these
(case-insensitive) suffixes
-o, --output-dir=<outputPath>
specify where to write the downloaded data
-h, --help Show this help message and exit.
-V, --version Print version information and exit.
Optional: specify a config file by running postman with '@/path/to/config.txt'.
A detailed documentation can be found at
https://github.com/qbicsoftware/postman-cli#readme.
The download command allows you to download data from our storage to your machine.
Use the --output-dir option to specify a location on your client the location will be interpreted relative to your working directory.
File integrity check
Postman computes the CRC32 checksum for all input streams using the native Java utility class CRC32. Postman favours CheckedInputStream
over the traditional InputStream, and promotes the CRC checksum computation.
Computed CRC32 checksums are compared with CRC32 checksums stored on our servers.
When the checksum does not match, then the download failed. Each download is attempted multiple times.
When all attempts fail, the mismatching checksum is recorded in your log directory in the checksum-mismatch.log file.
The checksum-mismatch.log file contains one line for each file that was not downloaded.
Client software for downloading datasets from QBiC's data management system openBIS (https://wiki-bsse.ethz.ch/display/bis/Home).
We are making use of the V3 API of openBIS (https://wiki-bsse.ethz.ch/display/openBISDoc1605/openBIS+V3+API) in order to interact with the data management system from the command line, in order to provide a quick data retrieval on the server or cluster resources, where the download via the qPortal is impractical.
How to run
Download
You can download the compiled and executable Java binaries as JAR of postman from the GitHub release page: https://github.com/qbicsoftware/postman-cli/releases.
If you want to build from source, check out the commit of your choice and execute
mvn clean package
. We only recommend this if you are familiar with Java build systems though, as we cannot give you support here. In the normal case, the binary of a stable release is sufficient.Requirements
You need to have Java JRE or JDK installed (openJDK is fine), at least version 17. And the client's host must have allowance to connect to the server, which is determined by our firewall settings. If you are unsure, if your client is allowed to connect, contact us at support@qbic.zendesk.com.
Configuration
Postman uses picocli file arguments. Therefore, a config file has to be passed with the '@' prefix to its path:
Example:
The structure of the configuration file is:
For example: To set the ApplicationServerURL to another URL we have to use:
-as [URL]
To use our openBIS URL we write the following lines in the config file:
(Anything beginning with '#' is a comment)
A default file is provided here: default-config.
How to use
Options
Just execute the following command to get an overview of the available options.
Printed file sizes
File sizes printed by postman Kilobyte, Megabyte, Gigabyte, Terabyte and Petabyte use base 2.
How to specify your log directory
By default, postman will create log files in a directory
./logs/
in your working directory. You can specify where logs are written by setting the system propertylog.path
to the desired directory.How to provide your credentials
After gaining access by applying through support@qbic.zendesk.com, you can log in using your credentials. The username is provided by us or if you have a uni-tuebingen account by the university of Tuebingen.
Provide your username:
Use the option
-u / --user
to provide us with your username.Provide your password:
Please never send your password over email. We will never ask you for it!
When using the application, you can either:
--password
--password:prop my.awesome.property
--password:env MY_PASSWORD
How to provide QBiC identifiers
To specify which data you want to list or download, you need to provide us with QBiC identifiers. A QBiC project identifier begins with
Q
followed by four characters (QTEST
). QBiC sample identifiers contain their project identifier. You can provide identifiers either using the command line directly:Please note: When you use the
*
character to search for all files in a project, escape your identifier using quotation marks.Provide identifiers using a file:
You can provide identifiers using a file containing the identifiers. Lines starting with
#
are ignored.How to filter files by suffix
Both the
download
and thelist
command allow you to filter files by their suffix. The suffixes provided are not case-sensitive..TXT
and.txt
will have the same effect. Multiple suffixes can be provided separated by a comma. A suffix does not have to be the file ending but can be any substring from the end of a file's name.If you only want to download
fastq
andfastq.gz
files you can run postman withlist
The
list
command comes with some special options. You can change how your output looks by--exact-filesize
--without-header
--with-checksum
TSV
formatLEGACY
formatdownload
The
download
command allows you to download data from our storage to your machine.Use the
--output-dir
option to specify a location on your client the location will be interpreted relative to your working directory.File integrity check
Postman computes the CRC32 checksum for all input streams using the native Java utility class CRC32. Postman favours
CheckedInputStream
over the traditional InputStream, and promotes the CRC checksum computation.Computed CRC32 checksums are compared with CRC32 checksums stored on our servers. When the checksum does not match, then the download failed. Each download is attempted multiple times. When all attempts fail, the mismatching checksum is recorded in your log directory in the
checksum-mismatch.log
file.The
checksum-mismatch.log
file contains one line for each file that was not downloaded.In addition, Postman writes the CRC32 checksum in an additional file
<file-name-of-checked-file>.crc32
and stores it together with the according file.Advanced Options
postman
-Dlog.path
: provide the log directory-Dlog.level
: provide the log level to use for logging--source-sample-type <sample-type>
: specify which sample type to consider as source sample type.--server-timeout <millis>
: the server timeout in millisecondsdownload
--download-attempts <download-attempts>
provide the maximal amount attempted downloads--buffer-size <buffer-size>
provide a custom buffer size. Please only specify values that are a multiple of1024
.