search

MWATelescope / manta-ray-client

Python API and helper scripts to interact with the MWA ASVO.
9 stars 7 forks source link

readme

Manta-ray Client (MWA ASVO Command Line Client)

Description

Python API and helper script (mwa_client) to interact with the MWA ASVO.

Giant Squid is the preferred CLI client for the MWA ASVO- check it out here Giant Squid

For general help on using the MWA ASVO, please visit: MWA ASVO wiki.

NOTE FOR HPC USERS

Please read this wiki article if you are running manta-ray-client on HPC systems.

mwa_client

mwa_client is a helper script which provides the following functions:

There are three types of MWA ASVO jobs:

Installation Options

You must have an account on the MWA ASVO website

Set your API key as an environment variables in linux (usually in your profile / .bashrc). You can get your API key from Your Profile page on the MWA ASVO website.

~$ export MWA_ASVO_API_KEY=<api key>

Then you may install natively on your computer OR install via Docker.

Installation Natively on your computer

Clone the repository

~$ git clone https://github.com/ICRAR/manta-ray-client.git

Create a virtual environment

python3 -m venv env

or if you are still using python2.7 you will need to use virtualenv (See Setting up Python, Pip, and Virtualenv (external link) for information on installing virtualenv)

~$ virtualenv -p /usr/bin/python2.7 env

Activate the virtual environment

~$ source env/bin/activate
(env)~$

Install mwa_client and all required packages

(env)~$ cd manta-ray-client
(env)~/manta-ray-client$ pip3 install -r requirements.txt
(env)~/manta-ray-client$ python3 setup.py install

Installation using Docker

If you prefer, you can also run the manta-ray-client as a Docker container instead of installing it locally. This assumes you have docker installed on your machine. If not please see the Get Docker (external link) page for instructions.

Clone the repository

~$ git clone https://github.com/mwatelescope/manta-ray-client.git

Build the image

~$ cd manta-ray-client
~/manta-ray-client$ docker build --tag manta-ray-client:latest .

Use The Container

Once the image is built, you can run the mwa_client directly. The below command will:

~$ docker run --name my_mwa_client --entrypoint="" --volume=/your/host/data/path/:/data --rm=true -e MWA_ASVO_API_KEY manta-ray-client:latest mwa_client -w all -d /data

Or you can open a shell within the container itself and then run as many mwa_client commands as you like, interactively, then exit to leave the container:

~$ docker run -it --name my_mwa_client --entrypoint="" --volume=/your/host/data/path/:/data --rm=true -e MWA_ASVO_API_KEY manta-ray-client:latest /bin/bash
root@c197566f86d9:/# mwa_client -l
...
root@c197566f86d9:/# exit
~$

You will get a prompt like the one above and from there you can run mwa_client commands as normal.

Examples

mwa_client -c csvfile -d destdir           Submit jobs in the csv file, monitor them, then download the files, then exit
mwa_client -c csvfile -s                   Submit jobs in the csv file, then exit
mwa_client -d destdir -w JOBID             Download the job id (assuming it is ready to download), then exit
mwa_client -d destdir -w all               Download any ready to download jobs, then exit
mwa_client -d destdir -w all -e error_file Download any ready to download jobs, then exit, writing any errors to error_file
mwa_client -l                              List all of your jobs and their status, then exit

Help

optional arguments:
  -h, --help            Show this help message and exit
  -s, --submit-only     Submit job(s) from csv file then exit (-d is ignored)
  -l, --list-only       List the user's active job(s) and exit immediately
                        (-s, -c & -d are ignored)
  -w DOWNLOAD_JOB_ID, --download-only DOWNLOAD_JOB_ID
                        Download the job id (-w DOWNLOAD_JOB_ID), if it is ready;
                        or all downloadable jobs (-w all | -w 0), then exit (-s, -c & -l are ignored)
  -c FILE, --csv FILE   csv job file
  -d DIR, --dir DIR     Download directory
  -e ERRFILE, --error-file ERRFILE, --errfile ERRFILE
                        Write errors in json format to an error file
  -v, --verbose         Verbose output
  -ar, --allow-resubmit Will allow a job with the same parameters and an existing job in your queue in Completed, Error or Cancelled status to be resubmitted. Default is to not allow resubmission if the new job matches the parameters of an existing job in your queue.

Job States

Each job submitted will transition through the following states:

Submitting Jobs

Users can submit multiple jobs using a CSV file (see below for instructions).

CSV Format

Each row is a single job and each CSV element must be a key=value pair. Whitespace (blank rows) and comments (lines beginning with #) are allowed. Please see the included example.csv for several full working examples.

Conversion Job Options

Please note that some options are only available depending on the choice of preprocessor (explained below).

Flags / Optional Options

Birli currently supports the options below. For more info on the Birli preprocessor, please visit the repository. Any other flags passed will be ignored.

Calibration

Pointing options

If the centre options is omitted, the job will default to using the observations phase centre.

Example line in csv file

obs_id=1110103576, job_type=c, avg_time_res=8, avg_freq_res=40, flag_edge_width=80, output=ms, apply_di_cal=true, no_rfi=true

Download Job Options

Example lines in csv file

obs_id=1110103576, job_type=d, download_type=vis, delivery=acacia
obs_id=1110105120, job_type=d, download_type=vis_meta, delivery=astro
obs_id=1110105120, job_type=d, download_type=vis_meta, delivery=scratch

Voltage Job Options

Note that voltage jobs will always be left on /astro or /scratch, and you will therefore need a Pawsey account to submit them. Please get in contact if you're interested in accessing VCS data.

Example lines in csv file

obs_id=1323776840, job_type=v, offset=0, duration=1200

Understanding and using the error file output

You can get a machine readable error file in JSON format by specifying "-e" | "--error-file" | "--errfile" on the command line. This might be useful if you are trying to automate the download and processing of many observations and you don't want to try and parse the human readable standard output.

An example of the format is below, with two jobs with errors:

[
  {
    "obs_id": "1216295963",
    "job_id": 28979,
    "result": "Error: an error message"
  },
  {
    "obs_id": "1216298341",
    "job_id": 28980,
    "result": "Error: some error message"
  }
]

Since this is JSON, in python you could simply use the below code to iterate through any errors by deserialising the JSON string:

import json

# Open the error file mwa_client produced when using -e
with open("error.txt", "r") as f:
    # Read the JSON from the file into a string
    json_string = f.read()

    # Deserialise the JSON into a python list of objects
    result_list = json.loads(json_string)

    # Iterate through all of the errors
    for r in result_list:
        print("Job:{0} ObsId:{1} Result:{2}", r['job_id'], r['obs_id'], r['result'])