QutEcoacoustics / baw-workers

Workers that can process various long-running or intensive tasks.
Apache License 2.0
3 stars 0 forks source link

Baw::Workers

Bioacoustics Workbench workers. Provides workers and file storage.

Workers can process various long-running or intensive tasks. File storage provides helper methods for calculating paths to original and cached files.

Build Status Dependency Status Code Climate Test Coverage Documentation Status Documentation

Installation

Add this line to your application's Gemfile:

gem 'baw-workers', git: 'https://github.com/QutBioacoustics/baw-workers.git'

or clone the repository to the current directory:

git clone https://github.com/QutBioacoustics/baw-workers.git

And then execute:

$ bundle install

Actions

This project provides four actions. Actions are classes that implement a potentially long-running process.

Analysis

Runs analysers over audio files. This action analyses an entire single audio file.

  1. Resque jobs can be queued from baw-server and processed later by a Resque dequeue worker.
  2. A directory can be analysed manually by providing the settings for a single audio file in yaml format for the the analysis_config_file parameter.

Audio Check

Runs checks on original audio recording files. This action checks an entire single audio file.

Harvest

Harvests audio files to be accessible by baw-server via the file storage system.

Media

Cuts audio files and generates spectrograms.

Dependencies

You may need to install some additional tools for working with audio and images, and for processing long-running tasks. See baw-audio-tools for more information.

File Storage

There are classes for working with file storage paths:

Running Workers

A worker runs an action. Actions are simply a process to follow. Actions can get input from the settings file when run standalone and/or from Resque jobs (when running as a Resque dequeue worker).

Then answer these questions:

Based on the answers to these questions

Once you've got your config file, then a worker can be started. Workers are run using rake tasks. A list of the available rake tasks can be obtained by running this command in the baw-workers cloned directory or the directory containing your Gemfile:

bundle exec rake -T

There are two steps that a worker can run:

There are three ways to run a worker:

Configuration Hints

Some things to check and look out for when creating and modifying worker config files.

settings.resque.queues_to_process

This setting is only needed when running a Resque dequeue worker. It specifies a priority array of the Resque queues to reserve jobs from. The jobs in a queue specify the action class that will be used to process that job.

settings.resque.connection

The connection settings are passed directly to Resque to configure the Redis connection.

settings.resque.namespace

The Redis namespace. This should usually be left as 'resque'.

settings.resque.background_pid_file

Specify a background_pid_file to have a Resque dequeue worker run in the background. The output_log_file and error_log_file settings will only be used when a Resque dequeue worker is running in the background.

settings.actions

Each action has some settings specific to that action. An action is the actual processing that job arguments will be used to carry out.

Every action has a queue setting. The queue is the name of the queue the action will add jobs to when running as a Resque enqueue worker. See the Actions section below for more information about action-specific settings.

settings.endpoints and settings.api

These settings must match the equivalent baw-server settings.

log_level settings

Each log_level setting is independent of the others.

Examples for running a worker

Replace 'settings_file' with the full path to the settings file to use for the worker. Other parameters are described in the Actions section below.

Standalone

bundle exec rake baw:analysis:standalone:from_files[settings_file,analysis_config_file]  # Analyse audio files directly
bundle exec rake baw:audio_check:standalone:from_csv[settings_file,csv_file]             # Enqueue audio recording file checks from a csv file to be processed directly
bundle exec rake baw:harvest:standalone:from_files[settings_file,harvest_dir]            # Harvest audio files directly
# media action can only be run as a Resque dequeue worker

Resque enqueue

bundle exec rake baw:analysis:resque:from_files[settings_file,analysis_config_file]      # Enqueue files to analyse using Resque
bundle exec rake baw:audio_check:resque:from_csv[settings_file,csv_file]                 # Enqueue audio recording file checks from a csv file to be processed using Resque worker
bundle exec rake baw:harvest:resque:from_files[settings_file,harvest_dir]                # Enqueue files to harvest using Resque
# media action can only be run as a Resque dequeue worker

Resque dequeue

A Resque dequeue worker can process any queue with any type of job.

bundle exec rake baw:worker:current[settings_file]                                       # List running workers
bundle exec rake baw:worker:run[settings_file]                                           # Run a resque:work with the specified settings file
bundle exec rake baw:worker:setup[settings_file]                                         # Run a resque:work with the specified settings file
bundle exec rake baw:worker:stop_all[settings_file]                                      # Quit running workers

Contributing

  1. Fork this repo
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create a new pull request