search

veny / GearmaNode

Node.js library for the Gearman distributed job system with support for multiple servers
Apache License 2.0
101 stars 35 forks source link

readme 

        ____                                 _   _           _
       / ___| ___  __ _ _ __ _ __ ___   __ _| \ | | ___   __| | ___
      | |  _ / _ \/ _` | '__| '_ ` _ \ / _` |  \| |/ _ \ / _` |/ _ \
      | |_| |  __/ (_| | |  | | | | | | (_| | |\  | (_) | (_| |  __/
       \____|\___|\__,_|_|  |_| |_| |_|\__,_|_| \_|\___/ \__,_|\___|

Node.js library for the Gearman distributed job system with support for multiple servers.

npm version Build Status

Breaking API change

Features

Installation

> npm install gearmanode

Changelog

See version.js for detailed changelog.

Usage

var gearmanode = require('gearmanode');
var client = gearmanode.client();

var job = client.submitJob('reverse', 'hello world!');
job.on('workData', function(data) {
    console.log('WORK_DATA >>> ' + data);
});
job.on('complete', function() {
    console.log('RESULT >>> ' + job.response);
    client.close();
});
var gearmanode = require('gearmanode');
var worker = gearmanode.worker();

worker.addFunction('reverse', function (job) {
    job.sendWorkData(job.payload); // mirror input as partial result
    job.workComplete(job.payload.toString().split("").reverse().join(""));
});

TOC

See Geaman Manual to understand generic Gearman concepts. See example folder for more detailed samples.

Client

The client is responsible for creating a job to be run and sending it to a job server. The job server will find a suitable worker that can run the job and forwards the job on. -- Gearman Documentation --

Instance of class Client must be created to connect a Gearman job server(s) and to make requests to perform some function on provided data.

var gearmanode = require('gearmanode');
var client = gearmanode.client();

By default, the job server is expected on localhost:4730. Following options can be used for detailed configuration of the client:

// special port
client = gearmanode.client({port: 4732});

// two servers: foo.com:4731, bar.com:4732
client = gearmanode.client({servers: [{host: 'foo.com', port: 4731}, {host: 'bar.com', port: 4732}]});

// two servers with default values: foo.com:4730, localhost:4731
client = gearmanode.client({servers: [{host: 'foo.com'}, {port: 4731}]});

Submit job

Client submits job to a Gearman server and futher processed by a worker via client#submitJob(name, payload, options) where name is name of registered function a worker is to execute, payload is data to be processed and options are additional options as follows:

// by default foreground job with normal priority
var job = client.submitJob('reverse', 'hello world!');

// background job
var job = client.submitJob('reverse', 'hello world!', {background: true});

// full configured job
var job = client.submitJob('reverse', 'hello world!', {background: false, priority: 'HIGH', unique: 'FooBazBar', toStringEncoding: 'ascii'});

Client-side processing of job is managed via emitted events. See Job events for more info.

var client = gearmanode.client();
var job = client.submitJob('reverse', 'hi');
job.on('complete', function() {
    console.log('RESULT: ' + job.response);
    client.close();
});

A client object should be closed if no more needed to release all its associated resources and socket connections. See the sample above.

Client events

Worker

The worker performs the work requested by the client and sends a response to the client through the job server. -- Gearman Documentation --

Instance of class Worker must be created to connect a Gearman job server(s), where it then informs the server(s) of all different functions the Worker is capable of doing.

var gearmanode = require('gearmanode');
var worker = gearmanode.worker();

By default, the job server is expected on localhost:4730. Following options can be used for detailed configuration of the worker:

Register function

A function the worker is able to perform can be registered via worker#addFunction(name, callback, options) where name is a symbolic name of the function, callback is a function to be run when a job will be received and options are additional options as follows:

The worker function callback gets parameter Job which is:

worker.addFunction('reverse', function (job) {
    var rslt = job.payload.toString().split("").reverse().join("");
    job.workComplete(rslt);
});

// or with Timeout and conversion to String

worker.addFunction('reverse', function (job) {
    var rslt = job.payload.toString().split("").reverse().join("");
    job.workComplete(rslt);
}, {timeout: 10, toStringEncoding: 'ascii'});

It tries to connect to ALL job servers and fires error if one registration fails.

A registered function can be unregistered via worker#removeFunction. Call Worker#resetAbilities to notify the server(s) that the worker is no longer able to do any functions it previously registered.

Set Worker ID

This method sets the worker ID in all job servers so monitoring and reporting commands can uniquely identify the various workers. Parameter workerId has to be a non-blank string with no whitespaces.

worker.setWorkerId('FooBazBar');

Worker events

Job

The Job object is an encapsulation of job's attributes and interface for next communication with job server. Additionally is the object en emitter of events corresponding to job's life cycle (see Job events).

The job has following getters

and methods

Job events

Job server

Class JobServer represents an abstraction to Gearman job server (gearmand). Accessible job server(s) are stored in array jobServer on instance of Client/Worker. The class introduces following methods:

var client = gearmanode.client();
var js = client.jobServers[0];

js.once('echo', function(resp) {
    console.log('ECHO: response=' + resp);
    client.close();
});
js.echo('ping')

Job server events

Binary data

Both binary data and text with various encoding are supported. By default the data delivered to client and worker are Buffer objects. You can change this approach by providing toStringEncoding option in Client#submitJob or Worker#addFunction. See following snippets of code or test-all-stack.js for more inspiration.

// send text with default encoding; Job#response will be a Buffer object
client.submitJob('reverse', '123');

// send text with given encoding; Job#response will be a Buffer object
client.submitJob('reverse', Buffer('123', 'ascii').toString());

// send text with given encoding; Job#response will be a String object with ASCII encoding
client.submitJob('reverse', '123', {toStringEncoding: 'ascii'});
// and receive text on Worker; Job#payload will be a String object with ASCII encoding
worker.addFunction('reverse', function (job) {
    job.workComplete(job.payload.split("").reverse().join(""))
}, {toStringEncoding: 'ascii'});

// send binary data
client.submitJob('reverse', new Buffer([49, 50, 51]));

Multiple servers

Many of Gearman job servers can be started for both high-availability and load balancing.

Client is able to communicate with multiple servers with one of the following load balancing strategy:

// default load balancer
client = gearmanode.client({ servers: [{host: 'foo.com'}, {port: 4731}] });

// desired load balancer and recover time
client = gearmanode.client({ servers: [{host: 'foo.com'}, {port: 4731}], loadBalancing: 'RoundRobin', recoverTime: 10000 });

Worker can be initialized with multiple servers in order to register a function on each of them.

Error handling

Although exceptions are supported in JavaScript and they can be used to communicate an error, due to asynchronous concept of Node.js it can be a bad idea. According to Node.js best practices following error handling is introduced in GearmaNode.

Synchronous errors

A synchronous code returns an Error object if something goes wrong. This happens mostly in input value validation.

Asynchronous errors

In asynchronous code an error event will be emitted via EventEmitter on corresponding object if something goes wrong. This happens mostly by network communication failure or if a gearman service fails.

Configuration

Logger

Winston library is used for logging. See the project page for details.

The GearmaNode library registers following loggers:

You can configure the logger in this way:

gearmanode.Client.logger.transports.console.level = 'info';

Class diagram

Tests

> cd /path/to/repository
> mocha

Make sure before starting the tests:

Author

License