This is a queue-size-ware version of memcacheq (http://memcachedb.org/memcacheq/) from xunxin and agentzh originally developed for Yahoo! China's webpage information extraction cluster based on WebKit.

It's opensourced here meant to be useful for others and for the original author of memcacheq's potential consideration of merging back to the mainstream.

This fork adds the ability to limit and query the length of queues and to specify a length limit when creating a queue.

Here's the usage of this fork.

• Server side

  $ memcacheq -u my_user -p 5000 -B 4064 -A 4096 -r -c 1024 -m 64 -N \ -H /path/to/my/bdbdata/base

where -u indicates the user who runs this deamon, -p specifies a port number that this daemon listens to, -B specifies the message body length in bytes (but actually you can only store messages of size tens of bytes smaller that that), -A specifies the underlying BDB page size, -m specifies the in-memory cache size, -N uses BDB's no-sync feature to gain more speed at the cost of consistency, and -H specifies the on-disk storage location. See memcacheq's command-line usage for more information.

To start the memcacheq daemon:

If there's corruption in your memcacheq's underlying BDB database, try the following command:

$ /path/to/your/bdb/bin/db_recover -h /path/to/your/bdbdata/base

• Client side

We use pseudo libmemcached C code to demonstrate the client-side usage.

1. To create a queue:

   memcached_add(memc, queue_name, size_limit);

where "size_limit" is the maximum number of elements in that queue named "queue_name".

Note that the number of queues is limited by the underlying BDB queue storage.

2. To remove a queue:

   memcached_delete(memc, queue_name);

3. To insert an element to a queue:

   memcached_set(memc, queue_name, element_content);

Once the queue has already reached the length limit, it returns the standard memcached "NOT STORED" exception code.

4. To read an element from a queue:

   element_content = memcached_get(memc, queue_name);

Once the queue is already empty, it returns a standard memcached "NOT FOUND" exception code.

5. To monitor the queues' state in a certain memcacheq server (here we use a shell command to illustrate):

   $ echo stats queue | nc 10.62.100.35:11211

where the memcacheq daemon listens the 11211 port at 10.62.100.35. A typical instance of the output might be:

STAT bbsdetails 0 2000000
STAT bbslists 72 2000000
STAT comment 3 1234567
STAT done 0 1234567
STAT initial 25 1234567
STAT pagecat 5006 1234567
STAT preprocessed 10 500
END

where the second column in the listing specify the names of the queues, the third column the current size of the queues, and the forth column the size limit of the queues.

Note that the queue size is stored separately in contrast to real-time counting, so it might be kinda out of sync if the daemon exits abnormally and the -N option is specified when starting the daemon.

We usually use the command-line utilities as well as the high-level Perl 5 library provided by the Queue::Memcached::Buffered module opensourced here:

http://github.com/agentzh/queue-memcached-buffered

Below is the original memcacheq documentation in README:

=============================================== MemcacheQ - Simple Queue Service over Memcache

Features

• damn simple
• very fast
• multiple queue
• concurrent well
• memcache protocol compatible

Getting Started

Download

See: http://code.google.com/p/memcacheq/downloads/list

Installation

See: http://memcachedb.org/memcacheq/INSTALL.html

Please take a look at 'ChangLog' file in the distribution, see what's new.

Commands

Only two commands are used to operate the queue:

Append a message to the tail of queue::

set 0 \r\n