Popout
is a file-based queue for Java.
Don't forget to take a look at our benchmarks.
Include the dependency to your project's pom.xml file:
<dependencies>
...
<dependency>
<groupId>org.infobip.lib</groupId>
<artifactId>popout</artifactId>
<version>2.1.0</version>
</dependency>
...
</dependencies>
or Gradle:
compile 'org.infobip.lib:popout:2.1.0'
Let's create a minimal config FileQueue
instance:
FileQueue<String> queue = FileQueue.<String>synced().build();
The code above creates synced FileQueue
implementation with default config. The queue data writes on the disk in a small files, named WAL
-files. When the amount of that files is sufficiently (specified in the config, see below) that files merges to a big compressed
file, the next portions of WAL
-files to the next compressed
-file and etc.
The differences between the FileQueue
and the Java-default java.util.Queue
interfaces are the following:
FileQueue
.longSize
- returns the number of elements in this queue with wide range, than int
;FileQueue
.diskSize
- tells the amount of bytes, which the queue
takes on the disk;FileQueue
.flush
- flushes all this queue's data to the disk;FileQueue
.compress
- manually compress all WAL-files into a compressed file;FileQueue
.close
- flushes and closes the files descriptors of the queue.There are two main FileQueue
implementations:
synced - every add
operation is flushes on disk immediately and every poll
reads the items from the disk directly. There is no buffers or something in-memory. It suits for cases, when you don't want to lose your data at all and you don't care about performance. It is the most reliable kind of the FileQueue
;
batched - a concept of tail
and head
buffers is present here. You can specify a batchSize
option, which tells to the queue builder how many elements could be store in memory, before writing to the disk. Writes and reads to/from the disk operations are batched and it boosts the queue's performance, but you always should remember that in case of unexpected crash you could lose your head or tail data. This kind of queue suits well when your need more performant queue and you don't afraid to lose some amount of data, or you are ready to control it your self by periodically invoking the flush
method.
NOTICE: you also could instantiate WAL
maxCount
option andbatchSize
toInteger.MAX_VALUE
and useflush
andcompress
by yourself in fully manual manner.
More advanced FileQueue
usage:
Queue<Integer> queue = FileQueue.<Integer>batched()
// the name of the queue, used in file patterns
.name("popa")
// the default folder for all queue's files
.folder("/folder/where/store/queue/files")
// sets custom serializer
.serializer(Serializer.INTEGER)
// sets custom deserializer
.deserializer(Deserializer.INTEGER)
// set up the queue's limits settings
.limit(QueueLimit.queueLength()
.length(1_000_000)
.handler(myQueueLimitExceededHandler))
// restores from disk or not, during startup. If 'false' - the previous files will be removed
.restoreFromDisk(false)
// handler for corrupted data from disk
.corruptionHandler(new MyCorruptionHandler())
// WAL files configuration
.wal(WalFilesConfig.builder()
// the place where WAL files stores. Default is a queue's folder above
.folder("some/wal/files/folder")
// the maximum allowed amount of WAL files before compression
.maxCount(1000)
.build())
// compressed files config
.compressed(CompressedFilesConfig.builder()
// the place where compressed files stores. Default is a queue's folder above
.folder("some/compressed/files/folder")
// the maximum allowed compressed file's size
.maxSizeBytes(SizeUnit.MEGABYTES.toBytes(256))
.build())
// the amount of elements in one WAL file. only batched queue option
.batchSize(10_000)
.build();
Add some data to the queue to the end of the queue. FileQueue
accepts a generic type of arbitrary length:
queue.add("popa");
Read data at the head of the queue (without removing it):
String record = queue.peek();
In short, FileQueue
implements Queue
, Collection
and Iterable
interfaces and you are able to use all theirs methods:
// Queue's size
int queueSize = queue.size();
// Add many items
queue.addAll(asList("one", "two", "three"));
// Retrieves and removes the head of this queue,
String head = queue.poll();
// Remove all elements.
queue.clear();
// Use queue's iterator
Iterator<String> iterator = queue.iterator();
By default, queue uses standard Java's serialization/deserialization mechanism, but you could override it by implementing Serializer and Deserializer:
Queue<String> queue = FileQueue.<String>synced()
.serializer(<your_serializaer_impl>)
.deserializer(<your_deserializaer_impl>)
.build();
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.
For building the project you need only a Java compiler.
IMPORTANT: Popout requires Java version starting from 8
And, of course, you need to clone Popout from GitHub:
$> git clone https://github.com/infobip/popout
$> cd popout
For building routine automation, I am using maven.
To build the Popout project, do the following:
$> mvn clean package
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 35.491 s
[INFO] Finished at: 2019-01-18T23:25:12+03:00
[INFO] Final Memory: 50M/548M
[INFO] ------------------------------------------------------------------------
To run the project's test, do the following:
$> mvn clean test
...
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
...
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 32, Failures: 0, Errors: 0, Skipped: 0
[INFO]
...
Also, if you do package
or install
goals, the tests launch automatically.
To see what has changed in recent versions of Popout, see the changelog file.
Please read contributing file for details on my code of conduct, and the process for submitting pull requests to me.
We use SemVer for versioning. For the versions available, see the tags on this repository.
This project is licensed under the Apache License 2.0 License - see the license file for details