= Chronicle Bytes Chronicle Software :css-signature: demo :toc: macro :toclevels: 2 :icons: font
image:https://maven-badges.herokuapp.com/maven-central/net.openhft/chronicle-bytes/badge.svg[caption="",link=https://maven-badges.herokuapp.com/maven-central/net.openhft/chronicle-bytes] image:https://javadoc.io/badge2/net.openhft/chronicle-bytes/javadoc.svg[link="https://www.javadoc.io/doc/net.openhft/chronicle-bytes/latest/index.html"] //image:https://javadoc-badge.appspot.com/net.openhft/chronicle-bytes.svg?label=javadoc[JavaDoc, link=https://www.javadoc.io/doc/net.openhft/chronicle-bytes] image:https://img.shields.io/github/license/OpenHFT/Chronicle-Bytes[GitHub] image:https://img.shields.io/badge/release%20notes-subscribe-brightgreen[link="https://chronicle.software/release-notes/"] image:https://sonarcloud.io/api/project_badges/measure?project=OpenHFT_Chronicle-Bytes&metric=alert_status[link="https://sonarcloud.io/dashboard?id=OpenHFT_Chronicle-Bytes"]
image::images/Bytes_line.png[width=20%]
toc::[]
== Introduction
Chronicle Bytes is a comprehensive library dedicated to providing low-level memory access wrappers. It is constructed on the foundations of Chronicle Core, taking advantage of its direct memory and Operating System (OS) system call access.
In essence, Chronicle Bytes serves a similar purpose to Java's Non-blocking I/O (NIO) ByteBuffer, but with several additional features. The API offers:
=== Supported Data Types
The following table outlines the operations and data types supported by Chronicle Bytes:
|=== | Operation | Indexed or Streaming | Binary | Text | Read/Write Binary Primitives | Both | float, double, boolean and unsigned/signed byte, short, 24-bit int, int, long, incompleteLong | double, int, long, char, double with precision | Read/Write Text | Both | 8-bit/UTF-8 String with Length or Limit | 8-bit/UTF-8 String | Read/Write Other | Streaming | Histogram, Named Enum | BigDecimal, BigInteger, Date/Time/Zone, UUID, Hexadecimal | Data Driven Tests | From Files | No | Yes | CAS (Compare-and-Swap) | Indexed | int, long | | Volatile Read/Write | Indexed | byte, short, int, long, float, double | | Peek | Both | unsigned byte | | Stop Bit Compression | Streaming | int, long, double, float, char | | Search | From Start | indexOf String, findByte | | addAndGet | Indexed | float, double, int, long | | copy | From Start | write, copy | | hash | From Start | byteSum, fastHash | | BytesMarshallable | Streaming | Nested Data Structures, Expected Types Only. | |===
=== Explanation of Data Types
Here is an explanation for each operation supported by Chronicle Bytes:
|=== | Read/Write Binary Primitives | Reading and writing of primitive data structures stored in binary form | Read/Write Text | Reading and writing of text data | Read/Write Other | Reading and writing of other data types | Data Driven Tests | Refer to https://en.wikipedia.org/wiki/Data-driven_testing[Data-Driven Testing] | CAS (Compare-and-Swap) | An atomic instruction used in multithreading for synchronization. It compares the contents of a memory location with a given value and, only if they match, updates the contents of that memory location to a new given value | Volatile Read/Write | Refer to http://tutorials.jenkov.com/java-concurrency/volatile.html[Volatile Variables] | Peek | An operation that retrieves the value of the bytes without affecting its read position | Stop Bit Compression | Refer to https://github.com/OpenHFT/RFC/tree/master/Stop-Bit-Encoding[Stop Bit Encoding] | Search | Any algorithm that addresses the search problem i.e., retrieving information stored within a data structure | AddAndGet | An operation that atomically adds a given value to the current value | Copy | Transferring data from one structure to another | Hash | Refer to https://en.wikipedia.org/wiki/Hash_function[Hash Function] | Bytes Marshallable | A serialization function |===
=== Creating Bytes Instances
This section provides examples of how to create Bytes instances with different types of underlying data structures.
Bytes instance which wraps an on heap byte array:Bytes instance which wraps a pre-sized on heap byte array:Bytes instance which wraps an on heap ByteBuffer:Bytes instance which wraps a direct ByteBuffer:Bytes
Bytes instance which wraps some native memory:Bytes bytes = Bytes.allocateElasticDirect(64); long memoryAddress = bytes.address();
Bytes instance which will wrap some native memory when used:Bytes bytes = Bytes.allocateElasticDirect(); // use the bytes
=== Understanding Bytes Capacity, Read/Write Position and Limit
A Bytes buffer provides the following properties:
Bytes buffer.writePosition.[#img-Bytes]
.The diagram below illustrates a Bytes buffer with its read/write position markers.
image::docs/images/Figure1.png[450,450]
In the illustration above, note that readPosition() should always be less than or equals to writePosition() and greater than or equal to start().
Also, readLimit() should always be less than or equals to writeLimit() and greater than or equal to start().
=== BytesStore
BytesStore is a block of memory with fixed size into which you can write data and later read. You can create a BytesStore using the bytes in a CharSequence, String, ByteBuffer or another BytesStore.
//Create a BytesStore bs using bytes in a String. This gives you a BytesStore with //fixed size 18. BytesStore bs = BytesStore.from("This is an example");
You can see the buffer cursors of bs.
prints
You can write into a BytesStore from an offset, however if your data is larger than the capacity of BytesStore, the ByteStore will not extend.
NOTE: The returned object (BytesStore) is unchecked in terms of memory access, therefore the user code must make every effort not to exceed the underlying memory segment limit. Otherwise, the result is unspecified side effects including silently writing over other memory segments, and crashing the JVM.
//Write String "Another example.." into bs starting from offset 0. bs.writeUtf8(0,"Another example..");
==== Bytes versus BytesStore
The realCapacity of bb extends to 4096. Now, the cursors of bb Bytes are:
//Write another data starting from index 5 which needs more bytes therefore bb extends. bb.writePosition(5); bb.write("sentence was overwritten from index 5 using writePosition cursor");
//Prints out: "This sentence was overwritten from index 5 using writePosition cursor" System.out.println(bb.toString());
//Read bb from index 43 bb.readPosition(43); String st = bb.to8bitString();
=== bytesForRead() and bytesForWrite()
The bytesForRead() and bytesForWrite() methods can be used to create Bytes from a section of a BytesStore or a Bytes.
The size of the new Bytes depends on the limit and position cursors of the original Object.
. When you use bytesForWrite() readLimit and writePosition cursors are set to start.
. When you use bytesForRead(), readPosition is set to start, and readLimit and writePosition cursors are set to realCapacity.
The new Bytes is not elastic and whether you use bytesForRead() or bytesForWrite() you can read and write from/into the new Bytes using cursors.
//Create a BytesStore bs using bytes in a String. BytesStore bs = BytesStore.from("This is an example");
//Create a Bytes from bs using bytesForRead(). Bytes bfr = bs.bytesForRead();
The cursors of bs:
The cursors of bfr:
The cursors of bfw:
writeLimit is set to capacity and the new Bytes is elastic.. When you use bytesForRead() readPosition is set to start, and writeLimit is set to realCapacity and the new Bytes is not elastic.
Regardless if bytesForRead() or bytesForWrite() is used, you can both read and write from/into the new Bytes using cursors.
//Create a Bytes bb with default size (256 bytes) and write a text into it.
Bytes
//Create a Bytes bfr2 from bb using bytesForRead(). Bytes bfr2 = bb.bytesForRead();
The cursors of bb:
The cursors of bfr2:
The cursors of bfw2:
=== Flipping Bytes
The standard Java ByteBuffer needs to be flipped to switch between reading and writing.
Bytes holds a read position and a write position allowing you to write and immediately read without flipping.
NOTE: The writePosition is the readLimit.
=== Writing to a Hexadecimal dump
Writing to a hexadecimal dump is useful for documenting the format for messages written. We have used the hexadecimal dump here.
// only used for documentation HexDumpBytes bytes = new HexDumpBytes(); bytes.comment("true").writeBoolean(true); bytes.comment("s8").writeByte((byte) 1); bytes.comment("u8").writeUnsignedByte(2); bytes.comment("s16").writeShort((short) 3); bytes.comment("u16").writeUnsignedShort(4); bytes.comment("char").writeUnsignedShort('5'); // char bytes.comment("s24").writeInt24(-6_666_666); bytes.comment("u24").writeUnsignedInt24(16_666_666); bytes.comment("s32").writeInt(6); bytes.comment("u32").writeUnsignedShort(7); bytes.comment("s64").writeLong(8); bytes.comment("f32").writeFloat(9); bytes.comment("f64").writeDouble(10);
prints
to read this data you can use
=== Writing and reading using offsets
Instead of streaming the data, sometimes you need to control the placement of data, possibly at random.
Bytes
System.out.println(bytes.toHexString());
prints
NOTE: While HexDumpBytes supports the offset methods, you need to provide the offset in binary and the dump making it more complex to use.
==== Volatile read and ordered write
Chronicle Bytes supports variants of the write primitives which have a store barrier writeOrderedXxxx, and reads with a load barrier readVolatileXxxx
NOTE: write ordered doesn't stall the pipeline to wait for the write to occur, making it possible for a single thread to read an old value after the ordered write.
=== Working with text
You can also write and read text to Bytes for low level, direct to native memory text processing.
Bytes
prints
NOTE: There are fewer methods for text as 8, 16 and 24 bit can use methods for int, Unsigned int can use the long method.
=== Reading and Writing Strings
Chronicle Bytes supports two encodings, ISO-8859-1 and UTF-8. It also supports writing these as binary with a length prefix, and a string which should be terminated. Bytes expects Strings to be read to a buffer for further processing, possibly with a String pool.
HexDumpBytes bytes = new HexDumpBytes(); bytes.comment("write8bit").write8bit("£ 1"); bytes.comment("writeUtf8").writeUtf8("£ 1"); bytes.comment("append8bit").append8bit("£ 1").append('\n'); bytes.comment("appendUtf8").appendUtf8("£ 1").append('\n');
prints
Binary strings are prefixed with a https://github.com/OpenHFT/RFC/blob/master/Stop-Bit-Encoding/Stop-Bit-Encoding-1.0.adoc[Stop Bit Encoded] length.
HexDumpBytes bytes = new HexDumpBytes(); bytes.comment("write8bit").write8bit((String) null); bytes.comment("writeUtf8").writeUtf8(null);
System.out.println(bytes.toHexString());
prints
NOTE: 80 00 is the stop bit encoding for -1 or ~0
=== Compare and Set operation
In binary, you can atomically replace an int or long on condition that it is an expected value.
int and long are
[source,Java]HexDumpBytes bytes = new HexDumpBytes();
bytes.comment("s32").writeUtf8("s32"); long s32 = bytes.writePosition(); bytes.writeInt(0);
bytes.comment("s64").writeUtf8("s64"); long s64 = bytes.writePosition(); bytes.writeLong(0);
prints
assertTrue(bytes.compareAndSwapInt(s32, 0, Integer.MAX_VALUE)); assertTrue(bytes.compareAndSwapLong(s64, 0, Long.MAX_VALUE));
prints
INFO: You might wonder, how is the hex dump updated as well as the binary?
The readPosition actually holds the write position for both, which is why it has to be computed in this case.
=== Stop bit compression
Stop Bit encoding is one form of simple compression. For each 7 bits set, a byte is used with the high bit set when there is another byte to write.
See https://github.com/OpenHFT/RFC/blob/master/Stop-Bit-Encoding/Stop-Bit-Encoding-1.0.adoc[Stop Bit Encoding RFC] for more details
HexDumpBytes bytes = new HexDumpBytes();
for (long i : new long[]{ 0, -1, 127, -127, 128, -128, 1 << 14, 1 << 21, 1 << 28, 1L << 35, 1L << 42, 1L << 49, 1L << 56, Long.MAX_VALUE, Long.MIN_VALUE}) { bytes.comment(i + "L").writeStopBit(i); }
for (double d : new double[]{ 0.0, -0.0, 1.0, 1.0625, -128, -Double.MIN_NORMAL, Double.NEGATIVE_INFINITY, Double.NaN, Double.POSITIVE_INFINITY}) { bytes.comment(d + "").writeStopBit(d); }
prints
To read these you need either long x = bytes.readStopBit() or double d = bytes.readStopBitDouble()
=== BytesMarshallable objects
Chronicle Bytes supports serializing simple objects where the type is not stored.
This is similar toRawWire in Chronicle Wire.
class MyByteable implements BytesMarshallable { boolean flag; byte b; short s; char c; int i; float f; long l; double d;
public MyByteable(boolean flag, byte b, short s, char c, int i, float f, long l, double d) {
this.flag = flag;
this.b = b;
this.s = s;
this.c = c;
this.i = i;
this.f = f;
this.l = l;
this.d = d;
}class MyScalars implements BytesMarshallable { String s; BigInteger bi; BigDecimal bd; LocalDate date; LocalTime time; LocalDateTime dateTime; ZonedDateTime zonedDateTime; UUID uuid;
public MyScalars(String s, BigInteger bi, BigDecimal bd, LocalDate date, LocalTime time, LocalDateTime dateTime, ZonedDateTime zonedDateTime, UUID uuid) {
this.s = s;
this.bi = bi;
this.bd = bd;
this.date = date;
this.time = time;
this.dateTime = dateTime;
this.zonedDateTime = zonedDateTime;
this.uuid = uuid;
}prints
01 # mn1
4e # flag
01 # b
02 00 # s
33 # c
04 00 00 00 # i
00 00 b0 40 # f
06 00 00 00 00 00 00 00 # l
cd cc cc cc cc cc 1e 40 # d
# scalars
05 48 65 6c 6c 6f # s
01 31 # bi
02 31 30 # bd
0a 32 30 31 37 2d 31 31 2d 30 36 # date
0c 31 32 3a 33 35 3a 35 36 2e 37 37 35 # time
17 32 30 31 37 2d 31 31 2d 30 36 54 31 32 3a 33 # dateTime
35 3a 35 36 2e 37 37 35 27 32 30 31 37 2d 31 31 # zonedDateTime
2d 30 36 54 31 32 3a 33 35 3a 35 36 2e 37 37 35
5a 5b 45 75 72 6f 70 65 2f 4c 6f 6e 64 6f 6e 5d # uuid
24 30 30 30 30 30 30 30 31 2d 32 33 34 35 2d 36
37 38 39 2d 30 30 30 30 2d 30 30 30 30 30 30 61
62 63 64 65 6602 # mn2
59 # flag
0b # b
16 00 # s
54 # c
2c 00 00 00 # i
8f c2 b1 40 # f
42 00 00 00 00 00 00 00 # l
e1 7a 14 ae 47 71 53 40 # d
# scalars
05 57 6f 72 6c 64 # s
01 30 # bi
01 30 # bd
0a 32 30 31 36 2d 31 30 2d 30 35 # date
0c 30 31 3a 33 34 3a 35 36 2e 37 37 35 # time
17 32 30 31 36 2d 31 30 2d 30 35 54 30 31 3a 33 # dateTime
34 3a 35 36 2e 37 37 35 2c 32 30 31 36 2d 31 30 # zonedDateTime
2d 30 35 54 30 31 3a 33 34 3a 35 36 2e 37 37 35
2b 30 31 3a 30 30 5b 45 75 72 6f 70 65 2f 4c 6f
6e 64 6f 6e 5d 24 31 31 31 31 31 31 31 31 2d 31 # uuid
31 31 31 2d 31 31 31 31 2d 32 32 32 32 2d 32 32
32 32 32 32 32 32 32 32 32 32== Data driven tests
The purpose of a Lambda function is to create a simple, highly reproducible, easily testable component.
Once you have your data dumped as hexadecimal, you can create tests using that data, and make variations of those tests.
=== What do we mean by a Lambda function?
In this context a Lambda function is one which is entirely input driven and produces a list of messages (one or more outputs).
The simplest Lambda function is stateless, however this has limited application. They are useful for message translation.
If you need a stateful Lambda function, you can consider the input to the function to be every message it has ever consumed. Obviously this is inefficient, however with appropriate caches in your lamdba function, you can process and produce result incrementally.
=== Data in and out.
We module a Lambda function as having an interface for inputs and another for outputs. These interfaces can be the same.
interface IBytesMethod { @MethodId(0x81L) // <1> void myByteable(MyByteable byteable);
@MethodId(0x82L)
void myScalars(MyScalars scalars);
@MethodId(0x83L)
void myNested(MyNested nested);