= Instaclustr Icarus
image:https://img.shields.io/maven-central/v/com.instaclustr/icarus.svg?label=Maven%20Central[link=https://search.maven.org/search?q=g:%22com.instaclustr%22%20AND%20a:%22icarus%22] image:https://circleci.com/gh/instaclustr/icarus.svg?style=svg["Instaclustr",link="https://circleci.com/gh/instaclustr/icarus"]
Cloud-ready and Kubernetes-enabled Sidecar for Apache Cassandra
image::Icarus.png[Icarus,width=50%]
This repository is home of a sidecar for Apache Cassandra database. Sidecar is meant to be run alongside of Cassandra instance and talks to Cassandra via JMX.
Sidecar JAR is also integrating https://github.com/instaclustr/esop[Esop] for backup and restore seamlessly so it might be used in two modes—either as a long-running server application which responds to clients requests, or as a CLI application via which one can backup/restore a Cassandra node.
NOTE: Icarus is used in https://github.com/instaclustr/cassandra-operator[Instaclustr Kubernetes operator] as well as in https://github.com/Orange-OpenSource/casskop[CassKop]—Kubernetes operator for Cassandra from Orange where it is used for backup and restore of clusters.
== Overview
Icarus is a server-like application which acts as a gateway to the Cassandra node it is logically coupled with. In theory,
Icarus process can run wherever, but it needs to talk to a Cassandra node via JMX connection. If you do not want to expose
JMX remotely, Icarus connects to 127.0.0.1:7199 by default. The other prerequisite for backup and restore operations
is that Icarus has to be able to see the same directory structure as Cassandra does as it uploads/downloads SSTables and so on.
If you have a cluster of, let's say, 5 nodes, there should be one Icarus server running alongside each node.
Each Icarus is independent and will execute operations only against the particular node it is configured to interact with. However, for backups and restores, we need to have a cluster-wide orchestration layer which deals with the complicated logic of on-the-fly backup and restore for the whole cluster. This mode of operation is called "global". When a backup request is sent to one of sidecars, whichever Icarus does the job will act as a coordinator of that operation.
The job of a coordinator Icarus is to disperse individual requests to each node in a cluster (and to itself as well), and continually pull for results and advance the execution once all nodes finish their phases. Backup and restore only advances to the other phase when all nodes have finished the current phase a coordinator orchestrates. After all phases are passed successfully, the coordinator's operation transitions to completed state.
Currently, the coordination/orchestration layer is done only for backup and restore operations as they require such behavior inherently. All other operations are, as of now, executed only against one particular sidecar and only that particular Cassandra node it is configured to connect to is affected.
NOTE: For global operation cases, it is important to realize that there is not any main Icarus; all Icaruses are equal in their role— only if a global request is sent to one particular Icarus during backup or restore will it result in it being promoted to be a coordinator of a cluster-wide backup and restore operation.
The whole idea of how Icarus works in the case of backup/restore is depicted in the following image:
image::orchestration.svg[Icarus Workflow]
c1,c2,c3 are Cassandra nodes, forming a rings1. In the image, it acts
as a coordinator of the whole operations2 and s3 (it sends a request to itself too)s1 knows that other nodes finished a phase because it is continually retrieving these operations and checks its statuses (4)== Build and Release
If you want to build rpm or deb package, you need to enable rpm and/or deb Maven profile.
There are built RPM and DEB packages for each GitHub release as well as Icarus JAR. We are publishing each release to Maven central.
If you install RPM or DEB package, there is a bash script installed in /usr/local/bin/icarus
via which Icarus is interacted with. You can or course invoke Icarus via java -jar icarus.jar.
== CLI Tool or Server?
Whatever installation method you prefer, Icarus contains both an Esop and Icarus servers. The reason for CLI being embedded into Icarus is purely practical—Icarus needs to be able to do backup/restores and this implementation logic is located in Esop, so Icarus depends on this tool. Secondly, we do not need two separate JARs—this is particularly handy when Icarus is used in our Kubernetes operator for Cassandra for restoration purposes. It is invoked like a CLI tool but for the sidecar container; it is invoked like a server application which accepts user's operation requests from the operator.
NOTE: Icarus reuses Esop—that means that the configuration parameters for CLI of Esop have the same meaning for Icarus. Instead of feeding these flags via CLI in the case of Esop, we interact with Icarus via REST by sending a JSON body to the operations endpoint.
To execute Icarus:
To execute help for Icarus:
== Connecting to Cassandra
By default, Icarus uses 127.0.0.1:7199 for JMX connection to Cassandra. Please consult
Icarus help command to learn how to secure it, or how to point it to another location.
The connection to Cassandra via JMX is done lazily; it is only initiated if it is needed and closed afterwards, so each JMX interaction is a completely separate JMX connection.
== REST Interface
You may consult (and try) all operations via our https://instaclustr.github.io/instaclustr-icarus-go-client/[Swagger spec].
To walk-though a reader via an operation lifecycle, let's explain the most "complicated" operations—backups and restores. After the explanation, a user will be able to do a backup and restore against a running Cassandra node (or Cassandra cluster) as well as being able to similarly submit and query any other operation.
The default address and port for REST is 127.0.0.1:4567
=== Backup Request Submission
Let's say we have keyspaces test1 and test2 with tables testtable1,testtable2, and testtable3 respectively
which we want to make a backup of. In order to do that, we need to send a backup operation to one of the sidecars.
Let's say that we have three nodes, where hostname is node1,node2, and node3 respectively and it resolves to IP.
You need to send the following JSON as body for POST http://node1:4567/operations with application/json Content-Type and Accept headers.
Firstly, we said that this operation is of type backup—each operation has to have this field with its own type.
Next, we specify our bucket to backup the cluster to via storageLocation—we do not expose credentials in this request.
For example, the interaction with S3 has to be backed by some credentials.
Please refer to Esop documentation to learn how the credentials resolution for each supported cloud environment works.
Thirdly, we specify snapshotTag under which this backup will be taken and which we need to refer to upon restoration.
Additionally, we specify entities that tells us what keyspaces to actually backup. Icarus does not backup anything you
do not specify, so if you want to backup system_schema where table definitions are you have to do that yourself.
If you want to backup just one table (or only tables), you need to use ks1.t1,ks2.t2 format.
Last but not least, we say that this request is global by setting globalRequest: true. Once this request
is sent to node1 Icarus, that instance will act as a coordinator of the cluster-wide backup. If we did not specify it,
there would be a backup of just that node1 done.
After submission of this operation, we may check what operations there are by calling GET http://node1:4567/operations
(again JSON Accept).
We see that our Icarus running on node node1 (Cassandra runs there too, having same IP/hostname) is running two
backup operations. There are also other fields returned which the reader can consult by reading our Swagger spec.
The first fields returned are the same for every operation (type, id, creationTime, state, errors, progress, startTime).
The last operation is the global one (globalRequest:true). Internally, Icarus detected what the cluster topology is and
it executed other, individual, backup request, to each node in a cluster. This fact is reflected in having two backup operations.
If you check closely, you see that globalOperation for the first operation is set to false and storageLocation is updated with
name of a cluster, data center, and node id.
The similar response to the first operation would be found for the other two nodes too—node2 and node3, (with specific
fields updated to reflect each node).
Once an operation is finished or changes its state, we might see it from state field. Here, we see that such operation is COMPLETED.
An operation bumps over these states:
Progress of an operation might be checked via progress field—it varies from 0.0 to 1.0. If job has completed at
80%, the progress would be 0.8. It is up to each operation to update its progress.
If an operation fails, whatever exceptions are thrown, they are captured in errors field. errors is an array;
there might be 5 nodes and each might fail in its own way, so for each failure there would be a separate entry in errors array
having different hostname—distinguishing the source of that error from each other.
snapshotTag is updated as well—there is automatically appended host id as well as timestamp—timestamp is equal
for each node.
=== Restore Request Submission
After a cluster is backed up, we may want to restore it. The restoration logic is similar to the backup one when it comes to a coordination/orchestration. If you want to restore a keyspace on a running Cassandra cluster, you just send one JSON to whatever node with this payload:
Again, globalRequest is true, storageLocation will be updated on node id, test-cluster and dc1 needs to be there.
Other fields are self-explanatory—you can consult Swagger spec to learn more about them.
You may similarly GET operations to see there are multiple individual operations for each respective restoration phase.
There are restoration phases; these phases are passed per node, and each phase is initiated from coordinator only in case all nodes have passed that particular phase successfully.
DOWNLOAD—download SSTables from remote location locallyIMPORT—hardlinking (for Cassandra 3/4) or importing (for Cassandra 4 only) of SSTables from local destination of the respective nodeCLEANUP—optionally (by default turned on) delete downloaded SSTablesSSTables are truncated only after DOWNLOAD phase is over for all nodes.
import field specifies where downloaded SSTables will be located before they will be hardlinked or imported.
Each global restoration procedure has to start with restorationPhase: download. All other phases are handled automatically.
This is the advantage of Icarus. In the case of Esop, one would have to execute each phase individually. In Icarus' case, a coordinator
will take care of that transparently so in order to do a cluster-wide backup and restore on a running cluster, one just needs to
ever send one JSON to whichever node and say it is a global request.
NOTE: During the backup and restore operations, your whole cluster functions without any disruption. It is even possible to backup and restore while your cluster is running, there is no downtime for both operations. Please consult https://github.com/instaclustr/esop[Esop] readme to learn more about the nature of backups and restore and their configuration.
== Anatomy of an Operation
The interaction with Icarus via REST is conceptually driven around operations.
An operation has these features:
POST on /operationsGET /operations/{uuid} endpointGET /operations endpoint expire, by default, after one hour=== What Operations and Endpoints Are Available?
You may consult (and try) all operations via our https://instaclustr.github.io/instaclustr-icarus-go-client/[Swagger spec].
=== How to Create My Own Operation?
To implement an operation against a Cassandra node, you need to, in general, do the following steps. We will guide a reader through cleanup operation which is very easy to grasp. In this simple example, an operation consists of three classes:
CleanupOperation—the implementation of cleanup against Cassandra// each operation has to extend Operation class
// each operation has its operation request
public class CleanupOperation extends Operation
private final CassandraJMXService cassandraJMXService;
// Injection of necessary resources / instances
// for the interaction with Cassandra, you will very likely need JMX connection
// CassandraJMX service encapsulates this logic.
// request instance is injected there according to body in HTTP POST request upon
// operation submission
@Inject
public CleanupOperation(final CassandraJMXService cassandraJMXService,
@Assisted final CleanupOperationRequest request) {
super(request);
this.cassandraJMXService = cassandraJMXService;
}
// for brevity, constructor for JSON deserialisation is not shown here
// please consult the source code to know more
@Override
protected void run0() throws Exception {
// implement your operation, you have access to request from this method
cassandraJMXService.doWithStorageServiceMBean(
new FunctionWithEx<StorageServiceMBean, Integer>() {
@Override
public Integer apply(StorageServiceMBean ssMbean) throws Exception {
return ssMbean.forceKeyspaceCleanup(/* necessary arguments */);
}
});
}CleanupOperationRequest—this class represents your operation request sent to POST /operations. Feel free to
model your operation request as you please, but type field has to be there. Each operation request
is validated via Hibernate Validator so you may use javax.validation annotations on
fields.public class CleanupOperationRequest extends OperationRequest {
@NotEmpty
public final String keyspace;
public final Set<String> tables;
@Min(0)
public final int jobs;
@JsonCreator
public CleanupOperationRequest(
@JsonProperty("type") String type,
@JsonProperty("keyspace") String keyspace,
@JsonProperty("tables") Set<String> tables,
@JsonProperty("jobs") int jobs) {
this.jobs = jobs;
this.keyspace = keyspace;
this.tables = tables;
this.type = type;
}public class CleanupsModule extends AbstractModule {
@Override
protected void configure() {
installOperationBindings(binder(),
"cleanup",
CleanupOperationRequest.class,
CleanupOperation.class);
}Please note:
AbstractModuleinstallOperationBindings static method is from com.instaclustr.operations.OperationBindingscleanup is operation type. You have to specify type field in your JSON request. Each operation
is uniquely known and refered to through this type.After we have our operation module, we need to install it. This happens
via com.instaclustr.icarus.Icarus#operationModules method.
== Tests
There are cloud tests which are primarily focused on backup and restore against a cloud destination—being GCP, Azure, or S3. Clouds tests are disabled by default. You have to enable them like this:
== Usage
Please see https://www.instaclustr.com/support/documentation/announcements/instaclustr-open-source-project-status/ for Instaclustr support status of this project