ensemble servers to sync the deployed artifacts 1.x

[evmin]

With a fabric containing more than one ensemble servers an artifact deployed to one of them is not synced to the others.

So, in a case of a loss of this ensemble server, the deployed artifact is not available to the fabric anymore and the deployment needs to be repeated.

Alternatively the initial deployment should be done to all servers preemptively.

I would like the ensemble servers to be able to sync the deployed artifacts (Bundles, Features, KARs) between themselves automatically.

[jstrachan]

We use a maven proxy in the fabric to store the artefacts; all containers in a fabric should download artefacts from it (and there can be more than one). http://fabric8.io/#/site/book/doc/index.md?chapter=mavenProxy_md

Are you sure that using the maven proxy isn't enough?

[davsclaus]

As I can see from the code and as I understand this (correct me). Is that we have a node elected as the master maven proxy in the ZK registry. So when a node needs to access the maven proxy, it lookup in ZK the url for the master node, and accesses the master node to upload/download artefacts from it.

But the artefacts are stored locally (by default) on that node. So if the node goes down, then another node is elected as the new maven proxy master. And then those JAR files that was previously uploaded are no longer available, as the JARs is not distributed among the nodes.

Though what we have often seen is that companies have their own internal maven repository. So if each node in the fabric has access to that repository, then they dont need the internal repo inside fabric.

Maybe we can add a new functionality to have fabric automatic sync the fabric maven proxy nodes in the background, so it uploads the JARs to the other nodes. But this can be a bit tricky, if nodes crashes, then you would need to keep state what is "out of sync" and whatnot. Or just calculate a "diff" on the non-master nodes and see what the "diff" is and have the master sync the missing JARs.

[evmin]

Thank you, Claus, this is exactly the case.

The issue with using an external maven repo is that for a DR scenario it needs to be replicated. Quite a complex and expensive set up in a situation where the only task is to make sure the artifacts are available within the fabric.

At the moment we are using gluster for that purpose, but this kind of over engineered and prevents us from using OpenShift.

As for crashes - is the creation timestamp a valid option to identify which artefacts have been added after the crash? So when the crashed node comes back - it can explicitly inquire the other nodes to provide all deployment entities that are timestamped after the crash. Or something along these lines.

[davsclaus]

This is a good idea to make the built-in maven proxy HA.

[gnodet]

We should investigate reusing our clustered git infrastructure to store the maven proxy.

[evmin]

My impression was that having binary artifacts in a text versioning tool was something to avoid. What will be the performance impact of storing 200 MB of JARs (just measured our maven/proxy/downloads ) in git?

Especially in the light of https://issues.jboss.org/browse/FABRIC-1051?

[jstrachan]

@evmin we don't store binary artefacts in git; thats why we have the maven proxy; artefacts are downloaded from maven repositories; git stores a link to the artefacts (using maven coordinates; group id, artefact id, version etc).

[evmin]

Apologies, I misunderstood Guillaume's comment.

[jstrachan]

@evmin no worries; Guillaume's meaning to replicate the state of the maven proxy; so if the master fails we've backups. Another option would be to use Ceph or Gluster as the file system inside the maven proxies etc

[jstrachan]

@gnodet did you mean to put the mvn proxy stuff inside git?

I guess we could use a separate git repo to the profile repo; though I'd have thought something rsync'y or ceph/cluster or S3 (or OpenStack Swift might be a bit better: http://docs.openstack.org/developer/swift/

[gnodet]

I was actually thinking of storing the artifacts in git, but certainly not in the same git repo than the configuration. The artifacts in the maven repository almost never change (but for snapshots which are not really supposed to be used in production). So we don't really need anything complicated, especially the fact that git isn't the best tool for diffing binary files isn't really a problem imho, as we will mostly write once and then read. And for reading, we should be able to bypass the git layer if needed. My main idea is that we already have in fabric8 the needed infrastructure to maintain a clustered git proxy (electing the master, etc..), so I think the only thing we need to add is a way to maintain a small / fixed number of replicas instead of having each container cloning the git repo locally. As for other solutions, there may be some better than git, though I'm not aware of any usable rsync solution in java and I don't think the goal is to use a different backend such as S3, as users can already do that. And we also need it to run on openshift, so the most simple setup the better.

[jstrachan]

@gnodet ah ok! I wonder if we can figure out the git crack to prune old snapshots from the git history (so git only stores the latest snapshot image?). I guess a minor difference to the way the git repo works for profiles is we'd need to elect N nodes to be a 'puller' to keep a backup of the git repo.

Incidentally with the RHQ metrics project: https://github.com/rhq-project/rhq-metrics

its got a historical metrics service using cassandra which supports compression of values over time (like RRDTool and the like); its currently being refactored as a reusable service we can use to store huge numbers of metrics very efficiently (a single cassandra row can store the history of values of a metric in its columns & its easy to compress old values periodically etc).

I'm hoping we can soon provision the RHQ metrics service as an alternative to the insight-metrics; which hopefully will be a bit more disk efficient than storing massive numbers of metrics across many machines. We've already got the first cut of a profile to provision a cassandra distro: https://github.com/fabric8io/fabric8/tree/master/fabric/fabric8-karaf/src/main/resources/distro/fabric/import/fabric/profiles/containers/services

(one missing feature is we're not yet properly injecting the seeds value as the first N cassandra nodes created that are still running (which should be populated via a ZK query) https://github.com/fabric8io/fabric8/blob/master/fabric/fabric8-karaf/src/main/resources/distro/fabric/import/fabric/profiles/containers/services/cassandra.profile/io.fabric8.template.variables.properties#L3

Anyway; soon I can see folks choosing to use Cassandra for metrics (and ElasticSearch for logs/message audit); in which case Cassandra would make a pretty good solution as a replicated storage for the maven proxy; we'd just need the upload POSTS to cassandra (lazily creating the database on the fly if its not there like the RHQ metrics project does); then the GET if the local file system hasn't got the value, try querying from Cassandra.

[jstrachan]

so we don't forget I raised #1439 to remember the cassandra idea

[evmin]

Gentlemen,

Would it be OK if I provide a perspective of a single enterprise client? Background: Our fabric consists of 9 containers (including 3 ensemble nodes) on 5 hosts in 3 datacenters per environment. Three environments (DEV, TEST, PROD).

For monitoring and alerts, metrics collection and generic operations we are using JBoss ON with FUSE plugin, which starting with version 3.2.0 runs on Cassandra as a metric collector.

So I'd be cautious (at best) to have yet another Cassandra as a metric aggregation component within the infrastructure. Of course, if it is made optional, and the end user is left to choose which way to go with the metrics (internal or RHQ/JON) - this would not be an issue. But it still would look like a duplication of effort, with RHQ and its agents being available for use.

Another concern is the accessibility to the deployed artefacts - I have to regularly check the deployed artefacts in the maven/proxy/downloads for one or another reason. I am not sure how easy that would be with Cassandra.

And the last, but not the least, is the fact that the fabric in its current state is light and nimble. And I do like it this way. I could probably appreciate some extra persuasion in regards of including such heavy components as we perceive Cassandra to be.

Same goes probably for git based solution, yet with git I could probably clone the repo and it is not that heavyweight.

Myself I would lean towards James' (@jstrachan) interpretation - where the internal structure is used for the indexes. Could the sync in that case be done by a maven indexer via the standard maven upload endpoint?

Sure, I'm not in the position to comment on how to implement this or that feature - I am already grateful (a lot) for your approachability and openness about the requests for the new features. Just wanted to provide a bit of insight into how we use the fabric and I hope this was helpful.

Once again, thank you.

[jstrachan]

@evmin to be clear, I'm saying fabric8 would reuse whatever Cassandra cluster you wish; the idea behind RHQ metrics is to be able to reuse the exact same JBoss ON Cassandra cluster for folks who are using it. Also yes, its most definitely optional; whether you want to use any cassandra at all or have a single cassandra cluster for JBoss ON / RHQ or a separate one for the maven repo stuff etc.

Whatever the implementation of the maven proxy in fabric8 (git or cassandra) we need a way to browser it - see https://github.com/fabric8io/fabric8/issues/982 - which we could do via the HTTP API I'm hoping; with links to it via the maven/deploy page: https://github.com/fabric8io/fabric8/issues/594

But yeah am hoping folks can pick whatever solution makes the most sense (some unixy rsync thing, git, cassandra or if folks use OpenStack then Swift); but I also agree with keeping fabric8 light and nimble.

[jstrachan]

@gnodet another idea ;)

How about we populate in git the index of the maven repo? i.e. we keep a sorted list of all the artefacts in the repo in git. e.g. imagine this was in fabric/maven/repo/index.txt in the master branch or something:

com/acme/foo/1.0/foo-1.0.jar
com/acme/foo/1.0/foo-1.0.pom
cheese/something/2.2/something-2.2.jar
cheese/something/2.2/something-2.3.pom

Then the master mvn proxy writes (in order) to the end of this file; then any slave starts up, reads the file, checks its got the file locally, if not does a GET from the master and keeps going until it gets to the end of the file; then when git changes it keeps reading and GET'ing anything thats missing?

a simple rsync kinda mechanism; reusing the same git repo - which also acts as a kinda index of artefacts too.

I guess the downside is, it'll lead to quite a bit of git chat in the commit logs (though at least its on the master branch) while the mvn repo populates itself from upstream mvn repos ("downloading the internet").

Maybe we only do this real deployments (POSTs) to the maven proxy? - for just doing GETs where the maven proxy is just acting as a local cache (via pulling from upstream repos) we maybe don't have to do the git thing).

[evmin]

@jstrachan

By pure coincidence, I had a lengthy (and productive) discussion with Larry O'Leary today on how acceptable is to use Cassandra database provided with JON installations for other purposes. One of the key takeouts was that it is not at all welcome and is not supported by RedHat. It is provided as a bundled component that should not be maintained or accessed by the end users.

And I share this perspective - I would like to avoid maintaining yet another datastore (it would be difficult also due to political reasons for us). Therefore I am a bit concerned with the need to set up and maintain Cassandra just to enable the deployments sync. I'd rather achieve the same with Gluster - easier, external to fabric and a lot of added benefit for other use cases. Actually that we have done here (thugh it create a barrier for OpenShift adoption).

On the other hand, James, I like your ideas better and better. The latest suggestion with git synced index and slaves checking up on the master - that's how an ideal solution would look to me.

Regarding the git chatter/performance - would it still be a problem with the new git GC happening every 100 or so commits (https://issues.jboss.org/browse/FABRIC-1051)? My interpretation is that it should not probably be that big of a concern?

[davsclaus]

The git chatter is reduced on latest code - as the git fabric servlet, detects if its a push or pull (eg write vs read) and for writes it does an ++ to a shared ZK counter, that the other nodes listen and reacts upon. So they trigger when there was a write, and do a sync on their end.

So if the maven upload does a git write too, then the other containers will sync their git, and we have some logic that detects the git maven upload index was changed and do as James says to a HTTP GET to download these new artifacts.

[rajdavies]

@evmin - perhaps we should discuss this on irc ?

[jstrachan]

@evmin though I suspect if you were using a Red Hat product which includes both fabric8 and RHQ; then sharing its internal Cassandra database would probably be fine - but thats a decision for the Business Unit really.

But yeah, the git solution probably would work fine if we only wrote to git when folks do deployments. (We may one day need a compression mode where we zap all the old snapshots from the file).

[evmin]

@rajdavies happy to. What's the server/channel (apologies for my ignorance)?

[evmin]

@jstrachan

git should get the job done and yet keep the fabric slim. Love the idea

[rajdavies]

@evnmin - we hang out in freenode #fabric8

[geertschuring]

How about rsync for maven repo replication from master to slaves? Would also solve the case where an admin cleans up some snapshots from the master repo to keep its size down.

Another option would be to use an activemq topic. The master can announce uploaded maven artifacts there, and slaves can use persistent subscribers to keep track of which artifacts it hasn't downloaded yet.

[jstrachan]

Yeah; I guess when running on linux, running a periodic rsync command line in a slave might be reasonable.

I guess we maybe want to store POSTed stuff separately from the GET cache of upstream stuff; as there's no need to rsync all the GET cache stuff (which will be 90% of the volume I suspect) when we could just fetch those from the upstream respos/cache)

[jstrachan]

Another simple solution for folks who have NFS - use a shared NFS volume for the maven proxies? (We only elect one at a time so only one thread/process is writing to it at any time)

[evmin]

We use both Gluster and NFS. As I understood a typical limitation of an NFS was that it was limited to a single DC only. We run the fabric across three, hence the gluster was used to sync the mountpoint. But again - that precludes us from considering Openshift

[jstrachan]

BTW this isn't an issue in fabric8 V2 since the docker container images contain all the required artefacts already.

All thats required is a docker registry which supports replication/persistence which the default docker registry already does (e.g. via S3, swift, ceph, gcs, glance etc).

A work around until folks use OpenShift V3 is to use the Builder cartridge on OpenShift V2 which does a maven build inside OpenShift and syncs all the artifacts between all cartridge gears; then provides a maven repository onto all of its artefacts

[evmin]

We dynamically deploy OSGi CAMEL solution bundles to already provisioned JVM instances (ssh containers). So I am not 100% sure how docker would help. And we most likely will not be running openshift for another few years.

[jstrachan]

The docker registry already supports back end replicated storage to S3, swift, glance, google, ceph etc.

A work around is to just run a maven repository (eg nexus) or set up a http / rsync mirror and use httpd to host maven artifacts

On Saturday, October 18, 2014, evmin notifications@github.com wrote:

We dynamically deploy OSGi CAMEL solution bundles to already JVM instances (as they would need t run solution level dependencies). So I am not 100% sure how docker would help. And we most likely will not be running openshift for another few years.

— Reply to this email directly or view it on GitHub https://github.com/fabric8io/fabric8/issues/1280#issuecomment-59600714.

James

Red Hat

Twitter: @jstrachan Email: jstracha@redhat.com Blog: http://macstrac.blogspot.com/

hawtio: http:/ http://fusesource.com//hawt.io/ fabric8: http:/ http://fusesource.com//fabric8.io/

Open Source Integration