expose some hazelcast config in cluster.properties

[jdeolive]

At a minimum expose the following:

• hz cluster name to make it easier when setting up the session clustering in web.xml.
• the multicast subnet address

[smithkm]

I'm thinking it makes more sense to load a Hazelcast XML config file from the data dir rather than manipulating the config based on cluster.properties. It's simpler to implement and allows more flexibility in configuring the cluster. Also, the current values in cluster.properties can handle live changes, while HazelcastInstance can't be reconfigured after startup; so keeping them separate would help avoid confusion about some options updating without restart and others not.

[jdeolive]

That makes a lot of sense kevin. Great points. Let's go with this approach. If you could throw it in a file named "hazelcast.xml" or something that would be great. Also since we are going to be adding configuration files i think it makes sense to store all config under a directory called "cluster". So the structure woud look like.

data_dir/
     cluster/
         cluster.properties
         hazelcast.xml

[smithkm]

Making sure the Hazelcast instance is initialized from the config file before the session sharing filter is initialized is proving to be the most difficult part of this. Otherwise the session sharing will create its own instance with the Hazelcast default configuration. The session sharing will also occur (again with the default Hazelcast configuration) even if the rest of the cluster module is disabled in cluster.properties.

[jdeolive]

I see. A couple of things come to mind, and these are just ideas since i haven't really looked at any code so feel free to shoot them down :)

1. We could try subclassing the hazelcast filter, provided it is not final and has the necessary subclass hooks.
2. Try to implement a ServletContextListener instance to setup hazelcast. A listener would get a chance to execute before the servlet filter.

[smithkm]

Yeah, subclassing the filter was what I was looking at although I've been trying to delay the filter's initialization until Spring is setting things up and marking it as GeoServerFilter for auto discovery, rather that moving the hazelcast init forward. I've been looking up the config info via a GeoServerResourceLoader, which means waiting for Spring to create the resource loader.

I'm trying to figure out how to add the filter to the Spring context if and only if the 'enabled' property is set in cluster.properties I'm not sure but it looks like this might involve making a BeanFactory that returns the filter if it exists.

[smithkm]

Hmm, if there were a Listener counterpart to SpringDelegatingFilter, I think it would be possible to eliminate the need for a modified web.xml

[jdeolive]

Yeah... that might be tricky since the spring setup itself happens in a listener so it would have to be the last listener to update. Also depending on the listener and its dependencies triggering a plug-in look up that early in geoservers startup phase might cause issues. It could work though.

[jdeolive]

Also, if this turning out to be too much of an issue for now we can just have the session filter configuration be totally manual, and necessarily have to bootstrap itself based on the config in the data directory. With that said as it stands currently is it possible to manually configure the session filter to use the same hazelcast instance?

[smithkm]

Actually I think I've got it nailed. There is an HTTPSessionListener counterpart to SpringDelegatingFilter so I can load both the filter and the listener via GeoServer extension points and set everything up inside spring. It's not as pretty as it might have been with Servelet 3.0 but I think it should work.

[smithkm]

It looks like getting this to work with a stock web.xml is a no go. The Hazelcast filter has to go before all other filters and SpringDelegatingFilter is well down the chain. The same delegate based mechanism should still work for allowing Hazelcast to only be initialized if clustering is enabled though. The Delegator goes at the top of the web.xml, but its WebFilter delegate is set during Spring's initialization. If Hazelcast is enabled, the delegate gets set, and if not, it is null and the delegator acts as a no-op.

[smithkm]

OK, It seems to be working but is breaking on Wicket forms when the load balancer is not providing sticky sessions. I stay logged in as I bounce around the nodes, but forms like updating a layer will give a "Sorry, your session timed out..." message almost every time. I rolled back the changes and still get the same behaviour. With sticky sessions on though (whether I tell Hazelcast this or not) it works, although it would work even without the session sharing filter at that point. The filter then keeps you logged in if the node dies and you get moved to a new one, but won't prevent the Wicket error. Is this the way it's supposed to work?