Hook up AutoCluster to Hazelcast

[jtesser]

Currently our Autoclustr sets up the JGroups channel. We need to hook it up to hazelcast. We can still use TCP for this which is how the JGroups channel sets up. After we read the Hazelcast XML we can set the TCP properties and start the provider

There following is the overview of what needs to happen.

• The network tab page is broken when using Hazelcast as the transport because the JSONStatusNode casts an object as JgroupCacheTransport https://github.com/dotCMS/enterprise-2.x/blob/master/src/main/java/com/dotcms/enterprise/ClusterUtil.java#L79 We need to make the clusterUtil work with all transports. Figure out the information the status page actually needs and lets create an object that we add a method to for all CacheTransports to implement.

• Both AutoWire and Manual cluster config is handled here https://github.com/dotCMS/core/blob/master/dotCMS/src/main/java/com/dotmarketing/business/ChainableCacheAdministratorImpl.java#L103 It sets a bunch of system properties that Jgroups reads. We need to make these be dotCMS properties that all our CacheTransports can use on init() We will stick with TCP config for autowire. In the HazelCastTransport we need to figure out of the properties are commented out. IF they are don't try to use them just load the hazelcast config XML.
  The idea is that we will always load the config and then IF we have properties set in our system one of which might be autowire THEN we force the TCP BUT IF autowire is false AND there are no cache TCP settings in the dotmarketing-config.properties file THEN we only load the hazel xml and move on. IMPLEMENTATION NOTE : you can load the hazelcast xml as it works now in the HazelCastUtil and then programmatically something like Properties properties = new Properties(); fillProperties... XmlConfigBuilder builder = new XmlConfigBuilder(myxml); builder.setProperties(properties) .....

• HazelcastUtil needs to hold more then one HazelCastInstance Basically this https://github.com/dotCMS/core/blob/master/dotCMS/src/main/java/com/dotcms/cluster/business/HazelcastUtil.java#L16 needs to be a Map<String, HazelcastInstance> where string is the XMLFIlePath The default path is hazelcast-embedded.xml as it already is in the code To solve the properties case mentioned above we also need to overload the methods getHazel(...) which take the Properties object or a map of props

• The HazelCastCacheTransport needs to be renamed to HazelCastCacheTransportEmbedded. Add a Protected String to it for the xmlFilePath and create a second class that extends this class named HazelCastCacheTransportClient and just override the xmlFilePath When loading a client for the transport layer we should never try to autowire only do this in the Embedded. So we need to also override the init() in the HazelCastCacheTransportClient but other then the String and init() all code should be the same

• Need to make the HazelCastCacheProvidr follow the same pattern as the HazelCastCacheTransportClient above with the protected String and extending the Embedded Provider

• In the CacheProvider interface we need to provide a new boolean called isDistributed which both Hazel and Redis provider would return true. If true then we should NOT use the CacheTransport meaning we would removeLocalOnly because in distributed cache we expect the cache to handle cacheInvalidation (Look at call to Transport in ChainableCacheAdministratorImpl.remove(), implement cacheProviderAPI.isAllDistributed(region))

• Add support for property "CLUSTER_HAZELCAST_AUTOWIRE" (defaults to "true") which will allow to enable/disable the automatic configuration of the XMLs

[agomez-dotcms]

Hi Guys,

It looks this ticket is now covered by code adjustments made on branch issue-11124-hookup-autocluster-to-hazelcast: core and enterprise. This ticket was about learning a little bit of dotCMS pluggable Cluster Setup and Caching mechanisms, as well as, Hazelcast perse and its recently introduced implementation on dotCMS.

Code adjustments complement existing Hazelcast implementation, now working similarly as legacy JGroups -based implementation and at following levels:

1. Both Manual Clustering and Auto-Clustering configurations. Instructions and information on previous links are required and still hold for the Hazelcast implementation.
2. Both Embedded and Client Cluster Transports configured through "CACHE_INVALIDATION_TRANSPORT_CLASS" property on file "dotmarketing-config.properties" (Embedded is the default):

   CACHE_INVALIDATION_TRANSPORT_CLASS=com.dotcms.cache.transport.HazelcastCacheTransportEmbedded
   CACHE_INVALIDATION_TRANSPORT_CLASS=com.dotcms.cache.transport.HazelcastCacheTransportClient

3. Both Embedded and Client Cache Providers configured via Cache Chaining properties on file "dotmarketing-config.properties":

   cache.CACHE-REGION-NAME.chain=...,com.dotmarketing.business.cache.provider.hazelcast.HazelcastCacheProviderEmbedded,...
   cache.CACHE-REGION-NAME.chain=...,com.dotmarketing.business.cache.provider.hazelcast.HazelcastCacheProviderClient,...

4. Cluster Information on "System->Configuration->Network" tab and Performance Optimization for distributed cache providers (Hazelcast and Redis)
5. Bug-fixes here and there to support Manual and Auto Cluster on Fresh vs. Existing dotCMS instances as expected

When using Clustering with Hazelcast Embedded, Hazelcast network configuration on file "hazelcast-embedded.xml" is now automatically wired by default (re-configured) with corresponding ip-addresses and ports information of the cluster nodes (under/assuming/using TCP protocol). This auto-wiring process may be disabled by setting the newly introduced property "AUTOWIRE_CLUSTER_TRANSPORT" to "false" on file "dotcms-config-cluster.properties" (which defaults to "true"). In case of disabled, file "hazelcast-embedded.xml" is assumed to be pre-configured by User with network port and join configuration corresponding to the inherent Hazelcast instance and cluster members dotCMS is to be bound to; for instance:

<hazelcast>
  ...
  <network>
    ...
    <port auto-increment="true" port-count="100">7801</port>
    ...
    <join>
      ...
      <tcp-ip enabled="true">
        <interface>127.0.0.1</interface>
        <member-list>
          ...
          <member>127.0.0.1:7801</member>
          <member>127.0.0.1:7802</member>
          ...
        </member-list>
      </tcp-ip>
      ...
    </join>
    ...
  </network>
  ...
</hazelcast>

When using Hazelcast Client as either Cluster Transport or Cache Provider, Hazelcast Client configuration on file "hazelcast-client.xml" is assumed to be pre-configured by User with ip-address and port of the corresponding external Hazelcast instance dotCMS is to be bound to; for instance:

<hazelcast-client>
  ...
  <network>
    ...
    <cluster-members>
      ...
      <address>127.0.0.1:7801</address>
      ...
    </cluster-members>
    ...
  </network>
  ...
</hazelcast-client>

Instructions to quickly setup and run dotCMS with Hazelcast Client (against external Hazelcast node/instance) may be summarized as follows:

1. Setup Hazelcast In-Memory Data Grid: 1.1. Download Hazelcast from https://hazelcast.org/download/ 1.2. Uncompress Hazelcast to directory "", one Hazelcast node/instance per dotCMS instance in the cluster (on different directories if using nodes on same machine) 1.3. Copy contents of file "/webapps/ROOT/WEB-INF/classes/hazelcast-embedded.xml" into file "/bin/hazelcast.xml" 1.4. Assign a different port to each and every Hazelcast node/instance on file "/bin/hazelcast.xml":

   <hazelcast>
   ...
   <network>
   ...
   <port auto-increment="true" port-count="100">7801</port>
   ...
   </network>
   ...
   </hazelcast>

   1.5. Define network join configuration on each and every Hazelcast node/instance on file "/bin/hazelcast.xml". If you are using tcp-ip join schema, this may look like:

   <hazelcast>
   ...
   <network>
   ...
   <join>
     ...
     <tcp-ip enabled="true">
       <interface>127.0.0.1</interface>
       <member-list>
         ...
         <member>127.0.0.1:7801</member>
         <member>127.0.0.1:7802</member>
         ...
       </member-list>
     </tcp-ip>
     ...
   </join>
   ...
   </network>
   ...
   </hazelcast>

2. Configure "hazelcast-client.xml" on each and every dotCMS instance ("/webapps/ROOT/WEB-INF/classes/hazelcast-client.xml") as detailed above

3. Startup each and every Hazelcast node/instance by running "/bin/start.sh"

4. Startup each and every dotCMS instance! Make sure the associated Hazelcast nodes/instance(s) are up/clustered fine before running dotCMS

Finally, following issues were noticed and remain to be fixed. They may be separated from this ticket and will be addressed as particular issues:

1. Number of Node(s) in System->Configuration->Network
2. "java.lang.ClassNotFoundException: com.liferay.portlet.admin.model.AdminConfig" after System->Maintenance->Cache->Refresh Stats
3. "0" as Cache Stats in Hazelcast Client Provider
4. [NOT REPRODUCED ANYMORE] Re-Indexing often required on clustered instances
5. [NOT REPRODUCED ANYMORE] ERROR lang.System: bad serverId

Please do not hesitate to reach me in case of any question!

Thanks, Anibal

[agomez-dotcms]

PR Core: https://github.com/dotCMS/core/pull/11414 PR EE: https://github.com/dotCMS/enterprise-2.x/pull/464

[agomez-dotcms]

Hi Guys,

As part of review, following configuration changes on Auto-wire related properties were requested to be implemented:

1. Removed "CLUSTER_AUTOWIRE" in favor of the two properties below

2. Introduced "AUTOWIRE_CLUSTER_TRANSPORT" (default "true")

3. Introduced "AUTOWIRE_CLUSTER_ES" (default "true")

Remarkably, changes previously described (above) were also done in interest of covering features requested on issue #10001. Even the approach that was finally decided is not exactly the same suggested on issue #10001, it was discussed and decided as similar alternative to fulfill it. Finally, removal of CLUSTER_AUTOWIRE property is also tracked by #11468.

Thanks, Anibal

[DeanGonzalez]

Documented: https://auth.dotcms.com/docs/latest/hazelcast-cache-provider https://auth.dotcms.com/docs/latest/hazelcast-cluster-transport-configuration https://auth.dotcms.com/docs/latest/manual-cluster-configuration https://auth.dotcms.com/docs/latest/auto-clustering-configuration https://auth.dotcms.com/docs/latest/cache-chaining