infinispan / infinispan-operator

Infinispan Operator
https://infinispan.org/docs/infinispan-operator/main/operator.html
Apache License 2.0
48 stars 54 forks source link

Rolling Upgrade fails when memcached or RESP endpoint not defined on original cluster #2140

Closed ryanemerson closed 1 month ago

ryanemerson commented 1 month ago

Since #2119 was merged, the Memcached and RESP cache's are enabled by default. Unfortunately this breaks rolling upgrades, as the implementation is incorrectly using the "target" cluster as the source of truth for cache names when initiating the sync phase of the upgrade, process resulting in the following exception:

14:00:26,847 ERROR (blocking-thread--p3-t2) [org.infinispan.rest.RestRequestHandler] ISPN012005: An error occurred while responding to the client java.util.concurrent.CompletionException: org.infinispan.commons.CacheException: ISPN000397: Could not migrate data for cache memcachedCache, check remote store config in the target cluster. Make sure only one remote store is present and is pointing to the source cluster
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1770)
at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.infinispan.commons.CacheException: ISPN000397: Could not migrate data for cache memcachedCache, check remote store config in the target cluster. Make sure only one remote store is present and is pointing to the source cluster
at org.infinispan.persistence.remote.upgrade.HotRodTargetMigrator.synchronizeData(HotRodTargetMigrator.java:64)
at org.infinispan.upgrade.RollingUpgradeManager.synchronizeData(RollingUpgradeManager.java:74)
at org.infinispan.rest.resources.CacheResourceV2.lambda$syncData$8(CacheResourceV2.java:514)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
... 5 more

Instead the Rolling Upgrade process should always use the cacheNames defined on the source cluster as these are the caches that need to be synchronised with the new cluster.