Access denied on /opt/infinispan/server/data/___global.lck.

mxandeco commented 4 years ago

Following the manual install tutorial from https://github.com/infinispan/infinispan-simple-tutorials/tree/master/operator.

Tried versions 1.1.0 and 1.1.1, deploying on EKS i can confirm both pv and pvc were created properly, the pods won't start, and I can see the logs:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/opt/infinispan/bin/config-generator.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
19:26:22,229 INFO  (main) [BOOT] JVM OpenJDK 64-Bit Server VM Oracle Corporation 11.0.7+10-LTS
19:26:22,233 INFO  (main) [BOOT] JVM arguments = -Xmx200M -Xms200M -XX:MaxRAM=420M -Dsun.zip.disableMemoryMapping=true -XX:+UseSerialGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:+ExitOnOutOfMemoryError -XX:MetaspaceSize=32m -XX:MaxMetaspaceSize=96m -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dinfinispan.server.home.path=/opt/infinispan
19:26:22,238 INFO  (main) [BOOT] PID = 94
19:26:22,557 INFO  (main) [org.infinispan.SERVER] ISPN080000: Infinispan Server starting
19:26:22,558 INFO  (main) [org.infinispan.SERVER] ISPN080017: Server configuration: /opt/infinispan/server/conf/infinispan.xml
19:26:22,558 INFO  (main) [org.infinispan.SERVER] ISPN080032: Logging configuration: /opt/infinispan/server/conf/log4j2.xml
19:26:24,952 INFO  (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'query-dsl-filter-converter-factory'
19:26:25,016 INFO  (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'continuous-query-filter-converter-factory'
19:26:25,019 INFO  (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'jdk.nashorn.api.scripting.NashornScriptEngineFactory'
19:26:28,724 INFO  (main) [org.infinispan.CONTAINER] ISPN000128: Infinispan version: Infinispan 'Turia' 10.1.7.Final
19:26:28,726 WARN  (main) [org.infinispan.CONTAINER] ISPN000574: Global state cannot persisted because it is incomplete (usually caused by errors at startup).
19:26:28,816 FATAL (main) [org.infinispan.SERVER] ISPN080028: Infinispan Server failed to start java.util.concurrent.ExecutionException: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: ISPN000512: Cannot acquire lock '/opt/infinispan/server/data/___global.lck' for persistent global state
    at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
    at org.infinispan.server.Bootstrap.runInternal(Bootstrap.java:139)
    at org.infinispan.server.tool.Main.run(Main.java:98)
    at org.infinispan.server.Bootstrap.main(Bootstrap.java:39)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.infinispan.server.loader.Loader.main(Loader.java:71)
Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: ISPN000512: Cannot acquire lock '/opt/infinispan/server/data/___global.lck' for persistent global state
    at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:760)
    at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:726)
    at org.infinispan.server.SecurityActions.lambda$startCacheManager$1(SecurityActions.java:51)
    at org.infinispan.security.Security.doPrivileged(Security.java:46)
    at org.infinispan.server.SecurityActions.doPrivileged(SecurityActions.java:31)
    at org.infinispan.server.SecurityActions.startCacheManager(SecurityActions.java:54)
    at org.infinispan.server.Server.run(Server.java:286)
    ... 8 more
Caused by: org.infinispan.commons.CacheConfigurationException: ISPN000512: Cannot acquire lock '/opt/infinispan/server/data/___global.lck' for persistent global state
    at org.infinispan.globalstate.impl.GlobalStateManagerImpl.acquireGlobalLock(GlobalStateManagerImpl.java:87)
    at org.infinispan.globalstate.impl.GlobalStateManagerImpl.start(GlobalStateManagerImpl.java:64)
    at org.infinispan.globalstate.impl.CorePackageImpl$1.start(CorePackageImpl.java:34)
    at org.infinispan.globalstate.impl.CorePackageImpl$1.start(CorePackageImpl.java:27)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:587)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:578)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:547)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:770)
    at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:341)
    at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:237)
    at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:755)
    ... 14 more
Caused by: java.io.FileNotFoundException: /opt/infinispan/server/data/___global.lck (Permission denied)
    at java.base/java.io.FileOutputStream.open0(Native Method)
    at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
    at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237)
    at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:187)
    at org.infinispan.globalstate.impl.GlobalStateManagerImpl.acquireGlobalLock(GlobalStateManagerImpl.java:81)
    ... 25 more

19:26:28,819 INFO  (Thread-0) [org.infinispan.SERVER] ISPN080002: Infinispan Server stopping
19:26:28,832 INFO  (Thread-0) [org.infinispan.SERVER] ISPN080003: Infinispan Server stopped

rigazilla commented 4 years ago

I think is related to this: https://github.com/infinispan/infinispan/pull/8231 Atm workaround in this situation is cleanup the /opt/infinispan/server/data folder in the PV from any lock file before starting Infinispan.

mxandeco commented 4 years ago

Hi @rigazilla Im having the issue with brand new created infinispan crd, there is no "old" lock files in dir.

I have mounted the volume to a "dummy" pod and changed the fs permissions, to 777 as a test, and then infinispan pod started just fine. But I would have to do that for every new replica because a new pv is created =/.

I can't find an option to configure the security-context, custom pv or init container that i could use to fix the permissions.

Any other suggestions in how could i fix this without having to manually entering the volume and changing the permissions for each new replica?

Thanks

rigazilla commented 4 years ago

@mxandeco how would you configure the security-context in a deployment/pod? It would be great to have a feature for that allows to do that also in the infinispan CR.

In the meanwhile as a hack workaround you can try adding this entry in the env list of the operator.yaml:

     - name: MAKE_DATADIR_WRITABLE
       value: "true"

this should start a busybox init container that just set writable access to the data dir.

mxandeco commented 4 years ago

@rigazilla The MAKE_DATADIR_WRITABLE worked fine for me, maybe would be great to get that documented?

Regarding the:

security context. I thought I could try change the runAsUser or fsGroup to get the proper permission to write to the mounted volumes.
custom init, I thought i could write a init container to do what the init container is doing when MAKE_DATADIR_WRITABLE is enabled.

Thanks for the support.

rigazilla commented 4 years ago

@rigazilla The MAKE_DATADIR_WRITABLE worked fine for me, maybe would be great to get that documented?

I would prefer to have a clean solution for this, i.e. setting context correctly or understand which are the conditions that lead to a readonly pv. but yeah, in the meanwhile it would be good to have it documented, since seems to be the only available solution atm.

K890 commented 3 years ago

@mxandeco How did you add MAKE_DATADIR_WRITABLE to env section of operator.yml? I tried to change the file in the local and applied the spec on a fresh operator installation. This does applies the new env in the operators pod but it magically vanishes away as 'something' restarts the replicaset of the operators. I am getting it is being overrided by the original spec the git repo. I am sure I am missing something here.

mxandeco commented 3 years ago

@mxandeco How did you add MAKE_DATADIR_WRITABLE to env section of operator.yml? I tried to change the file in the local and applied the spec on a fresh operator installation. This does applies the new env in the operators pod but it magically vanishes away as 'something' restarts the replicaset of the operators. I am getting it is being overrided by the original spec the git repo. I am sure I am missing something here.

@K890 I used the config bellow:


apiVersion: apps/v1
kind: Deployment
metadata:
  name: infinispan-operator
  namespace: core
spec:
  replicas: 1
  selector:
    matchLabels:
      name: infinispan-operator
  template:
    metadata:
      labels:
        name: infinispan-operator
    spec:
      serviceAccountName: infinispan-operator
      containers:
      - name: infinispan-operator
        image: jboss/infinispan-operator:1.1.1
        ports:
        - containerPort: 60000
          name: metrics
        command:
        - infinispan-operator
        imagePullPolicy: Always
        env:
        - name: WATCH_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: OPERATOR_NAME
          value: "infinispan-operator"
        - name: MAKE_DATADIR_WRITABLE
          value: "true"

K890 commented 3 years ago

@mxandeco If I try to change the infinispan-operator spec directly by editing it, it is overridden in some time by the OLM/CSV. However, I was able to change this spec permanently by editing the csv resource infinispan-operator.v2.1.2 in the operators namespace and adding the MAKE_DATADIR_WRITABLE conf there.

P.S. If somebody gets an creationTimestamp= null error while updating the csv resource, copy-paste the other creationTimestamp which is not null to the place where it is null.

dicolasi commented 3 years ago

Unfortunately that does not work for me.

dmvolod commented 3 years ago

@dicolasi what is your target operator deployment platform and the operator version?

dicolasi commented 3 years ago



2021-05-11 14:19:13,319 WARN  [io.qua.qut.run.ContentTypes] (main) Unable to detect the content type for templates/jgroups-encrypt; using application/octet-stream
2021-05-11 14:19:13,321 INFO  [io.quarkus] (main) config-generator 2.1.3.Final native (powered by Quarkus 1.7.6.Final) started in 0.007s. 
2021-05-11 14:19:13,321 INFO  [io.quarkus] (main) Profile prod activated. 
2021-05-11 14:19:13,321 INFO  [io.quarkus] (main) Installed features: [cdi, qute]
2021-05-11 14:19:13,329 INFO  [io.quarkus] (main) config-generator stopped in 0.001s
14:19:18,099 INFO  (main) [BOOT] JVM OpenJDK 64-Bit Server VM Red Hat, Inc. 11.0.10+9-LTS
14:19:18,108 INFO  (main) [BOOT] JVM arguments = [-server, -Xmx200M, -Xms200M, -XX:MaxRAM=420M, -Dsun.zip.disableMemoryMapping=true, -XX:+UseSerialGC, -XX:MinHeapFreeRatio=5, -XX:MaxHeapFreeRatio=10, -XX:+ExitOnOutOfMemoryError, -XX:MetaspaceSize=32m, -XX:MaxMetaspaceSize=96m, -Djava.net.preferIPv4Stack=true, -Djava.awt.headless=true, -Dvisualvm.display.name=infinispan-server, -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager, -Dinfinispan.server.home.path=/opt/infinispan, -classpath, :/opt/infinispan/boot/infinispan-server-runtime-12.0.2.Final-loader.jar, org.infinispan.server.loader.Loader, org.infinispan.server.Bootstrap]
14:19:18,108 INFO  (main) [BOOT] PID = 97
14:19:18,215 INFO  (main) [org.infinispan.SERVER] ISPN080000: Infinispan Server starting
14:19:18,215 INFO  (main) [org.infinispan.SERVER] ISPN080017: Server configuration: infinispan.xml
14:19:18,216 INFO  (main) [org.infinispan.SERVER] ISPN080032: Logging configuration: /opt/infinispan/server/conf/log4j2.xml
14:19:21,401 INFO  (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'query-dsl-filter-converter-factory'
14:19:21,403 INFO  (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'continuous-query-filter-converter-factory'
14:19:21,406 INFO  (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'iteration-filter-converter-factory'
14:19:21,411 INFO  (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'jdk.nashorn.api.scripting.NashornScriptEngineFactory'
14:19:23,713 WARN  (main) [org.infinispan.PERSISTENCE] ISPN000554: jboss-marshalling is deprecated and planned for removal
14:19:23,901 INFO  (main) [org.infinispan.CONTAINER] ISPN000556: Starting user marshaller 'org.infinispan.commons.marshall.ImmutableProtoStreamMarshaller'
14:19:26,705 INFO  (main) [org.infinispan.query.remote.impl.ProtobufMetadataManagerImpl] ISPN028019: Registering protostream serialization context initializer: org.infinispan.query.core.stats.impl.PersistenceContextInitializerImpl
14:19:26,707 INFO  (main) [org.infinispan.CONTAINER] ISPN000128: Infinispan version: Infinispan 'Lockdown' 12.0.2.Final
14:19:26,709 WARN  (main) [org.infinispan.CONTAINER] ISPN000574: Global state cannot persisted because it is incomplete (usually caused by errors at startup).
14:19:26,714 INFO  (main) [org.infinispan.server.core.RequestTracer] OpenTracing integration is disabled
14:19:26,715 FATAL (main) [org.infinispan.SERVER] ISPN080028: Infinispan Server failed to start java.util.concurrent.ExecutionException: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: ISPN000512: Cannot acquire lock '/opt/infinispan/server/data/___global.lck' for persistent global state
    at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
    at org.infinispan.server.Bootstrap.runInternal(Bootstrap.java:158)
    at org.infinispan.server.tool.Main.run(Main.java:98)
    at org.infinispan.server.Bootstrap.main(Bootstrap.java:46)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.infinispan.server.loader.Loader.run(Loader.java:106)
    at org.infinispan.server.loader.Loader.main(Loader.java:51)
Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: ISPN000512: Cannot acquire lock '/opt/infinispan/server/data/___global.lck' for persistent global state
    at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:751)
    at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:717)
    at org.infinispan.server.SecurityActions.lambda$startCacheManager$1(SecurityActions.java:67)
    at org.infinispan.security.Security.doPrivileged(Security.java:45)
    at org.infinispan.server.SecurityActions.doPrivileged(SecurityActions.java:39)
    at org.infinispan.server.SecurityActions.startCacheManager(SecurityActions.java:70)
    at org.infinispan.server.Server.run(Server.java:351)
    ... 9 more
Caused by: org.infinispan.commons.CacheConfigurationException: ISPN000512: Cannot acquire lock '/opt/infinispan/server/data/___global.lck' for persistent global state
    at org.infinispan.globalstate.impl.GlobalStateManagerImpl.acquireGlobalLock(GlobalStateManagerImpl.java:87)
    at org.infinispan.globalstate.impl.GlobalStateManagerImpl.start(GlobalStateManagerImpl.java:64)
    at org.infinispan.globalstate.impl.CorePackageImpl$1.start(CorePackageImpl.java:34)
    at org.infinispan.globalstate.impl.CorePackageImpl$1.start(CorePackageImpl.java:27)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:604)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:595)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:564)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:787)
    at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:341)
    at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:237)
    at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:746)
    ... 15 more
Caused by: java.io.FileNotFoundException: /opt/infinispan/server/data/___global.lck (Permission denied)
    at java.base/java.io.FileOutputStream.open0(Native Method)
    at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
    at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237)
    at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:187)
    at org.infinispan.globalstate.impl.GlobalStateManagerImpl.acquireGlobalLock(GlobalStateManagerImpl.java:81)
    ... 26 more
14:19:26,795 INFO  (Thread-0) [org.infinispan.SERVER] ISPN080002: Infinispan Server stopping
14:19:26,800 INFO  (Thread-0) [org.infinispan.SERVER] ISPN080003: Infinispan Server stopped
stream closed

dicolasi commented 3 years ago

@dmvolod

infinispan-operator.v2.1.3

dicolasi commented 3 years ago

:pods Screenshot 2021-05-11 at 15 24 07

dmvolod commented 3 years ago

@dicolasi where you've configured MAKE_DATADIR_WRITABLE env variable to configure this option?

dicolasi commented 3 years ago

@dmvolod in the csvs.

dmvolod commented 3 years ago

in the csvs.

Can you validate, that's applied to the operator deployment and Pod

dicolasi commented 3 years ago

Interesting:

#
# clusterserviceversions.operators.coreos.com "infinispan-operator.v2.1.3" was not valid:
# * <nil>: Invalid value: "The edited file failed validation": [yaml: line 359: found a tab character that violates indentation, invalid character 'a' looking for beginning of value]

here the content of that bit:


          spec:
              containers:
              - command:
                - infinispan-operator
                env:
                - name: WATCH_NAMESPACE
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.annotations['olm.targetNamespaces']
                - name: POD_NAME
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
                - name: OPERATOR_NAME
                  value: infinispan-operator
                - name: RELATED_IMAGE_OPENJDK
                  value: quay.io/infinispan/server:12.0.2.Final
                - name: MAKE_DATADIR_WRITABLE
                  value: "true"

dmvolod commented 3 years ago

: Invalid value: "The edited file failed validation": [yaml: line 359: found a tab character that violates indentation, invalid character 'a' looking for beginning of value]

That's the reason why OLM couldn't apply configuration to the Deployment. Please have a look at the operator deployment and Pod also

dicolasi commented 3 years ago

what should I look for? The env variable is not there.

dmvolod commented 3 years ago

what should I look for? The env variable is not there.

kubectl get deployment infinispan-operator -o yaml -n <namespace>

dicolasi commented 3 years ago

@dmvolod just to clarify: is this a bug with the operator?

dmvolod commented 3 years ago

just to clarify: is this a bug with the operator?

Not sure, this is specific k8s platform limitation and should work

dicolasi commented 3 years ago

@dmvolod I am re-deploying the operator. Need 3 minutes.

dicolasi commented 3 years ago

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
  creationTimestamp: "2021-05-11T14:53:39Z"
  generation: 2
  labels:
    olm.deployment-spec-hash: 7b99645b7f
    olm.owner: infinispan-operator.v2.1.3
    olm.owner.kind: ClusterServiceVersion
    olm.owner.namespace: cache
    operators.coreos.com/infinispan.cache: ""
  name: infinispan-operator
  namespace: cache
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: ClusterServiceVersion
    name: infinispan-operator.v2.1.3
    uid: 530740bb-b151-4fb4-9239-7e0e6ade486d
  resourceVersion: "95761466"
  selfLink: /apis/apps/v1/namespaces/cache/deployments/infinispan-operator
  uid: 121843e1-633e-4c7d-a767-1c15c716c372
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      name: infinispan-operator
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        alm-examples: |-
          [
            {
              "apiVersion": "infinispan.org/v1",
              "kind": "Infinispan",
              "metadata": {
                "name": "example-infinispan"
              },
              "spec": {
                "replicas": 1
              }
            },
            {
              "apiVersion": "infinispan.org/v2alpha1",
              "kind": "Backup",
              "metadata": {
                "name": "example-backup"
              },
              "spec": {
                "cluster": "example-infinispan",
                "container": {
                  "cpu": "1000m",
                  "extraJvmOpts": "-Djava.property=me",
                  "memory": "1Gi"
                },
                "path": "asdasd"
              }
            },
            {
              "apiVersion": "infinispan.org/v2alpha1",
              "kind": "Cache",
              "metadata": {
                "name": "example-cache"
              },
              "spec": {
                "adminAuth": {
                  "secretName": "basic-auth"
                },
                "clusterName": "example-infinispan",
                "name": "mycache"
              }
            },
            {
              "apiVersion": "infinispan.org/v2alpha1",
              "kind": "Restore",
              "metadata": {
                "name": "example-restore"
              },
              "spec": {
                "cluster": "example-infinispan",
                "container": {
                  "cpu": "1000m",
                  "extraJvmOpts": "-Djava.property=me",
                  "memory": "1Gi"
                },
                "path": "asdasd"
              }
            },
            {
              "apiVersion": "infinispan.org/v2alpha1",
              "kind": "Batch",
              "metadata": {
                "name": "example-batch",
              },
              "spec": {
                "cluster": "example-infinispan",
                "config": "create cache --template=org.infinispan.DIST_SYNC batch-cache"
              }
            },
          ]
        capabilities: Full Lifecycle
        categories: Database
        certified: "false"
        containerImage: quay.io/infinispan/operator:2.1.2.Final
        createdAt: "2021-03-18T09:00:00Z"
        description: Create and manage Infinispan clusters.
        olm.operatorGroup: operatorgroup
        olm.operatorNamespace: cache
        olm.targetNamespaces: cache
        operatorframework.io/properties: '{"properties":[{"type":"olm.gvk","value":{"group":"infinispan.org","kind":"Backup","version":"v2alpha1"}},{"type":"olm.gvk","value":{"group":"infinispan.org","kind":"Batch","version":"v2alpha1"}},{"type":"olm.gvk","value":{"group":"infinispan.org","kind":"Cache","version":"v2alpha1"}},{"type":"olm.gvk","value":{"group":"infinispan.org","kind":"Infinispan","version":"v1"}},{"type":"olm.gvk","value":{"group":"infinispan.org","kind":"Restore","version":"v2alpha1"}},{"type":"olm.package","value":{"packageName":"infinispan","version":"2.1.3"}}]}'
        repository: https://github.com/infinispan/infinispan-operator
        support: Infinispan
      creationTimestamp: "2021-03-18T09:00:00Z"
      labels:
        name: infinispan-operator
    spec:
      containers:
      - command:
        - infinispan-operator
        env:
        - name: MAKE_DATADIR_WRITABLE
          value: "true"
        - name: WATCH_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.annotations['olm.targetNamespaces']
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: OPERATOR_NAME
          value: infinispan-operator
        - name: RELATED_IMAGE_OPENJDK
          value: quay.io/infinispan/server:12.0.2.Final
        - name: OPERATOR_CONDITION_NAME
          value: infinispan-operator.v2.1.3
        image: quay.io/infinispan/operator:2.1.2.Final
        imagePullPolicy: Always
        name: infinispan-operator
        ports:
        - containerPort: 60000
          name: metrics
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: infinispan-operator
      serviceAccountName: infinispan-operator
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2021-05-11T14:53:43Z"
    lastUpdateTime: "2021-05-11T14:53:43Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2021-05-11T14:53:40Z"
    lastUpdateTime: "2021-05-11T14:56:22Z"
    message: ReplicaSet "infinispan-operator-5d695dfc55" is progressing.
    reason: ReplicaSetUpdated
    status: "True"
    type: Progressing
  observedGeneration: 2
  readyReplicas: 1
  replicas: 2
  unavailableReplicas: 1
  updatedReplicas: 1

dicolasi commented 3 years ago

I have modified the csv and now deploy has the environment variable set. No success however.

dicolasi commented 3 years ago

Screenshot 2021-05-11 at 15 57 33

dicolasi commented 3 years ago

ok sorted! The trick was to remove the old stateful set.

dicolasi commented 3 years ago

Thanks for the support @dmvolod

dmvolod commented 3 years ago

Thanks for the support

You are welcome

dicolasi commented 2 years ago

@dmvolod still see the same error with a new installation:


2021-10-21 07:57:34,163 WARN  [io.qua.qut.run.ContentTypes[] (main) Unable to detect the content type for templates/jgroups-encrypt; using application/octet-stream
2021-10-21 07:57:34,163 INFO  [io.quarkus[] (main) config-generator 2.1.8.Final native (powered by Quarkus 1.7.6.Final) started in 0.075s. 
2021-10-21 07:57:34,164 INFO  [io.quarkus[] (main) Profile prod activated. 
2021-10-21 07:57:34,164 INFO  [io.quarkus[] (main) Installed features: [cdi, qute[]
2021-10-21 07:57:34,173 INFO  [io.quarkus[] (main) config-generator stopped in 0.000s
07:57:48,171 INFO  (main) [BOOT[] JVM OpenJDK 64-Bit Server VM Red Hat, Inc. 11.0.11+9-LTS
07:57:48,277 INFO  (main) [BOOT[] JVM arguments = [-server, -Dinfinispan.bind.address=0.0.0.0, -Xmx256m, -XX:+ExitOnOutOfMemoryError, -XX:MetaspaceSize=32m, -XX:MaxMetaspaceSize=96m, -Djava.net.preferIPv4Stack=true, -Djava.awt.headless=true, -Dvisualvm.display.name=infinispan-server, -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager, -Dinfinispan.server.home.path=/opt/infinispan, -classpath, :/opt/infinispan/boot/infinispan-server-runtime-12.1.7.Final-loader.jar, org.infinispan.server.loader.Loader, org.infinispan.server.Bootstrap]
07:57:48,367 INFO  (main) [BOOT[] PID = 85
07:58:02,768 WARN  (main) [org.infinispan.PERSISTENCE[] ISPN000554: jboss-marshalling is deprecated and planned for removal
07:58:14,470 WARN  (main) [org.infinispan.CONTAINER[] ISPN000574: Global state cannot persisted because it is incomplete (usually caused by errors at startup).
07:58:14,567 FATAL (main) [org.infinispan.SERVER[] ISPN080028: Infinispan Server failed to start java.util.concurrent.ExecutionException: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: ISPN000512: Cannot acquire lock '/opt/infinispan/server/data/___global.lck' for persistent global state
    at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
    at org.infinispan.server.Bootstrap.runInternal(Bootstrap.java:160)
    at org.infinispan.server.tool.Main.run(Main.java:98)
    at org.infinispan.server.Bootstrap.main(Bootstrap.java:47)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.infinispan.server.loader.Loader.run(Loader.java:106)
    at org.infinispan.server.loader.Loader.main(Loader.java:51)
Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: ISPN000512: Cannot acquire lock '/opt/infinispan/server/data/___global.lck' for persistent global state
    at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:755)
    at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:718)
    at org.infinispan.server.SecurityActions.lambda$startCacheManager$1(SecurityActions.java:67)
    at org.infinispan.security.Security.doPrivileged(Security.java:56)
    at org.infinispan.server.SecurityActions.doPrivileged(SecurityActions.java:39)
    at org.infinispan.server.SecurityActions.startCacheManager(SecurityActions.java:70)
    at org.infinispan.server.Server.run(Server.java:360)
    ... 9 more
Caused by: org.infinispan.commons.CacheConfigurationException: ISPN000512: Cannot acquire lock '/opt/infinispan/server/data/___global.lck' for persistent global state
    at org.infinispan.globalstate.impl.GlobalStateManagerImpl.acquireGlobalLock(GlobalStateManagerImpl.java:88)
    at org.infinispan.globalstate.impl.GlobalStateManagerImpl.start(GlobalStateManagerImpl.java:65)
    at org.infinispan.globalstate.impl.CorePackageImpl$1.start(CorePackageImpl.java:34)
    at org.infinispan.globalstate.impl.CorePackageImpl$1.start(CorePackageImpl.java:27)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:604)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:595)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:564)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30)
    at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:787)
    at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:354)
    at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:250)
    at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:750)
    ... 15 more
Caused by: java.io.FileNotFoundException: /opt/infinispan/server/data/___global.lck (Permission denied)
    at java.base/java.io.FileOutputStream.open0(Native Method)
    at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
    at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237)
    at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:187)
    at org.infinispan.globalstate.impl.GlobalStateManagerImpl.acquireGlobalLock(GlobalStateManagerImpl.java:82)
    ... 26 more

andrey-dubnik commented 1 year ago

Just hit the issue with the Operator in AKS. IT does look like adding the fsGroup to the STS makes a difference

e.g.

securityContext:
        fsGroup: 0

Current STS template does not have that configuration. I can raise PR to add it if this is Ok as there does not seem to be any apparent way of getting around that problem when deploying via OLM?

STS reference - https://github.com/infinispan/infinispan-operator/blob/9fe9afc914659ebbad6e2189062ee5c803a58344/pkg/reconcile/pipeline/infinispan/handler/provision/statefulsets.go#L89

ryanemerson commented 1 year ago

It's not possible for us to hard code the fsGroup in the Operator as different k8 providers treat this differently, e.g. Openshift vs AKS.

You can workaround this problem in OLM by adding the following to your subscription:

spec:
  ...
 config:
    env:
      - name: MAKE_DATADIR_WRITABLE
        value: "true"

See here for more details on OLM env variables.

andrey-dubnik commented 1 year ago

Thanks a lot @ryanemerson! The best engineering solution involves writing no code :)

PS - I would not opted for the hard code but rather than the config enhancement of the operator fortunately the provided solution works perfectly.

ramonmoraga commented 4 months ago

I have this error when i what use volumen to http://opt/infinispan/server/data/ (

[33m2024-05-16 10:56:49,052 WARN (main) [org.infinispan.CONTAINER] ISPN000574: Global state cannot persisted because it is incomplete (usually caused by errors at startup).[m | infineror -- | -- May 16, 2024 at 12:56 (UTC+2:00) | [1;31m2024-05-16 10:56:49,049 ERROR (main) [org.infinispan.CONFIG] ISPN000660: DefaultCacheManager start failed, stopping any running components org.infinispan.commons.CacheConfigurationException: ISPN000512: Cannot acquire lock '/opt/infinispan/server/data/___global.lck' for persistent global state

) Someone can healp me please

ryanemerson commented 4 months ago

@ramonmoraga That issue is not directly related to this one, let's continue the discussion on #2082

infinispan / infinispan-operator

Access denied on /opt/infinispan/server/data/___global.lck. #392