Unable to undeploy a rule after manager failure (sp-distributed)

[debelyoo]

Description: I'm deploying rules on a distributed setup of WSO2 stream processor (2 managers, 2 workers). I'm deploying the rules with the REST API.

Here is what I'm observing:

1. I deploy rule with REST API on manager1 (POST request on https://localhost:9543/siddhi-apps)
2. I see the rule deployed by querying the REST API on manager1 (GET https://localhost:9543/siddhi-apps). I see nothing on manager 2 (GET https://localhost:9544/siddhi-apps)
3. I kill manager1. The rule still runs on the worker but is not visible on manager2. GET https://localhost:9544/siddhi-apps returns an empty array
4. I restart manager1. The rule still runs but is not visible on either managers (manager1 and manager2). GET https://localhost:9543/siddhi-apps and GET https://localhost:9544/siddhi-apps return an empty array
5. Then, I want to remove the rule via the REST API (DELETE https://localhost:9543/siddhi-apps/myapp) but I can't

Suggested Labels: sp-distributed, rule deployment, rest API

Affected Product Version: 4.3.0

Steps to reproduce: Described above

[pcnfernando]

Hi @debelyoo , Can you please share the 2 manager and 2 worker deployment.yml files respectively residing in SP_HOME/conf/manager and SP_HOME/conf/worker.

Also, please verify whether you are kubernetes, docker-compose or local for the distributed deployment.

[debelyoo]

Hi @pcnfernando, I'm using a docker-compose deployment.

manager's deployment.yml

################################################################################
#   Copyright (c) 2018, WSO2 Inc. (http://www.wso2.org) All Rights Reserved
#
#   Licensed under the Apache License, Version 2.0 (the \"License\");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an \"AS IS\" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.
################################################################################

  # Carbon Configuration Parameters
wso2.carbon:
    # value to uniquely identify a server
  id: ${NODE_ID}
    # server name
  name: WSO2 Stream Processor
    # ports used by this server
  ports:
      # port offset
    offset: 0

wso2.transport.http:
  transportProperties:
    -
      name: "server.bootstrap.socket.timeout"
      value: 60
    -
      name: "client.bootstrap.socket.timeout"
      value: 60
    -
      name: "latency.metrics.enabled"
      value: true
  listenerConfigurations:
    -
      id: "default"
      host: "${NODE_IP}"
      port: 9190
    -
      id: "msf4j-https"
      host: "${NODE_IP}"
      port: 9543
      scheme: https
      keyStoreFile: "${carbon.home}/resources/security/wso2carbon.jks"
      keyStorePassword: wso2carbon
      certPass: wso2carbon
  senderConfigurations:
    -
      id: "http-sender"

 # Configuration used for the databridge communication
databridge.config:
    # No of worker threads to consume events
    # THIS IS A MANDATORY FIELD
  workerThreads: 10
    # Maximum amount of messages that can be queued internally in MB
    # THIS IS A MANDATORY FIELD
  maxEventBufferCapacity: 10000000
    # Queue size; the maximum number of events that can be stored in the queue
    # THIS IS A MANDATORY FIELD
  eventBufferSize: 2000
    # Keystore file path
    # THIS IS A MANDATORY FIELD
  keyStoreLocation : ${sys:carbon.home}/resources/security/wso2carbon.jks
    # Keystore password
    # THIS IS A MANDATORY FIELD
  keyStorePassword : wso2carbon
    # Session Timeout value in mins
    # THIS IS A MANDATORY FIELD
  clientTimeoutMin: 30
    # Data receiver configurations
    # THIS IS A MANDATORY FIELD
  dataReceivers:
  -
      # Data receiver configuration
    dataReceiver:
        # Data receiver type
        # THIS IS A MANDATORY FIELD
      type: Thrift
        # Data receiver properties
      properties:
        tcpPort: '7611'
        sslPort: '7711'

  -
      # Data receiver configuration
    dataReceiver:
        # Data receiver type
        # THIS IS A MANDATORY FIELD
      type: Binary
        # Data receiver properties
      properties:
        tcpPort: '9611'
        sslPort: '9711'
        tcpReceiverThreadPoolSize: '100'
        sslReceiverThreadPoolSize: '100'
        hostName: ${NODE_IP}

 # Configuration of the Data Agents - to publish events through databridge
data.agent.config:
    # Data agent configurations
    # THIS IS A MANDATORY FIELD
  agents:
  -
      # Data agent configuration
    agentConfiguration:
        # Data agent name
        # THIS IS A MANDATORY FIELD
      name: Thrift
        # Data endpoint class
        # THIS IS A MANDATORY FIELD
      dataEndpointClass: org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint
        # Data publisher strategy
      publishingStrategy: async
        # Trust store path
      trustStorePath: '${sys:carbon.home}/resources/security/client-truststore.jks'
        # Trust store password
      trustStorePassword: 'wso2carbon'
        # Queue Size
      queueSize: 32768
        # Batch Size
      batchSize: 200
        # Core pool size
      corePoolSize: 1
        # Socket timeout in milliseconds
      socketTimeoutMS: 30000
        # Maximum pool size
      maxPoolSize: 1
        # Keep alive time in pool
      keepAliveTimeInPool: 20
        # Reconnection interval
      reconnectionInterval: 30
        # Max transport pool size
      maxTransportPoolSize: 250
        # Max idle connections
      maxIdleConnections: 250
        # Eviction time interval
      evictionTimePeriod: 5500
        # Min idle time in pool
      minIdleTimeInPool: 5000
        # Secure max transport pool size
      secureMaxTransportPoolSize: 250
        # Secure max idle connections
      secureMaxIdleConnections: 250
        # secure eviction time period
      secureEvictionTimePeriod: 5500
        # Secure min idle time in pool
      secureMinIdleTimeInPool: 5000
        # SSL enabled protocols
      sslEnabledProtocols: TLSv1.1,TLSv1.2
        # Ciphers
      ciphers: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
  -
      # Data agent configuration
    agentConfiguration:
        # Data agent name
        # THIS IS A MANDATORY FIELD
      name: Binary
        # Data endpoint class
        # THIS IS A MANDATORY FIELD
      dataEndpointClass: org.wso2.carbon.databridge.agent.endpoint.binary.BinaryDataEndpoint
        # Data publisher strategy
      publishingStrategy: async
        # Trust store path
      trustStorePath: '${sys:carbon.home}/resources/security/client-truststore.jks'
        # Trust store password
      trustStorePassword: 'wso2carbon'
        # Queue Size
      queueSize: 32768
        # Batch Size
      batchSize: 200
        # Core pool size
      corePoolSize: 1
        # Socket timeout in milliseconds
      socketTimeoutMS: 30000
        # Maximum pool size
      maxPoolSize: 1
        # Keep alive time in pool
      keepAliveTimeInPool: 20
        # Reconnection interval
      reconnectionInterval: 30
        # Max transport pool size
      maxTransportPoolSize: 250
        # Max idle connections
      maxIdleConnections: 250
        # Eviction time interval
      evictionTimePeriod: 5500
        # Min idle time in pool
      minIdleTimeInPool: 5000
        # Secure max transport pool size
      secureMaxTransportPoolSize: 250
        # Secure max idle connections
      secureMaxIdleConnections: 250
        # secure eviction time period
      secureEvictionTimePeriod: 5500
        # Secure min idle time in pool
      secureMinIdleTimeInPool: 5000
        # SSL enabled protocols
      sslEnabledProtocols: TLSv1.1,TLSv1.2
        # Ciphers
      ciphers: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256

  # Deployment configuration parameters
wso2.artifact.deployment:
    # Scheduler update interval
  updateInterval: 5

 # Periodic Persistence Configuration
state.persistence:
  enabled: false
  intervalInMin: 1
  revisionsToKeep: 2
  persistenceStore: org.wso2.carbon.stream.processor.core.persistence.FileSystemPersistenceStore
  config:
    location: siddhi-app-persistence

 # Secure Vault Configuration
wso2.securevault:
  secretRepository:
    type: org.wso2.carbon.secvault.repository.DefaultSecretRepository
    parameters:
      privateKeyAlias: wso2carbon
      keystoreLocation: ${sys:carbon.home}/resources/security/securevault.jks
      secretPropertiesFile: ${sys:carbon.home}/conf/${sys:wso2.runtime}/secrets.properties
  masterKeyReader:
    type: org.wso2.carbon.secvault.reader.DefaultMasterKeyReader
    parameters:
      masterKeyReaderFile: ${sys:carbon.home}/conf/${sys:wso2.runtime}/master-keys.yaml
wso2.datasources:
  dataSources:
  - name: SP_MGT_DB
    description: The datasource used for registry and user manager
    jndiConfig:
      name: jdbc/WSO2ClusterDB
    definition:
      type: RDBMS
      configuration:
        jdbcUrl: 'jdbc:mysql://sp-rdbms:3306/SP_MGT_DB?useSSL=false'
        username: root
        password: root
        driverClassName: com.mysql.jdbc.Driver
        maxPoolSize: 50
        idleTimeout: 60000
        connectionTestQuery: SELECT 1
        validationTimeout: 30000
        isAutoCommit: false
  - name: WSO2_PERMISSIONS_DB
    description: The datasource used for permission feature
    jndiConfig:
      name: jdbc/PERMISSION_DB
      useJndiReference: true
    definition:
      type: RDBMS
      configuration:
        jdbcUrl: 'jdbc:h2:${sys:carbon.home}/wso2/${sys:wso2.runtime}/database/PERMISSION_DB;IFEXISTS=TRUE;DB_CLOSE_ON_EXIT=FALSE;LOCK_TIMEOUT=60000;MVCC=TRUE'
        username: wso2carbon
        password: wso2carbon
        driverClassName: org.h2.Driver
        maxPoolSize: 50
        idleTimeout: 60000
        connectionTestQuery: SELECT 1
        validationTimeout: 30000
        isAutoCommit: false

 # Cluster Configuration
cluster.config:
  enabled: true
  groupId:  sp-mgt
  coordinationStrategyClass: org.wso2.carbon.cluster.coordinator.rdbms.RDBMSCoordinationStrategy
  strategyConfig:
    datasource: SP_MGT_DB     # define a mysql datasource configured to the shared database
    heartbeatInterval: 1000
    heartbeatMaxRetry: 2
    eventPollingInterval: 1000

  # Deployment Configuration for Distributed Deployment
deployment.config:
  type: distributed
  httpsInterface:
    host: ${NODE_ID}
    port: 9543
  heartbeatInterval: 2000
  heartbeatMaxRetry: 2
  datasource: SP_MGT_DB           # define a mysql datasource in datasources and refer it from here.
  minResourceCount: 1
  bootstrapURLs: kafka:9092   # kafka urls
  zooKeeperURLs: zookeeper:2181   # zookeeper urls

worker's deployment.yml

################################################################################
#   Copyright (c) 2018, WSO2 Inc. (http://www.wso2.org) All Rights Reserved
#
#   Licensed under the Apache License, Version 2.0 (the \"License\");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an \"AS IS\" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.
################################################################################

  # Carbon Configuration Parameters
wso2.carbon:
    # value to uniquely identify a server
  id: ${NODE_ID}
    # server name
  name: WSO2 Stream Processor
    # ports used by this server
  ports:
      # port offset
    offset: 0

wso2.transport.http:
  transportProperties:
    -
      name: "server.bootstrap.socket.timeout"
      value: 60
    -
      name: "client.bootstrap.socket.timeout"
      value: 60
    -
      name: "latency.metrics.enabled"
      value: true

  listenerConfigurations:
    -
      id: "default"
      host: "${NODE_IP}"
      port: 9090
    -
      id: "msf4j-https"
      host: "${NODE_IP}"
      port: 9443
      scheme: https
      keyStoreFile: "${carbon.home}/resources/security/wso2carbon.jks"
      keyStorePassword: wso2carbon
      certPass: wso2carbon

  senderConfigurations:
    -
      id: "http-sender"

  # Configuration used for the databridge communication
databridge.config:
    # No of worker threads to consume events
    # THIS IS A MANDATORY FIELD
  workerThreads: 10
    # Maximum amount of messages that can be queued internally in MB
    # THIS IS A MANDATORY FIELD
  maxEventBufferCapacity: 10000000
    # Queue size; the maximum number of events that can be stored in the queue
    # THIS IS A MANDATORY FIELD
  eventBufferSize: 2000
    # Keystore file path
    # THIS IS A MANDATORY FIELD
  keyStoreLocation : ${sys:carbon.home}/resources/security/wso2carbon.jks
    # Keystore password
    # THIS IS A MANDATORY FIELD
  keyStorePassword : wso2carbon
    # Session Timeout value in mins
    # THIS IS A MANDATORY FIELD
  clientTimeoutMin: 30
    # Data receiver configurations
    # THIS IS A MANDATORY FIELD
  dataReceivers:
  -
      # Data receiver configuration
    dataReceiver:
        # Data receiver type
        # THIS IS A MANDATORY FIELD
      type: Thrift
        # Data receiver properties
      properties:
        tcpPort: '7611'
        sslPort: '7711'

  -
      # Data receiver configuration
    dataReceiver:
        # Data receiver type
        # THIS IS A MANDATORY FIELD
      type: Binary
        # Data receiver properties
      properties:
        tcpPort: '9611'
        sslPort: '9711'
        tcpReceiverThreadPoolSize: '100'
        sslReceiverThreadPoolSize: '100'
        hostName: ${NODE_IP}

  # Configuration of the Data Agents - to publish events through databridge
data.agent.config:
    # Data agent configurations
    # THIS IS A MANDATORY FIELD
  agents:
  -
      # Data agent configuration
    agentConfiguration:
        # Data agent name
        # THIS IS A MANDATORY FIELD
      name: Thrift
        # Data endpoint class
        # THIS IS A MANDATORY FIELD
      dataEndpointClass: org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint
        # Data publisher strategy
      publishingStrategy: async
        # Trust store path
      trustStorePath: '${sys:carbon.home}/resources/security/client-truststore.jks'
        # Trust store password
      trustStorePassword: 'wso2carbon'
        # Queue Size
      queueSize: 32768
        # Batch Size
      batchSize: 200
        # Core pool size
      corePoolSize: 1
        # Socket timeout in milliseconds
      socketTimeoutMS: 30000
        # Maximum pool size
      maxPoolSize: 1
        # Keep alive time in pool
      keepAliveTimeInPool: 20
        # Reconnection interval
      reconnectionInterval: 30
        # Max transport pool size
      maxTransportPoolSize: 250
        # Max idle connections
      maxIdleConnections: 250
        # Eviction time interval
      evictionTimePeriod: 5500
        # Min idle time in pool
      minIdleTimeInPool: 5000
        # Secure max transport pool size
      secureMaxTransportPoolSize: 250
        # Secure max idle connections
      secureMaxIdleConnections: 250
        # secure eviction time period
      secureEvictionTimePeriod: 5500
        # Secure min idle time in pool
      secureMinIdleTimeInPool: 5000
        # SSL enabled protocols
      sslEnabledProtocols: TLSv1.1,TLSv1.2
        # Ciphers
      ciphers: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
  -
      # Data agent configuration
    agentConfiguration:
        # Data agent name
        # THIS IS A MANDATORY FIELD
      name: Binary
        # Data endpoint class
        # THIS IS A MANDATORY FIELD
      dataEndpointClass: org.wso2.carbon.databridge.agent.endpoint.binary.BinaryDataEndpoint
        # Data publisher strategy
      publishingStrategy: async
        # Trust store path
      trustStorePath: '${sys:carbon.home}/resources/security/client-truststore.jks'
        # Trust store password
      trustStorePassword: 'wso2carbon'
        # Queue Size
      queueSize: 32768
        # Batch Size
      batchSize: 200
        # Core pool size
      corePoolSize: 1
        # Socket timeout in milliseconds
      socketTimeoutMS: 30000
        # Maximum pool size
      maxPoolSize: 1
        # Keep alive time in pool
      keepAliveTimeInPool: 20
        # Reconnection interval
      reconnectionInterval: 30
        # Max transport pool size
      maxTransportPoolSize: 250
        # Max idle connections
      maxIdleConnections: 250
        # Eviction time interval
      evictionTimePeriod: 5500
        # Min idle time in pool
      minIdleTimeInPool: 5000
        # Secure max transport pool size
      secureMaxTransportPoolSize: 250
        # Secure max idle connections
      secureMaxIdleConnections: 250
        # secure eviction time period
      secureEvictionTimePeriod: 5500
        # Secure min idle time in pool
      secureMinIdleTimeInPool: 5000
        # SSL enabled protocols
      sslEnabledProtocols: TLSv1.1,TLSv1.2
        # Ciphers
      ciphers: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256

wso2.metrics:
  # Enable Metrics
  enabled: true
  jmx:
      # Register MBean when initializing Metrics
      registerMBean: true
      # MBean Name
      name: org.wso2.carbon:type=Metrics
  levels:
    # The root level configured for Metrics
    rootLevel: INFO
    # Metric Levels
    levels:
      jvm.buffers: 'OFF'
      jvm.class-loading: INFO
      jvm.gc: DEBUG
      jvm.memory: INFO

wso2.metrics.jdbc:
  # Data Source Configurations for JDBC Reporters
  dataSource:
    # Default Data Source Configuration
    - &JDBC01
      # JNDI name of the data source to be used by the JDBC Reporter.
      # This data source should be defined in a *-datasources.xml file in conf/datasources directory.
      dataSourceName: java:comp/env/jdbc/WSO2MetricsDB
      # Schedule regular deletion of metrics data older than a set number of days.
      # It is recommended that you enable this job to ensure your metrics tables do not get extremely large.
      # Deleting data older than seven days should be sufficient.
      scheduledCleanup:
        # Enable scheduled cleanup to delete Metrics data in the database.
        enabled: false

        # The scheduled job will cleanup all data older than the specified days
        daysToKeep: 7

        # This is the period for each cleanup operation in seconds.
        scheduledCleanupPeriod: 86400

  # The JDBC Reporter is in the Metrics JDBC Core feature
  reporting:
    # The JDBC Reporter configurations will be ignored if the Metrics JDBC Core feature is not available in runtime
    jdbc:
      - # The name for the JDBC Reporter
        name: JDBC

        # Enable JDBC Reporter
        enabled: true

        # Source of Metrics, which will be used to identify each metric in database -->
        # Commented to use the hostname by default
        # source: Carbon

        # Alias referring to the Data Source configuration
        dataSource: *JDBC01

        # Polling Period in seconds.
        # This is the period for polling metrics from the metric registry and updating the database with the values
        pollingPeriod: 20

  # Deployment configuration parameters
wso2.artifact.deployment:
    # Scheduler update interval
  updateInterval: 5

  # Periodic Persistence Configuration
state.persistence:
  enabled: true
  intervalInMin: 1
  revisionsToKeep: 2
  persistenceStore: org.wso2.carbon.stream.processor.core.persistence.DBPersistenceStore
  config:
    datasource: SP_WORKER_STATE_DB   # A datasource with this name should be defined in wso2.datasources namespace
    table: PERSISTENCE_TABLE

  # Secure Vault Configuration
wso2.securevault:
  secretRepository:
    type: org.wso2.carbon.secvault.repository.DefaultSecretRepository
    parameters:
      privateKeyAlias: wso2carbon
      keystoreLocation: ${sys:carbon.home}/resources/security/securevault.jks
      secretPropertiesFile: ${sys:carbon.home}/conf/${sys:wso2.runtime}/secrets.properties
  masterKeyReader:
    type: org.wso2.carbon.secvault.reader.DefaultMasterKeyReader
    parameters:
      masterKeyReaderFile: ${sys:carbon.home}/conf/${sys:wso2.runtime}/master-keys.yaml

  # Datasource Configurations
wso2.datasources:
  dataSources:
  - name: SP_WORKER_STATE_DB
    description: The datasource used for registry and user manager
    definition:
      type: RDBMS
      configuration:
        jdbcUrl: 'jdbc:mysql://sp-rdbms:3306/SP_WORKER_STATE_DB?useSSL=false'
        username: root
        password: 'root'
        driverClassName: com.mysql.jdbc.Driver
        maxPoolSize: 50
        idleTimeout: 60000
        connectionTestQuery: SELECT 1
        validationTimeout: 30000
        isAutoCommit: false
  - definition:
        configuration:
          connectionTestQuery: "SELECT 1"
          driverClassName: org.h2.Driver
          idleTimeout: 60000
          isAutoCommit: false
          jdbcUrl: "jdbc:h2:${sys:carbon.home}/wso2/${sys:wso2.runtime}/database/WSO2_CARBON_DB;DB_CLOSE_ON_EXIT=FALSE;LOCK_TIMEOUT=60000"
          maxPoolSize: 50
          password: wso2carbon
          username: wso2carbon
          validationTimeout: 30000
        type: RDBMS
    description: "The datasource used for registry and user manager"
    name: WSO2_CARBON_DB

    # carbon metrics data source
  - name: WSO2_METRICS_DB
    description: The datasource used for dashboard feature
    jndiConfig:
        name: jdbc/WSO2MetricsDB
    definition:
        type: RDBMS
        configuration:
          jdbcUrl: 'jdbc:h2:${sys:carbon.home}/wso2/dashboard/database/metrics;AUTO_SERVER=TRUE'
          username: wso2carbon
          password: wso2carbon
          driverClassName: org.h2.Driver
          maxPoolSize: 50
          idleTimeout: 60000
          connectionTestQuery: SELECT 1
          validationTimeout: 30000
          isAutoCommit: false

  - name: WSO2_PERMISSIONS_DB
    description: The datasource used for permission feature
    jndiConfig:
        name: jdbc/PERMISSION_DB
        useJndiReference: true
    definition:
        type: RDBMS
        configuration:
          jdbcUrl: 'jdbc:h2:${sys:carbon.home}/wso2/${sys:wso2.runtime}/database/PERMISSION_DB;IFEXISTS=TRUE;DB_CLOSE_ON_EXIT=FALSE;LOCK_TIMEOUT=60000;MVCC=TRUE'
          username: wso2carbon
          password: wso2carbon
          driverClassName: org.h2.Driver
          maxPoolSize: 50
          idleTimeout: 60000
          connectionTestQuery: SELECT 1
          validationTimeout: 30000
          isAutoCommit: false

  # Cluster Configuration
cluster.config:
  enabled: false
  groupId:  sp
  coordinationStrategyClass: org.wso2.carbon.cluster.coordinator.rdbms.RDBMSCoordinationStrategy
  strategyConfig:
    datasource: WSO2_CARBON_DB
    heartbeatInterval: 1000
    heartbeatMaxRetry: 2
    eventPollingInterval: 1000

  # Sample of deployment.config for Two node HA
#deployment.config:
#  type: ha
#  liveSync:
#    enabled: true
#    advertisedHost: localhost
#    advertisedPort: 9090
#    username: admin
#    password: admin
#  outputSyncInterval: 10000
#  stateSyncGracePeriod: 60000
#  sinkQueueCapacity: 20000
#  sourceQueueCapacity: 20000
#  retryAppSyncPeriod: 60000

  # Sample of deployment.config for Distributed deployment
deployment.config:
  type: distributed
  httpsInterface:
    host: ${NODE_ID}
    port: 9443
    username: admin
    password: admin
  leaderRetryInterval: 10000
  resourceManagers:
    - host: wso2sp-manager-1
      port: 9543
      username: admin
      password: admin
    - host: wso2sp-manager-2
      port: 9544
      username: admin
      password: admin

[pcnfernando]

Hi @debelyoo ,

Thank you for the information. The issue you are experiencing is related to docker-sp#76.

Due to the fact that, the manager's deployment directory not being synced, you encountered this issue. This issue will be fixed with PR docker-sp#77.

Furthermore, you can use the docker-compose/sp-distributed/manager/siddhi-files directory to deploy the distributed apps once the PR is merged.

I will update the thread once this fix is available in master.