Splunk Operator: how to let varVolumeStorageConfig and etcVolumeStorageConfig to use custom defined volumes?

[yaroslav-nakonechnikov]

Please select the type of request

Feature Request

Tell us more

Describe the request Question about usage custom defined volumes in varVolumeStorageConfig and etcVolumeStorageConfig section.

Expected behavior There is a field which may be called volumeName, and if this field is not empty, it checks that volume name esists and starts to use it for varVolumeStorageConfig and etcVolumeStorageConfig.

Splunk setup on K8S It may not be related directly, but we run splunk-operator in EKS with managed nodes under Bottlerocket.

Additional context(optional) This may be useful to make it a bit simplier to use Instance store in AWS. For example i3en.* has quite big amount of space, which may be utilized instead of EBS. And using Instance Store is significantly faster then using EBS and then using special kubernetes drivers/provisioners - less api calls. Yes, i may be potentially dangerous, but in case only one pod sticks to node - it may be called safe.

[yaroslav-nakonechnikov]

looks like using env var helps:

"etcVolumeStorageConfig" = {
        "ephemeralStorage" = true
      }
      "varVolumeStorageConfig" = {
        "ephemeralStorage" = true
      }
      "extraEnv" = [{
        "name"  = "SPLUNK_DB"
        "value" = "/mnt/scratch/splunk-sh"
      }]
      "volumes" = [{
        "hostPath" = {
          "path" = "/mnt/scratch"
          "type" = "DirectoryOrCreate"
        }
        "name" = "scratch"
      }]

scratch - pre-created ephemeral space. but, now facing issue with with search-head... Somehow search-head deployer is being configured with same CRD, and it adds complexity to config.

[yaroslav-nakonechnikov]

we found unexpected behaviour there: at some point nodes are getting recreated due to:

The node was low on resource: ephemeral-storage. Threshold quantity: 8042683097, available: 5319692Ki. Container splunk was using 119968Ki, request is 0, has larger consumption of ephemeral-storage.

keep that in mind everyone who will start to utilize ephemeral storage.