Smartstore: Clarify smartstore documentation

[marcusschiesser]

Please select the type of request

Enhancement

Tell us more

Describe the request Setting up an indexer cluster took me 4 days, primarily because the incorrect documentation you can find at https://splunk.github.io/splunk-operator/SmartStore.html was leading me in the wrong direction - for the whole drama read: https://splunk-usergroups.slack.com/archives/C01M8M2R3AM/p1667274184903249?thread_ts=1666813185.357399&cid=C01M8M2R3AM

Expected behavior

1. Fix incorrect documentation:

   • It's not possible to authenticate to S3 via pod role see: https://github.com/splunk/splunk-operator/issues/627
   • I couldn't make node authentication work either and it seems like no one on Slack did either

2. Add more debugging help:

   • Add a simple AWS Python script to test the S3 connection, similar to https://github.com/splunk/splunk-operator/issues/627#issuecomment-1299534979
   • Add the necessary AWS policies for the S3 bucket as JSON

Splunk setup on K8S

• EKS 1.21
• Splunk Operator 1.10

[yaroslav-nakonechnikov]

@marcusschiesser so did you manage to install apps from s3 with native s3:// way? or how you did that?

[marcusschiesser]

@iaroslav-nakonechnikov yes it's working with secret key authentication, see https://splunk-usergroups.slack.com/archives/C01M8M2R3AM/p1667457690723269?thread_ts=1666813185.357399&cid=C01M8M2R3AM - but the documentation is claiming it's working with IAM roles which is not true.

[yaroslav-nakonechnikov]

aaah, we can't use secret keys... :(

so in our case this leads to #1030

ps. i don't have access to slack, sent request to splunk to open it.

[vivekr-splunk]

this will be an enhancement for splunk. We are working on supporting AWS IAM Service account for splunkd.

[yaroslav-nakonechnikov]

another thing which was found just today: splunk can work with s3 smartstore in EKS with node-profile with one exception: Nodes should have NDIS v1 turned on, otherwise it can't send requests to http://169.254.169.254

for us, for example, it was unexpected completely, as we are already using V2 by default everywhere. and why? Answer: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html