Support for Indexer Cluster and Distributed Environment

[VatsalJagani]

• See if we can provide support for Indexer and Distributed Environment such that users don't have to manually install App on Indexer.
• Need to make sure Geo-IP-location file and limits.conf replicate to Indexers and that limits.conf is taking precedence.

[VatsalJagani]

Currently following are the problems for distributing limits.conf and db file to Indexer Cluster:

• Currently neither the limits.conf nor the mmdb file is part of knowledge bundle.
• We need to add below two stanzas to enable them to be part of knowledge bundle.
  1. [replicationWhitelist]
  2. [replicationSettings:refineConf]
• But that will not work, which is why we instructed to install the App on the Indexers separately. (I forgot these reasons in yesterday's call.)
  • The replicated bundle will have some limits.conf [iplocation] db_path = /opt/splunk/etc/apps/splunk_maxmind_db_auto_update/local/mmdb/GeoLite2-City.mmdb
  • But the bundle's mmdb file will be under /opt/splunk/var/run/searchpeers/-1654771866/
  • Also, note that location of search peer's location is dynamic.
• Hence, we cannot use this process of replicating the configuration bundle to Indexers.

[sclapper]

If you make the following changes you can avoid using limits.conf, it will replicate to indexers and works in cloud. bin/mmdb_utils.py 22 MMDB_FILE_NAME = 'GeoIP2-City.mmdb' 25 DB_DIR_PATH = '/opt/splunk/etc/apps/search/lookups’ 117 #self.update_mmdb_location()

[VatsalJagani]

@sclapper - I don't think on indexer the location of the lookup would be same. It would be under /opt/splunk/var/run/ folder instead which would be a problem.

[sclapper]

/opt/splunk/etc/apps/search/lookups is just the path where you put GeoIP2-City.mmdb, just need to write the mmdb to ..search/lookups/GeoIP2-City.mmdb and it will replicate.

[VatsalJagani]

@sclapper - Yes that is accurate it will replicate, but the location on the indexer would not be /opt/splunk/etc/apps/search/lookups. It would something like this: /opt/splunk/var/run/searchpeers/splunkui111-1324253269/apps/search/lookups. Path would have dynamic number with it (replicated bundle number)

So how would you assign a path in the limits.conf for the indexer? Also, the path for the indexer and search head would be different so you need to write different limits.conf for search head and indexer, which is not possible in Splunk cloud.

[sclapper]

This method doesn’t require using limits.conf, Test it out.

[dave-safian-kyndryl]

When you update the database using this method (as a lookup) https://docs.splunk.com/Documentation/Splunk/9.0.1/SearchReference/Iplocation#Updating_the_IP_geolocation_database_file

It gets written to the splunk_dir/etc/apps/search/lookups dir and it gets distributed to the indexers. I believe this is a new feature in Splunkv9 but has been available in Splunk cloud for a while longer.

[VatsalJagani]

@sclapper, @kyndsafian - Thanks for your suggestion and comments. We're working on this to improve for both enterprise and cloud by using this new feature.

[VatsalJagani]

@sclapper , @kyndsafian - Thanks for your suggestion and guidance.

We have implemented the change here - https://github.com/CrossRealms/Splunk-App-Auto-Update-MaxMind-Database/pull/13 We have tested the approach it's working in SHC.

Currently, we are resolving one other issue with SHC - Other SHs opening the setup page even after App setup is completed on one SH, and app.conf is_configured parameter is also replicated to other SHs. Once that is resolved, we'll create a new release of the App.

Really appreciate your comments here.

[VatsalJagani]

App version 2.0.0 has been released with proper support of the Search Head Cluster, Distributed environment, and Splunk Cloud Classic and Victoria experience. https://splunkbase.splunk.com/app/5482

Thanks for everyone's suggestions and support. @sclapper , @kyndsafian