search

splunk / utbox

URL Toolbox (UTBox) is a set of building blocks for Splunk specially created for URL manipulation. UTBox has been created to be modular, easy to use and easy to deploy in any Splunk environments.
https://preview.splunkbase.splunk.com/app/2734/
Apache License 2.0
8 stars 6 forks source link

Add configuration mechanism for custom list that works via Splunk Lookup #5

Open dfederschmidt opened 1 year ago

dfederschmidt commented 1 year ago

Currently, supplying a custom list requires editing bin/suffix_list_custom.dat. As an administrator, I'd like to supply my custom list via a Splunk Lookup that does not require me to create a fork of the app.

pirxthepilot commented 1 year ago

Hi @dfederschmidt , I might get some free cycles to work on a PR, but I am not sure where do start. Do you have some docs you can point me out to? Thanks!

dfederschmidt commented 1 year ago

Hi @pirxthepilot - Thanks for considering to contribute!

In general, Splunk lookup table files can simply be CSV files in the lookups folder of an App. What we could do is we add a new type of TLD list. Currently, we support the values mozilla, iana and custom.

Specifically, we could add a new mechanism inside this function: https://github.com/splunk/utbox/blob/c156de526bb8e86aeccd9a44a89e09f7fb937ff3/utbox/bin/ut_parse_lib.py#L102-L110

What I'm thinking is that we could add a pattern such as lookup:my_custom_list.csv which would load a list located inside lookups/my_custom_list.csv. Probably we would need another function eg. def _loadLookupList() to load the file, similar to the other methods in https://github.com/splunk/utbox/blob/main/utbox/bin/ut_parse_lib.py

A limitation of this approach would be is that the lookup would have to be located inside of the app directory, so it's not possible to pass a lookup from another app context.

On how to develop on this app specifically, there is not really a whole lot of documentation right now - there is a brief mention on https://github.com/splunk/utbox#development-setup to get started. https://dev.splunk.com/ has the whole wealth of knowledge around how Splunk Apps work, but I think in this case, we don't need to dive too deep into that.