Add upload data styles to Splunk uploader

[Tatsuya-hasegawa]

Hi

I have added the post styles to Splunk by Splunk Uploader. It's very useful for msticpy Splunk uploader's users to post data by json or text style.

ASIS, all the data frame are posted with the csv style by using to_csv() method. It occurs event fragments by depending on the data to upload.

So, I developed a switch by using "sourcetype" argument. The main change is below.

__post_data
        for row in data.iterrows():
            if sourcetype=="csv":
                data = row[1].to_csv()  # type: ignore
            elif sourcetype=="json":
                data = row[1].to_json()  # type: ignore
            else:
                data = row[1].to_string()  # type: ignore

In addition, I renamed table_name to sourcetype at many locations. Because, table_name is not usual word in Splunk. On the other hand, pylint was failed in upload_file and upload_folder methods due to W0237 so I remain table_name in only these methods.

pylint...................................................................Failed msticpy/data/uploaders/splunk_uploader.py:166:4: W0237: Parameter 'table_name' has been renamed to 'sourcetype' in overriding 'SplunkUploader.upload_file' method (arguments-renamed) msticpy/data/uploaders/splunk_uploader.py:218:4: W0237: Parameter 'table_name' has been renamed to 'sourcetype' in overriding 'SplunkUploader.upload_folder' method (arguments-renamed)

Finally, my code commit has been passed by the commit test.

(base) hacket@hackeTlab msticpy % git commit -m "add_post_data_styles_to_splunk_uploader"
Check Yaml...........................................(no files to check)Skipped
Check JSON...........................................(no files to check)Skipped
Trim Trailing Whitespace.................................................Passed
black....................................................................Passed
pylint...................................................................Passed
flake8...................................................................Passed
isort (python)...........................................................Passed
pydocstyle...............................................................Passed
check_reqs_all...........................................................Passed
[mod_splunk_uploader_post 10646520] add_post_data_styles_to_splunk_uploader
 2 files changed, 41 insertions(+), 18 deletions(-)

Best regards,

[Tatsuya-hasegawa]

My Splunk upload test is like this.

csv example for the test data is bad!

json example for the test data is great!

text example for the test data is not bad!

Thanks in advance.

[ianhelle]

Looks like you need to fix a couple of tests as well because of the parameter change. You can add tests that use both table_name and source_type to check that they both work.

[Tatsuya-hasegawa]

Hi @ianhelle

Thanks for your check. I agree with your comments.

I wanted to rename the parameter of table_name to source_type(sourcetype). Unfortunately, pylint W0237 prevented it.

pylint...................................................................Failed msticpy/data/uploaders/splunk_uploader.py:166:4: W0237: Parameter 'table_name' has been renamed to 'sourcetype' in overriding 'SplunkUploader.upload_file' method (arguments-renamed) msticpy/data/uploaders/splunk_uploader.py:218:4: W0237: Parameter 'table_name' has been renamed to 'sourcetype' in overriding 'SplunkUploader.upload_folder' method (arguments-renamed)

I don't know the way of bypass this check. Would you teach me? In addition, regarding to your last comment of

You can add tests that use both table_name and source_type to check that they both work.

How do I write that test? Don't you mind if you give me a sample?

Sincerely,

[Tatsuya-hasegawa]

Dear @ianhelle

Thanks for looking at my PR. I have fixed codes to be along your comment, in addition, added the tests.

Could you check again ?

Best regards,

[Tatsuya-hasegawa]

Dear @ianhelle

Thank you for teaching me many times. I have modified along with your advice. I much appreciate your advice.

So, I think this will finally passed for CI workflows. My local code review is all green and my Splunk instance upload tests were also fine!

(base) hacket@hackeTlab msticpy % git commit -m "fix the new argument position and slightly modified index_name param default value"
Check Yaml...........................................(no files to check)Skipped
Check JSON...........................................(no files to check)Skipped
Trim Trailing Whitespace.................................................Passed
black....................................................................Passed
pylint...................................................................Passed
flake8...................................................................Passed
isort (python)...........................................................Passed
pydocstyle...............................................................Passed
check_reqs_all...........................................................Passed
[mod_splunk_uploader_post c40ed2e7] fix the new argument position and slightly modified index_name param default value
 1 file changed, 12 insertions(+), 6 deletions(-)

Best regards,