search

target / huntlib

A Python library to help with some common threat hunting data analysis operations
MIT License
138 stars 22 forks source link

SplunkDF.search_df() returns iterative results as individual frames #4

Closed gregbuehler closed 4 years ago

gregbuehler commented 5 years ago

Describe the bug SplunkDF.search_df() returns iterative results as frames.

To Reproduce

s = SplunkDF(
              host=splunk_server,
              username="myuser",
              password="mypass"
)

query = """
search index=foo | timechart span=1d sum(bar)
"""

df = s.search_df(spl=query, days=2, limit=0)
print(df)
                           _time  _span _spandays         sum(bar)
0    2019-01-30 00:00:00.000 PST  86400         1            NaN
1    2019-01-31 00:00:00.000 PST  86400         1     3406055529
2    2019-02-01 00:00:00.000 PST  86400         1    46301211318
3    2019-01-30 00:00:00.000 PST  86400         1            NaN
4    2019-01-31 00:00:00.000 PST  86400         1    20410198989
5    2019-02-01 00:00:00.000 PST  86400         1   204170618369
...
117  2019-01-30 00:00:00.000 PST  86400         1   702547096112
118  2019-01-31 00:00:00.000 PST  86400         1  2068766398830
119  2019-02-01 00:00:00.000 PST  86400         1  1314273591733
120  2019-01-30 00:00:00.000 PST  86400         1   708375398493
121  2019-01-31 00:00:00.000 PST  86400         1  2084630792268
122  2019-02-01 00:00:00.000 PST  86400         1  1314273591733

Expected behavior A finalized response is produced as the DataFrame contents instead of a frame per streaming result.

DavidJBianco commented 4 years ago

Now that I have unit tests for the SplunkDF class, I wanted to try to reproduce this issue with an eye towards fixing it. However, I'm unable to make it misbehave. Coming as close as I can to your original query (without having the dataset upon which it was based), the results always come back as a single DataFrame.

I know it's been a long time, but are you still experiencing this issue?

DavidJBianco commented 4 years ago

I'm going to close this issue since I wasn't able to reproduce it. Also, starting with v0.4.0, the SplunkDF code has been extensively re-written. If this continues to be an problem, please re-open this issue and I'll take another look.