search

splunk / splunk-app-flurry

Splunk Flurry App
Apache License 2.0
5 stars 5 forks source link

HTTP Error 401: Authorization Required -- Update authorization & bot-detection-evasion logic #5

Open dmduck opened 9 years ago

dmduck commented 9 years ago

Hello,

I am getting HTTP Error 401: Authorization Required every time I try to use the flurry data application. I have verified numerous times that my credentials are correct. Below is a snippet from the splunkd.log file. Has anyone else run into this, and is there a solution?

Thanks!

04-13-2015 10:57:39.082 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" Traceback (most recent call last): 04-13-2015 10:57:39.082 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/extract.py", line 236, in 04-13-2015 10:57:39.082 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" conn.login() 04-13-2015 10:57:39.082 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/extract.py", line 65, in login 04-13-2015 10:57:39.082 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" self.browser.open('https://dev.flurry.com/secure/login.do') 04-13-2015 10:57:39.083 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/mechanize/_mechanize.py", line 203, in open 04-13-2015 10:57:39.083 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" return self._mech_open(url, data, timeout=timeout) 04-13-2015 10:57:39.083 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/mechanize/_mechanize.py", line 255, in _mech_open 04-13-2015 10:57:39.083 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" raise response 04-13-2015 10:57:39.083 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" mechanize._response.httperror_seek_wrapper: HTTP Error 401: Authorization Required 04-13-2015 10:57:49.191 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" Traceback (most recent call last): 04-13-2015 10:57:49.191 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/extract.py", line 236, in 04-13-2015 10:57:49.191 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" conn.login() 04-13-2015 10:57:49.192 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/extract.py", line 65, in login 04-13-2015 10:57:49.192 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" self.browser.open('https://dev.flurry.com/secure/login.do') 04-13-2015 10:57:49.192 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/mechanize/_mechanize.py", line 203, in open 04-13-2015 10:57:49.192 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" return self._mech_open(url, data, timeout=timeout) 04-13-2015 10:57:49.192 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/mechanize/_mechanize.py", line 255, in _mech_open 04-13-2015 10:57:49.192 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" raise response 04-13-2015 10:57:49.192 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" mechanize._response.httperror_seek_wrapper: HTTP Error 401: Authorization Required

davidfstr commented 9 years ago

I am seeing a different error myself in splunkd.log:

04-15-2015 11:21:39.296 -0700 INFO  ExecProcessor - New scheduled exec process: python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py
04-15-2015 11:21:39.296 -0700 INFO  ExecProcessor -     interval: 600000 ms
04-15-2015 11:21:39.296 -0700 INFO  ExecProcessor - New scheduled exec process: /Applications/splunk_621_flurry/bin/splunkd instrument-resource-usage
04-15-2015 11:21:39.296 -0700 INFO  ExecProcessor -     interval: 0 ms
04-15-2015 11:25:55.092 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py" Traceback (most recent call last):
04-15-2015 11:25:55.093 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"   File "/Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py", line 241, in <module>
04-15-2015 11:25:55.093 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"     flurry_csv_stream = conn.download_log(year, month, day, offset)
04-15-2015 11:25:55.093 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"   File "/Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py", line 102, in download_log
04-15-2015 11:25:55.093 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"     resp = self.browser.open(url)
04-15-2015 11:25:55.093 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"   File "/Applications/splunk_621_flurry/etc/apps/flurry/bin/mechanize/_mechanize.py", line 203, in open
04-15-2015 11:25:55.094 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"     return self._mech_open(url, data, timeout=timeout)
04-15-2015 11:25:55.094 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"   File "/Applications/splunk_621_flurry/etc/apps/flurry/bin/mechanize/_mechanize.py", line 255, in _mech_open
04-15-2015 11:25:55.094 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"     raise response
04-15-2015 11:25:55.094 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py" mechanize._response.httperror_seek_wrapper: HTTP Error 404: Not Found

Regardless, both my error trace and your error trace indicate that either Flurry has changed their bot detection system or rate limiting system. The actual (private) API being accessed appears to still work, namely: https://dev.flurry.com/eventsLogCsv.do

I no longer have enough cycles to continue updating this app to adjust to Flurry's API changes. So I will mark this item as Community Support.

hmasanari commented 9 years ago

Hi, How about setting "Next Session ID [Advanced]" as well as your credentials? I got the same problem for many times without entering it, but the app seems to start again after I entered and saved bigger Session ID. I found the biggest Session ID on the time as below; index="flurry" | stats max(Session)

And I entered a number next to the search result. Thanks.