User Defined Data Source Display Issue

[stevelodin]

Attempting a user defined data source. I think the Wizard is pretty good. When I create it, the 2nd screen in the wizard shows a legit response and displays in the upper left this: Displayed PACKETMAIL IPREP: 8.8.8.8 UNIQUE_LOOKUPS: 110 DSHIELD SOURCES: Attacks 6, Count 80

That is what I would expect. When I run it on an address in Chrome,, the popup shows: PACKETMAIL IPREP: 8.8.8.8 UNIQUE_LOOKUPS: undefined

I tried comparing to the code for others like AlienVault, ThreatCrowd, etc. but I couldn't find a code-related issue. I tried to move it to the top of the list but couldn't figure out how to do that. At a loss as to how to fix it. Thanks for any advice.

Code is: { "lookupName": "PacketMail IPRep", "userDefined": { "api_key": { "title": "API Key", "value": "MY_KEY_DATA_HERE" } }, "lookupVariable": "PACKETMAILIPREP", "lookupType": "IPV4", "lookupUrl": "https://www.packetmail.net/iprep.php/${PINCH.HOVERITEM}?apikey=${PINCH.USERDEFINED.api_key.value}", "httpHeaders": "", "httpType": "GET", "dataType": "json", "dataSchema": { "0_PacketMail IPRep": { "title": "PacketMail IPRep", "mapping": "${PINCH.LINKURL}", "order": 0, "linkTitle": "${PINCH.HOVERITEM}", "linkUrl": "https://www.packetmail.net/iprep.php/${PINCH.HOVERITEM}?apikey=${PINCH.USERDEFINED.api_key.value}" }, "1_Unique_Lookups": { "title": "Unique_Lookups", "mapping": "${PINCH.RESPONSE.Unique_Lookups}", "order": 1 }, "2_DShield Sources": { "title": "DShield Sources", "mapping": "${PINCH.RESPONSE.dshield_sources.context}", "order": 2 } }, "disabled": false, "iocs": true, "authorizationType": "DEFAULT", "requestGroup": "INTERNET", "sample": "8.8.8.8", "order": 35 }

[cloudtracer]

Hi Steve,

I can't say for certain since I don't have a PacketMail API key, but if I had to wager a guess it might be the spaces in the "0_PacketMail IPRep" and "2_DShield Sources" attributes. I fixed them up and added shared the request below with the URL. Let me know if this was the issue. The wizard is still pretty bare bones and this could be a bug with how it puts the data schema attributes together.

chrome-extension://ljdgplocfnmnofbhpkjclbefmjoikgke/src/options/wizard.html?RL=eyJsb29rdXBOYW1lIjoiUGFja2V0TWFpbCBJUFJlcCIsInVzZXJEZWZpbmVkIjp7ImFwaV9rZXkiOnsidGl0bGUiOiJBUEkgS2V5IiwidmFsdWUiOiJZT1VSREFUQUhFUkUifX0sImxvb2t1cFZhcmlhYmxlIjoiUEFDS0VUTUFJTElQUkVQIiwibG9va3VwVHlwZSI6IklQVjQiLCJsb29rdXBVcmwiOiJodHRwczovL3d3dy5wYWNrZXRtYWlsLm5ldC9pcHJlcC5waHAvJHtQSU5DSC5IT1ZFUklURU19P2FwaWtleT0ke1BJTkNILlVTRVJERUZJTkVELmFwaV9rZXkudmFsdWV9IiwiaHR0cEhlYWRlcnMiOiIiLCJodHRwVHlwZSI6IkdFVCIsImRhdGFUeXBlIjoianNvbiIsImRhdGFTY2hlbWEiOnsiMF9QYWNrZXRNYWlsX0lQUmVwIjp7InRpdGxlIjoiUGFja2V0TWFpbCBJUFJlcCIsIm1hcHBpbmciOiIke1BJTkNILkxJTktVUkx9Iiwib3JkZXIiOjAsImxpbmtUaXRsZSI6IiR7UElOQ0guSE9WRVJJVEVNfSIsImxpbmtVcmwiOiJodHRwczovL3d3dy5wYWNrZXRtYWlsLm5ldC9pcHJlcC5waHAvJHtQSU5DSC5IT1ZFUklURU19P2FwaWtleT0ke1BJTkNILlVTRVJERUZJTkVELmFwaV9rZXkudmFsdWV9In0sIjFfVW5pcXVlX0xvb2t1cHMiOnsidGl0bGUiOiJVbmlxdWVfTG9va3VwcyIsIm1hcHBpbmciOiIke1BJTkNILlJFU1BPTlNFLlVuaXF1ZV9Mb29rdXBzfSIsIm9yZGVyIjoxfSwiMl9EU2hpZWxkX1NvdXJjZXMiOnsidGl0bGUiOiJEU2hpZWxkIFNvdXJjZXMiLCJtYXBwaW5nIjoiJHtQSU5DSC5SRVNQT05TRS5kc2hpZWxkX3NvdXJjZXMuY29udGV4dH0iLCJvcmRlciI6Mn19LCJkaXNhYmxlZCI6ZmFsc2UsImlvY3MiOnRydWUsImF1dGhvcml6YXRpb25UeXBlIjoiREVGQVVMVCIsInJlcXVlc3RHcm91cCI6IklOVEVSTkVUIiwic2FtcGxlIjoiOC44LjguOCIsIm9yZGVyIjo0NH0=

[cloudtracer]

Actually, I tried testing out if the spaces actually cause any problems and they don't.

Perhaps try a different IP address or deleting your history in the developers section. API results by default are cached for 24 hours and the PacketMail API appears to actually send a legitimate API response (to let you know you shouldn't be using it). Its possible that response is cached.

[stevelodin]

I tried your version (with my API key) and I tried slimming down the code to remove those spaces. Neither worked. Code below. I will try the history deletion idea.

{ "lookupName": "PacketMail IPRep", "userDefined": { "api_key": { "title": "API Key", "value": "MY_KEY_GOES_HERE" } }, "lookupVariable": "PACKETMAILIPREP", "lookupType": "IPV4", "lookupUrl": "https://www.packetmail.net/iprep.php/${PINCH.HOVERITEM}?apikey=${PINCH.USERDEFINED.api_key.value}", "httpHeaders": "", "httpType": "GET", "dataType": "json", "dataSchema": { "IPRep": { "title": "IPRep", "mapping": "${PINCH.LINKURL}", "order": 0, "linkTitle": "${PINCH.HOVERITEM}", "linkUrl": "https://www.packetmail.net/iprep.php/${PINCH.HOVERITEM}?apikey=${PINCH.USERDEFINED.api_key.value}" }, "Lookups": { "title": "Lookups", "mapping": "${PINCH.RESPONSE.Unique_Lookups}", "order": 1 }, "Sources": { "title": "Sources", "mapping": "${PINCH.RESPONSE.dshield_sources.context}", "order": 2 } }, "disabled": false, "iocs": true, "authorizationType": "DEFAULT", "requestGroup": "INTERNET", "sample": "8.8.8.8", "order": 35 }

[cloudtracer]

I have a strong feeling its the cache that is causing the problem at this point. This is the first time the successful API caching is causing an issue, I'll have to make it configurable in the next release.

Also, if you are trying to move the request to the top of the popover you can do it like this in the developers option section. I'll have to make this easier in the future.

[stevelodin]

Deleted history, exited browser, re-started browser, Tried new badguy ip address (123.123.123.123 known bad guy in China), still no luck.

Thanks for the assistance, I'm stumped. I may try one of the other shared contributions to see if that works for me. Here is the code where it seems to be returning "undefined".

Here is the Wizard showing the Step 2 response and the display text, which seems to be what I want. :-)

Even if I try 8.8.8.8 I get the same undefined.

Should I try to uninstall and reinstall the whole extension? Would I need to delete something in the chrome directory structure on disk?

Cheers, Steve

[cloudtracer]

I'm stumped as well, I don't think uninstalling will help. Everything looks right to me.

I get the "Lookups: undefined" as well when I try it, but that's because I don't have a valid API key. Which makes me think that this issue has something to do with the API key, or the passing of it. Is there anything non-traditional about the PacketMail API key? Perhaps some kind of special characters it uses?

Either way, one other thing you can try is to disable your other IPV4 lookups (just to remove noise) and then set the debug level in the developers options to "warn", which might give you more hints in the Chrome Developers tools (hit F12). At the very least, with the debugging set at warn it should dump the API response to the developers console somewhere in the "ThreatPinchHTTPResponseHandler" prints.

I wouldn't set the debugger any higher than warn, after that its a lot of noise that is really only valuable for debugging much more complex issues with the filtering and advanced config settings.

[cloudtracer]

Hi Steve,

The maintainer of PacketMail was kind enough to set me up with an API key to test with. It looks like the root cause of the issue is that the wizard is creating the dataType as "json" instead of the expected "JSON" for specific circumstances.

You can use the raw edit button to update the integration in the meantime, but I'll be pushing a patch for this issue later tonight.

[cloudtracer]

Fixed in 2.0.9, should be available shortly.

[stevelodin]

Thanks for the fix. Working now and I showed my team the "product" earlier this week.

Steve

On Thu, May 25, 2017 at 4:31 PM, cloudtracer notifications@github.com wrote:

Fixed in 2.0.9, should be available shortly.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/cloudtracer/ThreatPinchLookup/issues/41#issuecomment-304115853, or mute the thread https://github.com/notifications/unsubscribe-auth/AA7TKX5fT-FROcm1Eg9MjPgavJhCjL06ks5r9eUYgaJpZM4Nl4Kk .

-- Cell: +1-317-840-9088 LinkedIn: http://www.linkedin.com/in/stevelodin Twitter: http://twitter.com/stevelodin