json.loads on datafield invalidates JSON object

[bowesmana]

The handling of datafield in POST requests does not work. Using latest 2.02 version of WebTools, code does this

` if 'datafield' in options:

try:
    data = json.loads(result[options['datafield']])
except:
    data = str(result[options['datafield']])

` however, this request, using the ITSI REST API fails.

| makeresults count=1 | eval header="{\"Authorization\":\"Bearer <token>\",\"content-type\":\"application/json\"}" | eval data="{\"title\": \"Maintenance window test-006\", \"start_time\": 1626259120, \"end_time\": 1626260000, \"objects\": [{\"object_type\": \"entity\", \"_key\": \"145b4987-4d17-412a-8da4-a443976e41b7\"}]}" | curl method=post uri="https://localhost:8089/servicesNS/nobody/SA-ITOA/maintenance_services_interface/maintenance_calendar" debug=true headerfield=header datafield=data | table curl* as the json.loads converts the original JSON datafield contents to

{'title': 'Maintenance window test-006', 'start_time': 1626259120, 'end_time': 1626260000, 'objects': [{'object_type': 'entity', '_key': '145b4987-4d17-412a-8da4-a443976e41b7'}]}

instead of

{"title": "Maintenance window test-006", "start_time": 1626259120, "end_time": 1626260000, "objects": [{"object_type": "entity", "_key": "145b4987-4d17-412a-8da4-a443976e41b7"}]}

which would occur if using the code in the except handler.

It's related to this community conversation

https://community.splunk.com/t5/All-Apps-and-Add-ons/Problem-with-TA-Webtools-curl-POST-method-with-JSON-datafield/m-p/504521#M62129

I am not sure what the fix is, because if I use the example given in the WebTools documentation of

| makeresults count=1 | eval message="{\"name\":\"restart_link\",\"value\":\"Hello World\",\"severity\":\"warn\"}" | curl method=post uri=https://localhost:8089/services/messages/new datafield=message splunkauth=true | table curl*

this fails if the code using the str() variant, but not when using the loads() variant.

However, is does seem wrong that loads() is effectively converting double quotes to single quotes, therefore giving invalid JSON.

Here's a bit of a hack solution that introduces a new option to force it to interpret the data as str(), i.e. it retains the existing default - line 273

`

if datafield in options, set datafield = options['datafield']

                if 'datafield' in options:

                    stringData = False
                    try:
                        if 'stringdata' in options:
                            if options['stringdata'].lower() in ("yes", "true", "t", "1"):
                                stringData = True
                        if stringData:
                            data = str(result[options['datafield']])
                        else:
                            data = json.loads(result[options['datafield']])
                    except:
                        data = str(result[options['datafield']])
                        #result['curl_data_str'] = data
                else:
                    data = None

`

[Serinu]

@bentleymi : The issue continues to exist in v3.0.2. Is there any chance this will be fixed in the future? A workaround, that we used successfully in our current use case instead of bowesmana's hack:

# -------[ FIND ]---------------------
def post(uri,sessionKey,cert,headers=None,payload=None,user=None,password=None,timeout=60):
# -------[ ADD, AFTER ]---------------
    if(payload is not None):
        payload = json.dumps(payload)

[bentleymi]

I like @bowesmana fix the most so far. Reason being, some APIs prefer strings, others prefer json, etc. Thank you both for your contributions!

[Serinu]

@bentleymi : I see. Then maybe you can add the switch to the next release, so that the way of payload-translation becomes controllable from the search as a command parameter.

In the past, I had some other issues with the json.loads() as well ... but I cannot remember exactly, what it was. I believe it had to do something with null pointers or so. Means, the difference is not just about single vs. double quotes, but also certain value types, when translated from a python dictionary into something 'visible'.

[mghocke]

It's actually a bit worse. I am running Splunk 9.0.4.1 and TA-webtools 3.0.2. This code snippet

| makeresults
| eval header=json_object("Content-Type","application/json")
| eval body=json_object("key", "value")
| curl method=post datafield=body uri=https://echo.zuplo.io debug=true headerfield=header

results in this:

{
  "url": "https://echo.zuplo.io/",
  "method": "POST",
  "query": {},
  "body": "key=value",
  "headers": {
    "accept": "*/*",
    "accept-encoding": "gzip",
    "connection": "Keep-Alive",
    "content-length": "9",
    "content-type": "application/json",
    "host": "echo.zuplo.io",
    "true-client-ip": "70.175.138.95",
    "user-agent": "python-requests/2.25.0",
    "x-forwarded-proto": "https",
    "x-real-ip": "70.175.138.95"
  }
}

The message body was converted into form data even though the data is JSON and I even explicitly set the content type. The only way to make this work at all was to go into curl.py and remove the json.loads branch altogether.

We did not have the same problem with 2.0.2 of this add-on running on Splunk 8.2.6. We used data= instead of datafield= but that stopped working after we upgraded to 9.0.3:

| makeresults
| eval header=json_object("Content-Type","application/json")
| eval body=json_object("key", "value")
| curl method=post data=body uri=https://echo.zuplo.io debug=true headerfield=header

results in

{
  "url": "https://echo.zuplo.io/",
  "method": "POST",
  "query": {},
  "body": "",
  "headers": {
    "accept": "*/*",
    "accept-encoding": "gzip",
    "connection": "Keep-Alive",
    "content-length": "0",
    "content-type": "application/json",
    "host": "echo.zuplo.io",
    "true-client-ip": "70.175.138.95",
    "user-agent": "python-requests/2.25.0",
    "x-forwarded-proto": "https",
    "x-real-ip": "70.175.138.95"
  }
}

Notice that absolutely no message body made it into the message. For now we decided to run the hacked code in production and stick with datafield instead.

[kyung123a]

I solved it by adding the value "json=json_payload".

def post(uri,sessionKey,verifyssl,cert,headers=None,payload=None,user=None,password=None,timeout=60):
    try:
        data=None
        json_payload=None
        if payload is not None:
            if isinstance(payload, dict):
                json_payload = payload
            else:
                data = payload
        if sessionKey == None:
            if user == None and password == None:
                r = requests.post(uri,data=data,json=json_payload,verify=verifyssl,cert=cert,headers=headers,timeout=timeout)
            else:
                r = requests.post(uri,auth=(user,password),data=data,json=json_payload,verify=verifyssl,cert=cert,headers=headers,timeout=timeout)
        else:
            headers = {}
            headers["Authorization"] =  'Splunk %s' % sessionKey
            r = requests.post(uri,data=data,json=json_payload,verify=verifyssl,cert=cert,headers=headers,timeout=timeout)
        return(getResponse(r,uri))
    except requests.exceptions.RequestException as e:
        return(getException(e,uri))