TrickyHarry commented 6 years ago

I am using jq v1.5 in a Linux environment making a system call within AWK to jq. I have a rather complex AWK script that has invoked a curl script that has a large JSON block returned to it. I then try to parse the JSON via jq and I get the error.

The AWK script is making a system call to bash to invoke jq and I am reading the response. I tried almost every combination of quotating to eliminating the issue. I have used this successfully on a simpler JSON response and I was happy with its response, just this complex block is giving me grief. I wrote the JSON to a file, within AWK and was then able to invoke jq from the command line to parse the data - giving me the correct results. Once I had written out the string into the file I was able to invoke jq from the command line via cat stored_file | jq -c '.items[]' jq -c '.items[]' stored_file

I got the same error, trying to parse the individual JSON blocks

run_curl_cmd = sprintf("curl %s -X GET %s/%s/oauth/clients\n", _curl_prefix, _curl_url, pf_api_root);

Run the main REST_API Call and collect the data ....

while ( (run_curl_cmd | getline _resp) > 0) { loop_1: resp_data = _resp loop_2: response_success_code = _resp rIdx++ } close(run_curl_cmd)

Parse the JSON response

Hoping to use:

run_jq_cmd = sprintf("echo %s | jq -c '.items[]' ", resp_data); while( run_jq_cmd | getline _element ) > 0 ){ custom_process( _element ) } close(run_jq_cmd) I got the parse error at the external call.

I wrote out the string to a file and then I would be able to "cat" the file into jq - same parsing error I tried to replace the ' with " and I then tried to escape the single d-quote: \" run_jq_cmd = sprintf("echo %s | jq -c '.items[]' ", resp_data); run_jq_cmd = sprintf("echo %s | jq -c \".items[]\" ", resp_data); run_jq_cmd = sprintf("cat %s | jq -c '.items[]' ", resp_data_file); run_jq_cmd = sprintf("cat %s | jq -c \".items[]\" ", resp_data_file); run_jq_cmd = sprintf("cat %s | jq -c '.items[]' ", resp_data_file); run_jq_cmd = sprintf("jq -c \".items[]\" ./%s", resp_data_file); run_jq_cmd = sprintf("jq -c \".items[]\" ./%s", resp_data_file);

The following worked: printf("jq '.items[]' $1") > "local_jq.sh" and the following also worked printf("jq \".items[]\" $1") > "local_jq.sh" system("chmod +x local_jq.sh") run_jq_cmd = sprintf("./local_jq.sh %s", resp_data_file);

Extract from curl message response: Response[01]: [function] = [pf-admin-api/v1/oauth/clients] Response[02]: [Resp_Data] = [{"items":[{"clientId":"pa_rs","redirectUris":[],"grantTypes":["ACCESS_TOKEN_VALIDATION"],"name":"PingAccess Resource Server","description":"","logoUrl":"","refreshRolling":"SERVER_DEFAULT","persistentGrantExpirationType":"SERVER_DEFAULT","persistentGrantExpirationTime":0,"persistentGrantExpirationTimeUnit":"DAYS","bypassApprovalPage":false,"restrictScopes":false,"restrictedScopes":[],"validateUsingAllEligibleAtms":false,"oidcPolicy":{"grantAccessSessionRevocationApi":false,"pingAccessLogoutCapable":false},"clientAuth":{"type":"SECRET","encryptedSecret":"OBF:JWE:eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2Iiwia2lkIjoieXBRZnV4MUtyciIsInZlcnNpb24iOiI4LjQuNC4wIn0..L62K4Ax3_KRIAR6f6G361A.pgLXYHRjHvO0OpIgcIOIJnvHyPlEnpqdp1LkorVh7Rc5-hEhooyS4Nyg23hTztdPmCCga3yyIfhYTpzqUv3_3GkN7J15N-tEzlIVnRV4VMI.S3U7VmPh2uj6kQhwhQly6g", .... ] Response[03]: [Resp_Code]=[200]

I had the same error processing an element of the array. Here is the entire JSON: Entry Extracted[01]: [{"clientId":"pa_rs","redirectUris":[],"grantTypes":["ACCESS_TOKEN_VALIDATION"],"name":"PingAccess Resource Server","description":"","logoUrl":"","refreshRolling":"SERVER_DEFAULT","persistentGrantExpirationType":"SERVER_DEFAULT","persistentGrantExpirationTime":0,"persistentGrantExpirationTimeUnit":"DAYS","bypassApprovalPage":false,"restrictScopes":false,"restrictedScopes":[],"validateUsingAllEligibleAtms":false,"oidcPolicy":{"grantAccessSessionRevocationApi":false,"pingAccessLogoutCapable":false},"clientAuth":{"type":"SECRET","encryptedSecret":"OBF:JWE:eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2Iiwia2lkIjoieXBRZnV4MUtyciIsInZlcnNpb24iOiI4LjQuNC4wIn0..L62K4Ax3_KRIAR6f6G361A.pgLXYHRjHvO0OpIgcIOIJnvHyPlEnpqdp1LkorVh7Rc5-hEhooyS4Nyg23hTztdPmCCga3yyIfhYTpzqUv3_3GkN7J15N-tEzlIVnRV4VMI.S3U7VmPh2uj6kQhwhQly6g","enforceReplayPrevention":false},"requireSignedRequests":false}] Here I wanted to extract the clientId: jq '.clientId' json_file

I tried the above methods but same error parse error: Invalid numeric literal at line 1, column 9
I encapsulated the JSON string by d-quotes, the error column just incremented by 1

TrickyHarry commented 6 years ago

I extracted the core of my larger script into a separate file to test it separately. testme.txt invoke the file above file by: awk -f testme.txt

awk -f testme.txt Debug statement 1:[curl -w " %{http_code} " -sS -k --user Administrator:AnyPassw0rd -H "Content-type:application/json" -H "Accept:application/json" -H "x-xsrf-header:PingFederate" -X GET https://ping1demo:9999/pf-admin-api/v1/oauth/clients].

Response[02]: [REST_Resp_2] = [{"items":[{"clientId":"pa_rs","redirectUris":[],"grantTypes":["ACCESS_TOKEN_VALIDATION"],"name":"PingAccess Resource Server","description":"","logoUrl":"","refreshRolling":"SERVER_DEFAULT","persistentGrantExpirationType":"SERVER_DEFAULT","persistentGrantExpirationTime":0,"persistentGrantExpirationTimeUnit":"DAYS","bypassApprovalPage":false,"restrictScopes":false,"restrictedScopes":[],"validateUsingAllEligibleAtms":false,"oidcPolicy":{"grantAccessSessionRevocationApi":false,"pingAccessLogoutCapable":false},"clientAuth":{"type":"SECRET","encryptedSecret":"OBF:JWE:eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2Iiwia2lkIjoieXBRZnV4MUtyciIsInZlcnNpb24iOiI4LjQuNC4wIn0..XyCvaBqd-DHn94-LjIdYQA.kdC1J62Uw7nnvFiB8QosLI7pgntIt74o8hYgbzqCWULPxDBpDk7nHhsX2CN1fwVjTiFIGX-_UrSdWIupO0NoSrNMSiFzsJL6jCOrrZPVofY.AVlHxd38ERLy2oSzEFw-BA","enforceReplayPrevention":false},"requireSignedRequests":false},{"clientId":"api_implicit","redirectUris":["https://localhost:3000/PingAccessQuickStart/api-demo","https://ping1demo:3000/PingAccessQuickStart/api-demo"],"grantTypes":["IMPLICIT"],"name":"Implicit API Client","description":"","logoUrl":"","refreshRolling":"SERVER_DEFAULT","persistentGrantExpirationType":"SERVER_DEFAULT" ......."enforceReplayPrevention":false},"requireSignedRequests":false}]}]

Response[03]: [REST_Resp_3] = [200]

parse error: Invalid numeric literal at line 1, column 6

pkoppstein commented 6 years ago

Setting jsonQ to "jq -c '.items[]'" is fine. Using echo to pipe JSON into another command, however, is tricky. It would probably be better to use printf but it can be done:

BEGIN {
 jsonQ = "jq -c '.items[]'";
 _response_="{\"items\": [1,2,3]}";
 print _response_;
 parse_cmd = sprintf("echo \'%s\' | %s", _response_, jsonQ)
 print parse_cmd;
 parse_cmd | getline it
 print it
}

TrickyHarry commented 6 years ago


Thanks very much - you provided the answer to the problem and it was the handling of the
'echo of the string itself ```
- **Escaping the single-quote: [\']**
    parse_cmd = sprintf("echo \'%s\' | %s", _response_, jsonQ)
----
I knew that the fix would have been simple, but I just could not work it out ... I just never thought about escaping the single-quote !!

I now have the sequence of the double parsing working ..
a. Extract 1 Object of the array
b. Extract 1 attribute from the Object - I did need the object itself as I need to update it also.
===============
Trimmed Extracted[15]: [{**"clientId":"pa_wam",**"redirectUris":["https://ping1demo:3000/pa/oidc/cb","https://ping1demo.demo1.com:3000/pa/oidc/cb"],"grantTypes":["AUTHORIZATION_CODE"],"name":"PingAccess WAM","description":"","logoUrl":"","refreshRolling":"SERVER_DEFAULT","persistentGrantExpirationType":"SERVER_DEFAULT","persistentGrantExpirationTime":0,"persistentGrantExpirationTimeUnit":"DAYS","bypassApprovalPage":true,"restrictScopes":false,"restrictedScopes":[],"defaultAccessTokenManagerRef":{"id":"paToken","location":"https://ping1demo:9999/pf-admin-api/v1/oauth/accessTokenManagers/paToken"},"validateUsingAllEligibleAtms":false,"oidcPolicy":{"idTokenSigningAlgorithm":"ES256","policyGroup":{"id":"PA-WAM","location":"https://ping1demo:9999/pf-admin-api/v1/oauth/openIdConnect/policies/PA-WAM"},"grantAccessSessionRevocationApi":false,"pingAccessLogoutCapable":true},"clientAuth":{"type":"SECRET","encryptedSecret":"OBF:JWE:eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2Iiwia2lkIjoieXBRZnV4MUtyciIsInZlcnNpb24iOiI4LjQuNC4wIn0..tUDRYOLEEeJaeNa65aGa3A.j7HV2BhD3KPaQtstXYJwkQZ26TcwnGuVrLOwAXKmTJKr2JfsZj9TmP_1Zkv_2OGRWQlThG7Ov1YRgLnbzuGMZ-SU6WDhtO4pklEdwq51M9c.ncFeD5GYmJuxtTByg3V8Rg","enforceReplayPrevention":false},"requireSignedRequests":false}]
------
Within clientID while-loop, read **clientId:["pa_wam"]**
Updated OAuth ClientId Entry: **[{"clientId":"pa_wam_t3"**,"redirectUris":["https://ping1demo:3000/pa/oidc/cb","https://ping1demo.demo1.com:3000/pa/oidc/cb"],"

I have another question but I will raise this as a separate question.

pkoppstein commented 6 years ago

I have another question but I will raise this as a separate question.

Please ask usage questions at stackoverflow.com using the jq tag: https://stackoverflow.com/questions/tagged/jq

That way, you are more likely to get better and/or faster responses, and others will be more likely to benefit as well.

Please also follow the mcve guidelines at http://stackoverflow.com/help/mcve

jqlang / jq

parse error: Invalid numeric literal at line 1, column 9 #1674

I got the same error, trying to parse the individual JSON blocks

Run the main REST_API Call and collect the data ....

Parse the JSON response

Hoping to use:

I extracted the core of my larger script into a separate file to test it separately. testme.txt invoke the file above file by: awk -f testme.txt