threatgrid / jqpipe-go

A Go Wrapper for JQ
BSD 2-Clause "Simplified" License
12 stars 8 forks source link

"unexplained jq failure" occurring non-deterministically #6

Closed estroz closed 7 years ago

estroz commented 7 years ago

Bug report

I am using jqpipe-go to evaluate jq syntax expressions on JSON data. Every time I ingest a large volume of JSON data using my program and evaluate a syntax expression (stored in a config file), jqpipe-go.Eval() will return "unexplained jq failure" errors in a non-deterministic manner. These errors are not related to what the JSON data contains, because they'll occur when running my program on static tests.

Example expression: "{createdBy: .userIdentity.arn}"

Example JSON object:

{
  "eventVersion": "1.04",
  "userIdentity": {
    "type": "IAMUser",
    "principalId": "EXAMPLEFW34KYSCWPHAXX",
    "arn": "arn:aws:iam::123456789101:user/test-user",
    "accountId": "123456789101",
    "userName": "test-user"
  },
  "eventTime": "2017-06-01T14:39:35Z",
  "eventSource": "s3.amazonaws.com",
  "eventName": "CreateBucket",
  "awsRegion": "us-west-2",
  "sourceIPAddress": "12.34.56.78",
  "userAgent": "[aws-sdk-go/1.8.25 (go1.8.1; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.9.6-dev]",
  "requestParameters": {
    "CreateBucketConfiguration": {
      "LocationConstraint": "us-west-2"
    },
    "bucketName": "aaws-pr-751",
    "x-amz-acl": ["private"]
  },
  "responseElements": null,
  "requestID": "example",
  "eventID": "example-bdf8-4cfb-9241-ae263f85c203",
  "eventType": "AwsApiCall",
  "recipientAccountId": "123456789101"
}

Do you have any insight into why I'm seeing non-deterministic errors? The full program can be found here.

estroz commented 7 years ago

Relates to #3