bazaarvoice / jolt

JSON to JSON transformation library written in Java.
Apache License 2.0
1.54k stars 329 forks source link

Concatenation Issue #285

Closed abhishek1510 closed 7 years ago

abhishek1510 commented 7 years ago

I'm trying to transform a JSON which needs 1) Concatenation and appending strings 2) Filter only first field out of array of fields 3) Sum the quantities : (Manipulation is not allowed as per documentation)

Input:

{
    "data": [{
        "IDF": "123",
        "FirstName": "George",
        "LastName": "Kosher",
        "PaymentInfo": [{
            "Type": "ABC",
            "Text": "Soft",
            "Amount": 3
        }, {
            "Type": "ABC",
            "Text": "Text",
            "Amount": 5
        }],
        "PaymentCard": [{
            "CardNumber": "12345",
            "CardType": "Credit"
        }, {
            "CardNumber": "56789",
            "CardType": "Credit"
        }]
    }, {
        "IDF": "456",
        "FirstName": "Mill",
        "LastName": "King",
        "PaymentInfo": [{
            "Type": "ABC",
            "InstructionText": "Hard",
            "Amount": 6
        }, {
            "Type": "ABC",
            "InstructionText": "Text",
            "Amount": 8
        }],
        "PaymentCard": [{
            "CardNumber": "12345",
            "CardType": "Credit"
        }, {
            "CardNumber": "56789",
            "CardType": "Credit"
        }]
    }]
}

Desired Output:

{
    "data": [{
        "IDF": "123",
        "Name": "George Kosher", //concatenation of FirstName+LastName
        "PaymentText": "Soft,Text", //Appending PaymentInfo.InstructionText
        "CardType": "Credit",           //Just read first any cardtype value.
        "Amount": 8                       //Sum of Amount
    }, {

        "IDF": "456",
        "Name": "Mill King",
        "PaymentText": "Hard,Text",
        "CardType": "Credit",
        "Amount": 14
    }]
}

Spec created:

[{
        "operation": "shift",
        "spec": {
            "data": {

                "*": {
                    "IDF": "data[&1].OrderNumber",
                    "Name": "=concat(@(1,FirstName),' ',@(1,LastName))",
                    "PaymentText": "data[&1].PaymentInfo[&1].Text",
                    "CardType": {
                        "0": "data[&2].PaymentCard[&1].UOM"
                    }
                }
            }
        }
    }

]

I'm a newbie to JOLT and trying my hands first time. So pardon if any grave mistakes in spec. Just pointing me to rectify will be of great help

milosimpson commented 7 years ago

Spec : gets close, but you will need to do some of the work.

[
  {
    "operation": "shift",
    "spec": {
      "data": {
        "*": { // data arrayf
          "*": "data[&1].&", // pass thru stuff
          "PaymentInfo": {
            "*": {
              "Amount": "data[&3].Amount[]",
              "Text": "data[&3].PaymentText[]",
              "InstructionText": "data[&3].PaymentText[]"
            }
          },
          "PaymentCard": {
            "0": {
              "CardType": "data[&3].CardType"
            }
          }
        }
      }
    }
  },
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "data": {
        "*": { // data array
          "Name": "=concat(@(1,FirstName),' ',@(1,LastName))",
          "Amount": "=sum" // should work on Jolt 0.0.24
        }
      }
    }
  },
  {
    "operation": "remove",
    "spec": {
      "data": {
        "*": { // data array
          "FirstName": "",
          "LastName": ""
        }
      }
    }
  }
]

Generates

{
  "data" : [ {
    "IDF" : "123",
    "Amount" : [ 3, 5 ],
    "PaymentText" : [ "Soft", "Text" ],
    "CardType" : "Credit",
    "Name" : "George Kosher"
  }, {
    "IDF" : "456",
    "Amount" : [ 6, 8 ],
    "PaymentText" : [ "Hard", "Text" ],
    "CardType" : "Credit",
    "Name" : "Mill King"
  } ]
}

Note the demo site is using Jolt 0.0.22, the Amount sum should work on Jolt 0.0.24.
Don't have a solution for the "PaymentText" : [ "Hard", "Text" ] concatenation.

abhishek1510 commented 7 years ago

Thanks a lot. This works!! Remaining manipulation can be done from java side.

Thank you

sinoysiby commented 7 years ago

@abhishek1510 @milosimpson I have a JSON { "test":{ "ssn" : "xyz" } }

I need to transform it as

{ "test":{ "ssn" : "00000xyz" } }

can anyone help in this

milosimpson commented 7 years ago

Spec

[
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "test": {
        "ssn": "=concat('0000',@(1,ssn))"
      }
    }
  }
]
sinoysiby commented 7 years ago

I got it using some trial and error method. Thank you anyways for the help and concern.

yarabsyp commented 6 years ago

Hello is there a way to padding value with 0(zero at max 4 padding) . input :+1: "size" : "1" output size:"0001"

size value could be two or three or four digits "size" : "10" output size:"0010"

can anyone help us to find a spec for this

sreenivas123456 commented 6 years ago

@abhishek1510 @milosimpson I have a JSON

{ 
  "type":"put",
  "externalserviceid":"12345"
 }

I need to transform it as

{  
   "putserviceid":"12345"
}

can anyone help in this

milosimpson commented 6 years ago

This may break / need to be tweaked if used in a larger context, but given your input, this makes the otuput.

Spec

[
  {
    "operation": "shift",
    "spec": {
      // get put and serviceid into a nested structure
      "externalserviceid": "@(1,type).serviceid"
    }
  },
  {
    "operation": "shift",
    "spec": {
      // match put
      "*": {
        // match serviceid 
        //  &1 references "put"   
        //  &0 references "serviceid"
        "*": "&1&0"
      }
    }
  }
]
naveenvasu commented 6 years ago

How do I concat string at different levels of input and desired output ?

input json:- { "page": { "id": "abcdefghi" } }

desired output json:- "page": { "header": { "headerLayout": { "attributes": { "url": "/Page_Layout/abcdefghi" } } } }

spec:- { "operation": "modify-overwrite-beta", "spec": { "page": { "header": { "headerLayout": { "attributes": { "url": "=concat('/Page_Layout/', @(1, id))" } } } } } }

@milosimpson Could you please help me here ? Thanks.

mpanwala-wyn commented 10 months ago

This is the input I have:

[{ "sku": "00074690055268", "uri": "https://db7f7bac-bc42-4e29-aca1-4e64f39ca51b.jpeg", "isDefault": false, "Main_Image_Rank": "2" },{ "sku": "00074690055268", "uri": "https://081381ad-89ce-40ba-b4c6-892093fc376a.jpeg", "isDefault": false, "Main_Image_Rank": "12" } ]

Now what I need to do is if "Main_Image_Rank" is having one digit in a value then append 0 or keep as it is. so in this case output should be:

[{ "sku": "00074690055268", "uri": "https://db7f7bac-bc42-4e29-aca1-4e64f39ca51b.jpeg", "isDefault": false, "Main_Image_Rank": "02" },{ "sku": "00074690055268", "uri": "https://081381ad-89ce-40ba-b4c6-892093fc376a.jpeg", "isDefault": false, "Main_Image_Rank": "12" } ]

Suggest how to do this in NiFi using jolt transform json?