ideally processed input json should be:
{"RequestId":"1697836f-7825-4555-8e6c-9d8e89c5b8fa","Source":"MPAA","mapAliases":[{"inputJobTitle":"OWNER/IT MANAGER","correlationId":"0797dc4d-cd13-4023-8688-f61eae882ede"},{"inputJobTitle":"Adminsitrador de Base de Datos","correlationId":"8d232dff-3239-41de-80fa-0bc20dca2947"},{"inputJobTitle":"SENIOR DB ENGINEER","correlationId":"84de8e8e-b99f-4c0d-b78b-9652882ebb4b"},{"inputJobTitle":"INFO TECH SPECIALISTS","correlationId":"1209feb1-d06b-4654-b405-a1e30f95d211"}]}
when the column contains complex data type (object, array of objects) then the input json is malformed.
input dataframe element (databricks):
processed input json becomes:
{"RequestId":"96625e72-3064-4388-96fe-dbf34d728214","Source":"MPAA","mapAliases":"WrappedArray([SMALL BUSINESS MARKETING,e9c9db08-2c48-4668-83c7-e39a3864471b], [pedagog,c1f7c0ee-8fba-4df8-92d4-024529dc1a42], [BASIS SPECIALIST,78f1b493-6fd4-4512-884b-be6e603a467f], [COMPUTERPROGRAMMING,ec01cdfa-270f-475d-92c5-1345a4190f70])"}ideally processed input json should be:
{"RequestId":"1697836f-7825-4555-8e6c-9d8e89c5b8fa","Source":"MPAA","mapAliases":[{"inputJobTitle":"OWNER/IT MANAGER","correlationId":"0797dc4d-cd13-4023-8688-f61eae882ede"},{"inputJobTitle":"Adminsitrador de Base de Datos","correlationId":"8d232dff-3239-41de-80fa-0bc20dca2947"},{"inputJobTitle":"SENIOR DB ENGINEER","correlationId":"84de8e8e-b99f-4c0d-b78b-9652882ebb4b"},{"inputJobTitle":"INFO TECH SPECIALISTS","correlationId":"1209feb1-d06b-4654-b405-a1e30f95d211"}]}