search

syndesisio / syndesis

This project is archived. A flexible, customizable, open source platform that provides core integration capabilities as a service.
https://syndesis.io/
Apache License 2.0
598 stars 202 forks source link

API Connector - Reponse mapping #3382

Closed pszuster closed 6 years ago

pszuster commented 6 years ago

This is a...


[ ] Feature request
[ ] Regression (a behavior that used to work and stopped working in a new release)
[X] Bug report  
[ ] Documentation issue or request

The problem

When trying to map an API Connector's output to another datashape with a DataMapper I get the following error in the log:

{"exchange":"i-LJtjJ_t7gG7dxPzNpC1z","step":"-LJtGGNaGeegiSb7Dn6a","id":"i-LJtjJhW7gG7dxPzNpC5z","message":"Body: [{\"parameters\":{\"id\":\"quoting\"},\"body\":{\"lookup\":\"ksession\",\"commands\":[{\"insert\":{\"return-object\":true,\"out-identifier\":\"quote1\",\"object\":{\"com.redhat.insurancequoting.Quote\":{\"price\":0,\"driver\":{\"age\":20,\"fines\":2},\"vehicle\":{\"mileage\":500,\"modelYear\":2017}}}}}]}}] LOQUEMANDO:"}
--
  | 2018-08-14 18:40:31.585  INFO 1 --- [ Session Task-2] -LJtE092GeegiSb7Dn6_                     : Body: [{"parameters":{"id":"quoting"},"body":{"lookup":"ksession","commands":[{"insert":{"return-object":true,"out-identifier":"quote1","object":{"com.redhat.insurancequoting.Quote":{"price":0,"driver":{"age":20,"fines":2},"vehicle":{"mileage":500,"modelYear":2017}}}}}]}}] LOQUEMANDO:
  | {"exchange":"i-LJtjJ_t7gG7dxPzNpC1z","step":"-LJtGGNaGeegiSb7Dn6a","id":"i-LJtjJhU7gG7dxPzNpC4z","duration":2347192}
  | {"exchange":"i-LJtjJ_t7gG7dxPzNpC1z","step":"none","id":"i-LJtjJhX7gG7dxPzNpC6z","duration":408509716}
  | {"exchange":"i-LJtjJ_t7gG7dxPzNpC1z","step":"-LJtEHtjGeegiSb7Dn6_","id":"i-LJtjJnv7gG7dxPzNpC7z","duration":119778385}
  | {"exchange":"i-LJtjJ_t7gG7dxPzNpC1z","step":"-LJtiCAkGeegiSb7Dn6b","id":"i-LJtjJpn7gG7dxPzNpC9z","duration":349888}
  | 2018-08-14 18:40:32.124 ERROR 1 --- [ Session Task-2] o.a.camel.processor.DefaultErrorHandler  : Failed delivery for (MessageId: i-LJtjJpn7gG7dxPzNpC8z on ExchangeId: i-LJtjJ_t7gG7dxPzNpC1z). Exhausted after delivery attempt: 1 caught: java.lang.NullPointerException
  |  
  | Message History
  | ---------------------------------------------------------------------------------------------------------------------------------------
  | RouteId              ProcessorId          Processor                                                                        Elapsed (ms)
  | [-LJtE092GeegiSb7Dn] [-LJtE092GeegiSb7Dn] [sjms-sjms-1://queue:cola1                                                     ] [      1016]
  | [-LJtE092GeegiSb7Dn] [setHeader1        ] [setHeader[Syndesis.STEP_ID]                                                   ] [         5]
  | [-LJtE092GeegiSb7Dn] [process1          ] [Processor@0x6aa63c8a                                                          ] [         2]
  | [-LJtE092GeegiSb7Dn] [step:i-LJtjBLP7gG7] [pipeline                                                                      ] [       446]
  | [-LJtE092GeegiSb7Dn] [setHeader2        ] [setHeader[Syndesis.STEP_ID]                                                   ] [         0]
  | [-LJtE092GeegiSb7Dn] [to1               ] [atlas:mapping-step-2.json?sourceMapName=Syndesis.CAPTURED_OUT_MESSAGES_MAP    ] [       442]
  | [-LJtE092GeegiSb7Dn] [process2          ] [Processor@0x6aa63c8a                                                          ] [         0]
  | [-LJtE092GeegiSb7Dn] [step:-LJtGGNaGeegi] [pipeline                                                                      ] [         3]
  | [-LJtE092GeegiSb7Dn] [setHeader3        ] [setHeader[Syndesis.STEP_ID]                                                   ] [         0]
  | [-LJtE092GeegiSb7Dn] [log1              ] [log                                                                           ] [         2]
  | [-LJtE092GeegiSb7Dn] [process3          ] [Processor@0x6aa63c8a                                                          ] [         0]
  | [-LJtE092GeegiSb7Dn] [step:-LJtgB73Geegi] [pipeline                                                                      ] [       409]
  | [-LJtE092GeegiSb7Dn] [setHeader4        ] [setHeader[Syndesis.STEP_ID]                                                   ] [         0]
  | [-LJtE092GeegiSb7Dn] [process4          ] [Processor@0x6a9ab242                                                          ] [       407]
  | [-LJtE092GeegiSb7Dn] [process5          ] [Processor@0x6aa63c8a                                                          ] [         0]
  | [-LJtE092GeegiSb7Dn] [step:-LJtEHtjGeegi] [pipeline                                                                      ] [       120]
  | [-LJtE092GeegiSb7Dn] [setHeader5        ] [setHeader[Syndesis.STEP_ID]                                                   ] [         0]
  | [-LJtE092GeegiSb7Dn] [to2               ] [swagger-operation-5?operationId=manageContainer                               ] [       118]
  | [-LJtE092GeegiSb7Dn] [process6          ] [Processor@0x6aa63c8a                                                          ] [         0]
  | [-LJtE092GeegiSb7Dn] [step:-LJtiCAkGeegi] [pipeline                                                                      ] [         1]
  | [-LJtE092GeegiSb7Dn] [setHeader6        ] [setHeader[Syndesis.STEP_ID]                                                   ] [         0]
  | [-LJtE092GeegiSb7Dn] [to3               ] [mock-6                                                                        ] [         1]
  | [-LJtE092GeegiSb7Dn] [process7          ] [Processor@0x6aa63c8a                                                          ] [         0]
  | [-LJtE092GeegiSb7Dn] [step:-LJtiPSlGeegi] [pipeline                                                                      ] [         0]
  | [-LJtE092GeegiSb7Dn] [setHeader7        ] [setHeader[Syndesis.STEP_ID]                                                   ] [         0]
  | [-LJtE092GeegiSb7Dn] [to4               ] [atlas:mapping-step-7.json?sourceMapName=Syndesis.CAPTURED_OUT_MESSAGES_MAP    ] [         2]
  |  
  | Stacktrace
  | ---------------------------------------------------------------------------------------------------------------------------------------
  |  
  | java.lang.NullPointerException: null
  | at io.atlasmap.json.module.JsonModule.getCollectionSize(JsonModule.java:218) ~[atlas-json-module-1.34.5.fuse-000001-redhat-3.jar!/:1.34.5.fuse-000001-redhat-3]
  | at io.atlasmap.core.DefaultAtlasContext.extractCollectionMappings(DefaultAtlasContext.java:333) ~[atlas-core-1.34.5.fuse-000001-redhat-3.jar!/:1.34.5.fuse-000001-redhat-3]
  | at io.atlasmap.core.DefaultAtlasContext.process(DefaultAtlasContext.java:252) ~[atlas-core-1.34.5.fuse-000001-redhat-3.jar!/:1.34.5.fuse-000001-redhat-3]
  | at org.apache.camel.component.atlasmap.AtlasEndpoint.onExchange(AtlasEndpoint.java:194) ~[camel-atlasmap-1.34.5.fuse-000001-redhat-3.jar!/:na]
  | at org.apache.camel.impl.ProcessorEndpoint$1.process(ProcessorEndpoint.java:71) ~[camel-core-2.21.0.fuse-000112-redhat-3.jar!/:2.21.0.fuse-000112-redhat-3]
  | at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) ~[camel-core-2.21.0.fuse-000112-redhat-3.jar!/:2.21.0.fuse-000112-redhat-3]
  | at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148) ~[camel-core-2.21.0.fuse-000112-redhat-3.jar!/:2.21.0.fuse-000112-redhat-3]
  | at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:110) [camel-core-2.21.0.fuse-000112-redhat-3.jar!/:2.21.0.fuse-000112-redhat-3]
  | at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.21.0.fuse-000112-redhat-3.jar!/:2.21.0.fuse-000112-redhat-3]
  | at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.21.0.fuse-000112-redhat-3.jar!/:2.21.0.fuse-000112-redhat-3]
  | at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.21.0.fuse-000112-redhat-3.jar!/:2.21.0.fuse-000112-redhat-3]
  | at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [camel-core-2.21.0.fuse-000112-redhat-3.jar!/:2.21.0.fuse-000112-redhat-3]
  | at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:110) [camel-core-2.21.0.fuse-000112-redhat-3.jar!/:2.21.0.fuse-000112-redhat-3]
  | at io.syndesis.integration.runtime.logging.ActivityTrackingInterceptStrategy$EventProcessor.process(ActivityTrackingInterceptStrategy.java:77) [integration-runtime-1.3.12.fuse-000001-redhat-2.jar!/:1.3.12.fuse-000001-redhat-2]
  | at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:110) [camel-core-2.21.0.fuse-000112-redhat-3.jar!/:2.21.0.fuse-000112-redhat-3]
  | at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.21.0.fuse-000112-redhat-3.jar!/:2.21.0.fuse-000112-redhat-3]
  | at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.21.0.fuse-000112-redhat-3.jar!/:2.21.0.fuse-000112-redhat-3]
  | at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.21.0.fuse-000112-redhat-3.jar!/:2.21.0.fuse-000112-redhat-3]
  | at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [camel-core-2.21.0.fuse-000112-redhat-3.jar!/:2.21.0.fuse-000112-redhat-3]
  | at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.21.0.fuse-000112-redhat-3.jar!/:2.21.0.fuse-000112-redhat-3]
  | at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.21.0.fuse-000112-redhat-3.jar!/:2.21.0.fuse-000112-redhat-3]
  | at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [camel-core-2.21.0.fuse-000112-redhat-3.jar!/:2.21.0.fuse-000112-redhat-3]
  | at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109) [camel-core-2.21.0.fuse-000112-redhat-3.jar!/:2.21.0.fuse-000112-redhat-3]
  | at org.apache.camel.processor.Pipeline.process(Pipeline.java:80) [camel-core-2.21.0.fuse-000112-redhat-3.jar!/:2.21.0.fuse-000112-redhat-3]
  | at org.apache.camel.component.sjms.consumer.InOnlyMessageHandler.handleMessage(InOnlyMessageHandler.java:65) [camel-sjms-2.21.0.fuse-000112-redhat-3.jar!/:2.21.0.fuse-000112-redhat-3]
  | at org.apache.camel.component.sjms.consumer.AbstractMessageHandler.onMessage(AbstractMessageHandler.java:86) [camel-sjms-2.21.0.fuse-000112-redhat-3.jar!/:2.21.0.fuse-000112-redhat-3]
  | at org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:1401) [activemq-client-5.14.5.jar!/:5.14.5]
  | at org.apache.activemq.ActiveMQSessionExecutor.dispatch(ActiveMQSessionExecutor.java:131) [activemq-client-5.14.5.jar!/:5.14.5]
  | at org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:202) [activemq-client-5.14.5.jar!/:5.14.5]
  | at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133) [activemq-client-5.14.5.jar!/:5.14.5]
  | at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48) [activemq-client-5.14.5.jar!/:5.14.5]
  | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
  | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
  | at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]

Export of the integration: amqToRHDMtoAmq-export.zip

Syndesis version: 1.3.12.fuse-000001-redhat-2

Expected behavior

Be able to map an API connector's output to another dataformat.

Screenshot

image

Request and Response Data

API Endpoints and Schemas

Tasks involved / Steps to Reproduce

1.Create connection to AMQ 2.Create connection to API Insurance Quoting.json.txt

  1. Create integration from AMQ -> DataMapper -> API -> DataMapper -> AMQ
zregvart commented 6 years ago

@pszuster what is the reason for changing the data shape after RHDM-Insurance API call? This can only lead to issues as the data shape will not correspond to response from the API and the mapper will operate under false pretences.

pszuster commented 6 years ago

@pszuster what is the reason for changing the data shape after RHDM-Insurance API call? This can only lead to issues as the data shape will not correspond to response from the API and the mapper will operate under false pretences.

@zregvart removing the" DataShape" extension doesn't change the behaviour (it shouldn't be in that reproducer sample). It is there because the combination of adding a script step that converts body to string and that extension makes the integration work.

zregvart commented 6 years ago

@pszuster there seems to be an inconsistency between the the error and the integration export you provided. The error contains 7 steps and the integration contains 5 -- I think matching what you reference in steps to reproduce (AMQ -> DataMapper -> API -> DataMapper -> AMQ). This makes it really hard to troubleshoot, can you provide logs from the integration that has the 5 steps you outlined in the steps to reproduce?

zregvart commented 6 years ago

I can reproduce this using AMQ (data shape: JSON Instance {"data": "here"}) -> mapper -> API (TODO API, action: create task) -> mapper -> AMQ (data shape: JSON Instance {"other": "data"}) on 1.4.x. I'll try to debug a bit more and find a solution.

pszuster commented 6 years ago

I'll try to debug a bit more and find a solution.

I created a workaround that makes it work reproducing-export_workaround.zip

zregvart commented 6 years ago

I found the issue with message type conversion when using SJMS (ActiveMQ) connector. In combinations where type conversion is required the integration will fail. One of such cases is handling the response stream form the API connector in the mapper.

I would not call this a blocker, having a script as demonstrated here, will help as it performs type conversion before the mapper.

cc @syndesisio/qe @gaughan @kcbabo @paoloantinori @heiko-braun

zregvart commented 6 years ago

@pszuster this should be now fixed on the version in master, you can test it from master using the quick installation instructions or use a daily build -- this should be included in tomorrow's version: 1.5.4-20180821.