feat(runtime): Prefetch connector catalog

[rhuss]

When creating project files, a CamelCatalog and CamelConnectorCatalog is required to calculate the proper endpoint URL

Up to now, this is only done via Grape which does a live introspection and download of connector jars.

A new plugin goal syndesis:extract-connector-descriptor has been introduced to fetch this information offline. By adding this to META-INF/camel/camel-meta.json, the runtime can query this file and build up a CamelCatalog from this information only. This avoids to reach out to the internet, which can have unexpected side effects related to the performance.

---

This is still work in progress. Actually the generation works nicely and has been introduced to the runtime pom.xml During startup of the ConnectorCatalog this /META-INF/camel/camel-meta.json is parsed and the CamelCatalog as well as the CamelConnectorCatalog is build up.

It was tried to mimic the way how the MavenArtifactProvider resolves the required data, however it still fails.

When using the catalog on a prefetched connector, the following error happens:

2017-09-23 18:17:50.608 ERROR [-,,,] 1 --- [pool-5-thread-1] i.s.c.i.online.ActivateHandler           : Integration -KujxQDBjnvVpgWd0MgV : Failure

java.lang.IllegalArgumentException: Endpoint with scheme twitter-mention-connector has no syntax defined in the json schema
    at org.apache.camel.catalog.AbstractCamelCatalog.doAsEndpointUri(AbstractCamelCatalog.java:892) ~[camel-catalog-2.20.0.fuse-000101.jar!/:2.20.0.fuse-000101]
    at org.apache.camel.catalog.AbstractCamelCatalog.asEndpointUri(AbstractCamelCatalog.java:869) ~[camel-catalog-2.20.0.fuse-000101.jar!/:2.20.0.fuse-000101]
    at io.syndesis.connector.catalog.ConnectorCatalog.buildEndpointUri(ConnectorCatalog.java:164) ~[connector-catalog-0.1-SNAPSHOT.jar!/:0.1-SNAPSHOT]
    at io.syndesis.project.converter.visitor.EndpointStepVisitor.createEndpoint(EndpointStepVisitor.java:129) ~[project-generator-0.1-SNAPSHOT.jar!/:0.1-SNAPSHOT]
    at io.syndesis.project.converter.visitor.EndpointStepVisitor.createEndpoint(EndpointStepVisitor.java:125) ~[project-generator-0.1-SNAPSHOT.jar!/:0.1-SNAPSHOT]
    at io.syndesis.project.converter.visitor.EndpointStepVisitor.visit(EndpointStepVisitor.java:64) ~[project-generator-0.1-SNAPSHOT.jar!/:0.1-SNAPSHOT]
    at io.syndesis.project.converter.DefaultProjectGenerator.visitStep(DefaultProjectGenerator.java:219) ~[project-generator-0.1-SNAPSHOT.jar!/:0.1-SNAPSHOT]
    at io.syndesis.project.converter.DefaultProjectGenerator.lambda$generateFlowYaml$4(DefaultProjectGenerator.java:207) ~[project-generator-0.1-SNAPSHOT.jar!/:0.1-SNAPSHOT]
    at java.util.Optional.ifPresent(Optional.java:159) ~[na:1.8.0_131]
    at io.syndesis.project.converter.DefaultProjectGenerator.generateFlowYaml(DefaultProjectGenerator.java:184) ~[project-generator-0.1-SNAPSHOT.jar!/:0.1-SNAPSHOT]
    at io.syndesis.project.converter.DefaultProjectGenerator.generate(DefaultProjectGenerator.java:157) ~[project-generator-0.1-SNAPSHOT.jar!/:0.1-SNAPSHOT]
    at io.syndesis.controllers.integration.online.ActivateHandler.createProjectFiles(ActivateHandler.java:251) ~[controllers-0.1-SNAPSHOT.jar!/:0.1-SNAPSHOT]
    at io.syndesis.controllers.integration.online.ActivateHandler.execute(ActivateHandler.java:123) ~[controllers-0.1-SNAPSHOT.jar!/:0.1-SNAPSHOT]
    at io.syndesis.controllers.integration.IntegrationController.lambda$callStatusChangeHandler$10(IntegrationController.java:175) [controllers-0.1-SNAPSHOT.jar!/:0.1-SNAPSHOT]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_131]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_131]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_131]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_131]
    at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]

The following connectors have been preloaded from the plugin's dependencies:

2017-09-23 18:15:32.127  INFO [-,,,] 1 --- [           main] i.s.connector.catalog.ConnectorCatalog   : Prefetched meta for org.apache.camel:camel-salesforce:2.20.0.fuse-000101
2017-09-23 18:15:32.128  INFO [-,,,] 1 --- [           main] i.s.connector.catalog.ConnectorCatalog   : Prefetched meta for io.syndesis:salesforce-on-update-connector:0.5.5
2017-09-23 18:15:32.130  INFO [-,,,] 1 --- [           main] i.s.connector.catalog.ConnectorCatalog   : Prefetched meta for io.syndesis:salesforce-on-delete-connector:0.5.5
2017-09-23 18:15:32.135  INFO [-,,,] 1 --- [           main] i.s.connector.catalog.ConnectorCatalog   : Prefetched meta for io.syndesis:salesforce-update-sobject-connector:0.5.5
2017-09-23 18:15:32.143  INFO [-,,,] 1 --- [           main] i.s.connector.catalog.ConnectorCatalog   : Prefetched meta for io.syndesis:salesforce-upsert-sobject-connector:0.5.5
2017-09-23 18:15:32.156  INFO [-,,,] 1 --- [           main] i.s.connector.catalog.ConnectorCatalog   : Prefetched meta for io.syndesis:salesforce-create-sobject-connector:0.5.5
2017-09-23 18:15:32.160  INFO [-,,,] 1 --- [           main] i.s.connector.catalog.ConnectorCatalog   : Prefetched meta for io.syndesis:salesforce-get-sobject-connector:0.5.5
2017-09-23 18:15:32.163  INFO [-,,,] 1 --- [           main] i.s.connector.catalog.ConnectorCatalog   : Prefetched meta for io.syndesis:salesforce-get-sobject-with-id-connector:0.5.5
2017-09-23 18:15:32.166  INFO [-,,,] 1 --- [           main] i.s.connector.catalog.ConnectorCatalog   : Prefetched meta for io.syndesis:salesforce-delete-sobject-connector:0.5.5
2017-09-23 18:15:32.167  INFO [-,,,] 1 --- [           main] i.s.connector.catalog.ConnectorCatalog   : Prefetched meta for io.syndesis:salesforce-delete-sobject-with-id-connector:0.5.5
2017-09-23 18:15:32.170  INFO [-,,,] 1 --- [           main] i.s.connector.catalog.ConnectorCatalog   : Prefetched meta for io.syndesis:sql-stored-connector:0.5.5
2017-09-23 18:15:32.172  INFO [-,,,] 1 --- [           main] i.s.connector.catalog.ConnectorCatalog   : Prefetched meta for org.apache.camel:camel-sql:2.20.0.fuse-000101

Don't think its a major issue, but happy for any hint to fix this as I'm on the road tomorrow and this is still considered to be a blocker for TP1. 'will continue tomorrow, but who will fix this before me, will get (a) one pint of franconian beer and (b) a mini-jar of fresh Jolokia pepper powder from 2017 ;-)

@lburgazzoli @jimmidyson @zregvart @davsclaus is this good enough of an offer ;-) ?

[jimmibot]

@rhuss, thanks! @jimmidyson, @zregvart and @chirino, please review this.

[rhuss]

The issue here is that JSonSchemaHelper.parseJsonSchema() can only parse pretty-printed JSON with a very special format, but I put it on a single line (as I parse it first and then serialize it back as I'm working here with JsonNode objects.

Come on guys, this can't be true. If this can't be changed (should be easy though, happy to send a PR), then please don't call this JsonSchema. Call it CamelSchema or whatever. But not JsonSchema and not Json.

'wonder really why nobody already stumbled upon this. // @davsclaus @zregvart

[davsclaus]

@rhuss for the 110 time there is a CAMEL ticket about it. And we do not want to have a runtime dependency on a heavy json parser like jackson that causes all sorts of classpath issues when other libraries uses different version of it etc.

Recently a json spec was passed which will make it easier to plugin and use a lighter library in the future, or even shade a library into camel-core if necessary.

[davsclaus]

And btw @rhuss it seems the bot wants the beer it reported a gazillion issues.

[rhuss]

@davsclaus thx

[davsclaus]

The CAMEL ticket is: https://issues.apache.org/jira/browse/CAMEL-10969 and https://issues.apache.org/jira/browse/CAMEL-11005

[rhuss]

And btw @rhuss it seems the bot wants the beer it reported a gazillion issues.

good idea, will fix it, @davsclaus . Then it's not so evident anymore that I reused quite a bit from the camel code ;-P (its not so easy to build up a catalog programmatially, but happy for any pointer how this can be done more easily)

[davsclaus]

@rhuss okay working on using a better parser, json-simple is maintained in v2 which we can use.

There is a branch here https://github.com/apache/camel/tree/json-simple

And you can turn on|off prettyPrint mode on the camel-connector-maven-plugin (default on) https://github.com/apache/camel/blob/json-simple/connectors/camel-connector-maven-plugin/src/main/java/org/apache/camel/maven/connector/ConnectorMojo.java#L81

And you have json in pretty print such as (its hardcoded to use tab in json-simple so we cannot change to use 2 spaces etc) https://github.com/apache/camel/blob/json-simple/connectors/examples/foo-connector/src/main/resources/camel-connector-schema.json

The API is the same as before, you just use JSonSchemaHelper from camel-catalog https://github.com/apache/camel/blob/json-simple/platforms/camel-catalog/src/main/java/org/apache/camel/catalog/JSonSchemaHelper.java#L42

... which now supports:

• old single-line mode
• pretty printed mode
• or any json mode (assuming the parser works well)

Mind that outputting in pretty print mode, may affect older tooling such as Fuse IDE that may use an older version of camel-catalog with its parser. But if they upgrade to use Camel 2.20.0 then they should be okay.

We are running tests on the branch to see if it causes and problems, if not then its very likely to be included in Camel 2.20.0 as json-simple is shaded into camel-core and camel-catalog and its a single JAR so it does not drag in a lot of complexity/crap or potential problems with using jackson etc.

[davsclaus]

@rhuss @lburgazzoli @jimmidyson @zregvart @gashcrumb okay so the master branch of Apache Camel now comes with a better json parser embedded (based on simple-json v2) which supports json files as before but also in pretty print mode (eg real json schema). The API from camel-catalog and camel-core is 100% the same.

The camel-catalog-maven-plugin now outputs the generated json schema it spits out in pretty print mode, but you can turn it off to make it as it was before.

And the camel-connector component can then now read any kind of json schema file you throw at it.

The ticket at Apache Camel about this is: https://issues.apache.org/jira/browse/CAMEL-10969

[rhuss]

@davsclaus A.w.e.s.o.m.e !! Thanks a lot, that make things so much easier and more robust. I'm going to revert back to the original variation of this PR when the camel upstream change bubbled up to a pre-release version.

[rhuss]

retest this please

[rhuss]

retest this please

[rhuss]

@zregvart @jimmidyson @KurtStam @iocanel Time for a review ? This PR is ready for prime time and helps in creating project files (and avoids external download for creating the CamelCatalog)

[rhuss]

@zregvart are we good to merge ? I tried to adressed some of your issue, but struggled. Let's do this either later or ...

[pure-bot[bot]]

Pull request approved by @zregvart - applying approved label