search

odpi / egeria-connector-ibm-information-server

IBM Information Server connectors for Egeria: repository proxy connector for IGC, data engine proxy connector for DataStage.
https://odpi.github.io/egeria-connector-ibm-information-server
Apache License 2.0
27 stars 20 forks source link

Egeria-IGC-connectivity issues - Part I - Connectivity to IGC can't be restored after IGC restart #21

Closed jan-frommann closed 5 years ago

jan-frommann commented 5 years ago

Hi ODPI Egeria team,

I deployed Egeria on a dedicated Kubernetes cluster on the IBM Cloud (Yellow zone). Between the cluster and the IGC (Blue Zone) is a firewall, which was configured to allow communication. Everything seemed to work fine until the IGC was restarted. From that point on, it was neither possible to exchange information (Glossary or assets), nor has the connection ever recovered. Purging and reinstalling the helm chart restores the working state, but naturally leads to losing all assets, that have been transferred up to this point. Here is a piece of the IGC pod log:

2019-06-12 15:13:20.623 ERROR 1 --- [       Thread-4] o.o.o.a.r.i.clientlibrary.IGCRestClient  : Request failed -- check IGC environment connectivity and authentication details.

org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 Internal Server Error
    at org.springframework.web.client.HttpServerErrorException.create(HttpServerErrorException.java:79) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:124) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:102) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:778) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:736) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:670) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:579) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.makeRequest(IGCRestClient.java:435) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.makeRequest(IGCRestClient.java:467) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.searchJson(IGCRestClient.java:558) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.search(IGCRestClient.java:569) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.model.common.Reference.getAssetWithSubsetOfProperties(Reference.java:134) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.model.common.Reference.getAssetWithSubsetOfProperties(Reference.java:151) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.repositoryconnector.IGCOMRSMetadataCollection.getFullAssetDetails(IGCOMRSMetadataCollection.java:4176) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processAsset(IGCOMRSRepositoryEventMapper.java:556) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processAssetEventV115(IGCOMRSRepositoryEventMapper.java:354) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processEventV115(IGCOMRSRepositoryEventMapper.java:278) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processEvent(IGCOMRSRepositoryEventMapper.java:238) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper$IGCKafkaConsumerThread.run(IGCOMRSRepositoryEventMapper.java:204) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_201]

2019-06-12 15:13:20.624 ERROR 1 --- [       Thread-4] o.o.o.a.r.i.clientlibrary.IGCRestClient  : Unable to complete request -- check IGC environment connectivity and authentication details.
2019-06-12 15:13:20.624 ERROR 1 --- [       Thread-4] o.o.a.r.i.e.IGCOMRSRepositoryEventMapper : Failed trying to consume IGC events from Kafka.

java.lang.NullPointerException: Unable to complete request -- check IGC environment connectivity and authentication details.
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.makeRequest(IGCRestClient.java:477) ~[igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.searchJson(IGCRestClient.java:558) ~[igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.search(IGCRestClient.java:569) ~[igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.model.common.Reference.getAssetWithSubsetOfProperties(Reference.java:134) ~[igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.model.common.Reference.getAssetWithSubsetOfProperties(Reference.java:151) ~[igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.repositoryconnector.IGCOMRSMetadataCollection.getFullAssetDetails(IGCOMRSMetadataCollection.java:4176) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processAsset(IGCOMRSRepositoryEventMapper.java:556) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processAssetEventV115(IGCOMRSRepositoryEventMapper.java:354) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processEventV115(IGCOMRSRepositoryEventMapper.java:278) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processEvent(IGCOMRSRepositoryEventMapper.java:238) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper$IGCKafkaConsumerThread.run(IGCOMRSRepositoryEventMapper.java:204) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_201]

2019-06-12 15:13:20.625  WARN 1 --- [       Thread-4] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] Synchronous auto-commit of offsets {InfosphereEvents-0=OffsetAndMetadata{offset=4398, leaderEpoch=null, metadata=''}} failed: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.
2019-06-12 15:13:20.625  INFO 1 --- [       Thread-4] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] Revoking previously assigned partitions [InfosphereEvents-0]
2019-06-12 15:13:20.625  INFO 1 --- [       Thread-4] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] (Re-)joining group
2019-06-12 15:13:20.625  INFO 1 --- [       Thread-4] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] Group coordinator xxxxxxx.xxx.xxx:59092 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
2019-06-12 15:13:21.221  INFO 1 --- [       Thread-4] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] Discovered group coordinator xxxxxxx.xxx.xxx:59092 (id: 2147483646 rack: null)
2019-06-12 15:13:21.222  INFO 1 --- [       Thread-4] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] (Re-)joining group
2019-06-12 15:13:21.817  INFO 1 --- [       Thread-4] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] Successfully joined group with generation 1
2019-06-12 15:13:21.817  INFO 1 --- [       Thread-4] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] Setting newly assigned partitions [InfosphereEvents-0]
2019-06-12 15:13:26.308  WARN 1 --- [       Thread-4] o.o.o.a.r.i.clientlibrary.IGCRestClient  : Request failed -- session may have expired, retrying...

org.springframework.web.client.HttpClientErrorException$BadRequest: 400 Bad Request
    at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:79) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:122) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:102) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:778) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:736) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:670) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:579) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.makeRequest(IGCRestClient.java:435) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.makeRequest(IGCRestClient.java:467) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.searchJson(IGCRestClient.java:558) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.search(IGCRestClient.java:569) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.model.common.Reference.getAssetWithSubsetOfProperties(Reference.java:134) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.model.common.Reference.getAssetWithSubsetOfProperties(Reference.java:151) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.repositoryconnector.IGCOMRSMetadataCollection.getFullAssetDetails(IGCOMRSMetadataCollection.java:4176) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processAsset(IGCOMRSRepositoryEventMapper.java:556) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processAssetEventV115(IGCOMRSRepositoryEventMapper.java:354) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processEventV115(IGCOMRSRepositoryEventMapper.java:278) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processEvent(IGCOMRSRepositoryEventMapper.java:238) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper$IGCKafkaConsumerThread.run(IGCOMRSRepositoryEventMapper.java:204) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_201]

2019-06-12 15:13:47.991  WARN 1 --- [       Thread-4] o.o.o.a.r.i.clientlibrary.IGCRestClient  : Request failed -- session may have expired, retrying...

org.springframework.web.client.HttpClientErrorException$BadRequest: 400 Bad Request
    at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:79) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:122) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:102) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:778) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:736) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:670) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:579) ~[spring-web-5.1.5.RELEASE.jar!/:5.1.5.RELEASE]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.makeRequest(IGCRestClient.java:435) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.openNewSessionWithRequest(IGCRestClient.java:215) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.makeRequest(IGCRestClient.java:444) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.makeRequest(IGCRestClient.java:467) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.searchJson(IGCRestClient.java:558) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.search(IGCRestClient.java:569) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.model.common.Reference.getAssetWithSubsetOfProperties(Reference.java:134) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.model.common.Reference.getAssetWithSubsetOfProperties(Reference.java:151) [igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.repositoryconnector.IGCOMRSMetadataCollection.getFullAssetDetails(IGCOMRSMetadataCollection.java:4176) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processAsset(IGCOMRSRepositoryEventMapper.java:556) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processAssetEventV115(IGCOMRSRepositoryEventMapper.java:354) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processEventV115(IGCOMRSRepositoryEventMapper.java:278) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processEvent(IGCOMRSRepositoryEventMapper.java:238) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper$IGCKafkaConsumerThread.run(IGCOMRSRepositoryEventMapper.java:204) [igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_201]

2019-06-12 15:13:47.991 ERROR 1 --- [       Thread-4] o.o.o.a.r.i.clientlibrary.IGCRestClient  : Opening a new session already attempted without success -- giving up on POST to https://xxxxxxx.xxx.xxx:444/ibm/iis/igc-rest/v1/search with {"types":["database"],"properties":["name","short_description","long_description","host","labels","stewards","assigned_to_terms","implements_rules","governed_by_rules","database_schemas","alias_(business_name)","location","dbms","dbms_server_instance","dbms_vendor","dbms_version","database_type","imported_from","data_connections","data_connection_mappings","mapped_to_mdm_models","read_by_(static)","written_by_(static)","read_by_(design)","written_by_(design)","read_by_(operational)","written_by_(operational)","read_by_(user_defined)","written_by_(user_defined)","impacted_by","impacts_on","bi_reports","bi_models","suggested_term_assignments","data_policies","blueprint_elements","in_collections","created_by","created_on","modified_by","modified_on"],"pageSize":1000,"where":{"conditions":[{"property":"_id","operator":"=","value":"b1c497ce.6e83759b.9g85t7kh2.9ge978u.81ro06.344cvqapg9q7d4ofhmog8"}],"operator":"and"}}
2019-06-12 15:13:47.991 ERROR 1 --- [       Thread-4] o.o.o.a.r.i.clientlibrary.IGCRestClient  : Unable to complete request -- check IGC environment connectivity and authentication details.
2019-06-12 15:13:47.992 ERROR 1 --- [       Thread-4] o.o.a.r.i.e.IGCOMRSRepositoryEventMapper : Failed trying to consume IGC events from Kafka.

java.lang.NullPointerException: Unable to complete request -- check IGC environment connectivity and authentication details.
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.makeRequest(IGCRestClient.java:477) ~[igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.searchJson(IGCRestClient.java:558) ~[igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient.search(IGCRestClient.java:569) ~[igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.model.common.Reference.getAssetWithSubsetOfProperties(Reference.java:134) ~[igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.model.common.Reference.getAssetWithSubsetOfProperties(Reference.java:151) ~[igc-rest-client-library-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.repositoryconnector.IGCOMRSMetadataCollection.getFullAssetDetails(IGCOMRSMetadataCollection.java:4176) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processAsset(IGCOMRSRepositoryEventMapper.java:556) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processAssetEventV115(IGCOMRSRepositoryEventMapper.java:354) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processEventV115(IGCOMRSRepositoryEventMapper.java:278) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper.processEvent(IGCOMRSRepositoryEventMapper.java:238) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.adapters.repositoryservices.igc.eventmapper.IGCOMRSRepositoryEventMapper$IGCKafkaConsumerThread.run(IGCOMRSRepositoryEventMapper.java:204) ~[igc-repository-connector-1.1-SNAPSHOT.jar!/:na]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_201]

(@cmgrote edited for log formatting)

cmgrote commented 5 years ago

Thanks for raising this -- I'll have a look ASAP (probably a little later this week). For now, transferring to separate IGC connector repository.

cmgrote commented 5 years ago

@jan-frommann couple of follow-up questions:

  1. From the canonical names in the stack traces (eg.org.odpi.openmetadata.adapters.repositoryservices.igc.clientlibrary.IGCRestClient), it appears that you're using the old version of the IGC connector. In the latest version, these should be eg. org.odpi.egeria.connectors.ibm.igc.clientlibrary.IGCRestClient. Any chance you could reproduce with this latest version of the connector to see if the problem has already been addressed? (See instructions at https://github.com/odpi/egeria-connector-ibm-igc -- specifically copying the packaged version of the connector you either build yourself or download from the releases of this repository into a particular location, and referencing this at OMAG Server Platform startup using the -Dloader.path=... parameter.) If you're using the latest version of the vdc helm charts from the core Egeria repository, I think this should already be done for you (ie. should already be using this new connector) (?)
  2. Generally a 400: Bad request response back from IGC would indicate that the request being made by the connector includes some parameters that the IGC environment doesn't understand / recognise. This will be the case if, for example, the connector thinks it is using an IGC environment at version x but the IGC environment is actually version y. The auto-detection should handle this, but only occurs at startup of the connector (the OMAG Server Platform admin API's instance call)... I assume when you restart your IGC environment you're not actually switching it to a different version or the like? (If so, this is likely the culprit, and was never really an anticipated scenario to try to work with.) You could force re-detection by restarting the connector itself: you should be able to do this without needing to re-deploy the entire Helm chart, but to do so you'll need to first save the server configuration from the connector's pod. Copy the *.registrystore and omag.* files out of the pod to back them up, delete the pod (to have it re-created), then copy these files into that pod, and then re-run the OMAG Server Platform admin API's instance call against the proxy and it should pickup these configurations again automatically.
cmgrote commented 5 years ago

Trying to reproduce with the following setup:

Steps taken:

  1. Startup IGC on host A.
  2. Startup Egeria (docker container) on host B using: 
    docker run -d --name egeria --mount type=bind,source="$LOADER_PATH",target=/opt/egeria/connectors -p 8080:8080 -e LOADER_PATH=/opt/egeria/connectors -e STRICT_SSL=false -e LOGGING_LEVEL_ROOT=INFO odpi/egeria
  3. Configure Egeria using the sample payloads for the IGC connector setup.
  4. Create a simple metadata object in IGC (a new term with name "test123").
  5. Observe from logs of Egeria that the term is picked up by the event mapper.
  6. Run a search via the Egeria API (eg. /instances/entities/by-property where displayName contains Test) to confirm that the new GlossaryTerm can be found through the connector (1 result).
  7. Shutdown IGC on host A. Reboot host A to ensure no stale threads or processes were left behind.
  8. Observe from logs of Egeria that the Kafka connection for the event mapper has gone offline: 
    org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] Connection to node 1 (hostname/xxx.xxx.xxx.xxx:xxxx) could not be established. Broker may not be available.
  9. Startup IGC on host A.
  10. Observe from logs of Egeria that the Kafka connection for the event mapper has automatically been restored: 
    o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] Group coordinator hostname:xxxx (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-1, groupId=IGCOMRSRepositoryEventMapper_consumer] Discovered group coordinator hostname:xxxx (id: 2147483646 rack: null)
  11. Create another simple metadata object in IGC (another new term with name "test321").
  12. Note from logs of Egeria that the previous session it held is no longer valid, due to the restart of the IGC system, but that it automatically retries with a new session: 
    o.o.e.c.i.i.clientlibrary.IGCRestClient  : Request failed -- session may have expired, retrying...
  13. Observe from logs of Egeria that the new term is picked up by the event mapper.
  14. Run a search via the Egeria API (eg. /instances/entities/by-property where displayName contains Test) to confirm that both the newest and older GlossaryTerms can be found through the connector (2 results).

Note that at no point was the Egeria runtime restarted or reconfigured.

As such, I'm unable to reproduce the issue. If still not working, can you provide updated logs (using the latest publicly-available odpi/egeria docker image)?

cmgrote commented 5 years ago

I believe this is now resolved: please re-open with further details (updated logs) if still an issue.