search

odpi / egeria-connector-xtdb

Pluggable repository for Egeria, using XTDB (formerly "Crux") as the back-end to natively support historical metadata.
https://odpi.github.io/egeria-docs/connectors/repository/xtdb/
Apache License 2.0
15 stars 7 forks source link

OMAG-REPOSITORY-HANDLER-404-018 Method updateSchemaAttribute for service Data Manager OMAS is not supported by any of the metadata repositories connected to {mds-server} #457

Open juergenhemelt opened 1 year ago

juergenhemelt commented 1 year ago

I get the above exception when calling TopicIntegratorContext.updateSchemaAttribute when using a metadata repository server with XTDB connector.

cmgrote commented 1 year ago

@mandy-chessell can you help troubleshoot this one? Looks like an error being thrown by the OMAS — hoping you might have an idea what method(s) it would be trying to call on the underlying repository that would be new / changed and thus not (yet) available in XTDB repository...

(Or is there a chance that this is a configuration issue where the OMAS hasn't first been stood up somewhere, since updateSchemaAttribute afaik must be an OMAS-level method (I don't recognise it as an OMRS-level method)?)

mandy-chessell commented 1 year ago

@cmgrote I suspect it is caused by the new classification methods not being implemented - a look at the console log would confirm it.

@juergenhemelt could you supply details of the version of the XTDB connector that you are running and also the other messages logged on the console around the same time.

juergenhemelt commented 1 year ago

This is an excerpt of the audit log of the calling integration connector:

2023-02-10 10:31:31.907 ERROR 1 --- [ionDaemonThread] o.o.o.c.ffdc.RESTExceptionHandler        : FFDC Response: VoidResponse{exceptionClassName='org.odpi.openmetadata.frameworks.connectors.ffdc.PropertyServerException', exceptionCausedBy='null', actionDescription='updateSchemaAttribute', relatedHTTPCode=404, exceptionErrorMessage='OMAG-REPOSITORY-HANDLER-404-018 Method updateSchemaAttribute for service Data Manager OMAS is not supported by any of the metadata repositories connected to mds1', exceptionErrorMessageId='OMAG-REPOSITORY-HANDLER-404-018', exceptionErrorMessageParameters=[updateSchemaAttribute, Data Manager OMAS, mds1], exceptionSystemAction='The system is unable to process the request because none of the members of the connected cohort(s) support this function.', exceptionUserAction='Add an Egeria native metadata repository to one of the connected cohorts.  This will provide the support that you need.', exceptionProperties=null}
2023-02-10 10:31:31.907 ERROR 1 --- [ionDaemonThread] o.o.o.c.ffdc.RESTExceptionHandler        : Property Server Error Exception returned by REST Call

org.odpi.openmetadata.frameworks.connectors.ffdc.PropertyServerException: OMAG-REPOSITORY-HANDLER-404-018 Method updateSchemaAttribute for service Data Manager OMAS is not supported by any of the metadata repositories connected to mds1
    at org.odpi.openmetadata.commonservices.ffdc.RESTExceptionHandler.throwPropertyServerException(RESTExceptionHandler.java:312) ~[ffdc-services-3.10.jar!/:na]
    at org.odpi.openmetadata.commonservices.ffdc.RESTExceptionHandler.detectAndThrowStandardExceptions(RESTExceptionHandler.java:155) ~[ffdc-services-3.10.jar!/:na]
    at org.odpi.openmetadata.commonservices.ffdc.rest.FFDCRESTClient.callVoidPostRESTCall(FFDCRESTClient.java:309) ~[ffdc-services-3.10.jar!/:na]
    at org.odpi.openmetadata.accessservices.datamanager.client.SchemaManagerClient.updateSchemaAttribute(SchemaManagerClient.java:1260) ~[data-manager-client-3.10.jar!/:na]
    at org.odpi.openmetadata.integrationservices.topic.connector.TopicIntegratorContext.updateSchemaAttribute(TopicIntegratorContext.java:1046) ~[topic-integrator-api-3.10.jar!/:na]
    at org.odpi.openmetadata.adapters.connectors.integration.eventschema.mapper.SchemaAttributeMapper.createEgeriaSchemaAttribute(SchemaAttributeMapper.java:200) ~[evtschema-1.0-SNAPSHOT.jar:na]
    at org.odpi.openmetadata.adapters.connectors.integration.eventschema.mapper.SchemaAttributeMapper.map(SchemaAttributeMapper.java:226) ~[evtschema-1.0-SNAPSHOT.jar:na]
    at org.odpi.openmetadata.adapters.connectors.integration.eventschema.EventSchemaIntegrationConnector.addSchema(EventSchemaIntegrationConnector.java:168) ~[evtschema-1.0-SNAPSHOT.jar:na]
    at org.odpi.openmetadata.adapters.connectors.integration.eventschema.EventSchemaIntegrationConnector.getSchemaRegistryContent(EventSchemaIntegrationConnector.java:139) ~[evtschema-1.0-SNAPSHOT.jar:na]
    at org.odpi.openmetadata.adapters.connectors.integration.eventschema.EventSchemaIntegrationConnector.refresh(EventSchemaIntegrationConnector.java:109) ~[evtschema-1.0-SNAPSHOT.jar:na]
    at org.odpi.openmetadata.governanceservers.integrationdaemonservices.handlers.IntegrationConnectorHandler.refreshConnector(IntegrationConnectorHandler.java:487) ~[integration-daemon-services-server-3.10.jar!/:na]
    at org.odpi.openmetadata.governanceservers.integrationdaemonservices.threads.IntegrationDaemonThread.run(IntegrationDaemonThread.java:106) ~[integration-daemon-services-server-3.10.jar!/:na]
    at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

PropertyServerException{reportedHTTPCode=404, reportingClassName='org.odpi.openmetadata.commonservices.ffdc.RESTExceptionHandler', reportingActionDescription='updateSchemaAttribute', reportedErrorMessage='OMAG-REPOSITORY-HANDLER-404-018 Method updateSchemaAttribute for service Data Manager OMAS is not supported by any of the metadata repositories connected to mds1', reportedErrorMessageId='OMAG-REPOSITORY-HANDLER-404-018', reportedErrorMessageParameters=[updateSchemaAttribute, Data Manager OMAS, mds1], reportedSystemAction='The system is unable to process the request because none of the members of the connected cohort(s) support this function.', reportedUserAction='Add an Egeria native metadata repository to one of the connected cohorts.  This will provide the support that you need.', reportedCaughtException=null, reportedCaughtExceptionClassName='null', relatedProperties=null}
    at org.odpi.openmetadata.commonservices.ffdc.RESTExceptionHandler.throwPropertyServerException(RESTExceptionHandler.java:312)
    at org.odpi.openmetadata.commonservices.ffdc.RESTExceptionHandler.detectAndThrowStandardExceptions(RESTExceptionHandler.java:155)
    at org.odpi.openmetadata.commonservices.ffdc.rest.FFDCRESTClient.callVoidPostRESTCall(FFDCRESTClient.java:309)
    at org.odpi.openmetadata.accessservices.datamanager.client.SchemaManagerClient.updateSchemaAttribute(SchemaManagerClient.java:1260)
    at org.odpi.openmetadata.integrationservices.topic.connector.TopicIntegratorContext.updateSchemaAttribute(TopicIntegratorContext.java:1046)
    at org.odpi.openmetadata.adapters.connectors.integration.eventschema.mapper.SchemaAttributeMapper.createEgeriaSchemaAttribute(SchemaAttributeMapper.java:200)
    at org.odpi.openmetadata.adapters.connectors.integration.eventschema.mapper.SchemaAttributeMapper.map(SchemaAttributeMapper.java:226)
    at org.odpi.openmetadata.adapters.connectors.integration.eventschema.EventSchemaIntegrationConnector.addSchema(EventSchemaIntegrationConnector.java:168)
    at org.odpi.openmetadata.adapters.connectors.integration.eventschema.EventSchemaIntegrationConnector.getSchemaRegistryContent(EventSchemaIntegrationConnector.java:139)
    at org.odpi.openmetadata.adapters.connectors.integration.eventschema.EventSchemaIntegrationConnector.refresh(EventSchemaIntegrationConnector.java:109)
    at org.odpi.openmetadata.governanceservers.integrationdaemonservices.handlers.IntegrationConnectorHandler.refreshConnector(IntegrationConnectorHandler.java:487)
    at org.odpi.openmetadata.governanceservers.integrationdaemonservices.threads.IntegrationDaemonThread.run(IntegrationDaemonThread.java:106)
    at java.base/java.lang.Thread.run(Thread.java:829)

Hope this helps.

mandy-chessell commented 1 year ago

Sorry, I was not clear - we need the audit log from the metadata server where XTDB is located.

juergenhemelt commented 1 year ago

Here it is:

Fri Feb 10 13:10:13 GMT 2023 mds1 Event OMRS-AUDIT-8009 The Open Metadata Repository Services (OMRS) has sent event of type Instance Event to the cohort topic mds1.openmetadata.repositoryservices.enterprise.mds1.OMRSTopic
Fri Feb 10 13:10:13 GMT 2023 mds1 Event OMRS-AUDIT-8009 The Open Metadata Repository Services (OMRS) has sent event of type Instance Event to the cohort topic egeria.omag.openmetadata.repositoryservices.cohort.mds.OMRSTopic.instances
Fri Feb 10 13:10:13 GMT 2023 mds1 Error OMAG-REPOSITORY-HANDLER-0010 Method reclassifyEntity(LatestChange) called from updateSchemaAttribute for service Data Manager OMAS is using function that not supported by any of the metadata repositories connected to mds1 - error message is: OMRS-METADATA-COLLECTION-501-001 OMRSMetadataInstanceStore method updateEntityProxyClassification for OMRS Connector org.odpi.egeria.connectors.juxt.xtdb.repositoryconnector.XtdbOMRSMetadataCollection to repository type mds1 is not implemented
2023-02-10 13:10:13.012 ERROR 1 --- [nio-9443-exec-6] o.o.o.c.ffdc.RESTExceptionHandler        : Exception from updateSchemaAttribute being packaged for return on REST call

org.odpi.openmetadata.frameworks.connectors.ffdc.PropertyServerException: OMAG-REPOSITORY-HANDLER-404-018 Method updateSchemaAttribute for service Data Manager OMAS is not supported by any of the metadata repositories connected to mds1
    at org.odpi.openmetadata.commonservices.repositoryhandler.RepositoryErrorHandler.handleRepositoryError(RepositoryErrorHandler.java:759) ~[repository-handler-3.15.jar!/:na]
    at org.odpi.openmetadata.commonservices.repositoryhandler.RepositoryHandler.reclassifyEntity(RepositoryHandler.java:1649) ~[repository-handler-3.15.jar!/:na]
    at org.odpi.openmetadata.commonservices.generichandlers.OpenMetadataAPIGenericHandler.addLatestChangeToAnchor(OpenMetadataAPIGenericHandler.java:3646) ~[generic-handlers-3.15.jar!/:na]
    at org.odpi.openmetadata.commonservices.generichandlers.OpenMetadataAPIGenericHandler.updateBeanInRepository(OpenMetadataAPIGenericHandler.java:6372) ~[generic-handlers-3.15.jar!/:na]
    at org.odpi.openmetadata.commonservices.generichandlers.OpenMetadataAPIGenericHandler.updateBeanInRepository(OpenMetadataAPIGenericHandler.java:6224) ~[generic-handlers-3.15.jar!/:na]
    at org.odpi.openmetadata.commonservices.generichandlers.SchemaAttributeHandler.updateSchemaAttribute(SchemaAttributeHandler.java:1820) ~[generic-handlers-3.15.jar!/:na]
    at org.odpi.openmetadata.accessservices.datamanager.server.SchemaManagerRESTServices.updateSchemaAttribute(SchemaManagerRESTServices.java:1834) ~[data-manager-server-3.15.jar!/:na]
    at org.odpi.openmetadata.accessservices.datamanager.server.spring.SchemaManagerResource.updateSchemaAttribute(SchemaManagerResource.java:626) ~[data-manager-spring-3.15.jar!/:na]
    at jdk.internal.reflect.GeneratedMethodAccessor438.invoke(Unknown Source) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-5.3.24.jar!/:5.3.24]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) ~[spring-web-5.3.24.jar!/:5.3.24]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) ~[spring-webmvc-5.3.24.jar!/:5.3.24]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) ~[spring-webmvc-5.3.24.jar!/:5.3.24]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.24.jar!/:5.3.24]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.24.jar!/:5.3.24]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1071) ~[spring-webmvc-5.3.24.jar!/:5.3.24]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:964) ~[spring-webmvc-5.3.24.jar!/:5.3.24]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.3.24.jar!/:5.3.24]
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) ~[spring-webmvc-5.3.24.jar!/:5.3.24]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:696) ~[tomcat-embed-core-9.0.70.jar!/:4.0.1]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.24.jar!/:5.3.24]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:779) ~[tomcat-embed-core-9.0.70.jar!/:4.0.1]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.70.jar!/:na]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.odpi.openmetadata.http.HttpRequestHeadersFilter.doFilter(HttpRequestHeadersFilter.java:52) ~[http-helper-3.15.jar!/:na]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.24.jar!/:5.3.24]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.24.jar!/:5.3.24]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.3.24.jar!/:5.3.24]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.24.jar!/:5.3.24]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) ~[spring-boot-actuator-2.7.7.jar!/:2.7.7]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.24.jar!/:5.3.24]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.3.24.jar!/:5.3.24]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.24.jar!/:5.3.24]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:177) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:769) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:891) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1784) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.70.jar!/:na]
    at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]
cmgrote commented 1 year ago

Interesting — looks like I missed the separate update method for an entity proxy... Will add it to my backlog.

cmgrote commented 1 year ago

Actually looks like multiple new methods I missed:

@mandy-chessell — in case there are any others that jump to mind that I might have missed? 🙏

cmgrote commented 1 year ago

@mandy-chessell Am I correct to assume that the expected behavior of saveClassificationReferenceCopy is to create the entity if it does not exist? (As it doesn't look like we allow an EntityNotKnownException to be thrown by either the EntityDetail-based or EntityProxy-based methods)?

Looking at the in-memory repository implementation there seems to be different behavior between the two methods:

^^ Just want to confirm that's intentional?

mandy-chessell commented 1 year ago

Simple answer is yes (if the repository is saving reference copies of that type). So if only the entity proxy is passed (as in the new methods) then if the repo wants to save the classification, it saves the entity proxy first to provide an attachment point for the classification.

The reason we needed the new methods stems from the lack of OMRS events from IGC. This meant that no reference copies for IGC metadata were saved in any other member of the cohort. It was not safe to push the reference copy of the entity via the enterprise connector because the lack of a delete event from IGC could resulted in entities being returned after they had been deleted. Without reference copies, we could not save new classifications in these other repositories for IGC entities because the reference copy for the entity was not available. We needed to be able to save the entity proxy so we could attach the classification to it. (Entity proxies are ok to save because they are not retrieved by getEntityDetail()).

Unfortunately, this then required new OMRS event types to send details of the new classification passing the entity proxy rather than the entity detail (which of course is not available.)

cmgrote commented 1 year ago

These methods should all be implemented now in the latest snapshot. I've also run them through the CTS (locally), but I'm not sure that we have test coverage of them yet judging by the Store classifications independently of entity profile having an UNKNOWN_STATUS (?) (I checked the public runs we have of other repos like in-memory, and it looks like this is the same for those...)

@juergenhemelt any chance you'd be open to retrying your tests with the latest snapshot build here? I just want to have some level of confidence this has actually solved the problem before I cut the release 🙏

You can download the jar file containing the current latest build directly from this URL: https://oss.sonatype.org/content/repositories/snapshots/org/odpi/egeria/egeria-connector-xtdb/3.15-SNAPSHOT/egeria-connector-xtdb-3.15-20230212.225744-2-jar-with-dependencies.jar

mandy-chessell commented 1 year ago

@cmgrote do you have getHomeClassifications implemented? This was added to retrieve the classifications homed in a different repository to the entity

juergenhemelt commented 1 year ago

@cmgrote I downloaded and installed the JAR. After a restart of the platform the exceptions do not appear any more. So looks good from my site 👍

cmgrote commented 1 year ago

do you have getHomeClassifications implemented? This was added to retrieve the classifications homed in a different repository to the entity

Yes, List<Classification> getHomeClassifications(String, String) was already implemented in the connector. (Actually I just noticed there's a point-in-time variant, which wasn't implemented, so just implementing that now as well.)