DataStage Proxy does not pick up updated processes

[popa-raluca]

Can be reproduced with the minimal flows - https://github.com/odpi/egeria/blob/master/open-metadata-resources/open-metadata-deployment/sample-data/minimal/README.md

The IGC governance rule is created when we run flows 1-3 When trying to deploy flow 4 separately, the updates from flow 4 are not picked up by DataStage proxy. Only manually removing the rule from IGC will trigger the processing for flow 4

[popa-raluca]

I attached the debug log for DataStage proxy. It looks like it finds some changes for the lineage mappings, but it sends an empty payload to DE OMAS

dsproxy-lineageMappings-changes.txt

[cmgrote]

I haven't been able to reproduce this on my side with the latest egeria-feature-openlineage branch + master of the connector (including PR #350).

Instead, I see it failing at flow3 due to what appear to be a number of exceptions related to ProcessPorts (?) These also show up at flow2 (though not at flow1); however, they do not seem to block / put flow2 into an infinite retry loop (while for flow3 they do appear to enter into an infinite retry loop):

2020-07-16 12:51:00.574 ERROR 22918 --- [RSTopicListener] o.o.o.a.a.h.ProcessContextHandler        : No relationships Process Port has been found for the entity with guid e4f957a1-4cb6-45cd-91d1-cbc84fffbe57
2020-07-16 12:51:00.584 ERROR 22918 --- [RSTopicListener] o.o.o.f.c.ffdc.OCFCheckedExceptionBase   : Null error message Id passed to an exception
2020-07-16 12:51:00.659 ERROR 22918 --- [RSTopicListener] .o.o.a.a.l.AssetLineageOMRSTopicListener : The following exception occurred:
OCFCheckedExceptionBase{reportedHTTPCode=404, reportingClassName='org.odpi.openmetadata.accessservices.assetlineage.handlers.ProcessContextHandler', reportingActionDescription='Retrieving Relationship', reportedErrorMessage='Cannot get the relationships from repository, relationship does not exist.', reportedErrorMessageId='null', reportedErrorMessageParameters=null, reportedSystemAction='The system is unable to get the relationships for the queried entity.', reportedUserAction='Check if the relationship queried is available on the repository in the cohort.', reportedCaughtException=null, reportedCaughtExceptionClassName='null', relatedProperties=null}

While processing OMRSTopic event:
OMRSInstanceEvent{instanceEventType=OMRSInstanceEventType{ordinal=1, name='NewEntityEvent', description='A new entity has been created.'}, typeDefGUID='d8f33bd7-afa9-4a11-a8c7-07dcec83c050', typeDefName='Process', instanceGUID='e4f957a1-4cb6-45cd-91d1-cbc84fffbe57', originalEntity=null, entity=EntityDetail{type=InstanceType{typeDefName='Process', typeDefCategory=TypeDefCategory{typeCode=6, typeName='EntityDef', typeDescription='An object or concept of interest.'}, typeDefGUID='d8f33bd7-afa9-4a11-a8c7-07dcec83c050', typeDefVersion=1, typeDefDescription='Well-defined sequence of activities performed by people or software components.', typeDefDescriptionGUID='null', typeDefSuperTypes=[TypeDefLink{guid='896d14c2-7522-4f6c-8519-757711943fe6', name='Asset', status=TypeDefStatus{ordinal=1, name='ActiveTypeDef', description='TypeDef available and in use.  This is the default value equivalent to null'}, replacedByTypeGUID='null', replacedByTypeName='null'}, TypeDefLink{guid='a32316b8-dc8c-48c5-b12b-71c1b2a080bf', name='Referenceable', status=TypeDefStatus{ordinal=1, name='ActiveTypeDef', description='TypeDef available and in use.  This is the default value equivalent to null'}, replacedByTypeGUID='null', replacedByTypeName='null'}], validStatusList=null, validInstanceProperties=[qualifiedName, additionalProperties, name, description, owner, ownerType, zoneMembership, latestChange, displayName, formula]}, entityProperties=InstanceProperties{propertyNames=java.util.LinkedHashMap$LinkedKeyIterator@6d83095b, propertyCount=4, instanceProperties={name=PrimitivePropertyValue{primitiveValue=Database_Output, primitiveDefCategory=PrimitiveDefCategory{code=11, name='string', javaClassName='java.lang.String', guid='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4'}, instancePropertyCategory=InstancePropertyCategory{typeCode=1, typeName='Primitive', typeDescription='A primitive type.'}, typeGUID='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4', typeName='string'}, description=PrimitivePropertyValue{primitiveValue=, primitiveDefCategory=PrimitiveDefCategory{code=11, name='string', javaClassName='java.lang.String', guid='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4'}, instancePropertyCategory=InstancePropertyCategory{typeCode=1, typeName='Primitive', typeDescription='A primitive type.'}, typeGUID='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4', typeName='string'}, qualifiedName=PrimitivePropertyValue{primitiveValue=(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow2::(stage)=Database_Output, primitiveDefCategory=PrimitiveDefCategory{code=11, name='string', javaClassName='java.lang.String', guid='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4'}, instancePropertyCategory=InstancePropertyCategory{typeCode=1, typeName='Primitive', typeDescription='A primitive type.'}, typeGUID='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4', typeName='string'}, displayName=PrimitivePropertyValue{primitiveValue=Database_Output, primitiveDefCategory=PrimitiveDefCategory{code=11, name='string', javaClassName='java.lang.String', guid='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4'}, instancePropertyCategory=InstancePropertyCategory{typeCode=1, typeName='Primitive', typeDescription='A primitive type.'}, typeGUID='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4', typeName='string'}}}, classifications=null, instanceURL='null', GUID='e4f957a1-4cb6-45cd-91d1-cbc84fffbe57', instanceProvenanceType=InstanceProvenanceType{ordinal=6, name='External Source', description='The instance is maintained by an external technology.  The metadata collection id is the guid of the technology's descriptive entity.'}, metadataCollectionId='85761afa-3dc3-4947-bdc2-7a772a98dacc', metadataCollectionName='ibm-datastage@infosvr:9446', instanceLicense='null', status=InstanceStatus{ordinal=1, name='Draft', description='The content is incomplete.'}, createdBy='OMAGServer', updatedBy='null', createTime=Thu Jul 16 12:50:56 BST 2020, updateTime=null, version=1, statusOnDelete=null}, originalRelationship=null, relationship=null, instanceBatch=null, homeMetadataCollectionId='null', originalHomeMetadataCollectionId='null', originalTypeDefSummary=null, originalInstanceGUID='null', errorCode=OMRSInstanceEventErrorCode{ordinal=0, name='No Error', description='There has been no error detected and so the error code is not in use.', encoding=null}, eventTimestamp=Thu Jul 16 12:50:56 BST 2020, eventDirection=OMRSEventDirection{ordinal=1, name='Inbound Event ', description='Event from a remote member of the open metadata repository cluster.'}, eventCategory=OMRSEventCategory{ordinal=3, name='Instance Event', description='Event used to manage the replication of metadata instances within the metadata repository cohort'}, eventOriginator=OMRSEventOriginator{metadataCollectionId='8795350c-1cec-41d0-bc3c-49a0c23b20f5', serverName='omas', serverType='Open Metadata and Governance Server', organizationName='null'}, genericErrorCode=null, errorMessage='null', targetMetadataCollectionId='null', targetRemoteConnection=null, targetTypeDefSummary=null, targetAttributeTypeDef=null, targetInstanceGUID='null', otherOrigin=null, otherMetadataCollectionId='null', otherTypeDefSummary=null, otherTypeDef=null, otherAttributeTypeDef=null, otherInstanceGUID='null', OMRSEventV1=org.odpi.openmetadata.repositoryservices.events.beans.v1.OMRSEventV1@3b1caa71}

org.odpi.openmetadata.accessservices.assetlineage.ffdc.exception.AssetLineageException: Cannot get the relationships from repository, relationship does not exist.
        at org.odpi.openmetadata.accessservices.assetlineage.handlers.ProcessContextHandler.hasEntitiesLinkedWithProcessPort(ProcessContextHandler.java:123) ~[asset-lineage-server-1.8-SNAPSHOT.jar!/:na]
        at org.odpi.openmetadata.accessservices.assetlineage.handlers.ProcessContextHandler.checkIfAllRelationshipsExist(ProcessContextHandler.java:96) ~[asset-lineage-server-1.8-SNAPSHOT.jar!/:na]
        at org.odpi.openmetadata.accessservices.assetlineage.handlers.ProcessContextHandler.getProcessContext(ProcessContextHandler.java:91) ~[asset-lineage-server-1.8-SNAPSHOT.jar!/:na]
        at org.odpi.openmetadata.accessservices.assetlineage.outtopic.AssetLineagePublisher.publishProcessContext(AssetLineagePublisher.java:77) ~[asset-lineage-server-1.8-SNAPSHOT.jar!/:na]
        at org.odpi.openmetadata.accessservices.assetlineage.listeners.AssetLineageOMRSTopicListener.processNewEntity(AssetLineageOMRSTopicListener.java:171) ~[asset-lineage-server-1.8-SNAPSHOT.jar!/:na]
        at org.odpi.openmetadata.accessservices.assetlineage.listeners.AssetLineageOMRSTopicListener.processInstanceEvent(AssetLineageOMRSTopicListener.java:126) ~[asset-lineage-server-1.8-SNAPSHOT.jar!/:na]
        at org.odpi.openmetadata.repositoryservices.connectors.omrstopic.OMRSTopicListenerWrapper.processInstanceEvent(OMRSTopicListenerWrapper.java:146) [repository-services-apis-1.8-SNAPSHOT.jar!/:na]
        at org.odpi.openmetadata.repositoryservices.connectors.omrstopic.OMRSTopicConnector.processOMRSEvent(OMRSTopicConnector.java:508) [repository-services-apis-1.8-SNAPSHOT.jar!/:na]
        at org.odpi.openmetadata.repositoryservices.connectors.omrstopic.OMRSTopicConnector.lambda$processEvent$0(OMRSTopicConnector.java:445) [repository-services-apis-1.8-SNAPSHOT.jar!/:na]
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) ~[na:1.8.0_252]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[na:1.8.0_252]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[na:1.8.0_252]
        at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) ~[na:1.8.0_252]
        at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) ~[na:1.8.0_252]
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[na:1.8.0_252]
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) ~[na:1.8.0_252]
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) ~[na:1.8.0_252]
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) ~[na:1.8.0_252]

Thu Jul 16 12:51:00 BST 2020 omas Exception OMAS-ASSET-LINEAGE-0005 An exception occurred while processing incoming event OMRSInstanceEvent{instanceEventType=OMRSInstanceEventType{ordinal=1, name='NewEntityEvent', description='A new entity has been created.'}, typeDefGUID='d8f33bd7-afa9-4a11-a8c7-07dcec83c050', typeDefName='Process', instanceGUID='e4f957a1-4cb6-45cd-91d1-cbc84fffbe57', originalEntity=null, entity=EntityDetail{type=InstanceType{typeDefName='Process', typeDefCategory=TypeDefCategory{typeCode=6, typeName='EntityDef', typeDescription='An object or concept of interest.'}, typeDefGUID='d8f33bd7-afa9-4a11-a8c7-07dcec83c050', typeDefVersion=1, typeDefDescription='Well-defined sequence of activities performed by people or software components.', typeDefDescriptionGUID='null', typeDefSuperTypes=[TypeDefLink{guid='896d14c2-7522-4f6c-8519-757711943fe6', name='Asset', status=TypeDefStatus{ordinal=1, name='ActiveTypeDef', description='TypeDef available and in use.  This is the default value equivalent to null'}, replacedByTypeGUID='null', replacedByTypeName='null'}, TypeDefLink{guid='a32316b8-dc8c-48c5-b12b-71c1b2a080bf', name='Referenceable', status=TypeDefStatus{ordinal=1, name='ActiveTypeDef', description='TypeDef available and in use.  This is the default value equivalent to null'}, replacedByTypeGUID='null', replacedByTypeName='null'}], validStatusList=null, validInstanceProperties=[qualifiedName, additionalProperties, name, description, owner, ownerType, zoneMembership, latestChange, displayName, formula]}, entityProperties=InstanceProperties{propertyNames=java.util.LinkedHashMap$LinkedKeyIterator@797d79fb, propertyCount=4, instanceProperties={name=PrimitivePropertyValue{primitiveValue=Database_Output, primitiveDefCategory=PrimitiveDefCategory{code=11, name='string', javaClassName='java.lang.String', guid='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4'}, instancePropertyCategory=InstancePropertyCategory{typeCode=1, typeName='Primitive', typeDescription='A primitive type.'}, typeGUID='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4', typeName='string'}, description=PrimitivePropertyValue{primitiveValue=, primitiveDefCategory=PrimitiveDefCategory{code=11, name='string', javaClassName='java.lang.String', guid='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4'}, instancePropertyCategory=InstancePropertyCategory{typeCode=1, typeName='Primitive', typeDescription='A primitive type.'}, typeGUID='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4', typeName='string'}, qualifiedName=PrimitivePropertyValue{primitiveValue=(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow2::(stage)=Database_Output, primitiveDefCategory=PrimitiveDefCategory{code=11, name='string', javaClassName='java.lang.String', guid='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4'}, instancePropertyCategory=InstancePropertyCategory{typeCode=1, typeName='Primitive', typeDescription='A primitive type.'}, typeGUID='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4', typeName='string'}, displayName=PrimitivePropertyValue{primitiveValue=Database_Output, primitiveDefCategory=PrimitiveDefCategory{code=11, name='string', javaClassName='java.lang.String', guid='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4'}, instancePropertyCategory=InstancePropertyCategory{typeCode=1, typeName='Primitive', typeDescription='A primitive type.'}, typeGUID='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4', typeName='string'}}}, classifications=null, instanceURL='null', GUID='e4f957a1-4cb6-45cd-91d1-cbc84fffbe57', instanceProvenanceType=InstanceProvenanceType{ordinal=6, name='External Source', description='The instance is maintained by an external technology.  The metadata collection id is the guid of the technology's descriptive entity.'}, metadataCollectionId='85761afa-3dc3-4947-bdc2-7a772a98dacc', metadataCollectionName='ibm-datastage@infosvr:9446', instanceLicense='null', status=InstanceStatus{ordinal=1, name='Draft', description='The content is incomplete.'}, createdBy='OMAGServer', updatedBy='null', createTime=Thu Jul 16 12:50:56 BST 2020, updateTime=null, version=1, statusOnDelete=null}, originalRelationship=null, relationship=null, instanceBatch=null, homeMetadataCollectionId='null', originalHomeMetadataCollectionId='null', originalTypeDefSummary=null, originalInstanceGUID='null', errorCode=OMRSInstanceEventErrorCode{ordinal=0, name='No Error', description='There has been no error detected and so the error code is not in use.', encoding=null}, eventTimestamp=Thu Jul 16 12:50:56 BST 2020, eventDirection=OMRSEventDirection{ordinal=1, name='Inbound Event ', description='Event from a remote member of the open metadata repository cluster.'}, eventCategory=OMRSEventCategory{ordinal=3, name='Instance Event', description='Event used to manage the replication of metadata instances within the metadata repository cohort'}, eventOriginator=OMRSEventOriginator{metadataCollectionId='8795350c-1cec-41d0-bc3c-49a0c23b20f5', serverName='omas', serverType='Open Metadata and Governance Server', organizationName='null'}, genericErrorCode=null, errorMessage='null', targetMetadataCollectionId='null', targetRemoteConnection=null, targetTypeDefSummary=null, targetAttributeTypeDef=null, targetInstanceGUID='null', otherOrigin=null, otherMetadataCollectionId='null', otherTypeDefSummary=null, otherTypeDef=null, otherAttributeTypeDef=null, otherInstanceGUID='null', OMRSEventV1=org.odpi.openmetadata.repositoryservices.events.beans.v1.OMRSEventV1@2144d526}
Thu Jul 16 12:51:00 BST 2020 omas Exception OMAS-ASSET-LINEAGE-0005 Supplementary information: log record id d47eed8d-6a35-472c-bdb0-4edd62bbd367 org.odpi.openmetadata.accessservices.assetlineage.ffdc.exception.AssetLineageException returned message of Cannot get the relationships from repository, relationship does not exist. and stacktrace of
OCFCheckedExceptionBase{reportedHTTPCode=404, reportingClassName='org.odpi.openmetadata.accessservices.assetlineage.handlers.ProcessContextHandler', reportingActionDescription='Retrieving Relationship', reportedErrorMessage='Cannot get the relationships from repository, relationship does not exist.', reportedErrorMessageId='null', reportedErrorMessageParameters=null, reportedSystemAction='The system is unable to get the relationships for the queried entity.', reportedUserAction='Check if the relationship queried is available on the repository in the cohort.', reportedCaughtException=null, reportedCaughtExceptionClassName='null', relatedProperties=null}
        at org.odpi.openmetadata.accessservices.assetlineage.handlers.ProcessContextHandler.hasEntitiesLinkedWithProcessPort(ProcessContextHandler.java:123)
        at org.odpi.openmetadata.accessservices.assetlineage.handlers.ProcessContextHandler.checkIfAllRelationshipsExist(ProcessContextHandler.java:96)
        at org.odpi.openmetadata.accessservices.assetlineage.handlers.ProcessContextHandler.getProcessContext(ProcessContextHandler.java:91)
        at org.odpi.openmetadata.accessservices.assetlineage.outtopic.AssetLineagePublisher.publishProcessContext(AssetLineagePublisher.java:77)
        at org.odpi.openmetadata.accessservices.assetlineage.listeners.AssetLineageOMRSTopicListener.processNewEntity(AssetLineageOMRSTopicListener.java:171)
        at org.odpi.openmetadata.accessservices.assetlineage.listeners.AssetLineageOMRSTopicListener.processInstanceEvent(AssetLineageOMRSTopicListener.java:126)
        at org.odpi.openmetadata.repositoryservices.connectors.omrstopic.OMRSTopicListenerWrapper.processInstanceEvent(OMRSTopicListenerWrapper.java:146)
        at org.odpi.openmetadata.repositoryservices.connectors.omrstopic.OMRSTopicConnector.processOMRSEvent(OMRSTopicConnector.java:508)
        at org.odpi.openmetadata.repositoryservices.connectors.omrstopic.OMRSTopicConnector.lambda$processEvent$0(OMRSTopicConnector.java:445)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
        at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
        at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

Any idea if there are changes / fixes to this problem that might simply not yet be merged back into egeria-feature-openlineage branch?

[cmgrote]

Is this still an issue in the latest merged feature branch? I was not able to reproduce it there.

[popa-raluca]

I think we still need to test this. We'll update the issue afterward accordingly

[popa-raluca]

Closing this issue, as is not really a problem with the governance rule. We need to review the scenario for testing the update at some point, maybe using a DataStge Designer client for updating the processes