ProcessHierarchy ordering issue?

[cmgrote]

There is some underlying issue with the ProcessHierarchy processing of the DE OMAS -- unclear whether it's related to ordering, or potentially something with the underlying persistence (eg. the graph repository).

When attempting to load the new process hierarchy payloads produced by https://github.com/odpi/egeria-connector-ibm-information-server/pull/293, the loading eventually hits an exception that blocks any further progress:

2020-04-02 16:43:04.130 ERROR 9525 --- [      Thread-16] o.o.o.c.ffdc.RESTExceptionHandler        : Detected Invalid Parameter Exception in REST Response

org.odpi.openmetadata.frameworks.connectors.ffdc.InvalidParameterException: OMAS-DATA-ENGINE-400-008 Process with qualifiedName (host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1 was not found
    at org.odpi.openmetadata.commonservices.ffdc.RESTExceptionHandler.throwInvalidParameterException(RESTExceptionHandler.java:213) [ffdc-services-1.7-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.commonservices.ffdc.RESTExceptionHandler.detectAndThrowInvalidParameterException(RESTExceptionHandler.java:130) [ffdc-services-1.7-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.commonservices.ffdc.RESTExceptionHandler.detectAndThrowInvalidParameterException(RESTExceptionHandler.java:146) [ffdc-services-1.7-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.accessservices.dataengine.client.DataEngineImpl.callProcessListPostRESTCall(DataEngineImpl.java:260) [data-engine-client-1.7-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.accessservices.dataengine.client.DataEngineImpl.createOrUpdateProcess(DataEngineImpl.java:113) [data-engine-client-1.7-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.governanceservers.dataengineproxy.processor.DataEngineProxyChangePoller.upsertProcesses(DataEngineProxyChangePoller.java:187) [data-engine-proxy-services-server-1.7-SNAPSHOT.jar!/:na]
    at org.odpi.openmetadata.governanceservers.dataengineproxy.processor.DataEngineProxyChangePoller.run(DataEngineProxyChangePoller.java:106) [data-engine-proxy-services-server-1.7-SNAPSHOT.jar!/:na]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_242]

... which implies that is is unable to find a job with a particular qualified name.

An example of the payloads that are actually sent across (in the order they are sent over -- note that the one listed as "not found" above is sent last, after the child processes that refer to this as their parent):

[
{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=Database_Output","displayName":"Database_Output","description":"","owner":"Administrator IIS","name":"Database_Output","portImplementations":[{"displayName":"EMPLNAME","qualifiedName":"(host_(engine))=INFOSVR::(data_connection)=MINIMAL::(database_schema)=db2inst1::(database_table)=EMPLNAME(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=Database_Output","updateSemantic":"REPLACE","type":"OUTPUT_PORT","schema":{"qualifiedName":"(host_(engine))=INFOSVR::(data_connection)=MINIMAL::(database_schema)=db2inst1::(database_table)=EMPLNAME(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=Database_Output","displayName":"EMPLNAME","author":"Administrator IIS","columns":[{"qualifiedName":"(host_(engine))=INFOSVR::(data_connection)=MINIMAL::(database_schema)=db2inst1::(database_table)=EMPLNAME::(database_column)=EMPID(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=Database_Output","displayName":"EMPID","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":0},{"qualifiedName":"(host_(engine))=INFOSVR::(data_connection)=MINIMAL::(database_schema)=db2inst1::(database_table)=EMPLNAME::(database_column)=FNAME(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=Database_Output","displayName":"FNAME","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":0},{"qualifiedName":"(host_(engine))=INFOSVR::(data_connection)=MINIMAL::(database_schema)=db2inst1::(database_table)=EMPLNAME::(database_column)=LOCID(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=Database_Output","displayName":"LOCID","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":0},{"qualifiedName":"(host_(engine))=INFOSVR::(data_connection)=MINIMAL::(database_schema)=db2inst1::(database_table)=EMPLNAME::(database_column)=SURNAME(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=Database_Output","displayName":"SURNAME","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":0}]}},{"displayName":"write_table","qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table_Database_Output","updateSemantic":"REPLACE","type":"INPUT_PORT","schema":{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table_Database_Output","displayName":"c2e76d84.78bf4d29.001n2era2.qa79oh0.lmnot0.gc3tsu1in7c8549p34van","author":"Administrator IIS","columns":[{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=FNAME_Database_Output","displayName":"FNAME","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":0,"dataType":"VARCHAR"},{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=LOCID_Database_Output","displayName":"LOCID","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":1,"dataType":"INTEGER"},{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=EMPID_Database_Output","displayName":"EMPID","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":2,"dataType":"INTEGER"},{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=SURNAME_Database_Output","displayName":"SURNAME","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":3,"dataType":"VARCHAR"}]}}],"lineageMappings":[{"sourceAttribute":"(host_(engine))=INFOSVR::(data_connection)=MINIMAL::(database_schema)=db2inst1::(database_table)=EMPLNAME::(database_column)=LOCID(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=Database_Output","targetAttribute":"(host_(engine))=INFOSVR::(data_connection)=MINIMAL::(database_schema)=db2inst1::(database_table)=EMPLNAME::(database_column)=LOCID"},{"sourceAttribute":"(host_(engine))=INFOSVR::(data_connection)=MINIMAL::(database_schema)=db2inst1::(database_table)=EMPLNAME::(database_column)=FNAME(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=Database_Output","targetAttribute":"(host_(engine))=INFOSVR::(data_connection)=MINIMAL::(database_schema)=db2inst1::(database_table)=EMPLNAME::(database_column)=FNAME"},{"sourceAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=FNAME_Database_Output","targetAttribute":"(host_(engine))=INFOSVR::(data_connection)=MINIMAL::(database_schema)=db2inst1::(database_table)=EMPLNAME::(database_column)=FNAME(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=Database_Output"},{"sourceAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=SURNAME_Database_Output","targetAttribute":"(host_(engine))=INFOSVR::(data_connection)=MINIMAL::(database_schema)=db2inst1::(database_table)=EMPLNAME::(database_column)=SURNAME(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=Database_Output"},{"sourceAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=EMPID_Database_Output","targetAttribute":"(host_(engine))=INFOSVR::(data_connection)=MINIMAL::(database_schema)=db2inst1::(database_table)=EMPLNAME::(database_column)=EMPID(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=Database_Output"},{"sourceAttribute":"(host_(engine))=INFOSVR::(data_connection)=MINIMAL::(database_schema)=db2inst1::(database_table)=EMPLNAME::(database_column)=SURNAME(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=Database_Output","targetAttribute":"(host_(engine))=INFOSVR::(data_connection)=MINIMAL::(database_schema)=db2inst1::(database_table)=EMPLNAME::(database_column)=SURNAME"},{"sourceAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=LOCID_Database_Output","targetAttribute":"(host_(engine))=INFOSVR::(data_connection)=MINIMAL::(database_schema)=db2inst1::(database_table)=EMPLNAME::(database_column)=LOCID(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=Database_Output"},{"sourceAttribute":"(host_(engine))=INFOSVR::(data_connection)=MINIMAL::(database_schema)=db2inst1::(database_table)=EMPLNAME::(database_column)=EMPID(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=Database_Output","targetAttribute":"(host_(engine))=INFOSVR::(data_connection)=MINIMAL::(database_schema)=db2inst1::(database_table)=EMPLNAME::(database_column)=EMPID"}],"updateSemantic":"REPLACE","parentProcesses":[{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1","containmentType":"OWNED"}]},
{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=File_Input","displayName":"File_Input","description":"","owner":"Administrator IIS","name":"File_Input","portImplementations":[{"displayName":"read_file","qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file_File_Input","updateSemantic":"REPLACE","type":"OUTPUT_PORT","schema":{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file_File_Input","displayName":"c2e76d84.78bf4d29.001n2era2.qa77kvs.vplgr4.88v8npq08psl63b03710d","author":"Administrator IIS","columns":[{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=Last_File_Input","displayName":"Last","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":0,"dataType":"VARCHAR"},{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=Location_File_Input","displayName":"Location","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":1,"dataType":"INTEGER"},{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=Id_File_Input","displayName":"Id","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":2,"dataType":"INTEGER"},{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=First_File_Input","displayName":"First","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":3,"dataType":"VARCHAR"}]}},{"displayName":"names","qualifiedName":"(host_(engine))=INFOSVR::(data_file_folder)=/::(data_file_folder)=data::(data_file_folder)=files::(data_file_folder)=minimal::(data_file)=names.csv::(data_file_record)=names(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=File_Input","updateSemantic":"REPLACE","type":"INPUT_PORT","schema":{"qualifiedName":"(host_(engine))=INFOSVR::(data_file_folder)=/::(data_file_folder)=data::(data_file_folder)=files::(data_file_folder)=minimal::(data_file)=names.csv::(data_file_record)=names(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=File_Input","displayName":"names","author":"Administrator IIS","columns":[{"qualifiedName":"(host_(engine))=INFOSVR::(data_file_folder)=/::(data_file_folder)=data::(data_file_folder)=files::(data_file_folder)=minimal::(data_file)=names.csv::(data_file_record)=names::(data_file_field)=First(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=File_Input","displayName":"First","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":2,"dataType":"STRING","defaultValue":""},{"qualifiedName":"(host_(engine))=INFOSVR::(data_file_folder)=/::(data_file_folder)=data::(data_file_folder)=files::(data_file_folder)=minimal::(data_file)=names.csv::(data_file_record)=names::(data_file_field)=Id(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=File_Input","displayName":"Id","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":1,"dataType":"INT64","defaultValue":""},{"qualifiedName":"(host_(engine))=INFOSVR::(data_file_folder)=/::(data_file_folder)=data::(data_file_folder)=files::(data_file_folder)=minimal::(data_file)=names.csv::(data_file_record)=names::(data_file_field)=Last(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=File_Input","displayName":"Last","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":3,"dataType":"STRING","defaultValue":""},{"qualifiedName":"(host_(engine))=INFOSVR::(data_file_folder)=/::(data_file_folder)=data::(data_file_folder)=files::(data_file_folder)=minimal::(data_file)=names.csv::(data_file_record)=names::(data_file_field)=Location(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=File_Input","displayName":"Location","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":4,"dataType":"INT64","defaultValue":""}]}}],"lineageMappings":[{"sourceAttribute":"(host_(engine))=INFOSVR::(data_file_folder)=/::(data_file_folder)=data::(data_file_folder)=files::(data_file_folder)=minimal::(data_file)=names.csv::(data_file_record)=names::(data_file_field)=Location","targetAttribute":"(host_(engine))=INFOSVR::(data_file_folder)=/::(data_file_folder)=data::(data_file_folder)=files::(data_file_folder)=minimal::(data_file)=names.csv::(data_file_record)=names::(data_file_field)=Location(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=File_Input"},{"sourceAttribute":"(host_(engine))=INFOSVR::(data_file_folder)=/::(data_file_folder)=data::(data_file_folder)=files::(data_file_folder)=minimal::(data_file)=names.csv::(data_file_record)=names::(data_file_field)=Id","targetAttribute":"(host_(engine))=INFOSVR::(data_file_folder)=/::(data_file_folder)=data::(data_file_folder)=files::(data_file_folder)=minimal::(data_file)=names.csv::(data_file_record)=names::(data_file_field)=Id(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=File_Input"},{"sourceAttribute":"(host_(engine))=INFOSVR::(data_file_folder)=/::(data_file_folder)=data::(data_file_folder)=files::(data_file_folder)=minimal::(data_file)=names.csv::(data_file_record)=names::(data_file_field)=Last","targetAttribute":"(host_(engine))=INFOSVR::(data_file_folder)=/::(data_file_folder)=data::(data_file_folder)=files::(data_file_folder)=minimal::(data_file)=names.csv::(data_file_record)=names::(data_file_field)=Last(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=File_Input"},{"sourceAttribute":"(host_(engine))=INFOSVR::(data_file_folder)=/::(data_file_folder)=data::(data_file_folder)=files::(data_file_folder)=minimal::(data_file)=names.csv::(data_file_record)=names::(data_file_field)=Id(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=File_Input","targetAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=Id_File_Input"},{"sourceAttribute":"(host_(engine))=INFOSVR::(data_file_folder)=/::(data_file_folder)=data::(data_file_folder)=files::(data_file_folder)=minimal::(data_file)=names.csv::(data_file_record)=names::(data_file_field)=Last(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=File_Input","targetAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=Last_File_Input"},{"sourceAttribute":"(host_(engine))=INFOSVR::(data_file_folder)=/::(data_file_folder)=data::(data_file_folder)=files::(data_file_folder)=minimal::(data_file)=names.csv::(data_file_record)=names::(data_file_field)=Location(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=File_Input","targetAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=Location_File_Input"},{"sourceAttribute":"(host_(engine))=INFOSVR::(data_file_folder)=/::(data_file_folder)=data::(data_file_folder)=files::(data_file_folder)=minimal::(data_file)=names.csv::(data_file_record)=names::(data_file_field)=First","targetAttribute":"(host_(engine))=INFOSVR::(data_file_folder)=/::(data_file_folder)=data::(data_file_folder)=files::(data_file_folder)=minimal::(data_file)=names.csv::(data_file_record)=names::(data_file_field)=First(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=File_Input"},{"sourceAttribute":"(host_(engine))=INFOSVR::(data_file_folder)=/::(data_file_folder)=data::(data_file_folder)=files::(data_file_folder)=minimal::(data_file)=names.csv::(data_file_record)=names::(data_file_field)=First(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=File_Input","targetAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=First_File_Input"}],"updateSemantic":"REPLACE","parentProcesses":[{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1","containmentType":"OWNED"}]},
{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=Rename","displayName":"Rename","description":"","owner":"Administrator IIS","name":"Rename","portImplementations":[{"displayName":"write_table","qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table_Rename","updateSemantic":"REPLACE","type":"OUTPUT_PORT","schema":{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table_Rename","displayName":"c2e76d84.78bf4d29.001n2era2.qa79oh0.lmnot0.gc3tsu1in7c8549p34van","author":"Administrator IIS","columns":[{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=FNAME_Rename","displayName":"FNAME","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":0,"dataType":"VARCHAR"},{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=LOCID_Rename","displayName":"LOCID","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":1,"dataType":"INTEGER"},{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=EMPID_Rename","displayName":"EMPID","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":2,"dataType":"INTEGER"},{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=SURNAME_Rename","displayName":"SURNAME","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":3,"dataType":"VARCHAR"}]}},{"displayName":"read_file","qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file_Rename","updateSemantic":"REPLACE","type":"INPUT_PORT","schema":{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file_Rename","displayName":"c2e76d84.78bf4d29.001n2era2.qa77kvs.vplgr4.88v8npq08psl63b03710d","author":"Administrator IIS","columns":[{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=Last_Rename","displayName":"Last","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":0,"dataType":"VARCHAR"},{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=Location_Rename","displayName":"Location","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":1,"dataType":"INTEGER"},{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=Id_Rename","displayName":"Id","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":2,"dataType":"INTEGER"},{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=First_Rename","displayName":"First","minCardinality":0,"maxCardinality":0,"allowsDuplicateValues":false,"orderedValues":false,"position":3,"dataType":"VARCHAR"}]}}],"lineageMappings":[{"sourceAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=Location_Rename","targetAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=LOCID_Rename"},{"sourceAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=Last_Rename","targetAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=SURNAME_Rename"},{"sourceAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=Id_Rename","targetAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=EMPID_Rename"},{"sourceAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=First_Rename","targetAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=FNAME_Rename"}],"updateSemantic":"REPLACE","parentProcesses":[{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1","containmentType":"OWNED"}]},
{"qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1","displayName":"flow1","description":"","owner":"Administrator IIS","name":"flow1","portAliases":[{"displayName":"File_Input","qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=File_Input","updateSemantic":"REPLACE","delegatesTo":"(host_(engine))=INFOSVR::(data_file_folder)=/::(data_file_folder)=data::(data_file_folder)=files::(data_file_folder)=minimal::(data_file)=names.csv::(data_file_record)=names(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=File_Input","type":"INPUT_PORT"},{"displayName":"Database_Output","qualifiedName":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=Database_Output","updateSemantic":"REPLACE","delegatesTo":"(host_(engine))=INFOSVR::(data_connection)=MINIMAL::(database_schema)=db2inst1::(database_table)=EMPLNAME(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(stage)=Database_Output","type":"OUTPUT_PORT"}],"lineageMappings":[{"sourceAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=EMPID_Rename","targetAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=EMPID_Database_Output"},{"sourceAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=FNAME_Rename","targetAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=FNAME_Database_Output"},{"sourceAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=Last_File_Input","targetAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=Last_Rename"},{"sourceAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=Id_File_Input","targetAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=Id_Rename"},{"sourceAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=LOCID_Rename","targetAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=LOCID_Database_Output"},{"sourceAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=SURNAME_Rename","targetAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=write_table::(ds_stage_column)=SURNAME_Database_Output"},{"sourceAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=Location_File_Input","targetAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=Location_Rename"},{"sourceAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=First_File_Input","targetAttribute":"(host_(engine))=INFOSVR::(transformation_project)=minimal::(dsjob)=flow1::(link)=read_file::(ds_stage_column)=First_Rename"}],"updateSemantic":"REPLACE"}
]

... however, note that the last one sent does have a qualifiedName that matches the one listed in the error as "not found". (The error appears well after all the payloads are sent, but I'm not sure whether it's before or after they might all have been processed.)

[popa-raluca]

The ProcessHierarchy relationships are added after the processes are created, so I think that it failed to create the parent process that wasn't found(or it failed to create an entity/relationship inside that process). I will do some investigations to see what went wrong..do you have more log details for DE OMAS processing? Do you have the igc events for creating the external entities available? Or maybe the payloads for the external schema types - if they are treated as virtualAssets and created with DE OMAS?

[cmgrote]

Attached (minimal-flows.zip) -- note that I'd tested this using the option on the DataStage connector to send its data assets as schema types. This might be the problem, given that it's only sending the schematypes (and not any information about the assets themselves)?

If so, probably helpful to have the ability to create assets via DE OMAS for the DataStage connector as well (the same as the Bluemetrix chaps suggested 😉).

[popa-raluca]

DE OMAS only needs schema types, AL OMAS would need the assets in order to build the context. So no problem for me to test :)

I did the test locally and i didn't have any issues, but I had an array of processes that were created together. From what i saw in DataEngineProxyChangePoller, each process is created in a separate request:

List<Process> changedProcesses = connector.getChangedProcesses(changesLastSynced, changesCutoff);
        if (changedProcesses != null) {
            for (Process changedProcess : changedProcesses) {
                dataEngineOMASClient.createOrUpdateProcess(userId, changedProcess);
            }
            log.info(" ... completing process changes.");
        }

The logic for handling parent processes doesn't work with DS proxy because it is not sending all the processes together in a single request. After creating the first process, it will directly try to create the ProcessHierarchy relationships - and it will fail, because the request for the forth process was not done yet.

Modifying the code above to use dataEngineOMASClient.createOrUpdateProcesses(userId, changedProcesses) should fix the issue.

Also, it should improve the overall performance, since the processes would be created in parallel by DE OMAS.

[cmgrote]

Cool -- I can make that change hopefully later today.

However, is it safe for us to assume that all engines will be able to always send the processes together in an array like this?

[popa-raluca]

Maybe not. For this use case, I can add an endpoint just for adding the ProcessHierarchy relationships

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.