search

open-metadata / OpenMetadata

OpenMetadata is a unified metadata platform for data discovery, data observability, and data governance powered by a central metadata repository, in-depth column level lineage, and seamless team collaboration.
https://open-metadata.org
Apache License 2.0
5.59k stars 1.05k forks source link

Airflow override lineage option is not working properly #18678

Open mramirsha opened 3 days ago

mramirsha commented 3 days ago

Affected module It impacts the Ingestion Framework?

Describe the bug When we are enabling the override lineage option for airflow ingestion pipeline we are facing the following error: Airflow ingestion pipelines log:

Failed to ingest override_lineage=True lineage_request=AddLineageRequest(edge=EntitiesEdge(fromEntity=EntityReference(id=Uuid(root=UUID('fdcc19e4-cebc-4c31-a937-8e3c3bed3fd2')), type='table', name=None, fullyQualifiedName=None, description=None, displayName=None, deleted=None, inherited=None, href=None), toEntity=EntityReference(id=Uuid(root=UUID('b3d86767-7aef-4753-b819-4d3d39f188fa')), type='table', name=None, fullyQualifiedName=None, description=None, displayName=None, deleted=None, inherited=None, href=None), description=None, lineageDetails=LineageDetails(sqlQuery=None, columnsLineage=None, pipeline=EntityReference(id=Uuid(root=UUID('9b900446-5c4f-4199-9da7-3e310bd5d327')), type='pipeline', name=None, fullyQualifiedName=None, description=None, displayName=None, deleted=None, inherited=None, href=None), description=None, source=<Source.PipelineLineage: 'PipelineLineage'>))) due to api request failure: 500 Server Error: Internal Server Error for url: https://openmetadata.sophie.free-now.com/api/v1/lineage/pipeline/9b900446-5c4f-4199-9da7-3e310bd5d327/type/PipelineLineage

OpenMetadata server log:

ERROR [2024-11-18 12:58:33,969] [dw-334 - DELETE /api/v1/lineage/pipeline/ee120d9c-d9ed-4907-b45b-62e14db1c2c9/type/PipelineLineage] o.o.s.r.l.LineageResource - Got exception: [UnableToExecuteStatementException] / message [org.postgresql.util.PSQLException: ERROR: syntax error at or near "ORDER"

  | Nov 18, 2024 @ 13:58:33.970 | Position: 182 [statement:"/* EntityRelationshipDAO.deleteLineageBySourcePipeline */ DELETE FROM entity_relationship WHERE  json->'pipeline'->>'id' =:toId OR toId = :toId AND relation = :relation AND json->>'source' = :source ORDER BY toId", arguments:{positional:{1:pipeline,2:PipelineLineage,3:13}, named:{toId:ee120d9c-d9ed-4907-b45b-62e14db1c2c9,toEntity:pipeline,source:PipelineLineage,relation:13}, finder:[]}]

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.LazyHandleSupplier.invokeInContext(LazyHandleSupplier.java:75)

  | Nov 18, 2024 @ 13:58:33.970 | at jdk.proxy2/jdk.proxy2.$Proxy175.deleteLineageBySourcePipeline(Unknown Source)

  | Nov 18, 2024 @ 13:58:33.970 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:722)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.Server.handle(Server.java:516)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)

  | Nov 18, 2024 @ 13:58:33.970 | at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:194)

  | Nov 18, 2024 @ 13:58:33.970 | at jdk.internal.reflect.GeneratedMethodAccessor48.invoke(Unknown Source)

  | Nov 18, 2024 @ 13:58:33.970 | ... 94 common frames omitted

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.statement.Update.execute(Update.java:57)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.sqlobject.statement.internal.SqlUpdateHandler.invoke(SqlUpdateHandler.java:32)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.Jdbi.callWithExtension(Jdbi.java:502)

  | Nov 18, 2024 @ 13:58:33.970 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)

  | Nov 18, 2024 @ 13:58:33.970 | at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlets.HeaderFilter.doFilter(HeaderFilter.java:117)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)

  | Nov 18, 2024 @ 13:58:33.970 | at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:318)

  | Nov 18, 2024 @ 13:58:33.970 | at io.dropwizard.jetty.ZipExceptionHandlingGzipHandler.handle(ZipExceptionHandlingGzipHandler.java:26)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)

  | Nov 18, 2024 @ 13:58:33.970 | at jdk.proxy2/jdk.proxy2.$Proxy170.execute(Unknown Source)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.statement.Update.execute(Update.java:69)

  | Nov 18, 2024 @ 13:58:33.970 | at org.openmetadata.service.jdbi3.LineageRepository.deleteLineageBySource(LineageRepository.java:399)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.sqlobject.SqlObjectFactory.lambda$attach$2(SqlObjectFactory.java:108)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.internal.OnDemandExtensions.lambda$invoke$5(OnDemandExtensions.java:99)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)

  | Nov 18, 2024 @ 13:58:33.970 | at org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:118)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.statement.SqlStatement.internalExecute(SqlStatement.java:1789)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.sqlobject.internal.SqlObjectInitData$1.invoke(SqlObjectInitData.java:135)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.internal.OnDemandExtensions.invoke(OnDemandExtensions.java:99)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.internal.OnDemandExtensions.lambda$createProxy$2(OnDemandExtensions.java:83)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.internal.Errors.process(Errors.java:244)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181)

  | Nov 18, 2024 @ 13:58:33.970 | at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2412)

  | Nov 18, 2024 @ 13:58:33.970 | at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:419)

  | Nov 18, 2024 @ 13:58:33.970 | at java.base/java.lang.reflect.Method.invoke(Method.java:569)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.statement.SqlLoggerUtil.wrap(SqlLoggerUtil.java:31)

  | Nov 18, 2024 @ 13:58:33.970 | Position: 182 [statement:"/* EntityRelationshipDAO.deleteLineageBySourcePipeline */ DELETE FROM entity_relationship WHERE  json->'pipeline'->>'id' =:toId OR toId = :toId AND relation = :relation AND json->>'source' = :source ORDER BY toId", arguments:{positional:{1:pipeline,2:PipelineLineage,3:13}, named:{toId:ee120d9c-d9ed-4907-b45b-62e14db1c2c9,toEntity:pipeline,source:PipelineLineage,relation:13}, finder:[]}]] / related resource location: [org.openmetadata.service.resources.lineage.LineageResource.deleteLineageByType](LineageResource.java:475)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.result.ResultProducers.lambda$returningUpdateCount$0(ResultProducers.java:48)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.internal.exceptions.Unchecked.lambda$function$4(Unchecked.java:76)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.Jdbi.withExtension(Jdbi.java:489)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.internal.Errors.process(Errors.java:292)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.internal.Errors.process(Errors.java:274)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:292)

  | Nov 18, 2024 @ 13:58:33.970 | at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)

  | Nov 18, 2024 @ 13:58:33.970 | Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at or near "ORDER"

  | Nov 18, 2024 @ 13:58:33.970 | at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:180)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.sqlobject.statement.internal.SqlUpdateHandler.lambda$configureReturner$2(SqlUpdateHandler.java:91)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.sqlobject.statement.internal.CustomizingStatementHandler.invoke(CustomizingStatementHandler.java:196)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.sqlobject.internal.SqlObjectInitData$1.call(SqlObjectInitData.java:141)

  | Nov 18, 2024 @ 13:58:33.970 | at org.openmetadata.service.resources.lineage.LineageResource.deleteLineageByType(LineageResource.java:475)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)

  | Nov 18, 2024 @ 13:58:33.970 | at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47)

  | Nov 18, 2024 @ 13:58:33.970 | at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)

  | Nov 18, 2024 @ 13:58:33.970 | at java.base/java.lang.Thread.run(Thread.java:840)

  | Nov 18, 2024 @ 13:58:33.970 | Position: 182

  | Nov 18, 2024 @ 13:58:33.970 | at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2725)

  | Nov 18, 2024 @ 13:58:33.970 | at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:502)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.statement.SqlStatement.internalExecute(SqlStatement.java:1787)

  | Nov 18, 2024 @ 13:58:33.970 | org.jdbi.v3.core.statement.UnableToExecuteStatementException: org.postgresql.util.PSQLException: ERROR: syntax error at or near "ORDER"

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.sqlobject.internal.SqlObjectInitData$1.lambda$invoke$0(SqlObjectInitData.java:135)

  | Nov 18, 2024 @ 13:58:33.970 | at org.jdbi.v3.core.internal.OnDemandExtensions.lambda$createProxy$3(OnDemandExtensions.java:83)

  | Nov 18, 2024 @ 13:58:33.970 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  | Nov 18, 2024 @ 13:58:33.970 | at java.base/java.lang.reflect.Method.invoke(Method.java:569)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)

  | Nov 18, 2024 @ 13:58:33.970 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)

  | Nov 18, 2024 @ 13:58:33.970 | at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)

  | Nov 18, 2024 @ 13:58:33.970 | at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:371)

  | Nov 18, 2024 @ 13:58:33.970 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

To Reproduce

Create an airflow connection in OpenMetadata and enable override lineage option and try to run the pipeline.(make sure that airflow has some lineage configured)

Expected behavior Overriding the airflow lineage data

Version:

Additional context It seems that OpenMetadata is using a ORDER BY in delete statement which is not valid.