search

MarquezProject / marquez

Collect, aggregate, and visualize a data ecosystem's metadata
https://marquezproject.ai
Apache License 2.0
1.78k stars 319 forks source link

Unable to check the column lineage and dataset lineage and got below error need some attention #2921

Open prabakaran-123 opened 1 month ago

prabakaran-123 commented 1 month ago

Read from oracle : oracle://oracledb.cydlut9zgajr.us-east-1.rds.amazonaws.com:1521

image image

boring-cyborg[bot] commented 1 month ago

Thanks for opening your first issue in the Marquez project! Please be sure to follow the issue template!

phixMe commented 1 month ago

Hello, Do you happen to have any logs on the server that indicate what your issue is?

prabakaran-123 commented 1 month ago

@phixMe Here I have attached the logs lhost:3000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" 17 marquez-api-1 | INFO [2024-10-14 17:51:20,631] marquez.logging.LoggingMdcFilter: status: 200 marquez-api-1 | 172.20.0.4 - - [14/Oct/2024:17:51:20 +0000] "GET /api/v1/namespaces/oracle%3A%2F%2Foracledb.cydlut9zgajr.us-east-1.rds.amazonaws.com%3A1521/jobs?limit=20&offset=0 HTTP/1.1" 200 26 "http://localhost:3000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" 11 marquez-api-1 | INFO [2024-10-14 17:51:25,712] marquez.logging.LoggingMdcFilter: status: 200 marquez-api-1 | 172.20.0.4 - - [14/Oct/2024:17:51:25 +0000] "GET /api/v1/namespaces/oracle%3A%2F%2Foracledb.cydlut9zgajr.us-east-1.rds.amazonaws.com%3A1521/datasets?limit=20&offset=0 HTTP/1.1" 200 963 "http://localhost:3000/datasets" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" 27 marquez-api-1 | ERROR [2024-10-14 17:51:27,057] io.dropwizard.jersey.errors.IllegalStateExceptionMapper: Error handling a request: faef7c3122f24251 marquez-api-1 | ! java.lang.IllegalStateException: No match available marquez-api-1 | ! at java.base/java.util.regex.Matcher.start(Matcher.java:450) marquez-api-1 | ! at marquez.service.models.NodeId.parts(NodeId.java:233) marquez-api-1 | ! at marquez.service.models.NodeId.asDatasetId(NodeId.java:257) marquez-api-1 | ! at marquez.api.BaseResource.throwIfNotExists(BaseResource.java:161) marquez-api-1 | ! at marquez.api.OpenLineageResource.getLineage(OpenLineageResource.java:118) marquez-api-1 | ! at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) marquez-api-1 | ! at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) marquez-api-1 | ! at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) marquez-api-1 | ! at java.base/java.lang.reflect.Method.invoke(Method.java:568) marquez-api-1 | ! at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) marquez-api-1 | ! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) marquez-api-1 | ! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) marquez-api-1 | ! at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176) marquez-api-1 | ! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) marquez-api-1 | ! at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) marquez-api-1 | ! at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) marquez-api-1 | ! at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) marquez-api-1 | ! at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) marquez-api-1 | ! at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) marquez-api-1 | ! at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) marquez-api-1 | ! at org.glassfish.jersey.internal.Errors.process(Errors.java:292) marquez-api-1 | ! at org.glassfish.jersey.internal.Errors.process(Errors.java:274) marquez-api-1 | ! at org.glassfish.jersey.internal.Errors.process(Errors.java:244) marquez-api-1 | ! at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) marquez-api-1 | ! at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) marquez-api-1 | ! at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) marquez-api-1 | ! at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) marquez-api-1 | ! at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) marquez-api-1 | ! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358) marquez-api-1 | ! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311) marquez-api-1 | ! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) marquez-api-1 | ! at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) marquez-api-1 | ! at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) marquez-api-1 | ! at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35) marquez-api-1 | ! at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) marquez-api-1 | ! at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) marquez-api-1 | ! at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47) marquez-api-1 | ! at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41) marquez-api-1 | ! at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) marquez-api-1 | ! at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) marquez-api-1 | ! at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) marquez-api-1 | ! at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) marquez-api-1 | ! at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) marquez-api-1 | ! at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) marquez-api-1 | ! at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) marquez-api-1 | ! at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) marquez-api-1 | ! at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) marquez-api-1 | ! at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) marquez-api-1 | ! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) marquez-api-1 | ! at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:322) marquez-api-1 | ! at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52) marquez-api-1 | ! at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) marquez-api-1 | ! at io.dropwizard.jetty.ZipExceptionHandlingGzipHandler.handle(ZipExceptionHandlingGzipHandler.java:26) marquez-api-1 | ! at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54) marquez-api-1 | ! at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181) marquez-api-1 | ! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) marquez-api-1 | ! at org.eclipse.jetty.server.Server.handle(Server.java:516) marquez-api-1 | ! at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) marquez-api-1 | ! at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) marquez-api-1 | ! at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) marquez-api-1 | ! at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) marquez-api-1 | ! at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) marquez-api-1 | ! at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) marquez-api-1 | ! at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) marquez-api-1 | ! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) marquez-api-1 | ! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) marquez-api-1 | ! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) marquez-api-1 | ! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) marquez-api-1 | ! at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) marquez-api-1 | ! at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) marquez-api-1 | ! at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) marquez-api-1 | ! at java.base/java.lang.Thread.run(Thread.java:840) marquez-api-1 | ERROR [2024-10-14 17:51:27,058] marquez.logging.LoggingMdcFilter: status: 500 marquez-api-1 | 172.20.0.4 - - [14/Oct/2024:17:51:27 +0000] "GET /api/v1/lineage?nodeId=dataset:oracle://oracledb.cydlut9zgajr.us-east-1.rds.amazonaws.com:1521:1521/ORCL.CDC_TEST_01&depth=2 HTTP/1.1" 500 110 "http://localhost:3000/lineage/dataset/oracle%3A%2F%2Foracledb.cydlut9zgajr.us-east-1.rds.amazonaws.com%3A1521/1521%2FORCL.CDC_TEST_01" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" 13

prabakaran-123 commented 1 month ago

@phixMe - need some inside on the below logs!

| ERROR [2024-10-14 17:55:05,492] marquez.logging.LoggingMdcFilter: status: 500 marquez-api-1 | 172.20.0.4 - - [14/Oct/2024:17:55:05 +0000] "GET /api/v1/lineage?nodeId=dataset:oracle://oracledb.cydlut9zgajr.us-east-1.rds.amazonaws.com:1521:1521/test_01_newcol&depth=2 HTTP/1.1" 500 110 "http://localhost:3000/lineage/dataset/oracle%3A%2F%2Foracledb.cydlut9zgajr.us-east-1.rds.amazonaws.com%3A1521/1521%2FORCL.cdc_test_01_newcol" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" 15 postgres-1 | 2024-10-14 17:55:11.090 UTC [27] LOG: checkpoint starting: time postgres-1 | 2024-10-14 17:55:22.092 UTC [27] LOG: checkpoint complete: wrote 110 buffers (0.7%); 0 WAL file(s) added, 0 removed, 0 recycled; write=10.943 s, sync=0.043 s, total=11.003 s; sync files=96, longest=0.004 s, average=0.001 s; distance=347 kB, estimate=347 kB; lsn=0/20E7A70, redo lsn=0/20E7A38 marquez-api-1 | INFO [2024-10-14 17:56:27,809] marquez.logging.LoggingMdcFilter: status: 200 marquez-api-1 | 172.20.0.4 - - [14/Oct/2024:17:56:27 +0000] "GET /api/v1/namespaces/oracle%3A%2F%2Foracledb.cydlut9zgajr.us-east-1.rds.amazonaws.com%3A1521/datasets?limit=20&offset=0 HTTP/1.1" 200 963 "http://localhost:3000/datasets" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" 28 marquez-api-1 | ERROR [2024-10-14 17:56:31,426] io.dropwizard.jersey.errors.IllegalStateExceptionMapper: Error handling a request: 8f2e3a6a39c832f3 marquez-api-1 | ! java.lang.IllegalStateException: No match available marquez-api-1 | ! at java.base/java.util.regex.Matcher.start(Matcher.java:450) marquez-api-1 | ! at marquez.service.models.NodeId.parts(NodeId.java:233) marquez-api-1 | ! at marquez.service.models.NodeId.asDatasetId(NodeId.java:257) marquez-api-1 | ! at marquez.service.ColumnLineageService.getColumnNodes(ColumnLineageService.java:144) marquez-api-1 | ! at marquez.service.ColumnLineageService.lineage(ColumnLineageService.java:49) marquez-api-1 | ! at marquez.api.ColumnLineageResource.getLineage(ColumnLineageResource.java:50) marquez-api-1 | ! at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) marquez-api-1 | ! at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) marquez-api-1 | ! at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) marquez-api-1 | ! at java.base/java.lang.reflect.Method.invoke(Method.java:568) marquez-api-1 | ! at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) marquez-api-1 | ! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) marquez-api-1 | ! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) marquez-api-1 | ! at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176) marquez-api-1 | ! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) marquez-api-1 | ! at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) marquez-api-1 | ! at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) marquez-api-1 | ! at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) marquez-api-1 | ! at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) marquez-api-1 | ! at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) marquez-api-1 | ! at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) marquez-api-1 | ! at org.glassfish.jersey.internal.Errors.process(Errors.java:292) marquez-api-1 | ! at org.glassfish.jersey.internal.Errors.process(Errors.java:274) marquez-api-1 | ! at org.glassfish.jersey.internal.Errors.process(Errors.java:244) marquez-api-1 | ! at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) marquez-api-1 | ! at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) marquez-api-1 | ! at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) marquez-api-1 | ! at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) marquez-api-1 | ! at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) marquez-api-1 | ! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358) marquez-api-1 | ! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311) marquez-api-1 | ! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) marquez-api-1 | ! at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) marquez-api-1 | ! at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) marquez-api-1 | ! at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35) marquez-api-1 | ! at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) marquez-api-1 | ! at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) marquez-api-1 | ! at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47) marquez-api-1 | ! at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41) marquez-api-1 | ! at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) marquez-api-1 | ! at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) marquez-api-1 | ! at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) marquez-api-1 | ! at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) marquez-api-1 | ! at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) marquez-api-1 | ! at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) marquez-api-1 | ! at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) marquez-api-1 | ! at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) marquez-api-1 | ! at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) marquez-api-1 | ! at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) marquez-api-1 | ! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) marquez-api-1 | ! at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:322) marquez-api-1 | ! at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52) marquez-api-1 | ! at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) marquez-api-1 | ! at io.dropwizard.jetty.ZipExceptionHandlingGzipHandler.handle(ZipExceptionHandlingGzipHandler.java:26) marquez-api-1 | ! at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54) marquez-api-1 | ! at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181) marquez-api-1 | ! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) marquez-api-1 | ! at org.eclipse.jetty.server.Server.handle(Server.java:516) marquez-api-1 | ! at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) marquez-api-1 | ! at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) marquez-api-1 | ! at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) marquez-api-1 | ! at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) marquez-api-1 | ! at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) marquez-api-1 | ! at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) marquez-api-1 | ! at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) marquez-api-1 | ! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) marquez-api-1 | ! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) marquez-api-1 | ! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) marquez-api-1 | ! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) marquez-api-1 | ! at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) marquez-api-1 | ! at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) marquez-api-1 | ! at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) marquez-api-1 | ! at java.base/java.lang.Thread.run(Thread.java:840) marquez-api-1 | ERROR [2024-10-14 17:56:31,426] marquez.logging.LoggingMdcFilter: status: 500 marquez-api-1 | 172.20.0.4 - - [14/Oct/2024:17:56:31 +0000] "GET /api/v1/column-lineage?nodeId=dataset:oracle://oracledb.cydlut9zgajr.us-east-1.rds.amazonaws.com:1521:1521/ORCL.cdc_test_01_newcol&depth=2&withDownstream=true HTTP/1.1" 500 110 "http://localhost:3000/datasets/column-level/oracle%253A%252F%252Foracledb.cydlut9zgajr.us-east-1.rds.amazonaws.com%253A1521/1521%2FORCL.cdc_test_01_newcol" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" 8

prabakaran-123 commented 4 weeks ago

@phixMe if the dataset name start with a number its creating issue and unable to view the lineage graph

image

phixMe commented 4 weeks ago

Yes, it looks like you have the same issue as: https://github.com/MarquezProject/marquez/issues/2806

prabakaran-123 commented 4 weeks ago

@phixMe Yes, is there any solution to fix it!

phixMe commented 4 weeks ago

This is logistically a complicated thing for us to fix because it is everywhere including our docs and website and promotional materials...

Essentially, we are in this difficult position where we cannot parse connection strings and use the : colon character as delimiters. There are a bunch of edge cases we are exposed to for example job:trino://trino-integration-test:1337:1234:more:colons it's really unclear what we consider the namespace we what we consider the job... Fixing this for one user might break it for another.

I think the only way to rectify this is to change the delimiter character for our notational, but it's a breaking change for all kinds of users.

phixMe commented 4 weeks ago

I put together a branch of what this would look like on the api server. it would need to be done on the clients as well of course, and maybe others would prefer a different delimiter. https://github.com/MarquezProject/marquez/compare/main...modify/node-id-delimiter

prabakaran-123 commented 4 weeks ago

@phixMe tried with the above branch which you mentioned:

could you please provide me sample for the node id with delimeter, as I tried end up with below error

ERROR [2024-10-17 15:16:52,766] io.dropwizard.jersey.errors.LoggingExceptionMapper: Error handling a request: 11abd68c1ed35ebb marquez-api-1 | ! java.lang.UnsupportedOperationException: Expected NodeId of type dataset with 3 parts. Got: dataset&oracle://oracledb.cydlut9zgajr.us-east-1.rds.amazonaws.com:1521:1521/orcl.cdc_test_01_newcol marquez-api-1 | ! at marquez.service.models.NodeId.parts(NodeId.java:218) marquez-api-1 | ! at marquez.service.models.NodeId.asDatasetId(NodeId.java:257) marquez-api-1 | ! at marquez.api.BaseResource.throwIfNotExists(BaseResource.java:161) marquez-api-1 | ! at marquez.api.OpenLineageResource.getLineage(OpenLineageResource.java:121) marquez-api-1 | ! at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) marquez-api-1 | ! at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) marquez-api-1 | ! at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) marquez-api-1 | ! at java.base/java.lang.reflect.Method.invoke(Method.java:569) marquez-api-1 | ! at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) marquez-api-1 | ! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) marquez-api-1 | ! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) marquez-api-1 | ! at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
marquez-api-1 | ! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) marquez-api-1 | ! at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) marquez-api-1 | ! at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) marquez-api-1 | ! at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) marquez-api-1 | ! at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) marquez-api-1 | ! at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) marquez-api-1 | ! at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) marquez-api-1 | ! at org.glassfish.jersey.internal.Errors.process(Errors.java:292) marquez-api-1 | ! at org.glassfish.jersey.internal.Errors.process(Errors.java:274) marquez-api-1 | ! at org.glassfish.jersey.internal.Errors.process(Errors.java:244) marquez-api-1 | ! at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) marquez-api-1 | ! at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) marquez-api-1 | ! at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) marquez-api-1 | ! at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) marquez-api-1 | ! at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) marquez-api-1 | ! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358) marquez-api-1 | ! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311) marquez-api-1 | ! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) marquez-api-1 | ! at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) marquez-api-1 | ! at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) marquez-api-1 | ! at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35) marquez-api-1 | ! at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) marquez-api-1 | ! at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) marquez-api-1 | ! at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47) marquez-api-1 | ! at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41) marquez-api-1 | ! at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) marquez-api-1 | ! at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) marquez-api-1 | ! at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) marquez-api-1 | ! at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) marquez-api-1 | ! at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) marquez-api-1 | ! at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) marquez-api-1 | ! at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) marquez-api-1 | ! at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) marquez-api-1 | ! at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) marquez-api-1 | ! at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) marquez-api-1 | ! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) marquez-api-1 | ! at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:322) marquez-api-1 | ! at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52) marquez-api-1 | ! at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) marquez-api-1 | ! at io.dropwizard.jetty.ZipExceptionHandlingGzipHandler.handle(ZipExceptionHandlingGzipHandler.java:26) marquez-api-1 | ! at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54) marquez-api-1 | ! at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181) marquez-api-1 | ! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) marquez-api-1 | ! at org.eclipse.jetty.server.Server.handle(Server.java:516) marquez-api-1 | ! at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) marquez-api-1 | ! at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) marquez-api-1 | ! at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) marquez-api-1 | ! at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) marquez-api-1 | ! at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) marquez-api-1 | ! at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) marquez-api-1 | ! at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) marquez-api-1 | ! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) marquez-api-1 | ! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) marquez-api-1 | ! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) marquez-api-1 | ! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) marquez-api-1 | ! at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) marquez-api-1 | ! at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) marquez-api-1 | ! at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) marquez-api-1 | ! at java.base/java.lang.Thread.run(Thread.java:840)

Requesting Url format : (http://localhost:5000/api/v1/lineage?depth=2&nodeId=dataset%26oracle://oracledb.cydlut9zgajr.us-east-1.rds.amazonaws.com:1521:1521/orcl.cdc_test_01_newcol)

phixMe commented 4 weeks ago

Yes, the UI would also need an update... I think the easiest way for you to get around this is to intercept the OpenLineage events and prepend a string to the job names, or rename the jobs that you are testing with to not have this property.

I don't think we can realistically merge anything here without quite a bit of forethought and communication, so that's probably the best I can recommend for now.