search

MarquezProject / marquez

Collect, aggregate, and visualize a data ecosystem's metadata
https://marquezproject.ai
Apache License 2.0
1.78k stars 320 forks source link

When I open the dataset, I get "Something went wrong while fetching dataset runs" #2880

Open wenjiechen-cwj opened 3 months ago

boring-cyborg[bot] commented 3 months ago

Thanks for opening your first issue in the Marquez project! Please be sure to follow the issue template!

wenjiechen-cwj commented 3 months ago

image

wenjiechen-cwj commented 3 months ago

This problem did not arise until the complexity of the data set increased.

wenjiechen-cwj commented 3 months ago

The logs is 172.18.0.5 - - [26/Aug/2024:04:02:48 +0000] "GET /api/v1/namespaces/default/datasets?limit=20&offset=0 HTTP/1.1" 200 30 "http://192.168.1.239:3000/datasets" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 27 INFO [2024-08-26 04:02:52,965] marquez.logging.LoggingMdcFilter: status: 200 INFO [2024-08-26 04:02:52,965] marquez.logging.LoggingMdcFilter: status: 200 172.18.0.5 - - [26/Aug/2024:04:02:52 +0000] "GET /api/v1/tags HTTP/1.1" 200 145 "http://192.168.1.239:3000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 10 172.18.0.5 - - [26/Aug/2024:04:02:52 +0000] "GET /api/v1/namespaces HTTP/1.1" 200 420 "http://192.168.1.239:3000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 10 INFO [2024-08-26 04:02:53,114] marquez.logging.LoggingMdcFilter: status: 200 172.18.0.5 - - [26/Aug/2024:04:02:53 +0000] "GET /api/v1/namespaces/default/jobs?limit=20&offset=0 HTTP/1.1" 200 11403 "http://192.168.1.239:3000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 67 INFO [2024-08-26 04:03:12,517] marquez.logging.LoggingMdcFilter: status: 200 172.18.0.5 - - [26/Aug/2024:04:03:12 +0000] "GET /api/v1/namespaces/default/datasets?limit=20&offset=0 HTTP/1.1" 200 30 "http://192.168.1.239:3000/datasets" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 18 ERROR [2024-08-26 04:03:18,847] io.dropwizard.jersey.errors.LoggingExceptionMapper: Error handling a request: b59ce4fe462660f5 ! java.lang.NullPointerException: namespace is marked non-null but is null ! at marquez.db.models.InputFieldNodeData.(InputFieldNodeData.java:16) ! at marquez.db.mappers.ColumnLineageNodeDataMapper.lambda$toInputFields$1(ColumnLineageNodeDataMapper.java:59) ! at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ! at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ! at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992) ! at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ! at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ! at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ! at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ! at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ! at marquez.db.mappers.ColumnLineageNodeDataMapper.toInputFields(ColumnLineageNodeDataMapper.java:60) ! at marquez.db.mappers.ColumnLineageNodeDataMapper.map(ColumnLineageNodeDataMapper.java:41) ! at marquez.db.mappers.ColumnLineageNodeDataMapper.map(ColumnLineageNodeDataMapper.java:28) ! at org.jdbi.v3.core.result.internal.ResultSetResultIterator.next(ResultSetResultIterator.java:100) ! at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) ! at org.jdbi.v3.core.result.ResultIterator.forEachRemaining(ResultIterator.java:39) ! at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845) ! at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ! at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ! at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ! at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ! at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ! at org.jdbi.v3.core.result.ResultIterable.collect(ResultIterable.java:341) ! at org.jdbi.v3.sqlobject.statement.internal.ResultReturner$CollectedResultReturner.mappedResult(ResultReturner.java:293) ! at org.jdbi.v3.sqlobject.statement.internal.SqlQueryHandler.lambda$configureReturner$0(SqlQueryHandler.java:65) ! at org.jdbi.v3.sqlobject.statement.internal.CustomizingStatementHandler.invoke(CustomizingStatementHandler.java:191) ! at org.jdbi.v3.sqlobject.statement.internal.SqlQueryHandler.invoke(SqlQueryHandler.java:27) ! at org.jdbi.v3.core.extension.ExtensionMetadata$ExtensionHandlerInvoker.lambda$invoke$0(ExtensionMetadata.java:345) ! at org.jdbi.v3.core.AbstractHandleSupplier.invokeInContext(AbstractHandleSupplier.java:36) ! at org.jdbi.v3.core.extension.ExtensionMetadata$ExtensionHandlerInvoker.call(ExtensionMetadata.java:363) ! at org.jdbi.v3.core.extension.ExtensionMetadata$ExtensionHandlerInvoker.invoke(ExtensionMetadata.java:346) ! at org.jdbi.v3.core.extension.ExtensionFactoryDelegate.lambda$attach$0(ExtensionFactoryDelegate.java:118) ! at org.jdbi.v3.core.internal.OnDemandExtensions.invoke(OnDemandExtensions.java:85) ! at org.jdbi.v3.core.internal.OnDemandExtensions.lambda$createProxy$2(OnDemandExtensions.java:71) ! at org.jdbi.v3.core.Jdbi.callWithExtension(Jdbi.java:560) ! at org.jdbi.v3.core.Jdbi.withExtension(Jdbi.java:547) ! at org.jdbi.v3.core.internal.OnDemandExtensions.lambda$createProxy$3(OnDemandExtensions.java:71) ! at jdk.proxy2/jdk.proxy2.$Proxy74.getLineageRowsForDatasets(Unknown Source) ! at marquez.service.DelegatingDaos$DelegatingColumnLineageDao.getLineageRowsForDatasets(DelegatingDaos.java:99) ! at marquez.service.ColumnLineageService.enrichWithColumnLineage(ColumnLineageService.java:209) ! at marquez.api.DatasetResource.list(DatasetResource.java:156) ! at jdk.internal.reflect.GeneratedMethodAccessor83.invoke(Unknown Source) ! at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ! at java.base/java.lang.reflect.Method.invoke(Method.java:569) ! at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) ! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) ! at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176) ! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) ! at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) ! at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) ! at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) ! at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) ! at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ! at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ! at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ! at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ! at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ! at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) ! at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) ! at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) ! at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) ! at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) ! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358) ! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311) ! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) ! at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ! at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) ! at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35) ! at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ! at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ! at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47) ! at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41) ! at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ! at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ! at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) ! at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ! at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ! at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ! at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) ! at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ! at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ! at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ! at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:322) ! at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52) ! at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) ! at io.dropwizard.jetty.ZipExceptionHandlingGzipHandler.handle(ZipExceptionHandlingGzipHandler.java:26) ! at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54) ! at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181) ! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ! at org.eclipse.jetty.server.Server.handle(Server.java:516) ! at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ! at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ! at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) ! at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) ! at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ! at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ! at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) ! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) ! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) ! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) ! at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) ! at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) ! at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) ! at java.base/java.lang.Thread.run(Thread.java:840) ERROR [2024-08-26 04:03:18,847] marquez.logging.LoggingMdcFilter: status: 500 172.18.0.5 - - [26/Aug/2024:04:03:18 +0000] "GET /api/v1/namespaces/hdfs%3A%2F%2Fnamenode%3A8020/datasets?limit=20&offset=0 HTTP/1.1" 500 110 "http://192.168.1.239:3000/datasets" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" 1515

wenjiechen-cwj commented 3 months ago

@swaggy111 Thanks for your reply!😃 Only .exe files in the zip file? My marquez is installed in docker on a linux server. How should I use it.

ChinoUkaegbu commented 3 months ago

The user appears to be spamming malware unfortunately, please do not click the link. (They had posted the exact same thing in a repository I was working on and searching their username shows a bunch of other issues they've commented this exact link in).

lmassaoy commented 2 months ago

Hey folks I'm facing the same issue :( We updated from 0.47.0 to 0.49.0 and started to have the same error message about java.lang.NullPointerException: namespace is marked non-null but is null when performing a list command on datasets (such as the one done by the Web UI).

phixMe commented 2 months ago

@lmassaoy Do you have the complete stack trace for that?

lmassaoy commented 2 months ago

@phixMe I'm afraid I don't have it anymore, since we reverted to 0.47.0. I shall try reproducing the error in my test environment and let you know the full stack trace. Thank you!

fromcloudbase commented 1 month ago

@phixMe
Hi, I'm encountering the same issue with docker image docker.io/marquezproject/marquez@sha256:da785450d051c079058b87d97ded04148d9ac5e2e8c3fcf4942ddad9cbb1e9ad.

10.0.1.7 - - [03/Oct/2024:07:35:32 +0000] "GET /api/v1/namespaces/jobs_trino_bigs2/datasets?limit=20&offset=0 HTTP/1.1" 200 30 "https://marquez.offuscated.domain.name:3000/datasets" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" 14
ERROR [2024-10-03 07:35:37,668] io.dropwizard.jersey.errors.LoggingExceptionMapper: Error handling a request: c0fc82db2cfd017e
! java.lang.NullPointerException: namespace is marked non-null but is null
! at marquez.db.models.InputFieldNodeData.<init>(InputFieldNodeData.java:16)
! at marquez.db.mappers.ColumnLineageNodeDataMapper.lambda$toInputFields$1(ColumnLineageNodeDataMapper.java:59)
! at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
! at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
! at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992)
! at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
! at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
! at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
! at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
! at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
! at marquez.db.mappers.ColumnLineageNodeDataMapper.toInputFields(ColumnLineageNodeDataMapper.java:60)
! at marquez.db.mappers.ColumnLineageNodeDataMapper.map(ColumnLineageNodeDataMapper.java:41)
! at marquez.db.mappers.ColumnLineageNodeDataMapper.map(ColumnLineageNodeDataMapper.java:28)
! at org.jdbi.v3.core.result.internal.ResultSetResultIterator.next(ResultSetResultIterator.java:100)
! at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
! at org.jdbi.v3.core.result.ResultIterator.forEachRemaining(ResultIterator.java:39)
! at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)
! at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
! at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
! at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
! at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
! at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
! at org.jdbi.v3.core.result.ResultIterable.collect(ResultIterable.java:341)
! at org.jdbi.v3.sqlobject.statement.internal.ResultReturner$CollectedResultReturner.mappedResult(ResultReturner.java:293)
! at org.jdbi.v3.sqlobject.statement.internal.SqlQueryHandler.lambda$configureReturner$0(SqlQueryHandler.java:65)
! at org.jdbi.v3.sqlobject.statement.internal.CustomizingStatementHandler.invoke(CustomizingStatementHandler.java:191)
! at org.jdbi.v3.sqlobject.statement.internal.SqlQueryHandler.invoke(SqlQueryHandler.java:27)
! at org.jdbi.v3.core.extension.ExtensionMetadata$ExtensionHandlerInvoker.lambda$invoke$0(ExtensionMetadata.java:345)
! at org.jdbi.v3.core.AbstractHandleSupplier.invokeInContext(AbstractHandleSupplier.java:36)
! at org.jdbi.v3.core.extension.ExtensionMetadata$ExtensionHandlerInvoker.call(ExtensionMetadata.java:363)
! at org.jdbi.v3.core.extension.ExtensionMetadata$ExtensionHandlerInvoker.invoke(ExtensionMetadata.java:346)
! at org.jdbi.v3.core.extension.ExtensionFactoryDelegate.lambda$attach$0(ExtensionFactoryDelegate.java:118)
! at org.jdbi.v3.core.internal.OnDemandExtensions.invoke(OnDemandExtensions.java:85)
! at org.jdbi.v3.core.internal.OnDemandExtensions.lambda$createProxy$2(OnDemandExtensions.java:71)
! at org.jdbi.v3.core.Jdbi.callWithExtension(Jdbi.java:560)
! at org.jdbi.v3.core.Jdbi.withExtension(Jdbi.java:547)
! at org.jdbi.v3.core.internal.OnDemandExtensions.lambda$createProxy$3(OnDemandExtensions.java:71)
! at jdk.proxy2/jdk.proxy2.$Proxy73.getLineageRowsForDatasets(Unknown Source)
! at marquez.service.DelegatingDaos$DelegatingColumnLineageDao.getLineageRowsForDatasets(DelegatingDaos.java:99)
! at marquez.service.ColumnLineageService.enrichWithColumnLineage(ColumnLineageService.java:209)
! at marquez.api.DatasetResource.list(DatasetResource.java:156)
! at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
! at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
! at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
! at java.base/java.lang.reflect.Method.invoke(Method.java:569)
! at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
! at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
! at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
! at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
! at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
! at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
! at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
! at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
! at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
! at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
! at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
! at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
! at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
! at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
! at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
! at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
! at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
! at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
! at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35)
! at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
! at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
! at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47)
! at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41)
! at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
! at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
! at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
! at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
! at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
! at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
! at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
! at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
! at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
! at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
! at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:322)
! at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
! at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
! at io.dropwizard.jetty.ZipExceptionHandlingGzipHandler.handle(ZipExceptionHandlingGzipHandler.java:26)
! at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
! at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181)
! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
! at org.eclipse.jetty.server.Server.handle(Server.java:516)
! at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
! at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
! at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
! at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
! at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
! at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
! at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
! at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
! at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
! at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
! at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
! at java.base/java.lang.Thread.run(Thread.java:840)
ERROR [2024-10-03 07:35:37,673] marquez.logging.LoggingMdcFilter: status: 500
10.0.1.7 - - [03/Oct/2024:07:35:37 +0000] "GET /api/v1/namespaces/trino%3A%2F%2Ftrino1.offuscated.domain.name%3A8443/datasets?limit=20&offset=0 HTTP/1.1" 500 110 "https://marquez.offuscated.domain.name:3000/datasets" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" 938
wslulciuc commented 1 month ago

Thanks @wenjiechen-cwj for reporting this issue and @fromcloudbase for confirming.

Looking at both stacktraces,

// @wenjiechen-cwj
! java.lang.NullPointerException: namespace is marked non-null but is null
! at marquez.db.models.InputFieldNodeData.(InputFieldNodeData.java:16)
! at marquez.db.mappers.ColumnLineageNodeDataMapper.lambda$toInputFields$1(ColumnLineageNodeDataMapper.java:59)
// @fromcloudbase
! java.lang.NullPointerException: namespace is marked non-null but is null
! at marquez.db.models.InputFieldNodeData.<init>(InputFieldNodeData.java:16)
! at marquez.db.mappers.ColumnLineageNodeDataMapper.lambda$toInputFields$1(ColumnLineageNodeDataMapper.java:59)

It looks to me that ColumnLineageNodeDataMapper is the cause of NullPointerException. On InputFieldNodeData instantiation, namespace, dataset, and field cannot be null;

We will look deeper into the root cause of the issue and get out a hot fix as soon as we can. @fromcloudbase if you can share your OpenLineage events that you used to get into this state, I'd be very helpful.

fromcloudbase commented 3 weeks ago

Sorry @wslulciuc I've played a moment with 0;49.0, thinking a time problem can be related to Trino's jobs namespace name. But in vain and I've since downgraded to 0.47.0 too. Thanks for your time and efforts.