search

opendatadiscovery / odd-platform

First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.
https://opendatadiscovery.org
Apache License 2.0
1.16k stars 96 forks source link

DatasetFields duplicate ODDRN #1671

Open Vladysl opened 2 months ago

Vladysl commented 2 months ago

Steps to reproduce: 1)Collector should ingest data without relationships 2)Some field should become PK 3)Relationship should be created between existed fields from step 2 4)Collector should ingest data 5) Go to RelationshipDetails page

How to resolve manually: You can find oddrn duplicates using this 

select sub_q.oddrn, sub_q.rows_count
from (select oddrn, count(oddrn) as rows_count
      from public.dataset_field
      group by oddrn) as sub_q
where sub_q.rows_count > 1;

You need to delete one of the dataset_fields with a duplicate oddrn(or change the oddrn for one of them to something else). You can delete dataset_field, but first, you need to delete each place that uses dataset_field as a foreign key. _dataset_field_to_term, dataset_field_metadata_value, dataset_structure, dataset_field_description_unhandled_term, tag_to_datasetfield and then delete from public.dataset_field where id = id_to_delete;

odd-platform-1 | java.lang.IllegalStateException: Duplicate key //postgresql/host/sample-postgresql/databases/sample_odd_platform/schemas/public/tables/users/columns/id (attempted merging values DatasetFieldPojo (14, id, //postgresql/host/sample-postgresql/databases/sample_odd_platform/schemas/public/tables/users/columns/id, null, 0, {}, {"is_nullable":true,"logical_type":"int8","type":"TYPE_INTEGER"}, null, null, null, null, false, false, null, null, null) and DatasetFieldPojo (62, id, //postgresql/host/sample-postgresql/databases/sample_odd_platform/schemas/public/tables/users/columns/id, null, 0, {}, {"is_nullable":false,"logical_type":"int8","type":"TYPE_INTEGER"}, null, null, null, null, true, false, null, null, null)) odd-platform-1 | at java.base/java.util.stream.Collectors.duplicateKeyException(Collectors.java:135) ~[na:na] odd-platform-1 | Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: odd-platform-1 | Error has been observed at the following site(s): odd-platform-1 | *__checkpoint ⇢ Handler org.opendatadiscovery.oddplatform.controller.DatasetController#getDataSetRelationships(Long, RelationshipsType, ServerWebExchange) [DispatcherHandler] odd-platform-1 | Original Stack Trace: odd-platform-1 | at java.base/java.util.stream.Collectors.duplicateKeyException(Collectors.java:135) ~[na:na] odd-platform-1 | at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:182) ~[na:na] odd-platform-1 | at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) ~[na:na] odd-platform-1 | at java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1707) ~[na:na] odd-platform-1 | at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[na:na] odd-platform-1 | at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[na:na] odd-platform-1 | at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[na:na] odd-platform-1 | at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:na] odd-platform-1 | at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[na:na] odd-platform-1 | at org.opendatadiscovery.oddplatform.repository.reactive.ReactiveRelationshipsRepositoryImpl.extractErdDetails(ReactiveRelationshipsRepositoryImpl.java:241) ~[classes/:na] odd-platform-1 | at org.opendatadiscovery.oddplatform.repository.reactive.ReactiveRelationshipsRepositoryImpl.lambda$getRelationsByDatasetIdAndType$2(ReactiveRelationshipsRepositoryImpl.java:146) ~[classes/:na] odd-platform-1 | at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106) ~[reactor-core-3.5.6.jar:3.5.6] odd-platform-1 | at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) ~[reactor-core-3.5.6.jar:3.5.6] odd-platform-1 | at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) ~[reactor-core-3.5.6.jar:3.5.6] odd-platform-1 | at reactor.core.publisher.FluxUsingWhen$UsingWhenSubscriber.onNext(FluxUsingWhen.java:345) ~[reactor-core-3.5.6.jar:3.5.6] odd-platform-1 | at org.jooq.impl.Internal$1.onNext(Internal.java:485) ~[jooq-3.18.4.jar:na] odd-platform-1 | at org.jooq.impl.R2DBC$Forwarding.onNext(R2DBC.java:245) ~[jooq-3.18.4.jar:na] odd-platform-1 | at reactor.core.publisher.StrictSubscriber.onNext(StrictSubscriber.java:89) ~[reactor-core-3.5.6.jar:3.5.6] odd-platform-1 | at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:128) ~[reactor-core-3.5.6.jar:3.5.6] odd-platform-1 | at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:250) ~[reactor-core-3.5.6.jar:3.5.6] odd-platform-1 | at reactor.core.publisher.FluxHandleFuseable$HandleFuseableSubscriber.onNext(FluxHandleFuseable.java:193) ~[reactor-core-3.5.6.jar:3.5.6] odd-platform-1 | at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337) ~[reactor-core-3.5.6.jar:3.5.6] odd-platform-1 | at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107) ~[reactor-core-3.5.6.jar:3.5.6] odd-platform-1 | at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onNext(FluxPeekFuseable.java:854) ~[reactor-core-3.5.6.jar:3.5.6] odd-platform-1 | at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onNext(FluxPeekFuseable.java:854) ~[reactor-core-3.5.6.jar:3.5.6] odd-platform-1 | at io.r2dbc.postgresql.util.FluxDiscardOnCancel$FluxDiscardOnCancelSubscriber.onNext(FluxDiscardOnCancel.java:91) ~[r2dbc-postgresql-1.0.1.RELEASE.jar:1.0.1.RELEASE] odd-platform-1 | at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onNext(FluxDoFinally.java:113) ~[reactor-core-3.5.6.jar:3.5.6] odd-platform-1 | at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:128) ~[reactor-core-3.5.6.jar:3.5.6] odd-platform-1 | at reactor.core.publisher.FluxCreate$BufferAsyncSink.drain(FluxCreate.java:814) ~[reactor-core-3.5.6.jar:3.5.6] odd-platform-1 | at reactor.core.publisher.FluxCreate$BufferAsyncSink.next(FluxCreate.java:739) ~[reactor-core-3.5.6.jar:3.5.6] odd-platform-1 | at reactor.core.publisher.FluxCreate$SerializedFluxSink.next(FluxCreate.java:161) ~[reactor-core-3.5.6.jar:3.5.6] odd-platform-1 | at io.r2dbc.postgresql.client.ReactorNettyClient$Conversation.emit(ReactorNettyClient.java:687) ~[r2dbc-postgresql-1.0.1.RELEASE.jar:1.0.1.RELEASE] odd-platform-1 | at io.r2dbc.postgresql.client.ReactorNettyClient$BackendMessageSubscriber.emit(ReactorNettyClient.java:939) ~[r2dbc-postgresql-1.0.1.RELEASE.jar:1.0.1.RELEASE] odd-platform-1 | at io.r2dbc.postgresql.client.ReactorNettyClient$BackendMessageSubscriber.onNext(ReactorNettyClient.java:813) ~[r2dbc-postgresql-1.0.1.RELEASE.jar:1.0.1.RELEASE] odd-platform-1 | at io.r2dbc.postgresql.client.ReactorNettyClient$BackendMessageSubscriber.onNext(ReactorNettyClient.java:719) ~[r2dbc-postgresql-1.0.1.RELEASE.jar:1.0.1.RELEASE] odd-platform-1 | at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:128) ~[reactor-core-3.5.6.jar:3.5.6] odd-platform-1 | at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onNext(FluxPeekFuseable.java:854) ~[reactor-core-3.5.6.jar:3.5.6] odd-platform-1 | at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:224) ~[reactor-core-3.5.6.jar:3.5.6] odd-platform-1 | at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:224) ~[reactor-core-3.5.6.jar:3.5.6] odd-platform-1 | at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:292) ~[reactor-netty-core-1.1.7.jar:1.1.7] odd-platform-1 | at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:401) ~[reactor-netty-core-1.1.7.jar:1.1.7] odd-platform-1 | at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:411) ~[reactor-netty-core-1.1.7.jar:1.1.7] odd-platform-1 | at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113) ~[reactor-netty-core-1.1.7.jar:1.1.7] odd-platform-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] odd-platform-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] odd-platform-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] odd-platform-1 | at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.92.Final.jar:4.1.92.Final] odd-platform-1 | at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:333) ~[netty-codec-4.1.92.Final.jar:4.1.92.Final] odd-platform-1 | at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:454) ~[netty-codec-4.1.92.Final.jar:4.1.92.Final] odd-platform-1 | at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.92.Final.jar:4.1.92.Final] odd-platform-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] odd-platform-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] odd-platform-1 | at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] odd-platform-1 | at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] odd-platform-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] odd-platform-1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] odd-platform-1 | at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final] odd-platform-1 | at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) ~[netty-transport-classes-epoll-4.1.92.Final.jar:4.1.92.Final] odd-platform-1 | at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499) ~[netty-transport-classes-epoll-4.1.92.Final.jar:4.1.92.Final] odd-platform-1 | at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) ~[netty-transport-classes-epoll-4.1.92.Final.jar:4.1.92.Final] odd-platform-1 | at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.92.Final.jar:4.1.92.Final] odd-platform-1 | at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.92.Final.jar:4.1.92.Final] odd-platform-1 | at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.92.Final.jar:4.1.92.Final] odd-platform-1 | at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na] odd-platform-1 |