facing this exception - Can't redefine: list

[saiyerniakhil]

SLF4J: Failed toString() invocation on an object of type [org.apache.avro.Schema$RecordSchema] Reported exception: org.apache.avro.SchemaParseException: Can't redefine: list at org.apache.avro.Schema$Names.put(Schema.java:1128) at org.apache.avro.Schema$NamedSchema.writeNameRef(Schema.java:562) at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:690) at org.apache.avro.Schema$ArraySchema.toJson(Schema.java:805) at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:882) at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:716) at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:701) at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:882) at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:716) at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:701) at org.apache.avro.Schema.toString(Schema.java:324) at org.apache.avro.Schema.toString(Schema.java:314) at org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299) at org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271) at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233) at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173) at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:151) at org.slf4j.impl.Log4jLoggerAdapter.debug(Log4jLoggerAdapter.java:252) at org.apache.avro.SchemaCompatibility$ReaderWriterCompatiblityChecker.getCompatibility(SchemaCompatibility.java:228) at org.apache.avro.SchemaCompatibility.checkReaderWriterCompatibility(SchemaCompatibility.java:61) at org.apache.parquet.avro.AvroRecordConverter.isElementType(AvroRecordConverter.java:866) at org.apache.parquet.avro.AvroRecordConverter$AvroCollectionConverter.<init>(AvroRecordConverter.java:475) at org.apache.parquet.avro.AvroRecordConverter.newConverter(AvroRecordConverter.java:289) at org.apache.parquet.avro.AvroRecordConverter.<init>(AvroRecordConverter.java:141) at org.apache.parquet.avro.AvroRecordConverter.<init>(AvroRecordConverter.java:95) at org.apache.parquet.avro.AvroRecordMaterializer.<init>(AvroRecordMaterializer.java:33) at org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:138) at org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:183) at org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:156) at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135) at org.eugene.core.parquet.ParquetReader.read(ParquetReader.java:26) at org.eugene.controller.ParquetDataParser.parseData(ParquetDataParser.java:20) at org.eugene.controller.Renderer.load(Renderer.java:48) at org.eugene.controller.Renderer.loadAndShow(Renderer.java:69) at org.eugene.ui.CustomizedMenuBar.lambda$new$0(CustomizedMenuBar.java:38) at com.sun.javafx.event.CompositeEventHandler.dispatchBubblingEvent(CompositeEventHandler.java:86) at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:238) at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:191) at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:58) at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114) at com.sun.javafx.event.EventUtil.fireEventImpl(EventUtil.java:74) at com.sun.javafx.event.EventUtil.fireEvent(EventUtil.java:49) at javafx.event.Event.fireEvent(Event.java:198) at javafx.scene.control.MenuItem.fire(MenuItem.java:462) at com.sun.javafx.scene.control.skin.ContextMenuContent$MenuItemContainer.doSelect(ContextMenuContent.java:1405) at com.sun.javafx.scene.control.skin.ContextMenuContent$MenuItemContainer.lambda$createChildren$6(ContextMenuContent.java:1358) at com.sun.javafx.event.CompositeEventHandler$NormalEventHandlerRecord.handleBubblingEvent(CompositeEventHandler.java:218) at com.sun.javafx.event.CompositeEventHandler.dispatchBubblingEvent(CompositeEventHandler.java:80) at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:238) at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:191) at com.sun.javafx.event.CompositeEventDispatcher.dispatchBubblingEvent(CompositeEventDispatcher.java:59) at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:58) at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114) at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56) at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114) at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56) at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114) at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56) at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114) at com.sun.javafx.event.EventUtil.fireEventImpl(EventUtil.java:74) at com.sun.javafx.event.EventUtil.fireEvent(EventUtil.java:54) at javafx.event.Event.fireEvent(Event.java:198) at javafx.scene.Scene$MouseHandler.process(Scene.java:3757) at javafx.scene.Scene$MouseHandler.access$1500(Scene.java:3485) at javafx.scene.Scene.impl_processMouseEvent(Scene.java:1762) at javafx.scene.Scene$ScenePeerListener.mouseEvent(Scene.java:2494) at com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run(GlassViewEventHandler.java:394) at com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run(GlassViewEventHandler.java:295) at java.security.AccessController.doPrivileged(Native Method) at com.sun.javafx.tk.quantum.GlassViewEventHandler.lambda$handleMouseEvent$2(GlassViewEventHandler.java:432) at com.sun.javafx.tk.quantum.QuantumToolkit.runWithoutRenderLock(QuantumToolkit.java:410) at com.sun.javafx.tk.quantum.GlassViewEventHandler.handleMouseEvent(GlassViewEventHandler.java:431) at com.sun.glass.ui.View.handleMouseEvent(View.java:555) at com.sun.glass.ui.View.notifyMouse(View.java:937) at com.sun.glass.ui.win.WinApplication._runLoop(Native Method) at com.sun.glass.ui.win.WinApplication.lambda$null$4(WinApplication.java:186) at java.lang.Thread.run(Thread.java:748)

[Eugene-Mark]

@saiyerniakhil Can you give us the link to download the sample file which brings this issue? Thanks.

[theShadow89]

The issue is related to AvroRecordConverter The Converter can't convert a parquet schema where a group field name is reused in an inner structure.

The problem should be fixed: https://github.com/apache/parquet-mr/pull/560

This is an exampel of parquet schema that rise the problem:

message root {
  optional binary name (STRING);
  optional binary surname (STRING);
  optional group activities (LIST) {
    repeated group list {
      optional binary element (STRING);
    }
  }
  optional group values (LIST) {
    repeated group list {
      optional group element {
        optional int64 bigint;
      }
    }
  }
}

[Eugene-Mark]

@theShadow89 Thanks for this helpful information. I've updated the dependency parquet-avro from 1.10 to 1.12, hope the update is included in this version.

[Eugene-Mark]

@saiyerniakhil It's appreciated if you could test the same file with commit 449ad158b3006226f5f244438bd2a2419dae596c

[Eugene-Mark]

Since there is no update whether the update has fixed the issue, closed as per.