search

djuang1 / parquet

Parquet extension for Mule 4.x
9 stars 5 forks source link

Streaming of files is returning errors #2

Open sraghuiyer opened 1 year ago

sraghuiyer commented 1 year ago

Hello @djuang1 , it's really handy component for mule use to change formats. While the filesystem file convert fine, I constantly get errors while using the streaming function, I'm attaching the error seen.

Hope you can spend sometime and figure the issue out.

In my specific case, I'm streaming some file from Azure datalake and converting them to json for downstream usage.

java.lang.IllegalArgumentException: INT96 not yet implemented. at org.apache.parquet.avro.AvroSchemaConverter$1.convertINT96(AvroSchemaConverter.java:279) at org.apache.parquet.avro.AvroSchemaConverter$1.convertINT96(AvroSchemaConverter.java:264)

skhan-goodrx commented 11 months ago

Hello @djuang1 , we are also facing the same issue. Could you please solve it? Thanks in advance.

djuang1 commented 11 months ago

For testing, can either of you provide an example data file I could use to test and debug?

djuang1 commented 11 months ago

Can you both try again? I was able to get a Parquet file that gave me the same issue.

pdeshmuGoodRx commented 11 months ago

Hi @djuang1,

Thanks for fixing the issue. We have tried to test couple of files and it worked fine,error message didnt appeared. However we observe JVM restarts while processing files when file size (number od records) increases.

Regards, Preetam Deshmukh