[5.5.3]: java.io.EOFException while reading some BUFR files (solution: just update the bufr tables)

JP95Git commented 1 year ago

Versions impacted by the bug

v5.x

What went wrong?

Hi,

I am unable to read BUFR files from these stations:

SOWR = Wroclaw (PL) or Leba (PL)
LFPW = Trappes (FR)

The files from SOWR are generated by MODEM: https://www.meteomodem.com/

An example file is attached: Z__C_SOWR_20230716010100_aero1,IUSD02_SOWR_160000____RSSO.bin

I was able to read this file with this online tool https://kunden.dwd.de/bufrviewer/ and with ecCodes ( https://confluence.ecmwf.int/display/ECC ). I Also attached a dump of this file as bufr_dump.dat generated by bufr_dump from ecCodes.

Stacktrace after opening the file in toolsUI 5.5.3 and click on "Show Data" is attached in "Relevant stack trace".

I get the same stack trace in toolsUI 5.5.3, 5.5.2, 5.4.1, 5.3.3.

While using NetCDF Java 5.3.3 in my own program I got a similar stack trace:

java.io.EOFException: null
    at ucar.nc2.iosp.BitReader.nextByte(BitReader.java:163)
    at ucar.nc2.iosp.BitReader.bits2UInt(BitReader.java:104)
    at ucar.nc2.iosp.bufr.MessageUncompressedDataReader.readNumericData(MessageUncompressedDataReader.java:267)
    at ucar.nc2.iosp.bufr.MessageUncompressedDataReader.readData(MessageUncompressedDataReader.java:244)
    at ucar.nc2.iosp.bufr.MessageUncompressedDataReader.makeArraySequenceUncompressed(MessageUncompressedDataReader.java:364)
    at ucar.nc2.iosp.bufr.MessageUncompressedDataReader.readData(MessageUncompressedDataReader.java:203)
    at ucar.nc2.iosp.bufr.MessageUncompressedDataReader.readData(MessageUncompressedDataReader.java:132)
    at ucar.nc2.iosp.bufr.MessageUncompressedDataReader.readEntireMessage(MessageUncompressedDataReader.java:88)
    at ucar.nc2.iosp.bufr.BufrIosp2$SeqIter.readMessage(BufrIosp2.java:266)
    at ucar.nc2.iosp.bufr.BufrIosp2$SeqIter.readNextMessage(BufrIosp2.java:255)
    at ucar.nc2.iosp.bufr.BufrIosp2$SeqIter.hasNext(BufrIosp2.java:211)
    at ucar.ma2.ArraySequence.extractMemberArrayFromIteration(ArraySequence.java:319)
    at ucar.ma2.ArraySequence.extractMemberArray(ArraySequence.java:88)
    at ucar.nc2.Variable.reallyRead(Variable.java:757)
    at ucar.nc2.Variable._read(Variable.java:731)
    at ucar.nc2.Variable.read(Variable.java:611)
    at ucar.nc2.dataset.VariableDS.reallyRead(VariableDS.java:437)
    at ucar.nc2.dataset.VariableDS._read(VariableDS.java:426)
    at ucar.nc2.Variable.read(Variable.java:611)
    at ucar.nc2.Variable.getScalarData(Variable.java:708)
    at ucar.nc2.Variable.readScalarString(Variable.java:697)
    -> My own code is shown here.

I reported this issue last year via email and got this response:

The netcdf-java library is not particularly robust about handling bufr files, and is very particular about how the bufr table needs to be formatted. An EOFException usually means that's something's not quite right with the bufr table. Unfortunately, Unidata doesn't currently have the resources to work on the bufr module, so it will probably be a while before we're able to investigate why this error occurs with some files. In the meantime, this is the information we have available on how the netcdf-java library handles bufr tables: https://docs.unidata.ucar.edu/netcdf-java/5.5/userguide/bufr_tables.html

Relevant stack trace

java.io.EOFException
    at ucar.nc2.iosp.BitReader.nextByte(BitReader.java:163)
    at ucar.nc2.iosp.BitReader.bits2UInt(BitReader.java:104)
    at ucar.nc2.iosp.bufr.MessageUncompressedDataReader.readNumericData(MessageUncompressedDataReader.java:267)
    at ucar.nc2.iosp.bufr.MessageUncompressedDataReader.readData(MessageUncompressedDataReader.java:244)
    at ucar.nc2.iosp.bufr.MessageUncompressedDataReader.makeArraySequenceUncompressed(MessageUncompressedDataReader.java:364)
    at ucar.nc2.iosp.bufr.MessageUncompressedDataReader.readData(MessageUncompressedDataReader.java:203)
    at ucar.nc2.iosp.bufr.MessageUncompressedDataReader.readData(MessageUncompressedDataReader.java:132)
    at ucar.nc2.iosp.bufr.MessageUncompressedDataReader.readEntireMessage(MessageUncompressedDataReader.java:88)
    at ucar.nc2.iosp.bufr.BufrIosp2$SeqIter.readMessage(BufrIosp2.java:266)
    at ucar.nc2.iosp.bufr.BufrIosp2$SeqIter.readNextMessage(BufrIosp2.java:255)
    at ucar.nc2.iosp.bufr.BufrIosp2$SeqIter.hasNext(BufrIosp2.java:211)
    at ucar.ma2.ArraySequence.extractMemberArrayFromIteration(ArraySequence.java:315)
    at ucar.ma2.ArraySequence.extractMemberArray(ArraySequence.java:83)
    at ucar.ma2.ArrayStructureMA.factoryMA(ArrayStructureMA.java:224)
    at ucar.nc2.dataset.StructureDS.convert(StructureDS.java:284)
    at ucar.nc2.dataset.SequenceDS.read(SequenceDS.java:89)
    at ucar.nc2.ui.op.DatasetViewer.showData(DatasetViewer.java:634)
    at ucar.nc2.ui.op.DatasetViewer.access$1200(DatasetViewer.java:56)
    at ucar.nc2.ui.op.DatasetViewer$NestedTable$3.actionPerformed(DatasetViewer.java:477)
    at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
    at javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
    at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
    at javax.swing.DefaultButtonModel.setPressed(Unknown Source)
    at javax.swing.AbstractButton.doClick(Unknown Source)
    at javax.swing.plaf.basic.BasicMenuItemUI.doClick(Unknown Source)
    at javax.swing.plaf.basic.BasicMenuItemUI$Handler.mouseReleased(Unknown Source)
    at java.awt.Component.processMouseEvent(Unknown Source)
    at javax.swing.JComponent.processMouseEvent(Unknown Source)
    at java.awt.Component.processEvent(Unknown Source)
    at java.awt.Container.processEvent(Unknown Source)
    at java.awt.Component.dispatchEventImpl(Unknown Source)
    at java.awt.Container.dispatchEventImpl(Unknown Source)
    at java.awt.Component.dispatchEvent(Unknown Source)
    at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
    at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
    at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
    at java.awt.Container.dispatchEventImpl(Unknown Source)
    at java.awt.Window.dispatchEventImpl(Unknown Source)
    at java.awt.Component.dispatchEvent(Unknown Source)
    at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
    at java.awt.EventQueue.access$500(Unknown Source)
    at java.awt.EventQueue$3.run(Unknown Source)
    at java.awt.EventQueue$3.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
    at java.awt.EventQueue$4.run(Unknown Source)
    at java.awt.EventQueue$4.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
    at java.awt.EventQueue.dispatchEvent(Unknown Source)
    at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
    at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
    at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
    at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
    at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
    at java.awt.EventDispatchThread.run(Unknown Source)

Relevant log messages

No response

If you have an example file that you can share, please attach it to this issue.

Attached is bufr.zip which contains the BUFR file Z__C_SOWR_20230716010100_aero1,IUSD02_SOWR_160000____RSSO.bin and the dump bufr_dump.dat. bufr.zip

Code of Conduct

[X] I agree to follow the UCAR/Unidata Code of Conduct

JP95Git commented 1 year ago

I was able to read the file by using the latest (version 40) of the BUFR tables from https://community.wmo.int/en/activity-areas/wis/latest-version

BufrTables.addLookupFile("path/to/bufr/bufr-tables.csv");

path/to/bufr/bufr-tables.csv WMO.any, 0, 0, -1, 0, -1, path/to/bufr/BUFRCREX_TableB_en.xml, wmo_xml , path/to/bufr/BUFR_TableD_en.xml, wmo_xml , localWmo

Works with netcdf-java 5.5.3 and netcdf-java 5.3.3

JP95Git commented 1 year ago

@haileyajohnson Because the stack trace was very misleading you should update the BUFR tables in netcdf-java, or else other users may encounter the same problem. Thanks.

Unidata / netcdf-java