Failed to parse Float - Githubissues

GustavoAlves1993 commented 3 years ago

I'm trying to consume data from a table but I'm getting the following error:

com.linuxense.javadbf.DBFException: Failed to parse Float: null
    at com.linuxense.javadbf.DBFUtils.readNumericStoredAsText(DBFUtils.java:82)
    at com.linuxense.javadbf.DBFReader.getFieldValue(DBFReader.java:459)
    at com.linuxense.javadbf.DBFReader.nextRecord(DBFReader.java:343)
    at com.linuxense.javadbf.DBFReader.nextRow(DBFReader.java:395)

is giving error when trying to convert a double value, does anyone know how to proceed with this error?

GustavoAlves1993 commented 3 years ago

does anyone have any idea how i can resolve it?

albfernandez commented 3 years ago

Hi, maybe more info would be useful to help you. The version of javadbf you're using, the full stacktrace, and (if possible) the file causing the error.

GustavoAlves1993 commented 3 years ago

hi, first thanks for the feedback

i'm using version 1.11.1 of the library

the full stacktracer

com.linuxense.javadbf.DBFException: Failed to parse Float: null
    at com.linuxense.javadbf.DBFUtils.readNumericStoredAsText(DBFUtils.java:82)
    at com.linuxense.javadbf.DBFReader.getFieldValue(DBFReader.java:459)
    at com.linuxense.javadbf.DBFReader.nextRecord(DBFReader.java:343)
    at com.linuxense.javadbf.DBFReader.nextRow(DBFReader.java:395)
    at conversorconnexone.dao.DbfDAO.coletaEstoquePrincipal(DbfDAO.java:212)
    at conversorconnexone.bo.DdfBO.coletaEstoquePrincipal(DdfBO.java:35)
    at conversorconnexone.ui.Principal.coletarDBF(Principal.java:489)
    at conversorconnexone.ui.Principal.conectarDBF(Principal.java:440)
    at conversorconnexone.ui.Principal.jbConverterMouseClicked(Principal.java:1263)
    at conversorconnexone.ui.Principal.access$000(Principal.java:31)
    at conversorconnexone.ui.Principal$1.mouseClicked(Principal.java:1228)
    at java.awt.AWTEventMulticaster.mouseClicked(AWTEventMulticaster.java:270)
    at java.awt.Component.processMouseEvent(Component.java:6536)
    at javax.swing.JComponent.processMouseEvent(JComponent.java:3324)
    at java.awt.Component.processEvent(Component.java:6298)
    at java.awt.Container.processEvent(Container.java:2236)
    at java.awt.Component.dispatchEventImpl(Component.java:4889)
    at java.awt.Container.dispatchEventImpl(Container.java:2294)
    at java.awt.Component.dispatchEvent(Component.java:4711)
    at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4888)
    at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4534)
    at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4466)
    at java.awt.Container.dispatchEventImpl(Container.java:2280)
    at java.awt.Window.dispatchEventImpl(Window.java:2746)
    at java.awt.Component.dispatchEvent(Component.java:4711)
    at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758)
    at java.awt.EventQueue.access$500(EventQueue.java:97)
    at java.awt.EventQueue$3.run(EventQueue.java:709)
    at java.awt.EventQueue$3.run(EventQueue.java:703)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
    at java.awt.EventQueue$4.run(EventQueue.java:731)
    at java.awt.EventQueue$4.run(EventQueue.java:729)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
    at java.awt.EventQueue.dispatchEvent(EventQueue.java:728)
    at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
    at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
    at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:109)
    at java.awt.WaitDispatchSupport$2.run(WaitDispatchSupport.java:184)
    at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311)
    at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:756)
    at java.awt.EventQueue.access$500(EventQueue.java:97)
    at java.awt.EventQueue$3.run(EventQueue.java:709)
    at java.awt.EventQueue$3.run(EventQueue.java:703)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
    at java.awt.EventQueue.dispatchEvent(EventQueue.java:726)
    at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
    at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
    at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
    at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)
Caused by: java.lang.NumberFormatException
    at java.math.BigDecimal.<init>(BigDecimal.java:494)
    at java.math.BigDecimal.<init>(BigDecimal.java:383)
    at java.math.BigDecimal.<init>(BigDecimal.java:806)
    at com.linuxense.javadbf.DBFUtils.readNumericStoredAsText(DBFUtils.java:77)
    ... 54 more

my class:

public ArrayList coletaEstoquePrincipal(DBFReader connDBF) throws Exception {
        ArrayList estoquePrincipal = new ArrayList();

        DBFRow row;        

        while ((row = connDBF.nextRow()) != null) {            
                estoquePrincipal.add(new Object[]{
                    row.getString("SEQUENCIAL"), //0
                    row.getString("CODIGO"), //1
                    row.getString("GRUPO"), //2
                    row.getString("SUBGRUPO"), //3
                    row.getString("DESCRICAO"), //4
                    row.getString("UNIDADE"), //5
                    row.getString("COR"), //6
                    row.getString("CUSTO"), //7
                    row.getString("MARGEM"), //8
                    row.getString("VENDA"), //9
                    row.getString("MINIMO"), //10
                    row.getString("ATUAL"), //11
                    row.getString("ICMS"), //12
                    row.getString("BARRA"), //13
                    row.getString("FORNECEDOR"), //14
                    row.getString("MARCA"), //15
                    row.getString("ULTCOMPRA"), //16
                    row.getString("NCM"), //17
                    row.getString("CSOSN"), //18
                    row.getString("CODFAB"), //19
                    row.getString("GTIN"), //20
                    row.getString("INATIVO") //21
                });           
        }

        return estoquePrincipal;
    }

the error happens on the line while ((row = connDBF.nextRow()) != null) and drops the application

can I send the file in your email?

albfernandez commented 3 years ago

you can post the file here if it doesn't contain sensible data or email me the file, as you like.

GustavoAlves1993 commented 3 years ago

I sent in your email

albfernandez commented 3 years ago

Hi I've tested and it works for me.

I've updated some changes to be more verbose on exceptions, I've attached here javadbf-1.12.0-SNAPSHOT.jar.zip

Can you try with this one to get a more detailed messages?

I've tried with https://github.com/albfernandez/javadbf/blob/main/src/test/java/com/linuxense/javadbf/testutils/DbfToTxtTest.java It's basic the same you're using. I use this method to read a dbf and print the results.

public static void writeToConsole(File file) throws FileNotFoundException {
        DBFReader reader = null;
        try {

            // create a DBFReader object
            reader = new DBFReader(new FileInputStream(file));

            // get the field count if you want for some reasons like the following

            int numberOfFields = reader.getFieldCount();

            // use this count to fetch all field information
            // if required

            for (int i = 0; i < numberOfFields; i++) {

                DBFField field = reader.getField(i);

                // do something with it if you want
                // refer the JavaDoc API reference for more details
                //
                System.out.println(field.getType() + " (" + field.getLength() + ") "+ field.getName());
            }

            // Now, lets us start reading the rows

            Object[] rowObjects;

            System.out.println("-------------------");
            while ((rowObjects = reader.nextRecord()) != null) {
                for (int i = 0; i < rowObjects.length; i++) {
                    System.out.println(rowObjects[i]);
                }
                System.out.println("-------------------");
            }

            // By now, we have iterated through all of the rows

        }
        finally {
            DBFUtils.close(reader);
        }
    }

GustavoAlves1993 commented 3 years ago

Alberto I replaced the library as you requested, and now the error presented is this:

com.linuxense.javadbf.DBFException: Error reading field SALDOFIN of record 2 Failed to parse Float value [0.00N] : null
    at com.linuxense.javadbf.DBFReader.nextRecord(DBFReader.java:383)
    at com.linuxense.javadbf.DBFReader.nextRow(DBFReader.java:400)
    at conversorconnexone.dao.DbfDAO.coletaEstoquePrincipal(DbfDAO.java:212)
    at conversorconnexone.bo.DdfBO.coletaEstoquePrincipal(DdfBO.java:35)
    at conversorconnexone.ui.Principal.coletarDBF(Principal.java:489)
    at conversorconnexone.ui.Principal.conectarDBF(Principal.java:440)
    at conversorconnexone.ui.Principal.jbConverterMouseClicked(Principal.java:1263)
    at conversorconnexone.ui.Principal.access$000(Principal.java:31)
    at conversorconnexone.ui.Principal$1.mouseClicked(Principal.java:1228)
    at java.awt.AWTEventMulticaster.mouseClicked(AWTEventMulticaster.java:270)
    at java.awt.Component.processMouseEvent(Component.java:6536)
    at javax.swing.JComponent.processMouseEvent(JComponent.java:3324)
    at java.awt.Component.processEvent(Component.java:6298)
    at java.awt.Container.processEvent(Container.java:2236)
    at java.awt.Component.dispatchEventImpl(Component.java:4889)
    at java.awt.Container.dispatchEventImpl(Container.java:2294)
    at java.awt.Component.dispatchEvent(Component.java:4711)
    at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4888)
    at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4534)
    at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4466)
    at java.awt.Container.dispatchEventImpl(Container.java:2280)
    at java.awt.Window.dispatchEventImpl(Window.java:2746)
    at java.awt.Component.dispatchEvent(Component.java:4711)
    at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758)
    at java.awt.EventQueue.access$500(EventQueue.java:97)
    at java.awt.EventQueue$3.run(EventQueue.java:709)
    at java.awt.EventQueue$3.run(EventQueue.java:703)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
    at java.awt.EventQueue$4.run(EventQueue.java:731)
    at java.awt.EventQueue$4.run(EventQueue.java:729)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
    at java.awt.EventQueue.dispatchEvent(EventQueue.java:728)
    at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
    at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
    at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:109)
    at java.awt.WaitDispatchSupport$2.run(WaitDispatchSupport.java:184)
    at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311)
    at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:756)
    at java.awt.EventQueue.access$500(EventQueue.java:97)
    at java.awt.EventQueue$3.run(EventQueue.java:709)
    at java.awt.EventQueue$3.run(EventQueue.java:703)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
    at java.awt.EventQueue.dispatchEvent(EventQueue.java:726)
    at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
    at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
    at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
    at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
    at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)
Caused by: com.linuxense.javadbf.DBFException: Failed to parse Float value [0.00N] : null
    at com.linuxense.javadbf.DBFUtils.readNumericStoredAsText(DBFUtils.java:82)
    at com.linuxense.javadbf.DBFReader.getFieldValue(DBFReader.java:464)
    at com.linuxense.javadbf.DBFReader.nextRecord(DBFReader.java:342)
    ... 52 more
Caused by: java.lang.NumberFormatException
    at java.math.BigDecimal.<init>(BigDecimal.java:494)
    at java.math.BigDecimal.<init>(BigDecimal.java:383)
    at java.math.BigDecimal.<init>(BigDecimal.java:806)
    at com.linuxense.javadbf.DBFUtils.readNumericStoredAsText(DBFUtils.java:80)
    ... 54 more

when I connect to the DBF table, DBFReader has 106765 records, when I try to populate the arraylist it will be with only 1098 rows, in other words, the error must be giving in the 1099 line of DBFReader, it gives the error and stops populating the array

DBF records

ArrayList size

albfernandez commented 3 years ago

I need some time to investigate. I was wrong, it doesn't work for me, it stops at 1098 record, but without exception. there's some "buggy" in the file, as I can see, but others programs can open it, so I'll support it Your problem seems related to #84, but a this moment I can't reproduce it

javadbf-1.12.0-SNAPSHOT.jar.zip

GustavoAlves1993 commented 3 years ago

I do not believe it is related to bug 84, because I have been using your library for a while and I have used it in tables with more records than this, for example one with more than 400 thousand records

I believe that it is incorrect filling in the double fields, or an excess of numbers that breaks the double, or alphanumeric characters

using while on mine it also doesn’t point out the error, I changed it and made a for loop and then it fires the exception, follow the example code

DBFRow row;

        for (int i=0; i<connDBF.getRecordCount(); i++) {
            row = connDBF.nextRow();
            System.out.println(i);
        }

albfernandez commented 3 years ago

I'm still not able to reproduce on my mahine, maybe some setup is diferent. How do you create dbfReader? it's a normal FileInputStream? What OS are you runing? JVM version ?

I'm testing in Linux with openjdk 8 and openjdk 11

GustavoAlves1993 commented 3 years ago

Hi, yes it's a normal FileInputStream

I'm using Windows 10 64 bits and JVM 1.8.0_281 and JDK 8 update 251 (64-bit)

ApurbaPandey commented 3 years ago

I am getting the same exception while reading the stream from S3 (S3ObjectInputStream) or HDFS (FSDataInputStream), but working fine with local files (FileInputStream).

albfernandez commented 3 years ago

have you tried to wrap your inputStream in a BufferedInputStream?

GustavoAlves1993 commented 3 years ago

yes, the problem occurs when I go through the DBFReader and consume the fields inside it, there is a field that is causing the library to drop and I still haven't found a solution for that

thigg commented 3 years ago

Its funny, I have a similar error right now, but for me it's not [0.00N] but [0.00T].

Looking at the hexdump at that location, its actually in the file and not only T, but also F. Funny is, that I wrote the rows with this library, 1.12.0. But this also happened with the old lib before the fork.

What I did was taking an already existing legacy file, opening it and writing 2 rows into it, which then have this error. The fields are both Doubles (boxed - of course) with 0.0

I attached the hexdump with a few lines for context:

0002a030  20 20 20 20 20 32 30 31  35 30 34 32 36 30 39 3a  |     2015042609:|
0002a040  34 35 3a 34 33 20 20 31  20 20 20 20 30 2e 30 30  |45:43  1    0.00|
0002a050  20 20 20 31 36 39 37 30  30 30 30 30 32 35 20 20  |   16970000025  |
0002a060  20 20 33 37 2e 33 38 20  20 20 20 33 37 2e 33 38  |  37.38    37.38|
0002a070  20 20 20 20 20 30 2e 30  30 20 20 20 20 20 30 2e  |     0.00     0.|
0002a080  30 30 20 20 20 20 20 30  2e 30 30 20 20 20 20 20  |00     0.00     |
0002a090  30 2e 30 30 20 20 20 20  20 20 20 20 20 32 30 31  |0.00         201|
0002a0a0  35 30 34 32 36 31 33 3a  35 39 3a 32 39 20 20 34  |5042613:59:29  4|
0002a0b0  20 20 20 20 30 2e 30 30  20 31 36 39 38 20 30 30  |    0.00 1698 00|
0002a0c0  30 30 30 30 31 20 20 20  20 31 30 2e 30 30 20 20  |00001    10.00  |
0002a0d0  20 20 20 30 2e 30 30 20  20 20 20 31 30 2e 30 30  |   0.00    10.00|
0002a0e0  20 20 20 20 20 30 2e 30  30 20 20 20 20 20 30 2e  |     0.00     0.|
0002a0f0  30 30 20 20 20 20 20 30  2e 30 30 20 20 20 20 20  |00     0.00     |
0002a100  30 2e 30 31 32 30 32 31  30 35 32 31 31 35 3a 31  |0.012021052115:1|
0002a110  34 3a 30 31 20 20 31 20  20 20 20 30 2e 30 30 54  |4:01  1    0.00T|
0002a120  20 31 36 39 39 20 30 30  30 30 30 30 32 20 20 20  | 1699 0000002   |
0002a130  20 31 30 2e 30 30 20 20  20 20 31 30 2e 30 30 20  | 10.00    10.00 |
0002a140  20 20 20 20 30 2e 30 30  20 20 20 20 20 30 2e 30  |    0.00     0.0|
0002a150  30 20 20 20 20 20 30 2e  30 30 20 20 20 20 20 30  |0     0.00     0|
0002a160  2e 30 30 20 20 20 20 20  30 2e 30 31 32 30 32 31  |.00     0.012021|
0002a170  30 35 32 31 31 35 3a 31  34 3a 30 31 20 20 31 20  |052115:14:01  1 |
0002a180  20 20 20 30 2e 30 30 46  1a                       |   0.00F.|

thigg commented 3 years ago

Noticed, that I see a T and a F here, and the next field is defined as B_CHECKED,L, apperantly these fields get mixed up somehow

thigg commented 3 years ago

Looking closley at it, it looks like there is a space between the rows in the existing format, which is missing here

albfernandez commented 3 years ago

I've released 1.13 with the proposed changed, check if it works for you

thigg commented 3 years ago

No, unfortunatley exatctly the same error

pablosv94 commented 3 years ago

Hello,

I had the same error and to solve it what I did was comment the DBFException that is thrown in readNumericStoredAsText to return null. First I tried to use the skip method but not wanting to see the deleted rows, it was re-entered by nextRecord and the error occurred again.

How do you see returning null instead of the exception? null is a value that is returned when something goes wrong reading fields and it shouldn't cause any problems.

Greetings and thanks for the effort!

pablosv94 commented 3 years ago

The purpose of return null in readNumericStoreAsText when try to create new BigDecimal(aux) is for continue reading the file, if exceptions throws the execution is stopped.

albfernandez / javadbf

Failed to parse Float #85