search

bigmlcom / bigml-java

Java bindings for the BigML API
12 stars 21 forks source link

Error in LocalDeepnet fillArray() #64

Closed LeoMtzIT closed 2 years ago

LeoMtzIT commented 2 years ago

Hi! When trying to make a prediction on a regression LocalDeepnet, an exception gets thrown at the fillArray() method, specifically a NullPointerException. This is due to the Deepnet being made from a dataset with almost 2000 fields, the final Deepnet uses only 1000, so when fillArray() tries to iterate over the full list of fields expanding items and finding missings and finds one, it tries to get the field object from the input_fields which doesn't contains it, it retrieves a null and tries to cast it to Number and invoke intVal() on it.

https://github.com/bigmlcom/bigml-java/blob/d8181d8c4a1609d93bca31c218ce706ab6eef475/src/main/java/org/bigml/binding/LocalDeepnet.java#L455-L456

Maybe I'm not yet fully understanding the function objective but I believe fillArray() should only iterate over input_fields as it should find missings on that list, but I'm not sure if the full list of fields is later needed, if that would be the case, then I think there might be an issue with the structure of the model data. The dataset has no missings and every input is categorical, but the objective field is numeric. The Deepnet was created through OptiML on the web platform and is being consumed with the bindings. Thank you in advance! I'm available to provide any more information about my implementation.

javinp commented 2 years ago

H @LeoMtzIT ,

We are iterating inputFields, what is correct.

I think the problem is we are not checking if 'summary.missing_count' is returning a value.

Are you running from the source code?. If so, can you change that line with this and see if problem persist?

Number missingCountNumber = (Number) Utils.getJSONObject(fields, fieldId + ".summary.missing_count"); int missingCount = missingCountNumber != null ? missingCountNumber.intValue() : 0;

We are preparing a new version, but it would be great to confirm with you this fixes the problem.

Thank you!

LeoMtzIT commented 2 years ago

H @LeoMtzIT ,

We are iterating inputFields, what is correct.

I think the problem is we are not checking if 'summary.missing_count' is returning a value.

Are you running from the source code?. If so, can you change that line with this and see if problem persist?

Number missingCountNumber = (Number) Utils.getJSONObject(fields, fieldId + ".summary.missing_count"); int missingCount = missingCountNumber != null ? missingCountNumber.intValue() : 0;

We are preparing a new version, but it would be great to confirm with you this fixes the problem.

Thank you!

Thanks to you! I am running from the source code and the change has fixed the issue! The predictions are working, nonetheless, the results for the same inputs are not the same when consuming the deepnet over the REST API. The unused fields are the same but the prediction is very different. I've been checking the inputs and everything seems to be equal on the LocalDeepnet and the body of the REST API request, I'd appreciate if you know about something that I could be missing but I understand this might be not the place. Anyways, the main issue has been resolved, thank you so much for the attention and quick response!

javinp commented 2 years ago

@LeoMtzIT glad to hear this problem is fixed!, there is a new version of the library available.

About the problem with the local prediction, if you send us an email to support@bigml.com we will investigate it. We will need your help to solve it providing specific info and this is not the place to do it.

I close this issue.

Thank you!