bigmlcom / bigml-java

Java bindings for the BigML API
11 stars 21 forks source link

Incompatibility with ensembles/models with more than 1000 fields #78

Open LeoMtzIT opened 6 months ago

LeoMtzIT commented 6 months ago

Hi, There's an incompatibility with models and ensembles (I don't know if its the same case with deepnets or any other kind of model), where if the ensemble or any of the models that compose it have more than 1000 fields, the rest is ignored, then when the MultiModel is created and tries to construct every LocalPredictiveModel it finds that there are missing fields and throws an Exception with the message "Some fields are missing to generate a local model. Please, provide a model with the complete list of fields.".

In every ensemble and model theres an property called "fields_meta" that indicates the total amount of fields and the count and offset of the current request, and the API supports indicating these parameters so pagination is used and the full list of fields can be retrieved. I supose this needs a kind of extensive change as the resources are all downloaded through the same method without checking what is being retrieve, It should check the kind of resource and check for the pagination data "fields_meta" so consecutive requests are done until the complete list of data (fields) is retrieved.

https://bigml.com/api/ensembles?id=filtering-and-paginating-fields-from-an-ensemble

Call Stack for the case of and ensemble with 107 models and 3877 max columns (same as max used fields, altough ensemble.importance size indicates the actual number of used fields, but I ignore if using that number could affect the construction):

java.lang.Exception: Some fields are missing to generate a local model. Please, provide a model with the complete list of fields.

at org.bigml.binding.LocalPredictiveModel.(LocalPredictiveModel.java:129) at org.bigml.binding.LocalPredictiveModel.(LocalPredictiveModel.java:100) at org.bigml.binding.MultiModel.(MultiModel.java:92) at org.bigml.binding.LocalEnsemble.init(LocalEnsemble.java:295) at org.bigml.binding.LocalEnsemble.(LocalEnsemble.java:154)