I am having an issue with big amounts of data

[simenlf]

Hello I love the library. It works great, however I am having some problems with large amounts of data. I get the message:

lp_set_mat_row: i = 1; ind[11] = 405; duplicate column indices not allowed glpk.js?eval:4 Error detected in file glpapi01.c at line 776

glpk.js?eval:71179 Uncaught abort() at Error at jsStackTrace (https://webidetesting5757664-a2d39b03e.dispatcher.hana.ondemand.com/webapp/lib/glpk.js?eval:952:20) at stackTrace (https://webidetesting5757664-a2d39b03e.dispatcher.hana.ondemand.com/webapp/lib/glpk.js?eval:967:29) at Object.abort (https://webidetesting5757664-a2d39b03e.dispatcher.hana.ondemand.com/webapp/lib/glpk.js?eval:71173:51) at _abort (https://webidetesting5757664-a2d39b03e.dispatcher.hana.ondemand.com/webapp/lib/glpk.js?eval:2849:25) at Array.xd (https://webidetesting5757664-a2d39b03e.dispatcher.hana.ondemand.com/webapp/lib/glpk.js?eval:37338:52) at $e (https://webidetesting5757664-a2d39b03e.dispatcher.hana.ondemand.com/webapp/lib/glpk.js?eval:41804:23) at Db (https://webidetesting5757664-a2d39b03e.dispatcher.hana.ondemand.com/webapp/lib/glpk.js?eval:51335:13) at ccallFunc (https://webidetesting5757664-a2d39b03e.dispatcher.hana.ondemand.com/webapp/lib/glpk.js?eval:532:26) at Object.Module.solve (https://webidetesting5757664-a2d39b03e.dispatcher.hana.ondemand.com/webapp/lib/glpk.js?eval:71232:10) at f.doMagic (https://webidetesting5757664-a2d39b03e.dispatcher.hana.ondemand.com/webapp/modules/material/material.controller.js?eval:390:49) If this abort() is unexpected, build with -s ASSERTIONS=1 which can give more information. abort @ glpk.js?eval:71179 _abort @ glpk.js?eval:2849 xd @ glpk.js?eval:37338 $e @ glpk.js?eval:41804 Db @ glpk.js?eval:51335 ccallFunc @ glpk.js?eval:532 Module.solve @ glpk.js?eval:71232 doMagic @ material.controller.js?eval:390 triggerOprimizerChange @ material.controller.js?eval:244 a.fireEvent @ EventProvider-dbg.js:229 a.fireEvent @ Element-dbg.js:427 (anonymous) @ ManagedObjectMetadata-dbg.js:428 B.ontap @ Button-dbg.js:263 a._handleEvent @ Element-dbg.js:162 U._handleEvent @ UIArea-dbg.js:786 dispatch @ jquery-dbg.js:4737 g @ jquery-mobile-custom-dbg.js:1972 q @ jquery-mobile-custom-dbg.js:2063 dispatch @ jquery-dbg.js:4737 c3.handle @ jquery-dbg.js:4549 trigger @ jquery-dbg.js:7819 (anonymous) @ jquery-dbg.js:7903 each @ jquery-dbg.js:365 each @ jquery-dbg.js:137 trigger @ jquery-dbg.js:7902 P @ jquery-mobile-custom-dbg.js:1543 R @ jquery-mobile-custom-dbg.js:1553 dispatch @ jquery-dbg.js:4737 c3.handle @ jquery-dbg.js:4549

[jvail]

Hi, "large amounts" of data might be a problem but probably your model is incorrect: Duplicate variable/constraint names? Just guessing...

[simenlf]

Yeah. You are right. That was what I was guessing, but I cannot find the issue in the json, and it works fine with smaller datasets. Is it ok for me to send you the dataset?

[jvail]

Can you upload it somewhere?

For "large", "professional" problems you should consider using the original glpk c implementation or another solver e.g. scip

[simenlf]

https://www.dropbox.com/s/wjsfqac57at72tk/json_in.json?dl=0 Hey! Sorry for beeing late on the reply. Somehow I am not getting notifications. Should have checked anyways...

If you take a look at the attached .json file you will see what I am attempting to do. It is not huge, but large. Hope to do this using your excellent .js lib. It works great apart from this.

[jvail]

Hi @simenlf,

the little script yields 208 duplicate names in the obj func. Could you retry after fixing this?

let fs = require('fs');
let duplicates = require('array-duplicates');
let lp = JSON.parse(fs.readFileSync('./lp.json').toString());
let vars = lp.objective.vars.map(d => {
    return d.name;
});
let cons = lp.subjectTo.map(d => {
    return d.name;
});
console.log(duplicates(vars).length)
console.log(duplicates(cons).length)

[simenlf]

Hey! Thanks for the reply. I am running this in a browser. Exactly where do you need me to do the change?

[jvail]

I don't know how you create the json but make sure the script creates unique variable names in the objective array or add them together by setting the coef to the sum of all unique variables.

[simenlf]

Hey Jvail. Want to make some money while helping me with this? Perhaps a few hours as a consultant? If you want, it would be great for me.. Pressed for time you see..

[jvail]

No problem @simenlf. If you share some code with we (quasigit@gmx.de or private repo) I'll take a look at it. If I can't figure out whats wrong after one hour i'll let you know my rate ;)

[simenlf]

Great! I have some meetings tomorrow morning, but I hope to be able to send over some json files to you after that.

[simenlf]

Hey. Have not forgot about you. We just need to make som modifications to some backend ettings before we get a representative results. This has to be done later tonight. So I will have some examples tomorrow.

[jvail]

Sure, just drop me a mail. I'll close this one.

[jvail]

Hi @simenlf, at least the "large data/lp" problem should be solved with the new release. I remember that I had some trouble with very large JSON files (> few hundred variables). Those kept failing because of cson JSON parsing. Since I removed the cson dependency there should be no size limits any more.

[simenlf]

Thanks @jvail! We will try it out. These weeks before xmas has been crazy... Looking forward to vacation