dictionary or sparse matrices support

[JoOkuma]

Hi,

I'm working on cell tracking using ILP. We've been using gurobi for a while but I'm looking for other alternatives that don't require installing/buying licenses.

The problem can reach millions of variables but it's very sparse and it can take a while to build it when using pure python loops.

Is any of the approaches below supported by python-mip? If not, how hard would it be to add this to the current code base, I might have some bandwidth to implement it.

• Adding variables from a dictionary like in gurobi, where the values of the dictionary are added to the objective function (not that important), and you can quickly query and perform sums or products on the dictionary keys, reference, these operations fast and convenient.
• Another alternative is to not use the dictionary and model with add_var_tensor and add constraints by using scipy sparse matrices, cvxpy supports this but it could not handle our problem, it would quickly run out of memory when preprocessing the data even when using gurobi backend, while gurobi's official API was using a fraction of the memory.

Other alternatives are welcome.

[christian2022]

It would be helpful, if you provide some executable example code, showing your issue.

[sebheger]

@JoOkuma thanks for posting your idea here. I would say the same as @christian2022 that we need to understand in more detail what is slow, what is your requirement in details to determine what is necessary and what need to be done

[JoOkuma]

Sorry for the late response. While implementing a reproducible example I found an efficient implementation by mapping the keys to an index array from 0 to N-1 and using the existing var_tensor API.

To query the keys I used the group_by operator from the pandas.DataFrame, when called multiple times it's much faster than selecting from a numpy array directly.

@sebheger and @christian2022, I'm ok with closing the issue since a solution was found.