Closed fido20160817 closed 1 year ago
The MemoryError for large number of points could be caused by the np.kron function in keypoint_guided_optimal_transport /linearprog.py for linear programming (LP) to solve OT. I suggest using Sinkhorn's algorithm for large number of points. Thanks for your question. We will work on a more efficient implementation of LP and update the code if progresses are achieved.
I set num = 200 for each mode in source and target and tried "sinkhorn", but the results were not right:
linear_programming:
sinkhorn (I try to adjust "reg" parameter to get getter results. but the wrong matching line still exists!):
any solution? or am I wrong?
This makes sense because the Sinkhorn solves the entropic OT which is an approximation of the original OT. Meanwhile, the dense transport plan of the entropic OT results in blurred transported samples by the barycentric mapping. Nevertheless, the Sinkhorn may be better for some real-world applications, e.g., these with noises.
We updated the code of linear programming on sparse matrixes in the function "lp" in "keypoint_guided_optimal_transport/linearprog.py", which is memory-efficient for a large number of data points. Please set "sparse=True" in function "lp" to use it as follows:
lp(p, q, C, Mask=None,sparse=True)
The main change is to replace numpy.kron by scipy.sparse.kron. Note that since linear programming is with high time cost, the lp for a large number of data points will take a lot of time.
great! Thanks for your quick reply!
Hi, when there is a large number of points in source and target, MemoryError happens, any solution?
numpy.core._exceptions.MemoryError: Unable to allocate 1.05 TiB for an array with shape (1, 5998, 4001, 5998) and data type float64