search

fikisipi / elkai

elkai is a Python library for solving travelling salesman problems (TSP) based on LKH 3
Other
177 stars 18 forks source link

RAM memory is not free #15

Open hnnam0906 opened 1 month ago

hnnam0906 commented 1 month ago

When I run elkai with a large number of tests, I see that the RAM memory increases and is not free. So when the RAM memory reaches the maximum of computer RAM (64GB/128BM) it must be restarted to free memory to be able to run.

image

As you can see in the above photo, i run 2 instances with pm2, 1 instance uses 26.6GB, the other has to restart 1 (The curved arrow icon column) to free the memory so that it can be run again.

fikisipi commented 1 month ago

I'll take a look when I have time. I had valgrind running with no memory leaks but it's been a long time. I'm sure there could be leaks now.

pm2 as far as I know is a npm project.

  1. How do you run the tests more specifically? Does one process have multiple threads?
  2. How big are your matrices?
  3. Assuming you need this fast and I cannot fix it, consider just creating a new interpreter once you run many tests.
hnnam0906 commented 1 month ago

Hi @fikisipi , Thanks for your response.

  1. I generate the distance matrices from coordinates of locations then run the tests with asyncio.run.
  2. The range of cities from 2000-5000, so the distance matrices should be 2k x 2k to 5k x 5k
  3. I gen a test with the period of one and half mins. Since they are large number of cities and your original source doesn't support the params MAX_TRIALS, TIME_LIMIT or TOTAL_TIME_LIMIT as inputs so i have to mod your func solve_tsp in the elkai/types.py file some thing like below so that i can exit when timeout and get the best result at this time. image
  4. In addition, when i run the library https://pypi.org/project/lkh/ (its source code is from the git https://github.com/ben-hudson/pylkh) which also calls to LKH 3.0.8 to solve (it uses the subprocess to call the LKH command, but not build the interface between Python and C) with the same test method, it doesn't face the issue with the memory. It will gradually increase the RAM to a certain point (in my case from 800MB to 1.6GB) and it keeps stable there. So i guess the problem should be somewhere from your lib.

I know in general your lib can find a better result than the above LKH lib but it faces the issue with the RAM so that i cannot use it stable. Please support to investigate and correct the issue.

If you need any info, don't hesitate to ask me.

Thanks

hnnam0906 commented 1 month ago

@fikisipi Hi, is there any update from your side? I'd love to use your lib but we need to resolve the above issue first.

hnnam0906 commented 3 weeks ago

Hi @fikisipi If using multiprocessing, it doesn't have any problems with the memory issue, It only happens when i use asyncio.