confusing results, the same usage memory in loops besides negative values

[alisheikholeslam]

@fabianp How could I get true results or interpret them? memory-profiler shows same usage values for lines in the loop.

import numpy as np
from scipy.spatial import cKDTree, distance
import os
from memory_profiler import profile

radii = np.loadtxt(os.path.join(os.path.join(os.environ['USERPROFILE']), 'Desktop', "data", 'radii.csv'))
poss = np.loadtxt(os.path.join(os.path.join(os.environ['USERPROFILE']), 'Desktop', "data", 'coordinates.csv'), delimiter=",")
print(len(radii))
rad_max = np.amax(np.hstack(radii))
dia_max = 2 * rad_max

@profile
def ends_gap_opt(poss, dia_max):
    particle_corsp_overlaps = []
    ends_ind = [np.empty([1, 2], dtype=np.int64)]

    kdtree = cKDTree(poss)

    for particle_idx in range(len(poss)):
        cur_point = poss[particle_idx]
        nears_i_ind = np.array(kdtree.query_ball_point(cur_point, r=dia_max), dtype=np.int64)
        assert len(nears_i_ind) > 0

        if len(nears_i_ind) <= 1:
            continue

        nears_i_ind = nears_i_ind[nears_i_ind != particle_idx]
        dist_i = distance.cdist(poss[nears_i_ind], cur_point[None, :]).squeeze()

        contact_check = dist_i - (radii[nears_i_ind] + radii[particle_idx])
        connected = contact_check[contact_check <= 0]

        particle_corsp_overlaps.append(connected)

radii.csv coordinates.csv

I have modified the memory_profiler.py as pull requests Fix: Large negative increments #350 and also, in another test using large negative increment values in line profiler #195, . Both solutions remove previous negative values from increments. Which of them is the true one?
The same mem usages are doubtful and seem to be wrong:

[bsjones109]

Hey, just a quick comment on this since I worked on #350 so I was looking at this a few weeks ago. Based on my recollection of the code, the reason you're seeing the similar results is because it updates those values with the max memory usage. So when it goes through the loop to the last line then loops back to the first, it updates the memory value of the first line of the loop with the change since the last line since that was the previous line that it evaluated. So I think the last line of a for loop will always show the same max memory value (the mem usage column) as the first line of the loop.

It's also related to how it tracks the changes to memory. It's looking at how much memory is being used by the process and how that memory changes from one line to the next. So if the process stays the same size from one line to the next, it won't register a change. Individual lines in your code may cause the memory to change by such a small amount that it doesn't register at the scale being tracked. Even comparing line 28 and 30, your program's mem usage only goes up by 0.2-0.3 MiB maximum at any point during the loop.

[nyngwang]

@alisheikholeslam @bsjones109 I have the same problem. Is this resolved? The result gives these hilarious numbers.

[v4ngelo]

Same here. Is there any solution or a way to understand this?