Wrong result when using List comprehension

[rioj7]

When I allocate a lot of objects with a List comprehension the increment column shows a very low number.

class Particle:
    def __init__(self, x, y, z):
        self.x = x
        self.y = y
        self.z = z

@profile
def allocateParticles():
    particles = [Particle(1, 0, 3) for i in range(100000)]
    for p in particles:
        p.x += 4

if __name__ == '__main__':
    particles = allocateParticles()

Gives the following result

$ python  -m memory_profiler list-comp.py
Filename: list-comp.py

Line #    Mem usage    Increment   Line Contents
================================================
     7   31.035 MiB   31.035 MiB   @profile
     8                             def allocateParticles():
     9   48.793 MiB    0.348 MiB       particles = [Particle(1, 0, 3) for i in range(100000)]
    10   48.793 MiB    0.000 MiB       for p in particles:
    11   48.793 MiB    0.000 MiB           p.x += 4

The increment on line 9 should be around 17MiB. Tracing through the code I found the problem and I could only reliably fix it in the show_results(), not in the data collection. Depending how may lines you use for the List comprehension you get different values for the previous line and the line with the call event.

I modified show_results() (line 754).

        prev_mem = 0      # ADD
        for (lineno, mem) in lines:
            if mem:
                inc = mem[0]
                mem = mem[1]
                inc = max(0,mem-prev_mem)      # ADD
                prev_mem = mem                 # ADD
                mem = template_mem.format(mem)
                inc = template_mem.format(inc)
            else:
                mem = u''
                inc = u''
            tmp = template.format(lineno, mem, inc, all_lines[lineno - 1])
            stream.write(unicode(tmp, 'UTF-8'))

With this modification I get the following result:

$ python -m memory_profiler list-comp.py
Filename: list-comp.py

Line #    Mem usage    Increment   Line Contents
================================================
     7   31.172 MiB   31.172 MiB   @profile
     8                             def allocateParticles():
     9   48.691 MiB   17.520 MiB       particles = [Particle(1, 0, 3) for i in range(100000)]
    10   48.691 MiB    0.000 MiB       for p in particles:
    11   48.691 MiB    0.000 MiB           p.x += 4

[fabianp]

Pull requests welcome:-)

On Wed, Jul 10, 2019, 11:29 rioj7 notifications@github.com wrote:

When I allocate a lot of objects with a List comprehension the increment column shows a very low number.

class Particle: def init(self, x, y, z): self.x = x self.y = y self.z = z

@profile def allocateParticles(): particles = [Particle(1, 0, 3) for i in range(100000)] for p in particles: p.x += 4

if name == 'main': particles = allocateParticles()

Gives the following result

$ python -m memory_profiler list-comp.py Filename: list-comp.py

Line # Mem usage Increment Line Contents

 7   31.035 MiB   31.035 MiB   @profile
 8                             def allocateParticles():
 9   48.793 MiB    0.348 MiB       particles = [Particle(1, 0, 3) for i in range(100000)]
10   48.793 MiB    0.000 MiB       for p in particles:
11   48.793 MiB    0.000 MiB           p.x += 4

The increment on line 9 should be around 17MiB. Tracing through the code I found the problem and I could only reliably fix it in the show_results(), not in the data collection. Depending how may lines you use for the List comprehension you get different values for the previous line and the line with the call event.

I modified show_results() (line 754).

    prev_mem = 0      # ADD
    for (lineno, mem) in lines:
        if mem:
            inc = mem[0]
            mem = mem[1]
            inc = max(0,mem-prev_mem)      # ADD
            prev_mem = mem                 # ADD
            mem = template_mem.format(mem)
            inc = template_mem.format(inc)
        else:
            mem = u''
            inc = u''
        tmp = template.format(lineno, mem, inc, all_lines[lineno - 1])
        stream.write(unicode(tmp, 'UTF-8'))

With this modification I get the following result:

$ python -m memory_profiler list-comp.py Filename: list-comp.py

Line # Mem usage Increment Line Contents

 7   31.172 MiB   31.172 MiB   @profile
 8                             def allocateParticles():
 9   48.691 MiB   17.520 MiB       particles = [Particle(1, 0, 3) for i in range(100000)]
10   48.691 MiB    0.000 MiB       for p in particles:
11   48.691 MiB    0.000 MiB           p.x += 4

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pythonprofilers/memory_profiler/issues/238?email_source=notifications&email_token=AACDZB6NBYWLRBEDL2V35OLP6X57NA5CNFSM4H7QTGFKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4G6MLHIQ, or mute the thread https://github.com/notifications/unsubscribe-auth/AACDZB6HCXMQM65Z6U3TBNTP6X57NANCNFSM4H7QTGFA .

[rioj7]

This method ignores the inc calculations in the data collection. So they can be removed.