search

EnesYildirim / netcdf4-python

Automatically exported from code.google.com/p/netcdf4-python
Other
0 stars 0 forks source link

memory increasing with numerous call to num2date ... #123

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago 
What steps will reproduce the problem?
1. run the attach code 
2.
3.

What is the expected output? What do you see instead?

I expect the size (in memory) of the program to be constant.
Tests shows the memory size of the program to increases with time.

What version of the product are you using? On what operating system?
I use netCDF4 version 0.9.8 with netcdf v4.2.3
I have the problem on a unix machine and a linux machine.

On mac OSX I do not seem to get the problem but I am not sure because the 
get_mem_usage function define in the test does not work.

Please provide any additional information below.

Original issue reported on code.google.com by sbi...@gmail.com on 30 Apr 2012 at 8:20

Attachments:

GoogleCodeExporter commented 8 years ago 
Can't reproduce this.  On my linux system, the memory usage equilibrates at 
324.715 MB.  I don't think there can be a memory leak, since this function is 
pure python code (so the python garbage collector should take care of deleting 
unused references).

Original comment by whitaker.jeffrey@gmail.com on 30 Apr 2012 at 8:38

GoogleCodeExporter commented 8 years ago 
Sorry, I think I lost the problem by trying to oversimplify the case. I also 
got mixed up between machines.

Anyways, attached is a piece of code and a netcdf file that reproduce the 
problem on my linux machines.
If I run it as is the memory increases.
If I comments the dates calculation the memory stays constant.

Hope it works this time.

Cheers.

Original comment by sbi...@gmail.com on 30 Apr 2012 at 9:15

Attachments:

GoogleCodeExporter commented 8 years ago 
Still don't see it.  The memory saturates at about 281 mb for me.   At the end 
I have

iteration=974  *** memory usage    281.273 MB 
iteration=975  *** memory usage    281.273 MB 
iteration=976  *** memory usage    281.273 MB 
iteration=977  *** memory usage    281.273 MB 
iteration=978  *** memory usage    281.273 MB 
iteration=979  *** memory usage    281.273 MB 
iteration=980  *** memory usage    281.273 MB 
iteration=981  *** memory usage    281.273 MB 
iteration=982  *** memory usage    281.273 MB 
iteration=983  *** memory usage    281.273 MB 
iteration=984  *** memory usage    281.273 MB 
iteration=985  *** memory usage    281.273 MB 
iteration=986  *** memory usage    281.273 MB 
iteration=987  *** memory usage    281.273 MB 
iteration=988  *** memory usage    281.273 MB 
iteration=989  *** memory usage    281.273 MB 
iteration=990  *** memory usage    281.273 MB 
iteration=991  *** memory usage    281.273 MB 
iteration=992  *** memory usage    281.273 MB 
iteration=993  *** memory usage    281.273 MB 
iteration=994  *** memory usage    281.273 MB 
iteration=995  *** memory usage    281.273 MB 
iteration=996  *** memory usage    281.273 MB 
iteration=997  *** memory usage    281.273 MB 
iteration=998  *** memory usage    281.273 MB 
iteration=999  *** memory usage    281.273 MB

Original comment by whitaker.jeffrey@gmail.com on 30 Apr 2012 at 10:56

GoogleCodeExporter commented 8 years ago 
About your output, is it not a bit surprising that the size of the program is 
that big given the fact that the time variable is only about 2KB?

Down there you can see the output I get, it just keeps on growing. If I skip 
the call to num2date, the size of the program stays at around 24MB. That is why 
I thought there maybe a problem with nc.num2date().

iteration=988  *** memory usage     99.359 MB
iteration=989  *** memory usage     99.449 MB
iteration=990  *** memory usage     99.535 MB
iteration=991  *** memory usage     99.621 MB
iteration=992  *** memory usage     99.707 MB
iteration=993  *** memory usage     99.785 MB
iteration=994  *** memory usage     99.871 MB
iteration=995  *** memory usage     99.957 MB
iteration=996  *** memory usage    100.043 MB
iteration=997  *** memory usage    100.129 MB
iteration=998  *** memory usage    100.219 MB
iteration=999  *** memory usage    100.305 MB

Original comment by sbi...@gmail.com on 1 May 2012 at 2:01

GoogleCodeExporter commented 8 years ago 
It's not just the size of the time variable.  When you open a dataset, there 
are various memory caches and data structures that are set up by the C library. 
 I think what you are seeing is just the overhead of having the file open.  I 
don't see any increase in memory usage with time, or a significant increase in 
memory usage if num2date is called.  Here what I see

with num2date call:

iteration=992  *** memory usage    324.570 MB 
iteration=993  *** memory usage    324.570 MB 
iteration=994  *** memory usage    324.570 MB 
iteration=995  *** memory usage    324.570 MB 
iteration=996  *** memory usage    324.570 MB 
iteration=997  *** memory usage    324.570 MB 
iteration=998  *** memory usage    324.570 MB 
iteration=999  *** memory usage    324.570 MB 

without num2date call:

iteration=990  *** memory usage    315.371 MB 
iteration=991  *** memory usage    315.371 MB 
iteration=992  *** memory usage    315.371 MB 
iteration=993  *** memory usage    315.371 MB 
iteration=994  *** memory usage    315.371 MB 
iteration=995  *** memory usage    315.371 MB 
iteration=996  *** memory usage    315.371 MB 
iteration=997  *** memory usage    315.371 MB 
iteration=998  *** memory usage    315.371 MB 
iteration=999  *** memory usage    315.371 MB 

Note that the actual number varies a lot from run to run, for me it can range 
from 220 to 370 MB.

Bottom line: it does not look like a memory leak to me.  There may be something 
different happening on your system, but I have no idea what it could be.

BTW: it is much faster if you read the time data from the netcdf variable, and 
then pass that data to num2date, i.e.

times = time[:]
dates = nc.num2date(times, time.units)

Original comment by whitaker.jeffrey@gmail.com on 1 May 2012 at 3:01

GoogleCodeExporter commented 8 years ago 
Hello,

 I have looked around and found the problem not happening on certain linux machines around here. I will ask the sys admin and try to find the cause of my problem. I will let you know of what I find in case it helps anybody.

Thanks for the help and for the speeding tip. It is quite faster indeed.

Seb.

Original comment by sbi...@gmail.com on 1 May 2012 at 7:05

GoogleCodeExporter commented 8 years ago 
If I speed my code by using 

dates = nc.num2date(time[:], time.units)

it goes faster AND takes care of my memory problem ...

Original comment by sbi...@gmail.com on 1 May 2012 at 8:00

GoogleCodeExporter commented 8 years ago 
I got a quest have a motorola xprt sprint logo boostmobile network says it has 
993mbsalot im using 8gig8gig sdcard when i check internal memory it says 0.97 
free app space, i think anyway does that mean ive still have alot of memory or 
is it low?

Original comment by KATCHISO...@gmail.com on 8 Aug 2012 at 3:19

GoogleCodeExporter commented 8 years ago

Original comment by whitaker.jeffrey@gmail.com on 26 Feb 2014 at 2:04