reduce memory footprint

[GoogleCodeExporter]

hi, I am using 'wfrog' in a underpowered machine; 'top' reports this memory 
usage

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
 3462 root      30  10 77476 1156  364 S   7.5  9.1  54:38.57 wfrog                            
 3528 root      20   0  9704 1364  728 S   0.0 10.8   0:08.97 python

the second process is a python process where I loaded 
some of the modules
involved in running 'wfrog' and I connected to the sqlite database;
as you see the VIRT size is still much much less.

Have you any hint or idea on how to reduce it?

Original issue reported on code.google.com by mennu...@debian.org on 29 Mar 2013 at 6:53

[GoogleCodeExporter]

let me add that the sqlite database contained only today's data when I did that 
experiment

Original comment by mennu...@debian.org on 29 Mar 2013 at 6:59

[GoogleCodeExporter]

Hi,

This is my wfrog box (ALIX 3D2 - 256Mb RAM) with 3 years of data (5 minutes 
samples) using a Firebird database and reporting to Wunderground, 
meteoclimatic, pwsweather and generating a web page via FTP (10 minutes update):

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 2873 root      20   0 42708  12m 4496 S  1.0  5.1  98:40.31 wfrog              
20745 root      20   0 56384  18m 4564 S  0.3  7.2  48:39.81 wfrog              
 2843 firebird  20   0 79584 5512 3396 S  0.0  2.1  20:13.56 fbserver

I am running render and logger in separate processes. Memory consumption looks 
coherent with your numbers. Most wfrog classes are instantiated through yaml 
configuration files, so it is likely that just doing imports of wfrog modules 
is not equivalent.

Jordi. 

Original comment by jordi.pu...@gmail.com on 29 Mar 2013 at 7:12

[GoogleCodeExporter]

hi,

I looked a bit into it; I tried to simplify the code, but it is too entangled, 
too many crossreferences; so instead I conjured a small script that only logs 
data. Note that the station type has to be configured by editing the initial 
part.

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
 1517 root      30  10 18320 1948  780 R   7.5 15.4   0:34.15 wfdriver_light 

Original comment by mennu...@debian.org on 27 Apr 2013 at 3:00

Attachments:

• wfdriver_light

[GoogleCodeExporter]

hi

there is a fundamental problem in Python

http://bugs.python.org/issue9072

"Please accept that Python indeed does not support unloading modules for 
severe, fundamental, insurmountable, technical problems, in 2.x."

So the more modules a program uses (even to just call a single method once), 
the larger its footprint

If you wish, take a loot into the 'lessmemory' branch
git.debian.org/git/collab-maint/wfrog.git
where I lost some time to no result

I started understanding the problem using the attached snippet.

Still something may be done. IMHO wfrog uses too many threads , and one thread 
just contains a time.sleep, while others just do message passing. Refactoring 
the code in wflogger.py may improve things a bit (threads are rumored to use 
~8MB memory each). my code example wfdriver_light shows that actually the 
logging of data may be done w/o any threads.

a.

Original comment by mennu...@debian.org on 1 May 2013 at 7:24

Attachments:

• vari_test.py