search

leginon-org / leginon-redmine-archive

1 stars 0 forks source link

EMDB to Model memory error #997

Open leginonbot opened 5 months ago

leginonbot commented 5 months ago

Author Name: Lauren Fisher (Lauren Fisher) Original Redmine Issue: 997, https://emg.nysbc.org/redmine/issues/997 Original Date: 2010-11-04

EMDB to Model fails to properly upload structures that are "very large". The job either crashes due to a memory error or the model density is inverted (i.e. the threshold is negative, and the structure is inside a box when viewed in chimera).

****Example output from /ami/data00/appion/10oct21z/models/emdb/emdb1571-10nov03o02/emdb1571-10nov03o02.appionsub.log:

!!! WARNING: Volume is very large EMAN 1.9 Cluster ($Date: 2009/02/18 05:12:22 $). Run '/usr/local/EMAN/bin/volume help' for detailed help. Min=-2.739318 Max=4.777329 Mean=-0.000000 Sigma=1.000000 Threshold=1.000000

Threshold Mass 1.000000 25473.826080 1.000000 25473.826080 -2.750000 25473.826080 -2.750000 25473.826080 -2.750000 25473.826080 -2.750000 25473.826080 -2.750000 25473.826080 -2.750000 25473.826080 -2.750000 25473.826080

Image has been modified so threshold=1 is 25473.83 kDa ... finished scaling by mass in 1 min 55 sec !!! WARNING: Volume is very large

****Example command line output from /ami/data00/appion/10oct21z/models/emdb/emdb1571-10nov04k27/modelFromEMDB.log:

... retrieving emdb file: ftp://ftp.ebi.ac.uk/pub/databases/emdb/structures/EMD-1571/map/emd_1571.map.gz Traceback (most recent call last): File "/ami/sw/packages/myami-2.1/appion/bin/modelFromEMDB.py", line 214, in ? emdbmodel.start() File "/ami/sw/packages/myami-2.1/appion/bin/modelFromEMDB.py", line 181, in start emdbfile = self.fetchEMDB(self.params['emdbid'], ccp4name) File "/ami/sw/packages/myami-2.1/appion/bin/modelFromEMDB.py", line 133, in fetchEMDB g = gzip.open(data,'r').read() File "/usr/lib64/python2.4/gzip.py", line 218, in read self._read(readsize) File "/usr/lib64/python2.4/gzip.py", line 278, in _read self._add_read_data( uncompress ) File "/usr/lib64/python2.4/gzip.py", line 295, in _add_read_data self.extrabuf = self.extrabuf + data MemoryError

leginonbot commented 5 months ago

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2010-11-04T23:11:49Z

It is failing on the part where it is trying to set threshold by using the mass. I am not really set up to test this, but we have an 800^3 volume that it is trying to read into memory, which is 440MiB zipped/1.9GiB unzipped... In this case it dies when it tries to unzip the file in memory, but reading it into memory should not kill the program; I assume we are running on guppy. So, maybe we should not unzip it in python and unzip it using the terminal. Later, I think I like to read it into memory to calculate some statistics

Also, maybe we should add a binning option to the program for people that want to use large models in a smaller form, but we want to work with the big ones too.