Closed GoogleCodeExporter closed 9 years ago
It seems to be simply getting out of memory resources even though there's like
16 Gigs physically in nmr. In the shell executing the python code there are no
limits set on memory:
jd:nmr/~/ limit
cputime unlimited
filesize unlimited
datasize 6144 kbytes
stacksize 8192 kbytes
coredumpsize 0 kbytes
memoryuse unlimited
descriptors 256
memorylocked unlimited
maxproc 266
Processes: 113 total, 5 running, 1 stuck, 107 sleeping... 481 threads
14:38:54
Load Avg: 0.38, 0.52, 0.97 CPU usage: 0.57% user, 0.79% sys, 98.64% idle
SharedLibs: num = 21, resident = 52M code, 2108K data, 4016K linkedit.
MemRegions: num = 18007, resident = 702M + 47M private, 314M shared.
PhysMem: 1072M wired, 1275M active, 128M inactive, 2473M used, 14G free.
VM: 8875M + 384M 1329704(0) pageins, 22(0) pageouts
I need some help with this.
Original comment by jurge...@gmail.com
on 9 Jun 2010 at 12:39
Alan, is this something we wouldn't face with a 64 bit Python?
With machines getting bigger, it might make sense to use it. How has your
experience with 64 bit CING been?
Original comment by jurge...@gmail.com
on 9 Jun 2010 at 12:50
considering
http://code.google.com/p/cing/wiki/MacInstallationGuideForFinkIn64Bits
Original comment by jurge...@gmail.com
on 9 Jun 2010 at 12:51
Let me try to run this example here.
Where can I get the input file?
Original comment by alanwil...@gmail.com
on 9 Jun 2010 at 12:58
pdb ;-)
then just initPdb cing with it and run
project.validate()
Original comment by jurge...@gmail.com
on 9 Jun 2010 at 1:04
Rev. 796
After 60 hours running for a real time cpu usage of ~170 min using 2.88 Gb Real
mem and 4.65 Virtual mem (swapping almost all the time), I stopped the
programme. It seemed to have reached further than Jurgen's test.
amadeus[2009]:~/Downloads% cing -n 2uva --initPDB 2UVA.pdb.txt --validate -v 9
/sw/lib/python2.6/site-packages/pytz/tzinfo.py:5: DeprecationWarning: the sets
module is deprecated
from sets import Set
================================================================================
======================
| CING: Common Interface for NMR structure Generation version 0.9
AW,JFD,GWV 2004-2010 |
================================================================================
======================
User: alan on: Unknown host (darwin/64bit) at: Fri
Jun 11 19:41:23 2010
DEBUG:
Version: 0.900
Revision: None (http://code.google.com/p/cing/source/detail?r=None)
Date: 21 April 2010
Database: INTERNAL_0
SML: 0.230
CING root: /Users/alan/workspace/cingWork
CING tmp: /var/folders/+m/+meU9wCwHYKwh16Rr5-GsE+++TM/-Tmp-/cing
Default verbosity: 3
DEBUG: options: {'test2': None, 'old': None, 'docdoc': None, 'export': False,
'nosave': None, 'xeasyPeaks': None, 'generatePeaks': None, 'xeasy': None,
'moleculeName': None, 'script': None, 'init': None, 'initCcpn': None, 'test':
None, 'new': None, 'shiftx': False, 'ensemble': None, 'ranges': None,
'superpose': False, 'ipython': None, 'validate': True, 'info': None, 'name':
'2uva', 'yasara': None, 'doc': None, 'verbosity': 9, 'pydoc': None, 'merge':
None, 'initPDB': '2UVA.pdb.txt', 'initBMRB': None}
DEBUG: args: []
==> Parsing pdbFile "2UVA.pdb.txt" ...
PDB parser: unknown record type: SPLIT
Non-standard (residues and their) atoms to add:
FMN : C1' C10 C2 C2' C3' C4 C4' C4A C5' C5A C6 C7 C7M C8 C8M
C9 C9A N1 N10 N3 N5 O1P O2 O2' O2P O3' O3P O4 O4' O5' P
[ 3079]
DEBUG: Residue.addDihedralD1: skipping non N-terminal residue without doublet
<Residue GLN19> (missing preceding neighbor but not N-terminal)
DEBUG: Residue.addDihedralD1: skipping residue <Residue GLY29> because of
missing atoms
DEBUG: and so on for a total of 1680 messages
==> Molecule 2UVA.pdb: No potential disulfide bridged residues found
Calculating rmsd's (residues: 19-2078,3079, models: 0-0)
----------- Project: 2uva -----------
created: Fri Jun 11 19:41:23 2010
molecules: [<Molecule "2UVA.pdb" (C:6,R:12366,A:273984,M:1)>]
peaks: []
distances: []
dihedrals: []
rdcs: []
coplanars: []
-------------------------------------
==> Executing script
"/Users/alan/workspace/cingWork/python/cing/Scripts/doValidate.py"
==> Running shiftx
runShiftx: non-protein residues [<Residue 2UVA.pdb.G.FMN3079 (355404)>,
<Residue 2UVA.pdb.H.FMN3079 (498582)>, <Residue 2UVA.pdb.I.FMN3079 (641760)>,
<Residue 2UVA.pdb.J.FMN3079 (784938)>, <Residue 2UVA.pdb.K.FMN3079 (928116)>,
<Residue 2UVA.pdb.L.FMN3079 (1071294)>] will be skipped.
0
==> calculating Q-factors for chemical shift
==> Calculating secondary structure by DSSP
WARNING: runWhatif: non-standard residue <Residue 2UVA.pdb.G.FMN3079 (355404)>
found and will be written out for What If
WARNING: runWhatif: non-standard residue <Residue 2UVA.pdb.H.FMN3079 (498582)>
found and will be written out for What If
WARNING: runWhatif: non-standard residue <Residue 2UVA.pdb.I.FMN3079 (641760)>
found and will be written out for What If
WARNING: runWhatif: non-standard residue <Residue 2UVA.pdb.J.FMN3079 (784938)>
found and will be written out for What If
WARNING: runWhatif: non-standard residue <Residue 2UVA.pdb.K.FMN3079 (928116)>
found and will be written out for What If
WARNING: runWhatif: non-standard residue <Residue 2UVA.pdb.L.FMN3079 (1071294)>
found and will be written out for What If
==> Running What If checks on 12366 residues for an estimated (7 residues/s):
1766 seconds; please wait
0ERROR: Whatif._parseCheckdb: file "2uva.cing/2UVA.pdb/Whatif/check_000.db" not
found.
ERROR:
runWhatif: Failed to parse check db check_000.db
==> Running procheck_nmr
No DR lists to export
No DR lists to export
==> Parsing procheck results
==> Finished procheck_nmr successfully
Failed to find ccpn attribute project. Happens when no CCPN project was read
first.
Failed to nmrStar.toNmrStarFile (fine if there wasn't a CCPN project to start
with)
RunTalosPlus: No resonances defined so no sense in running.
==> Analyzing restraints
==> validateRestraints, output to 2uva.cing/2UVA.pdb/Cing/restraints.txt
DEBUG: chk 1 ssType res <Residue GLN19> mdl 0 a2 203.02 a1 252.74
c_av 36 c_sd 37 ck 1 zk -0.95 h- 0 h+ 115 z-
-0.97 z+ 2.13
DEBUG: chk 0 ssType res <Residue SER20> mdl 0 a2 84.06 a1 -147.23
c_av 47 c_sd 45 ck 2 zk -1.00 h- 0 h+ 157 z-
-1.03 z+ 2.45
DEBUG: chk 2 ssType res <Residue SER20> mdl 0 a2 124.54 a1 339.03
c_av 3048 c_sd 2049 ck 165 zk -1.41 h- 8 h+ 8855 z-
-1.48 z+ 2.83
DEBUG: chk 0 ssType res <Residue LEU21> mdl 0 a2 127.61 a1 -138.90
c_av 28 c_sd 21 ck 4 zk -1.15 h- 0 h+ 72 z-
-1.37 z+ 2.11
DEBUG: chk 1 ssType res <Residue LEU21> mdl 0 a2 189.41 a1 279.79
c_av 205 c_sd 168 ck 6 zk -1.18 h- 0 h+ 487 z-
-1.22 z+ 1.67
DEBUG: chk 2 ssType res <Residue LEU21> mdl 0 a2 168.38 a1 124.54
c_av 5220 c_sd 3356 ck 739 zk -1.34 h- 6 h+ 14168 z-
-1.55 z+ 2.67
DEBUG: chk 0 ssType S res <Residue ARG22> mdl 0 a2 109.22 a1 -141.49
c_av 20 c_sd 12 ck 8 zk -1.00 h- 0 h+ 40 z-
-1.67 z+ 1.73
DEBUG: chk 1 ssType S res <Residue ARG22> mdl 0 a2 212.28 a1 289.02
c_av 24 c_sd 18 ck 0 zk -1.28 h- 0 h+ 57 z-
-1.28 z+ 1.80
DEBUG: chk 2 ssType S res <Residue ARG22> mdl 0 a2 244.57 a1 168.38
c_av 4626 c_sd 3576 ck 4808 zk 0.05 h- 0 h+ 12688 z-
-1.29 z+ 2.25
DEBUG: chk 0 ssType S res <Residue PRO23> mdl 0 a2 108.12 a1 -59.95
c_av 31 c_sd 21 ck 0 zk -1.48 h- 0 h+ 70 z-
-1.48 z+ 1.86
DEBUG: chk 1 ssType S res <Residue PRO23> mdl 0 a2 35.72 a1 334.82
c_av 58 c_sd 37 ck 31 zk -0.73 h- 0 h+ 99 z-
-1.58 z+ 1.12
DEBUG: chk 2 ssType S res <Residue PRO23> mdl 0 a2 171.62 a1 244.57
c_av 2840 c_sd 2178 ck 2669 zk -0.08 h- 0 h+ 7644 z-
-1.30 z+ 2.21
DEBUG: chk 0 ssType S res <Residue LEU24> mdl 0 a2 120.79 a1 -110.44
c_av 47 c_sd 31 ck 96 zk 1.61 h- 0 h+ 98 z-
-1.53 z+ 1.67
DEBUG: chk 1 ssType S res <Residue LEU24> mdl 0 a2 185.33 a1 253.36
c_av 126 c_sd 96 ck 1 zk -1.30 h- 0 h+ 256 z-
-1.31 z+ 1.35
...
DEBUG: chk 0 ssType res <Residue TYR2077> mdl 0 a2 7.26 a1 -98.99
c_av 10 c_sd 8 ck 12 zk 0.17 h- 0 h+ 31 z-
-1.38 z+ 2.71
DEBUG: chk 1 ssType res <Residue TYR2077> mdl 0 a2 97.45 a1 305.27
c_av 18 c_sd 16 ck 8 zk -0.63 h- 0 h+ 58 z-
-1.14 z+ 2.58
DEBUG: chk 2 ssType res <Residue TYR2077> mdl 0 a2 200.93 a1 44.72
c_av 1698 c_sd 1445 ck 2011 zk 0.22 h- 0 h+ 6298 z-
-1.18 z+ 3.18
DEBUG: chk 1 ssType res <Residue GLU2078> mdl 0 a2 183.38 a1 193.07
c_av 38 c_sd 40 ck 7 zk -0.77 h- 0 h+ 140 z-
-0.95 z+ 2.58
DEBUG: Expected exactly one but Found 0 histogram for <Residue GLU2078>;
skipping
DEBUG: Failed to getDsspSecStructConsensus from validateDihedralCombinations
for residue; skipping: <Residue FMN3079>
DEBUG: Failed to getDsspSecStructConsensus from validateDihedralCombinations
for residue; skipping: <Residue FMN3079>
DEBUG: Failed to getDsspSecStructConsensus from validateDihedralCombinations
for residue; skipping: <Residue FMN3079>
DEBUG: Failed to getDsspSecStructConsensus from validateDihedralCombinations
for residue; skipping: <Residue FMN3079>
DEBUG: Failed to getDsspSecStructConsensus from validateDihedralCombinations
for residue; skipping: <Residue FMN3079>
DEBUG: Failed to getDsspSecStructConsensus from validateDihedralCombinations
for residue; skipping: <Residue FMN3079>
Original comment by alanwil...@gmail.com
on 14 Jun 2010 at 7:42
Thanks for checking Alan! Looks like 64 bits isn't making the biggest
difference on your 3 G machine.
This problem might slip through to another time.
Original comment by jurge...@gmail.com
on 14 Jun 2010 at 7:49
Well, the difference I see is that 64 bits at least didn't crash (although I
stopped the test), so if I had like 8Gb of Ram, it probably would work much
faster and I would have the chance to see it finish.
Besides, which Mac OSX is the iCing server using?
Original comment by alanwil...@gmail.com
on 14 Jun 2010 at 8:56
Right.
It's using 10.5.8.
Thanks!
Original comment by jurge...@gmail.com
on 14 Jun 2010 at 9:06
This issue comes up for entry 2ku2 as well. 50 models with each ~250 AA.
==> RunTalosPlus: No resonances defined so no sense in running.
==> Analyzing restraints
==> validateRestraints, output to 2ku2.cing/2ku2/Cing/restraints.txt
Python(32801) malloc: *** mmap(size=2097152) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
CING started at : Tue Sep 21 01:05:23 2010
CING stopped at : Tue Sep 21 04:27:53 2010
CING took : 12149.965 s
Traceback (most recent call last):
File "/Users/jd/workspace35/cingStable/python/cing/Scripts/validateEntry.py", line 348, in <module>
status = main(*sys.argv[1:])
File "/Users/jd/workspace35/cingStable/python/cing/Scripts/validateEntry.py", line 255, in main
doWattos=doWattos, doTalos=doTalos):
File "/Users/jd/workspace35/cingStable/python/cing/core/classes.py", line 964, in validate
doWattos = doWattos, doTalos = doTalos)
File "/Users/jd/workspace35/cingStable/python/cing/core/validate.py", line 108, in validate
project.runCingChecks(toFile=True, ranges=ranges)
File "/Users/jd/workspace35/cingStable/python/cing/core/classes.py", line 967, in runCingChecks
return runCingChecks(self, toFile=toFile, ranges = ranges)
File "/Users/jd/workspace35/cingStable/python/cing/core/validate.py", line 71, in runCingChecks
project.checkForSaltbridges(toFile)
File "/Users/jd/workspace35/cingStable/python/cing/core/classes.py", line 970, in checkForSaltbridges
return checkForSaltbridges(self, cutoff = cutoff, toFile = toFile)
File "/Users/jd/workspace35/cingStable/python/cing/core/validate.py", line 772, in checkForSaltbridges
s = validateSaltbridge(res1,res2)
File "/Users/jd/workspace35/cingStable/python/cing/core/validate.py", line 971, in validateSaltbridge
'criteria: %(criterium1)s, %(criterium2)s\n'
File "/Users/jd/workspace35/cingStable/python/cing/Libs/NTutils.py", line 1270, in __init__
dict.__init__(self, *args, **kwds)
MemoryError
Python(32801) malloc: *** mmap(size=2097152) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Exception MemoryError: MemoryError() in <module 'threading' from
'/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threa
ding.pyc'> ignored
Python(32801) malloc: *** mmap(size=2097152) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Error in sys.exitfunc:
MemoryError
Original comment by jurge...@gmail.com
on 21 Sep 2010 at 12:54
- It's taking 2 Mb chunks of memory at a time. size=2097152.
- Python can easily go over 2 Gb. I had a test program go to 3 Gb.
- It's not hardware related as it crashes on the exact same spot.
- Also, memtester shows no mistakes after:
memtester 1G 1
- It's weird that this happens so early in the CING validation.
- The CCPN file reads fine into Analysis. It shows only the hydrogen bond
restraints without assigned atoms. Maybe that's the problem?
Original comment by jurge...@gmail.com
on 22 Sep 2010 at 3:34
Geerten, can you have a quick peak at your saltbridge checking code. Does it
explode with large entries?
I had a brief look but couldn't spot any such case.
Original comment by jurge...@gmail.com
on 4 Jan 2011 at 11:53
Trying debug again with with:
cing -n 2ku2 --initPDB pdb2ku2_001.ent --validateFastest -v 9
Works fine with just one model:
DEBUG: >> <Residue G.ASP225> <Residue G.LYS231>
DEBUG: ------------------ Saltbridge ------------------
residues: <Residue G.ASP225> <Residue G.LYS231>
r (av,sd,min,max): (9.3, nan, 9.3, 9.3)
theta (av,sd): (143.8, nan)
types: [('saltbridge', 0), ('C-C bridge', 0), ('N-O bridge', 0),
('ion pair', 1), ('not observed', 0)]
DEBUG: Ending checkForSaltbridges
==> Calculating rmsd's (ranges:
A.-3-233,B.-3-233,C.-3-233,D.-3-233,E.-3-233,F.-3-233,G.-3-233, models: 0-0)
==> Disulfide analysis, output to 2ku2.cing/pdb2ku2_001/Cing/disulfides.txt
==> Found assigned/overall/fraction for spins: 13C 0/8071/0.00 15N
0/2184/0.00 1H 0/11179/0.00
==> Only spins with fraction >= 0.85 will be flagged when missing: {'13C':
False, '1H': False, '15N': False}
==> Criticizing project: output to "2ku2.cing/pdb2ku2_001/Cing/ROG.txt"
==> summary, output to 2ku2.cing/pdb2ku2_001/Cing/summary.txt
==> Generating Macros
==> Initializing HTML objects
==> Generating CING HTML code.
0...
0..
Creating Procheck html
0.........1.......
Creating Wattos html
0
Html for chain A and its residues
0.........1.........2.........3.........4.........5.........6.........7.........
8.........9.........
0.........1.........2.........3.........4.........5.........6.........7.........
8.........9.........
0.........1.........2.........3......
Html for chain B and its residues
0.........1.........2.........3.........4.........5.........6.........7.........
8.........9.........
0.........1.........2.........3.........4.........5.........6.........7.........
8.........9.........
0.........1.........2.........3......
Html for chain C and its residues
0.........1.........2.........3.........4.........5.........6.........7.........
8.........9.........
0.........1.........2.........3.........4.........5.........6.........7.........
8.........9.........
0.........1.........2.........3......
Html for chain D and its residues
0.........1.........2.........3.........4.........5.........6.........7.........
8.........9.........
0.........1.........2.........3.........4.........5.........6.........7.........
8.........9.........
0.........1.........2.........3......
Html for chain E and its residues
0.........1.........2.........3.........4.........5.........6.........7.........
8.........9.........
0.........1.........2.........3.........4.........5.........6.........7.........
8.........9.........
0.........1.........2.........3......
Html for chain F and its residues
0.........1.........2.........3.........4.........5.........6.........7.........
8.........9.........
0.........1.........2.........3.........4.........5.........6.........7.........
8.........9.........
0.........1.........2.........3......
Html for chain G and its residues
0.........1.........2.........3.........4.........5.........6.........7.........
8.........9.........
0.........1.........2.........3.........4.........5.........6.........7.........
8.........9.........
0.........1.........2.........3......
Html for dihedrals by Project and Residue
Html for atoms and models
Html for peaks and restraints
==> Rendering HTML pages
Done with overall validation
-------------------------------------------------------
==> Saving <Project 2ku2>
==> Saved <Molecule "pdb2ku2_001" (C:7,R:1659,A:30121,M:1)> to
"2ku2.cing/Data/Molecules/pdb2ku2_001"
CING started at : Tue Feb 1 16:06:03 2011
CING stopped at : Tue Feb 1 16:09:57 2011
CING took : 234.916 s
It's taking plenty long now with all models.
Original comment by jurge...@gmail.com
on 1 Feb 2011 at 3:22
But 2ku2's PDB file is running through validateFastest without error. Strange.
Regular run from CCPN with all checks is still running on nmr.
Original comment by jurge...@gmail.com
on 2 Feb 2011 at 10:17
Don't know how but fixed:
http://nmr.cmbi.ru.nl/NRG-CING/data/ku/2ku2/log_validateEntry/2ku2_2011-02-02_05
-29-06.log
Leaving X-ray 2uva for later.
Original comment by jurge...@gmail.com
on 4 Feb 2011 at 3:20
Original issue reported on code.google.com by
jurge...@gmail.com
on 9 Jun 2010 at 12:31