Closed gforney closed 9 years ago
here's the error file:
Original issue reported on code.google.com by BlackstoneEngineering
on 2010-12-07 23:52:21
Here it is crashing under the serial version:
C:\Program Files (x86)\NIST\Mill\PorMillFixed>fds5_win_64.ex
e PorterdaleMill.fds 1>PorterdleMill.err
Fire Dynamics Simulator
Compilation Date : Fri, 29 Oct 2010
Version: 5.5.3; MPI Disabled; OpenMP Disabled
SVN Revision No. : 7031
Job TITLE : Mill Lofts Condo Atrium Smoke Control Sy
stem - revised design
Job ID string : PorterdaleMill
Time Step: -40, Simulation Time: 0.00 s
Time Step: -30, Simulation Time: 0.00 s
Time Step: -20, Simulation Time: 0.00 s
Time Step: -10, Simulation Time: 0.00 s
Time Step: -9, Simulation Time: 0.00 s
Time Step: -8, Simulation Time: 0.00 s
Time Step: -7, Simulation Time: 0.00 s
Time Step: -6, Simulation Time: 0.00 s
Time Step: -5, Simulation Time: 0.00 s
Time Step: -4, Simulation Time: 0.00 s
Time Step: -3, Simulation Time: 0.00 s
Time Step: -2, Simulation Time: 0.00 s
Time Step: -1, Simulation Time: 0.00 s
Time Step: 0, Simulation Time: 0.00 s
Time Step: 1, Simulation Time: 0.10 s
Time Step: 2, Simulation Time: 0.20 s
Time Step: 3, Simulation Time: 0.30 s
Time Step: 4, Simulation Time: 0.40 s
Time Step: 5, Simulation Time: 0.49 s
Time Step: 6, Simulation Time: 0.59 s
Time Step: 7, Simulation Time: 0.69 s
Time Step: 8, Simulation Time: 0.79 s
Time Step: 9, Simulation Time: 0.89 s
Time Step: 10, Simulation Time: 0.99 s
Time Step: 20, Simulation Time: 1.98 s
Time Step: 30, Simulation Time: 2.97 s
Time Step: 40, Simulation Time: 3.51 s
Time Step: 50, Simulation Time: 3.95 s
Time Step: 60, Simulation Time: 4.34 s
Time Step: 70, Simulation Time: 4.72 s
Time Step: 80, Simulation Time: 5.11 s
Time Step: 90, Simulation Time: 5.43 s
Time Step: 100, Simulation Time: 5.73 s
Time Step: 200, Simulation Time: 8.61 s
forrtl: severe (157): Program Exception - access violation
Image PC Routine Line
Source
fds5_win_64.exe 000000014041038A Unknown U
nknown Unknown
fds5_win_64.exe 0000000140412F97 Unknown U
nknown Unknown
fds5_win_64.exe 00000001404A7769 Unknown U
nknown Unknown
fds5_win_64.exe 000000014058DDDC Unknown U
nknown Unknown
fds5_win_64.exe 000000014056CE2B Unknown U
nknown Unknown
kernel32.dll 0000000076CABE3D Unknown U
nknown Unknown
ntdll.dll 0000000076EB6A51 Unknown U
nknown Unknown
And below the error report:
Original issue reported on code.google.com by BlackstoneEngineering
on 2010-12-08 01:06:04
This same 'fds' file ran fine under FDS 5.4.3...... gonna try deleting sprinkler input
with 64-bit MPI next!
Original issue reported on code.google.com by BlackstoneEngineering
on 2010-12-08 01:07:56
Addition to Comment #3 - ran fine under FDS 5.4.3 32-bit MPI...
Original issue reported on code.google.com by BlackstoneEngineering
on 2010-12-08 01:17:18
Actually at around 10-11s the smoke detector activates - sprinklers activate around
a minute or so...
Original issue reported on code.google.com by BlackstoneEngineering
on 2010-12-08 01:46:54
removed sprinkler input & it crashed at same point so probably something else causing
this issue...
Original issue reported on code.google.com by BlackstoneEngineering
on 2010-12-08 02:00:24
Seems to fun fine under FDS 5.4.3 64-bit MPI...
C:\Program Files (x86)\NIST\Mill\PorMillFixed>mpiexec -n 4 -
localonly -noprompt fds5_mpi_win_64.exe PorterdaleMill.fds
1>PorMillLofts.err
Process 3 of 3 is running on FireBat-PC
Process 0 of 3 is running on FireBat-PC
Process 2 of 3 is running on FireBat-PC
Process 1 of 3 is running on FireBat-PC
Mesh 1 is assigned to Process 0
Mesh 2 is assigned to Process 1
Mesh 3 is assigned to Process 2
Mesh 4 is assigned to Process 3
Mesh 5 is assigned to Process 3
Mesh 6 is assigned to Process 3
Mesh 7 is assigned to Process 3
Mesh 8 is assigned to Process 3
Mesh 9 is assigned to Process 3
Mesh 10 is assigned to Process 3
Mesh 11 is assigned to Process 3
Mesh 12 is assigned to Process 3
Mesh 13 is assigned to Process 3
Mesh 14 is assigned to Process 3
Fire Dynamics Simulator
Compilation Date : Thu, 03 Dec 2009
Version: 5.4.3; MPI Enabled; OpenMP Disabled
SVN Revision No. : 5210
Job TITLE : Mill Lofts Condo Atrium Smoke Control Sy
stem - revised design
Job ID string : PorterdaleMill
Time Step: -40, Simulation Time: 0.00 s
Time Step: -30, Simulation Time: 0.00 s
Time Step: -20, Simulation Time: 0.00 s
Time Step: -10, Simulation Time: 0.00 s
Time Step: -9, Simulation Time: 0.00 s
Time Step: -8, Simulation Time: 0.00 s
Time Step: -7, Simulation Time: 0.00 s
Time Step: -6, Simulation Time: 0.00 s
Time Step: -5, Simulation Time: 0.00 s
Time Step: -4, Simulation Time: 0.00 s
Time Step: -3, Simulation Time: 0.00 s
Time Step: -2, Simulation Time: 0.00 s
Time Step: -1, Simulation Time: 0.00 s
Time Step: 0, Simulation Time: 0.00 s
Time Step: 1, Simulation Time: 0.10 s
Time Step: 2, Simulation Time: 0.20 s
Time Step: 3, Simulation Time: 0.30 s
Time Step: 4, Simulation Time: 0.40 s
Time Step: 5, Simulation Time: 0.49 s
Time Step: 6, Simulation Time: 0.59 s
Time Step: 7, Simulation Time: 0.69 s
Time Step: 8, Simulation Time: 0.79 s
Time Step: 9, Simulation Time: 0.89 s
Time Step: 10, Simulation Time: 0.99 s
Time Step: 20, Simulation Time: 1.98 s
Time Step: 30, Simulation Time: 2.97 s
Time Step: 40, Simulation Time: 3.55 s
Time Step: 50, Simulation Time: 4.01 s
Time Step: 60, Simulation Time: 4.42 s
Time Step: 70, Simulation Time: 4.80 s
Time Step: 80, Simulation Time: 5.18 s
Time Step: 90, Simulation Time: 5.52 s
Time Step: 100, Simulation Time: 5.84 s
Time Step: 200, Simulation Time: 8.91 s
Time Step: 300, Simulation Time: 11.35 s
Time Step: 400, Simulation Time: 11.90 s
Time Step: 500, Simulation Time: 12.40 s
Time Step: 600, Simulation Time: 12.82 s
Time Step: 700, Simulation Time: 13.21 s
Time Step: 800, Simulation Time: 13.58 s
Time Step: 900, Simulation Time: 13.95 s
...and still running. So, somewhere between 5.4.3 and 5.5.3 the 64-bit MPI got broke,
again?
Original issue reported on code.google.com by BlackstoneEngineering
on 2010-12-08 05:50:18
Issue 1285 has been merged into this issue.
Original issue reported on code.google.com by mcgratta
on 2010-12-08 13:16:23
Does the 32 bit version work, either serial or MPI?
Original issue reported on code.google.com by mcgratta
on 2010-12-08 14:25:55
Comment 8 by project member mcgratta, Today (6 hours ago)
Issue 1285 has been merged into this issue.
- Thanks, Kevin... my bad ;-)
Comment 9 by project member mcgratta, Today (5 hours ago)
Does the 32 bit version work, either serial or MPI?
- it ran all the way on 32-bit MPI using FDS 5.4.3 on my XP Core2Quad machine (don't
mess with 32-bit serial...), but I haven't tried 32-bit MPI using FDS 5.5.3a... currently
only have 5.4.3 installed on the XP machine and frankly it works so well for me on
that machine I'm not planning on upgrading FDS on it. The main reason I'm fixated on
getting the 64-bit MPI working is I plan to build a monster machine using twin Core2Quad
64-bit m'board with 32Gb ram in the future which depends on 64-bit MPI running flawlessly
:-\
Original issue reported on code.google.com by BlackstoneEngineering
on 2010-12-08 19:51:10
Just download the 32 bit bundle. The executables will co-exist in the FDS directory.
You just need to change 64 to 32 in your command. If both 32 and 64 bit fail, then
it's probably a bug. If only 64 bit fails, it's much more difficult to debug.
Original issue reported on code.google.com by mcgratta
on 2010-12-08 19:55:05
Comment 11 by project member mcgratta, Yesterday (21 hours ago)
Just download the 32 bit bundle. The executables will co-exist in the FDS directory.
You just need to change 64 to 32 in your command. If both 32 and 64 bit fail, then
it's probably a bug. If only 64 bit fails, it's much more difficult to debug.
Tried that, but get the following error message: "This application has failed to start
because fmpich2.dll was not found. Re-installing the application may fix the problem"
Error probably due to my MPICH2 is for 64-bit?
Tried to run non-MPI 32-bit, but get "ERROR: Memory allocation failed for WALL in the
routine INIT"
Error probably due to the monster size of this simulation - way too much for 32-bit
single processor or something?
Original issue reported on code.google.com by BlackstoneEngineering
on 2010-12-09 17:21:47
You're right. I forgot about the 64 bit MPICH2 install. Well, I can try running the
case on my linux cluster.
Original issue reported on code.google.com by mcgratta
on 2010-12-09 17:43:51
FDS 5.5.0 64-bit MPI seems to run fine:
C:\Program Files (x86)\NIST\Mill\PorMillLofts>mpiexec -n 4 -
localonly -noprompt fds5_mpi_win_64.exe PorterdaleMill.fds
1>PorMillLofts.err
Process 2 of 3 is running on FireBat-PC
Process 0 of 3 is running on FireBat-PC
Process 3 of 3 is running on FireBat-PC
Process 1 of 3 is running on FireBat-PC
Mesh 1 is assigned to Process 0
Mesh 2 is assigned to Process 1
Mesh 3 is assigned to Process 2
Mesh 4 is assigned to Process 3
Mesh 5 is assigned to Process 3
Mesh 6 is assigned to Process 3
Mesh 7 is assigned to Process 3
Mesh 8 is assigned to Process 3
Mesh 9 is assigned to Process 3
Mesh 10 is assigned to Process 3
Mesh 11 is assigned to Process 3
Mesh 12 is assigned to Process 3
Mesh 13 is assigned to Process 3
Mesh 14 is assigned to Process 3
Fire Dynamics Simulator
Compilation Date : Mon, 05 Apr 2010
Version: 5.5.0; MPI Enabled; OpenMP Disabled
SVN Revision No. : 6004
Job TITLE : Mill Lofts Condo Atrium Smoke Control Sy
stem - revised design
Job ID string : PorterdaleMill
Time Step: -40, Simulation Time: 0.00 s
Time Step: -30, Simulation Time: 0.00 s
Time Step: -20, Simulation Time: 0.00 s
Time Step: -10, Simulation Time: 0.00 s
Time Step: -9, Simulation Time: 0.00 s
Time Step: -8, Simulation Time: 0.00 s
Time Step: -7, Simulation Time: 0.00 s
Time Step: -6, Simulation Time: 0.00 s
Time Step: -5, Simulation Time: 0.00 s
Time Step: -4, Simulation Time: 0.00 s
Time Step: -3, Simulation Time: 0.00 s
Time Step: -2, Simulation Time: 0.00 s
Time Step: -1, Simulation Time: 0.00 s
Time Step: 0, Simulation Time: 0.00 s
Time Step: 1, Simulation Time: 0.10 s
Time Step: 2, Simulation Time: 0.20 s
Time Step: 3, Simulation Time: 0.30 s
Time Step: 4, Simulation Time: 0.40 s
Time Step: 5, Simulation Time: 0.49 s
Time Step: 6, Simulation Time: 0.59 s
Time Step: 7, Simulation Time: 0.69 s
Time Step: 8, Simulation Time: 0.79 s
Time Step: 9, Simulation Time: 0.89 s
Time Step: 10, Simulation Time: 0.99 s
Time Step: 20, Simulation Time: 1.98 s
Time Step: 30, Simulation Time: 2.97 s
Time Step: 40, Simulation Time: 3.54 s
Time Step: 50, Simulation Time: 4.03 s
Time Step: 60, Simulation Time: 4.45 s
Time Step: 70, Simulation Time: 4.85 s
Time Step: 80, Simulation Time: 5.24 s
Time Step: 90, Simulation Time: 5.57 s
Time Step: 100, Simulation Time: 5.87 s
Time Step: 200, Simulation Time: 8.88 s
Time Step: 300, Simulation Time: 11.45 s
Time Step: 400, Simulation Time: 12.12 s
Time Step: 500, Simulation Time: 12.61 s
...and still going, so it does not appear to be a FDS 5.5.X vs. FDS 5.4.X thing...
Original issue reported on code.google.com by BlackstoneEngineering
on 2010-12-09 18:55:12
On a side note - does FDS 5.5.3a fix the WFDS issue of the green tree cones (object:Canopy)
burning away, yet?
(I guess this question is one for Ruddy...)
Original issue reported on code.google.com by BlackstoneEngineering
on 2010-12-09 20:05:49
I have reproduced the crash just beyond 300 time steps on my 32 bit linux cluster. I'm
going to remove evac stuff to see if that makes a difference.
Original issue reported on code.google.com by mcgratta
on 2010-12-09 20:40:24
I removed the people and the case runs beyond the point of failure. I'm going to turn
this case over to Timo.
Original issue reported on code.google.com by mcgratta
on 2010-12-09 21:51:12
Well, I try to run it in our Linux cluster.
SVN Revision No.: 7310 should probably work.
So, one should run it using 4 threads. I'll
try that.
TimoK
Original issue reported on code.google.com by tkorhon1
on 2010-12-10 07:49:29
Hi,
I did run the latest SVN version (7310) in an evacuation
drill mode and it completed successfully. Well, I had
to modify the input file a little bit. I took away the
hole/vent/obst lines that had "smoke1" or "smoke2"
devices. I should change the code so that in an evacuation
drill mode the devc_id:s, ctrl_id:s etc are skipped.
Process 0 of 0 is running on compute-0-0.local
Mesh 1 is assigned to Process 0
Mesh 2 is assigned to Process 0
Mesh 3 is assigned to Process 0
Mesh 4 is assigned to Process 0
Mesh 5 is assigned to Process 0
Mesh 6 is assigned to Process 0
Mesh 7 is assigned to Process 0
Mesh 8 is assigned to Process 0
Mesh 9 is assigned to Process 0
Mesh 10 is assigned to Process 0
Mesh 11 is assigned to Process 0
EVAC: Emesh 1 MainEvacGrid1 has 0 door flow fields
EVAC: Emesh 2 MainEvacGrid2 has 0 door flow fields
EVAC: Emesh 3 MainEvacGrid3 has 0 door flow fields
Fire Dynamics Simulator
Compilation Date : Thu, 09 Dec 2010
Version: 6.0.0; MPI Enabled; OpenMP Disabled
SVN Revision No. : 7310
Time Step: 24000, Simulation Time: 1200.00 s
Time Step: 24001, Simulation Time: 1200.05 s
STOP: FDS completed successfully
The fire+evacuation case is still running:
Time Step: 100, Simulation Time: 4.70 s
TimoK
Original issue reported on code.google.com by tkorhon1
on 2010-12-10 08:30:40
So, the latest version (7310) crashed. This seems to
do with the creation/removal things. Might be that
the evacuation meshes do something wrong with open/close.
So, I should take the devc_id/ctrl_id things away from
the evacuation meshes. I'll try this next, but it might be
that you'll get the answer on 20th Dec. I'm out of the
office next week.
Time Step: 300, Simulation Time: 8.79 s
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
fds5_mpi_intel_li 00000000008CEA7B init_mp_create_or 2610 init.f90
fds5_mpi_intel_li 00000000008D192E init_mp_open_and_ 2477 init.f90
fds5_mpi_intel_li 00000000009BE210 MAIN__ 747 main.f90
TimoK
Original issue reported on code.google.com by tkorhon1
on 2010-12-10 11:14:05
Well, an alternative inputs that might help:
Try to put EVACUATION=.FALSE. to all OBST/HOLE
lines, where you have devc_id/ctrl_id given
(it is false by default for vents). This way
the evacuation meshes should not have any time
dependent things. Evacuation meshes should not
have any time dependent stuff there.
Check your evacuation geometry, if you need some
of the above holes/obsts there, create duplicate
holes/obsts and put EVACUATION=.TRUE. for these
and do not give any devc_id or ctrl_id.
And this issue is not only for linux 60bit. It is
for all operating systems, because it is really a
bug in the code.
TimoK
Original issue reported on code.google.com by tkorhon1
on 2010-12-10 11:33:01
Timo -- is there something I need to address to fix this problem?
Original issue reported on code.google.com by mcgratta
on 2010-12-10 12:50:00
Hi Kevin,
No, I made already some fix in read.f90. And/or
I could make the fix by putting "if (evacuation_only(NM))
return" to the beginning of the "open/close" subroutines.
The problem (error) was removed obst/created hole, the
evacuation meshes do not have YY nor YYS arrays allocated:
I still need to think the best way. It might be that later
I want/need also obsts/holes to be created/removed in the
evacuation geometry.
IF (REMOVE) THEN
DO K=K1+1,K2
DO J=J1+1,J2
DO I=I1+1,I2
RHOS(I,J,K) = RHOA
RHO(I,J,K) = RHOA
IF (N_SPECIES>0) THEN
YY(I,J,K,1:N_SPECIES) = SPECIES(1:N_SPECIES)%YY0
YYS(I,J,K,1:N_SPECIES) = SPECIES(1:N_SPECIES)%YY0
ENDIF
ENDDO
ENDDO
ENDDO
ENDIF
Timo
Original issue reported on code.google.com by tkorhon1
on 2010-12-10 13:06:06
Hi, try the SVN version 7313. This should be fine
with ctrl_ids and devc_ids. And I tested also the
"input fix", i.e., adding EVACUATION=.FALSE. to the
OBST/HOLE lines, which have the devices/controls.
(This input check was done with an exe version that
crashed without this input fix.)
The input fix with "old" program version is still
running nicely:
Time Step: 600, Simulation Time: 11.34 s
The new version (SVN version 7313) is running nicely:
Time Step: 600, Simulation Time: 11.34 s
Previously I got the crash at 8.79 s. I'll check the
status of the runs after some ten days. But it seems
that this issue is now fixed.
Timo
PS. Next week I'm on holiday.
Original issue reported on code.google.com by tkorhon1
on 2010-12-10 15:17:43
I know you guys just love me coming up with a bug like this just before the holidays...
keeps life interesting, eh?
So Timo, will SVN version 7313, once eventually posted on the downloads page, be 'backward
compatible' with the FDS file that ran fine under FDS 5.4.3, 5.0.0, etc. (the "PorterdaleMill.fds"
that I attached to Issue 1284), or will it be required to be revised to 'work' (i.e.
does SVN 7313 lose backward compatibility due to this EVAC bug fix thus requiring the
edits to the input file to work with the latest FDS)?
And, if going forward we'll need to revise how we do the OBST/HOLE lines from the way
the current user manual instructs, will there be a reworked user manual instructing
us on this new requirement?
BatGirl
Original issue reported on code.google.com by BlackstoneEngineering
on 2010-12-10 16:30:09
Well, the svn 7313 does not need any new inputs for the
evacuation part. I need just to add to the new FDS+Evac
manual that the OBSTs/HOLEs in the evacuation meshes do
not use any DEVC_ID/CTRL_ID, so the user should add OBSTs
and HOLEs with EVACUATION=.TRUE. (or .FALSE.) where needed
it the removable OBSTs/HOLEs in the fire geometry are doing
something at the z-levels of the evacuation meshes. The CTRL_ID
DEVC_ID OBSTs and HOLEs will be in the evacuation meshes even
if the initial state of these would say that there is no obst
or no hole there at time T_BEGIN. So, for these obsts/holes
it would be nice to give EVACUATION=.FALSE., if they are not
wanted to be in the evacuation geometry (the time=T_BEGIN state
of the obsts/holes it is).
Well, I checked the cases running with the new source code
and the "old" version (7310) and got same error at
the same time point:
Time Step: 28800, Simulation Time: 18.70 s
Time Step: 28869, Simulation Time: 18.71 s
STOP: Numerical Instability
I'll put the runs there with MISC-line option
NO_EVACUATION=.TRUE.
These will just run the fire part, they do not read in
the evacuation meshes, i.e., they are identical to runs,
where the evacuation stuff is taken away from the input
files.
I'll change the status to "Started" once again, because
there seems to be a numerical instability here.
TimoK
Original issue reported on code.google.com by tkorhon1
on 2010-12-20 09:03:17
"Old" source code and NO_EVACUATION=.TRUE. on MISC-line:
Time Step: 43600, Simulation Time: 22.32 s
Time Step: 43700, Simulation Time: 22.33 s
Time Step: 43800, Simulation Time: 22.35 s
Time Step: 43900, Simulation Time: 22.37 s
"New" source code and NO_EVACUATION=.TRUE. on MISC-line:
Time Step: 43500, Simulation Time: 22.30 s
Time Step: 43600, Simulation Time: 22.32 s
Time Step: 43700, Simulation Time: 22.33 s
And both are still running. So, I'll check the stuff
why numerical instability with evacuation meshes is
there. The fire+evacuation had numerical instability
at 18.71 s. But these checks take some time, it is
running something like a day before 18.71 s.
TimoK
Original issue reported on code.google.com by tkorhon1
on 2010-12-22 07:19:35
Well, there is no numerical instability, if you put
NOT_RANDOM=.TRUE. on (some) PERS-namelist. The evacuation
part uses random numbers and the 5.5.3 version used random
numbers before the initial noise for fire meshes. I.e., the
initial noise in the fire and fire+evacuation calculations
were not the same and, thus, the fire meshes did not produce
exactly same results. And in your case, with one random initial
noise you did not get numerical instability and with the other
one you did get a numerical instability. You should check you
fire input carefully and see that your time step is not going
too short, when the vents are opened.
I also changed the source code so that now the fire calculation
is always exactly the same with and without the evacuation part.
How I did this? I was moving the random number initialization in
the evacuation part (and related things) after the fire mesh
"random" noise generation.
So, put NOT_RANDOM=.TRUE. and do your fire+evacuation calculation.
Then you get CHID_evac.fed (and CHID_evac.eff) file(s) that you
can use later, when you set EVACUATION_MC_MODE=.TRUE., then the
fire meshes are skipped and the smoke information is read from
the CHID_evac.fed file, if it is found on the disk. But as I said,
you should check you fire input. Something not nice is going on
when you open your vents / remove our obsts. The calculation should
not be sensible to the initial noise.
TimoK
Original issue reported on code.google.com by tkorhon1
on 2011-01-24 13:35:57
OK - I just glazed over with all that - hope the next user manual includes something
to address this issue in a more 'user friendly' manner.
Meanwhile, I'm just going to stay with FDS 5.4.3 MPI that worked so well on my XP Core2Quad
machine, and next try the 64-bit 5.4.3 MPI in my plan to build a monster machine using
twin Core2Quad 64-bit m'board with 32Gb ram in the future which depends on 64-bit MPI
running flawlessly and uncomplicated... also noticed the latest smokeview has changed
so the ability to adjust skip frames seems to be gone - disappointing developments
all the way around.
Yep - just gonna stick with FDS 5.4.3, then wait a few years to even consider start
re-learning the latest version of these programs at that time (kinda like I had to
do going from FDS4 to FDS5 - a total change requiring throwing out almost all one learned
just to use the next version... makes one not want to 'upgrade', and if it weren't
for the evac stuff, I'd still be on FDS4!!!)
On a side note - will Ruddy ever get the WFDS issue fixed so the cones will burn away
again, or do I just need to stick with an earlier version of FDS for that feature,
too?
Original issue reported on code.google.com by BlackstoneEngineering
on 2011-01-24 14:42:41
Well, I change the status of this issue to "Closed".
The FDS6 should not have this problem. And the latest
source codes are also nice.
TimoK
Original issue reported on code.google.com by tkorhon1
on 2011-03-24 15:27:21
Original issue reported on code.google.com by
BlackstoneEngineering
on 2010-12-07 22:54:31