wyldckat
commented
12 years ago | Notes | |
| (0000003) | |
| wyldckat | |
| 2011-07-18 09:29 | The case scenario I've been testing to try and reproduce this error is as follows: * Use the case "tutorials/incompressible/icoFoam/cavity" as a base for unit testing. * Edit the file "constant/polyMesh/blockMeshDict". * Change the line that has: hex (0 1 2 3 4 5 6 7) (20 20 1) simpleGrading (1 1 1) To: hex (0 1 2 3 4 5 6 7) (3000 2000 1) simpleGrading (1 1 1) * Save the file. * Run "blockMesh". * Import the OpenFOAM mesh in enGrid. The work machine has 24GB, although large chunks of contiguous memory might not always be available. VTK is the one that comes with ParaView 3.8.0 (from OpenFOAM 1.7.1). enGrid versions tested were 1.2 and the 1.3 in development. I've thought that it could be a CPU-GPU limitation, but in either modes it crashes when there are too many surfaces to display. This case of 3000x2000 generates around 12 million points. This still opens on the machine. In the next notes I'll post the attempts to isolate the error. |
| (0000004) | |
| wyldckat | |
| 2011-07-18 10:51 (edited on: 2011-07-18 10:52) |
enGrid with 3800x2000 did not open, crashing with a bad_alloc. ParaView is able to open the case and display with surfaces and lines. enGrid with 3500x2000 did open. ParaView with internal reader is also able to open the case and display with surfaces and lines. enGrid with 3700x2000 did open, taking about 24min to import and 10GB of RAM after its open. Next step: I'm going to try the same, but with VTK from ParaView 3.10.1. Note: These tests were made via VNC, but last time I tried in locus with GPU it had basically the same performance... expect time-wise, I think. |
| (0000005) | |
| wyldckat | |
| 2011-07-18 13:02 (edited on: 2011-07-18 13:04) |
Using VTK that comes with ParaView 3.10.1 did not solve the issue. Further debugging will be necessary to detect where the problem occurs. One last thing - this bug report seems to be associated with this forum thread: http://engits.eu/vanilla/index.php?p=/discussion/193/is-there-a-mesh-size-limit-when-importing-openfoam-case- [^] |
| (0000006) | |
| wyldckat | |
| 2011-07-18 14:55 | More test results, but now defining resolution in X-Y-Z: * 250x250x250 did not import in enGrid. * 240x240x240 did not import in enGrid. * 220x220x220 did not import in enGrid. * 210x210x210 did import. * 200x200x200 did import. |
| (0000007) | |
| unauthenticated | |
| 2011-07-18 16:50 | Hi Wyldckat, and thanks for checking this issue. this bug report is indeed associated with the forum thread. Let me know if I can help you by running some tests. Regards, Fabien |
| (0000008) | |
| ogloth | |
| 2011-07-21 10:32 | Thanks everyone for the testing efforts. I will look at this once the release is in a beta state. |
| (0000024) | |
| wyldckat | |
| 2011-08-31 12:03 | Traced this to a problem with reading files larger than 1GiB. enGrid uses a "pseudo-optimal" solution of reading the whole file into RAM, namely into a QString buffer. The problem with this is that it seems that QString is unable to handle stuff larger than 1GB, or perhaps it's 2GB, since it might load in 2byte Unicode mode. The trace-back is something like this: FoamReader::operate() buildMaps() FoamObject::readFile("constant/polyMesh/faces") m_Buffer = f.readAll(); <<-- this is where it breaks. Solution: a more dynamic reader is going to be necessary sooner or later... something that doesn't take so much memory when reading files. QTextStream provides a readLine() method, so this would be best for importing data. Quick solution: use another way to load the whole file... perhaps loading it into a QByteArray. |
| (0000025) | |
| wyldckat | |
| 2011-08-31 16:16 | QByteArray is limited to signed 32bit bytes: http://lists.trolltech.com/qt4-preview-feedback/2005-03/msg00193.html [^] It has been this way since its inception and support for 64bit is unlikely in the near future. In the commit 83047d0b06f456a9ec541b2e354c61fdc59c8d24 on the "windows-1.3" branch is a temporary fix. It uses QByteArray instead of QString for the buffer for reading OpenFOAM cases, therefore extending reading capabilities from 1GiB to 2GiB. Unfortunately, enGrid doesn't have an optimized architecture for handling such cases. Expectations is that the visual interaction will be reaaaaally slow, when the whole geometry is visible. Loading tests will follow in the next notes. |
| (0000026) | |
| wyldckat | |
| 2011-08-31 17:29 (edited on: 2011-08-31 17:30) |
With the fix from the previous note, a mesh based on the method explained in note 3, with a resolution of 7000x2000, resulted in: * 28018002 points; * 56009000 faces - this is the biggest file in the constant/polyMesh folder, with 2095559862 bytes, near the 2^31 limit (2GiB). * 20GB on RAM in enGrid. * It wasn't possible to save the imported mesh in ".egc + .egc.vtu" files due to the machine's memory limit of 24GB of RAM. This is entering in a whole other scale of working with geometries and meshes, which I don't know if enGrid is capable of handling, at least for now :( But it's something to definitely improve upon when it starts to handle meshes in parallel :D |
| (0000027) | |
| ayoros00 | |
| 2011-09-01 14:59 | Hi Bruno, Many thanks for the report. I'm not really familiar with Qt programmation, but how could I easily apply your quick QByteArray solution to Engrig-1.2 sources for Linux ? I've already made some small changes to the 1.2 version and I would be interested in increasing the limit of this realease. Thanks again, Regards, Fabien |
| (0000028) | |
| wyldckat | |
| 2011-09-01 15:08 (edited on: 2011-09-01 15:10) |
Hi Fabien, OK, this should work... but I haven't tested this yet: 1. In a terminal, go to where you have enGrid's source code, namely the folder "src". 2. Download the respective patch by running: wget "http://engrid.git.sourceforge.net/git/gitweb.cgi?p=engrid/engrid;a=patch;h=83047d0b06f456a9ec541b2e354c61fdc59c8d24" [^] -O patch_qbytearray (without the [^]) 3. Patch things up: patch -p3 < patch_qbytearray 4. If all goes well with the patch, run "make -j 2", where 2 is the number of cores/threads you have on your machine. 5. It's ready for trying it out :) Keep in mind that I haven't checked yet if this patch works on enGrid 1.2, because right now I'm wrapped around the issue 0000009... You can always look at the patch file (its simple text) and try to apply the patch manually. If it doesn't work, let me know that I'll see what would need to be modified! Best regards, Bruno |
| (0000029) | |
| ayoros00 | |
| 2011-09-02 08:42 | Great ! Things ran fine at first try (indeed after I remembered the export VTKINCDIR and VTKLIBDIR commands !) Very first tests show that engrid runs fine with a faces file of 1.3 GB, but crashes with a 3.2 GB, as expected. Just let me know if you need help for testing further. Regards, Fabien |
| (0000030) | |
| wyldckat | |
| 2011-09-07 10:53 | Hi Fabien, I'm glad it worked! As for additional testing, I was wondering if you were able to save the case after the import process. I didn't manage to do it, due to the shortage of RAM, so I'm wandering if it was just a matter of RAM. Additionally, after importing the OpenFOAM case, were you able to generate a mesh and then save and export the case back to OpenFOAM? Another question is: how slow did the enGrid interface get? Best regards, Bruno |
| (0000040) | |
| wyldckat | |
| 2011-09-08 11:15 | New installers released for enGrid 1.3beta on Windows: https://sourceforge.net/projects/wyldckat.u/files/enGrid/ [^] The ones with the name "enGrid_1.3beta_2011-09-08_setup_win*_MSVC2008.exe" have these temporary fixes. |
| (0000064) | |
| wyldckat | |
| 2011-12-26 16:00 | The commits listed here are now also in the Linux version, as of ac6df618bf91ae0226639ff661d01092b689700a on the release-1.3 branch. Nonetheless, this still needs a proper fix, since this only postpones the import size from 1GB to 2GB. |
Original Report:
Since I found Engrid very usefull, I tried it with different OpenFOAM meshes.
Everything goes fine with small meshes, but when it comes to bigger meshes (more than 16 million cells), Engrid crashes loading the OpenFOAM case.
To be sure it comes from mesh size, I tried with simple squares mesh with more or less cells.
Maybe this comes from my install, or from modifications I made.
I hope you can help me.
Thanks,
Fabien