Open vdalessa opened 3 years ago
TonkomoLLC
commented
3 years ago Hello, Valerio,
I am not sure I follow. If you type grep -r "forAll" * from $WM_PROJECT_DIR/src you will see many instances of forAll loops.
Does this address your question?
Best regards,
Eric
vdalessa
commented
3 years ago Hi Eric, thank you very much for the suggestion. I wil see on Wednesday. Tomorrow the system on which I am working has a programmed stop. Sincerely, Valerio
vdalessa
commented
3 years ago Dear Eric,
i have fixed my previous issue swicthing to another programming strategy. Now, the solver seems to be ok, but the it crashes when I uses forces computation. Library seems to be ok and I have already followed forces.H details. Could you help on this issue ?
Valerio
forces { type forces; functionObjectLibs ("libforces.so"); log yes; patches (parete); rhoInf 1.4; CofR (0 0 0); }
}
TonkomoLLC
commented
3 years ago Dear Valerio,
I am honestly not sure what the problem is. I recommend using OpenFOAM v2.3.1 tutorials as a template for the forces function object.
I ran the cavity example with some dummy parameters for the forces calculation. These are totally nonsense parameters, but merely show that the forces function object is running. Firstly, here is the controlDict for the OpenFOAM cavity tutorial, with the forces function object added:
Here is screen output:
Time = 0.5
Courant Number mean: 0.222158 max: 0.852134
smoothSolver: Solving for Ux, Initial residual = 2.57283e-07, Final residual = 2.57283e-07, No Iterations 0
smoothSolver: Solving for Uy, Initial residual = 5.52242e-07, Final residual = 5.52242e-07, No Iterations 0
AINVPCG: Solving for p, Initial residual = 9.29243e-07, Final residual = 9.29243e-07, No Iterations 0
time step continuity errors : sum local = 7.93191e-09, global = -1.29735e-18, cumulative = 4.76985e-18
AINVPCG: Solving for p, Initial residual = 9.74121e-07, Final residual = 9.74121e-07, No Iterations 0
time step continuity errors : sum local = 8.29978e-09, global = 1.43433e-19, cumulative = 4.91329e-18
ExecutionTime = 2.42 s ClockTime = 2 s
forceCoeffs forces output:
Cm = 2.9948e-08
Cd = 8.50524e-06
Cl = 4.63552e-317
Cl(f) = 2.9948e-08
Cl(r) = -2.9948e-08
End
Error at lnInclude/DeviceStreamI.H:11
driver shutting down
Error at lnInclude/DeviceStreamI.H:11
driver shutting downI do not know what the error is at the end, but it appears to not impact the results. I have not seen this error before.
I hope this feedback helps.
Best regards,
Eric
vdalessa
commented
3 years ago Dear Eric,
thank you very much for your very fast response. I attach the error that I obtain using your Dict. I am really interested in fixing this issue. So, I can share many other details about our solver that we have already pubilshed on Computers and Fluids ( in its cpu version).
Sincerely, Valerio
vdalessa@login02 $ mpirun -np 4 rhoEnergyFoamSuth2 -parallel /---------------------------------------------------------------------------\ | RapidCFD by simFlow (sim-flow.com) | *---------------------------------------------------------------------------*/ Build : dev Exec : rhoEnergyFoamSuth2 -parallel Date : Jun 29 2021 Time : 23:57:59 Host : "login02" PID : 106901 Case : /m100/home/userexternal/vdalessa/RapidCFD/cyl nProcs : 4 Slaves : 3 ( "login02.106902" "login02.106903" "login02.106904" )
Pstream initialized with: floatTransfer : 0 nProcsSimpleSum : 0 commsType : nonBlocking polling iterations : 0 sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE). fileModificationChecking : Monitoring run-time modified files using timeStampMaster allowSystemOperations : Allowing user-supplied system call operations
// * // Create time
Create mesh for time = 0
Reading thermophysical properties
Selecting thermodynamics package { type hePsiThermo; mixture pureMixture; transport const; thermo hConst; equationOfState perfectGas; specie specie; energy sensibleInternalEnergy; }
Reading field U
Creating turbulence model
Selecting turbulence model type laminar Reading thermophysicalProperties
Ls = 15 f = 1 a = 1 M = 0.2 LAD = 0.75 etatg = 40
gamma = 1.4 pinf = 1 Tinf = 1
rho0 = 1.4 Eref0 = 1.80571428571 Uinf_x = 0.2 Uinf_y = 0 Uinf_z = 0
sigma0 = 0.442096337855
Starting time loop Start Timing = 1 s
forceCoeffs forces: Not including porosity effects
Time = 0.0001
ExecutionTime = 1.63 s ClockTime = 2 s
vdalessa@login02 $
TonkomoLLC
commented
3 years ago Dear Valerio,
Hmm. I normally associate errors like cudaErrorInvalidDeviceFunction: invalid device function to be a compilation issue such as when the sm_# is not set correctly in wmake rules. What I am referring to is the -arch=sm_30 value in the wmake rules. The value 30 needs to be set to the appropriate value for your GPU. If you do change this value (and the corresponding -arch=sm_30 flag) in the c rules, then you'll need to wclean all from the $WM_PROJECT_DIR and do a fresh recompile from scratch. Maybe also the rules files you need to change will be in a different directory than linux64Nvcc since I see you are using a Power PC CPU and a PPC directory does not exist in the RapidCFD wmake rules directory.
Let's suppose the -arch=sm_## flag is set appropriately and that's not the root problem. Then I would run the cavity tutorial from the test case repo and replace system/controlDict with the version i sent above (with the function object included for forces). If this cavity test with forces function object works, then the error message you see is related to something specific with your cas (e.g., a setting in the forces settings within controlDict). However, if the cavity case with the forces function object fails, then the root cause for the error you provided is probably due to something with the compilation of RapidCFD because the forces function object clearly works on my system with cavity.
I hope this advice makes sense and I look forward to learning what happens.
By the way, is this the CPU version of the code you are working on? https://github.com/davidem88/rhoEnergyFoam
Good luck!
Cheers, Eric
vdalessa
commented
3 years ago Dear Eric, thank you very much for you kind response. Actually, I have tested cavity problem with your dict for forces computation and icoFoam crashes. So I have a compilation issue. I am attemping to work on Marconi 100 system https://www.hpc.cineca.it/hardware/marconi100 As you can see V100 cards are available. I attach the rules dir that I am using. https://github.com/vdalessa/L17/blob/master/linuxPPC64leNvcc.tgz Moreover, I have built RCFD using following modules
module load spectrum_mpi/10.4.0--binary module load gnu/8.4.0 module load hpc-sdk/2020--binary module load cuda/10.2
CPU starting point is the following: https://github.com/vdalessa/caafoam which shares several points in m2 version with rhoEnergyFoam. In this specific version .C file is still named from rhoEnergyFoam starting point. Please note that, in this moment, solvers runs correctly. My problems are strictly related to forces computations.
Could you take a look to my rules dir ??? Thanks again for your help. Valerio
TonkomoLLC
commented
3 years ago Dear Valerio,
Thanks for this information. I agree that your settings in the wmake rules look fine. And as you said, the whole code seems to run fine except for the forces function object.
I am not sure this will work, but let's just be sure. Please go to $WM_PROJECT_DIR/src/postProcessing/functionObjects/forces - the location of the forces function object code - and type wclean followed by wmake. This will recompile libforces.so. I am asking to do this just in case this code is compiled for a different compute capability, and maybe it was missed in a wclean of your RapidCFD installation.
After recompiling, please try the cavity case again (it's fast, plus I have access to it so we can troubleshoot on cavity together).
If cavity fails again, please confirm that you also try running cavity with just one GPU (no MPI).
If icoFoam & cavity still fail to run with forces, then I wonder if the next step is to get a fresh copy of RapidCFD, change the wmake rules again and whatever else you need to do accommodate the PPC CPU's, and recompile from scratch. I do not know the history of your RapidCFD compilation. If there are compiled components leftover from a previous compilation then this may fix the issue.
Another idea, especially if you are going to recompile, is to update your CUDA version to 11.1. I know 11.4 is out, but I have not tested > 11.1. Sometimes bugs get fixed in newer versions of CUDA.
Anyhow, I do not know specifically what is causing your problem, but maybe these are some ideas to try... even if these ideas do not fix the problem immediately, they may help narrow the specific cause of the problem.
Thanks for the background on your solver. This is interesting work and I hope that you have some success in the near future.
Best regards,
Eric
vdalessa
commented
3 years ago Dear Eric, I apologize for the late response. In these days I am really busy since I am co-char of an approcching conference www.aigeancona.univpm.it I hope you can understand. Honestly, I am really interested in fixing the forces issue. I think it could be very interesting run some large cases on M100 GPU. I hope you are still available from next week for guidance in fixing my issues. Kind regards, Valerio
TonkomoLLC
commented
3 years ago Dear Valerio,
No worries on the conference commitment. That looks like a great conference.
I am happy to correspond with you on the forces problem. Since I cannot reproduce the problem on my machine and I don't have the same hardware as you, at present I can only help guide your troubleshooting. I do not know the exact answer to resolve the error you are receiving.
Thanks for your understanding.
Best regards,
Eric
jiaqiwang969
commented
2 years ago grep -r "forAll" *
Hi,vdalessa, could you say something about how you solve it? I also meet the same problem.
vdalessa
commented
2 years ago Dear jiaqui, many thanks for you message and for your interest in my work. Unfortunately, I have stopped my efforts in this direction since I did not have enough access to GPU and I have found some funding on other projects. However, I am still really interested in this topic. I attached to a private email a version of the code that compiled for me. But I did not still solved the forces issue. Please, could you me keep updated about your work ? Sincerely, Valerio
Dear All, it is very clear that standard forAll (mesh.C , celli ) .... does not run in RapidCFD environment. However, looking source code I do not still understand a possible alternative. Could you help me ?
Valerio