Closed Bill-gitt closed 4 months ago
Bill-gitt
commented
3 years ago At least, does anyone tried to compile CVode with Visual studio (2013 version or later) and encountered similar problems where Fortran objects are not compiled because the Fortran and C objects must be in different projects?
balos1
commented
3 years ago There was a previous user report on the sundials email list about the issue with Visual Studio and Fortran projects. It seems that you have found a resolution for that though. I don't quite follow what you have done to allow it to compile, but If you had to change code in SUNDIALS, posting the diff or a pull request would be appreciated.
As far as the run-time issue in Debug mode... I believe this is because the FN_VGetArrayPointer function calls c_f_pointer with a shape of [1] here. In Debug mode there is likely a bounds check that does not like this.
You can try to update the FN_VGetArrayPointer function to:
function FN_VGetArrayPointer(v) &
result(swig_result)
use, intrinsic :: ISO_C_BINDING
real(C_DOUBLE), dimension(:), pointer :: swig_result
type(N_Vector), target, intent(inout) :: v
type(C_PTR) :: fresult
type(C_PTR) :: farg1
type(C_INT64_T) :: length
farg1 = c_loc(v)
fresult = swigc_FN_VGetArrayPointer(farg1)
length = FN_VGetLength(v)
call c_f_pointer(fresult, swig_result, [length])
end functionIf this works, then we can try to incorporate it into the next SUNDIALS release.
Bill-gitt
commented
3 years ago Dear Balos,
Yep we have a winner! After 1.5 month of efforts CVode on Fortran runs!
The code you proposed fixed the problem with out of boundaries exception. After updating the "FN_VGetArrayPointer" Function the example ("examples/cvode/F2003_serial/cv_analytic_sys_dns_f2003.f90") ran properly and after doing some debug the "yvec" is an array with size of 3 elements as it was supposed to be.
A couple of things to note:
function FN_VGetArrayPointer(v) &
result(swig_result)
use, intrinsic :: ISO_C_BINDING
real(C_DOUBLE), dimension(:), pointer :: swig_result
type(N_Vector), target, intent(inout) :: v
type(C_PTR) :: fresult
type(C_PTR) :: farg1
type(C_INT64_T) :: length farg1 = c_loc(v) fresult = swigc_FN_VGetArrayPointer(farg1) length = FN_VGetLength(v) call c_f_pointer(fresult, swig_result, [length]) end function
The line "type(C_INT64_T) :: length " is supposed to be "integer(C_INT64_T) :: length " else it won't compile, so the correct update would be: function FN_VGetArrayPointer(v) & result(swig_result) use, intrinsic :: ISO_C_BINDING real(C_DOUBLE), dimension(:), pointer :: swig_result type(N_Vector), target, intent(inout) :: v type(C_PTR) :: fresult type(C_PTR) :: farg1 integer(C_INT64_T) :: length
farg1 = c_loc(v) fresult = swigc_FN_VGetArrayPointer(farg1) length = FN_VGetLength(v) call c_f_pointer(fresult, swig_result, [length]) end function
2. The subroutine "c_f_pointer" is used in a similar way elsewhere is the code and may need an update/fix too.
Please have a look:
function FN_VGetDeviceArrayPointer(v) & result(swig_result) use, intrinsic :: ISO_C_BINDING real(C_DOUBLE), dimension(:), pointer :: swig_result type(N_Vector), target, intent(inout) :: v type(C_PTR) :: fresult type(C_PTR) :: farg1
farg1 = c_loc(v) fresult = swigc_FN_VGetDeviceArrayPointer(farg1) call c_f_pointer(fresult, swig_result, [1]) end function
Best regards,
BillDear Balos,
Regarding the steps I followed to compile under Visual Studio 2013 and 2019 unfortunately I didn't update any code. I just did some manual changes on the visual studio project that the Cmake-gui had generated. This means that every time the generate button is pressed on Cmake all the changes I did, must be redone. For the a simple version of Cvode (no examples or tests) this requires about 50 mins of copy paste. To fix properly Cvode, extensive changes on Cmake files are needed, so that the C and Fortran files are not anymore in the same project and instead are separated in different projects. The compilation order should be, first build the Fortran project and then the C targets.
Since the Sundials Cmake files seemed to me very complicated (so many targets with different options)
I hope this helped a bit on the process I followed to compile Cvode with Visual Studio. If there is a need for more clarification let me know. It took me some time compile Cvode and I hope this is useful somehow.
Cheers, Bill
balos1
commented
3 years ago @Bill-gitt Thanks for the detailed writeup that others can refer to for a workaround. We will try to fix both issues (the Visual Studio build, and FN_VGetArrayPointer problem) in a future SUNDIALS release.
Bill-gitt
commented
3 years ago Dear Balos,
It seems I have encountered another issue with the FCvode. It is related to its thread-safety, it seems independent threads access shared memory when executing memory release functions (FCVodeFree or FSUNMatDestroy etc...) and the program crashes. So, should I create a new issue in github or continue here in the current thread (it may be compile related)?
Regards, Bill
drreynolds
commented
3 years ago Dear Bill,
FCVODE (and all of the other F77 interfaces) are not threadsafe, nor can they be (the F77-C "glue" code relies on global variables). This was the main motivation for why we created the newer F2003 interfaces to each SUNDIALS package. If you need thread safety from Fortran, then your only option is to upgrade your codes to use the new interfaces. Daniel R. Reynolds Professor and Department Chair 214-768-4339 (tel:214-768-4339) Mathematics @ SMU (https://maps.google.com/?q=Mathematics%20%40%20SMU) http://people.smu.edu/dreynolds
On Sep 7 2021, at 7:45 am, Bill-gitt @.***> wrote:
[EXTERNAL SENDER]
Dear Balos, It seems I have encountered another issue with the FCvode. It is related to its thread-safety, it seems independent threads access shared memory when executing memory release functions (FCVodeFree or FSUNMatDestroy etc...) and the program crashes. So, should I create a new issue in github or continue here in the current thread (it may be compile related)? Regards, Bill
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub (https://github.com/LLNL/sundials/issues/67#issuecomment-914275930), or unsubscribe (https://github.com/notifications/unsubscribe-auth/AC2TOOKJJHSGP5AW4GEBK2TUAYCPNANCNFSM5BPNNUAQ). Triage notifications on the go with GitHub Mobile for iOS (https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675) or Android (https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub).
balos1
commented
3 years ago @drreynolds is correct that FCVODE (the FORTRAN 77 interface to CVODE) is not theadsafe, but I think you are referring to the FCVode function from the sundials_fcvode_mod module (i.e. the F2003 interfaces), correct @Bill-gitt?
If you are indeed referring to the FCVode function from the sundials_fcvode_mod module, please start a new issue for it.
Bill-gitt
commented
3 years ago Thank you for your replies @drreynolds and balos1.
-The reason I turned to Cvode from Intel ode solver was that FCvode (version 2003) was thread safe. It means a lot that Pr. Reynolds confirmed that Fcvode (version 2003) is thread safe. Unfortunately the past month I didn't succeed to parallelize my code through openmp. I am using threadprivate allocatable arrays to feed data to Fcvode inside openmp regions and I get inconsistent results with lots of crashes due to either Floating point exceptions or Memory access violations. My code works perfectly in serial compilation (absence of any openmp directives). So Its either the latest Intel OneApi Fortran compiler has made small changes in the latest openmp implementation that I am not aware of or using Fcvode forced my compiler to use F2003 Fortran that may be incompatible with existing code.
-At the moment I am trying to produce a simple example that clearly exhibits inconsistent results between serial and openmp version of my code. If I have something concrete I 'll create a new issue in hope for resolution. -I replied in the current thread cause I am not quite sure the compilation process for producing Fcvode is absolutely correct. I worry that internal optimizations of fcvode or Cvode may spawn extra threads inside openmp regions and create unexpected results. Though this shouldn't be the case.
Best Regards
Bill-gitt
commented
3 years ago Dear balos1,
After lots of effort, I finally managed to make Fcvode work properly inside openmp regions. This is a success I believe (unless something else comes up later). To give a bit of backstory so others may benefit from it, this is what happened. After trying to run simple Fcvode examples in parallel yielded random results. Most of the time everything worked out well but one out ten runs 1/10 the program could give erroneous results. Thusly I decided to use Intel inspector on my code. What I found out was eye opening, more than 25 warnings where revealed inside Fcvode libraries (irrelevant of the code I have written) that indicated failure of thread safely. I knew though that Fcvode should be thread safe and thusly the reason for failing should be in compilation. Searching a bit further and testing lots of different compilation flags I found out what was missing. The directive "qopenmp" was missing from the compilation of Fcvode interface. Searching online in Intel I found this
(1) “Static local data” is shared by each thread, i.e. each thread may access the same data location! This is potentially unsafe. (2) “Automatic data” is independent between each thread; it has its own, independent copy on its own stack. With Intel Fortran when compiled and running serially, the defaults are local scalar variables are automatic and local arrays are static. But when compiling with -qopenmp, local arrays are automatic by default. It is the same as compiling with -auto. This may require an increase in the maximum stack size. If the stack size is too small, there will be a segmentation fault at runtime.
Which means that when not using qopenmp local arrays are not thread safe...
After using qopenmp and running intel inspector again the 25 warnings were now only 2 which seemed nothing serious. So for anyone that wants to run Fcvode inside openmp regions make sure that you compile code with qopenmp.
That's it for now. I have managed to compile Fcvode with Intel Visual studio and use its thread safety features, successfully.
Best regards
balos1
commented
3 years ago @Bill-gitt Thanks for posting your findings.
Hello dear developers,
Summary: I am working for a University project where I have access to Windows 10, Visual studio 2013 and Intel Parallel studio 2017 or Visual Studio 2019 Community along Intel OneApi (includes Fortran Classic 2021).
For about 1 month now I am trying to compile the CVode with Fortran Interface, from Sundials 5.7 package. I hope it has thread-safely even when the Fortran interface is used. Anyways after lots of effort I managed to compile and build CVode. Though when I run the Fortran examples in debug mode the program crashes.
The steps I followed for compilation:
The reason for this kind of errors after lots of search, is caused because the visual studio 2013 & 2019 cant handle a mix C and Fortran files in same project. As very well stated here in order for the compilation to succeed a Fortran project containing all .f90 files must be created and then added to the current project created from Cmake in Build_dir. Furthermore all "....._mod_static" targets must have all their "..._mod.f90.obj" objects change their location and point to the output directory of the Fortran project created above. For example the "sundials_fcvode_mod_static" contains many objects like "fcvode_mod.f90.obj", fnvector_serial_mod.f90.obj and so on. The location of these objects is changed to the output of the custom Fortran project.
A trick needed above is that all .f90 files must be renamed to duplicate their extension. For example the the fcvode_mod.f90 must be renamed to fcvode_mod.f90.f90 . This is needed because the produced fortran obj files retain inside them ".f90" and be in form "fcvode_mod.f90.obj". This is how the Cmake writes the ALL_BUILD.vcxproj file.
After doing all the above the code All_Build target runs well and compiles. Though as expected the Cmake_Fortan_Flags: "/W1 /nologo /fpp /libs:dll /threads" found in the Cmake are wrongly passed to cl compiler and produce warnings of type
cl : command line warning D9002: ignoring unknown option '/fpp'. I assume these warnings can be ignored. A bigger output of All_Build target output can be seen:Time to build the Install target. A necessary step that is needed is that a folder inside the "build_dir" must be created with the name "Fortran_Static". Here all mod files found in the custom Fortran project must be copied. This is a required step since at the last step of the INSTALL target the data of "Fortran_Static" are copied to the "Install_dir/fortran".
A word of caution its good to have the "Install_dir" in a directory for which the user has access to write. The "c:\program files (x86)\" is not one of them. In which case the user might have to call a cmd.exe with admin rights and the call "msbuild INSTALL.vcxproj".
After all the above steps I got the Install_dir with the folder containing "Fortran", "Include" and "lib" folders. The real problem starts now. After including the install_dir properly to one of the included examples (cv_analytic_sys_dns_f2003.f90) I got the following erroneous output:
fvec = matmul(Amat, yvec(1:neq)). Furthermore "yvec" gets its size from a call to the "FN_VGetArrayPointer".yvec => FN_VGetArrayPointer(sunvec_y).Which is exactly the expected output shown in the out file . Thusly the Cvode ran properly under release mode but not under debug. Moreover it seems that the "FN_VGetArrayPointer()" function shows properly the head of the array, but it ignores the rest of the body.
If I were to run the cv_analytic_fp_f2003.f90 example that has only one ode equation to solve and not a system of equations( as above) then no error will occur in the debug mode. The "yvec" properly is array with only one value. Obviously all the examples that have more than one ode to solve crash as also the test "test_fnvector_serial_mod" crashes. The test contains the files "test_fnvector_serial_mod.f90", "test_nvector.f90", "test_utilities.f90" and is part of the ALL_Build (If Fortran tests are included in Cmake). Some small output of the test_fnvector_serial is:
To catch some questions before asked, I pay attention to build and release modes in Fortran and All_build projects to be compatible (despite showing output from a release mode and running example in debug mode). Also the Fortran project is in 64bit mode well.
So please, I have spent too much time trying to compile Sundials/CVode under Visual studio with Intel Fortran environment, any useful input on how to fix the "Yvec" size would be greatly appreciated!!!!