Problems for installing CSU_RadarTools on Windows 7

[morstan]

Previously i was able to install the module and run on Windows using Anaconda3 with python 3.4, for some reasons i reintsall my Anaconda and python 3.4. I tried to install CSU_Radartools again but since there's a new fortran source file being added, i was unable to build the modules through python setpu.py install with the following errors:

D:\Python\CSU_RadarTools-master>python setup.py install running install running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src build_src building extension "calc_kdp_ray_fir" sources f2py options: [] f2py:> build\src.win-amd64-3.4\calc_kdp_ray_firmodule.c IOError: [Errno 2] No such file or directory: './calc_kdp_ray_fir.f'. Skipping file "./calc_kdp_ray_fir.f". Reading fortran codes... Post-processing... Post-processing (stage 2)... Building modules... adding 'build\src.win-amd64-3.4\fortranobject.c' to sources. adding 'build\src.win-amd64-3.4' to include_dirs. adding 'build\src.win-amd64-3.4\calc_kdp_ray_fir-f2pywrappers.f' to sources. build_src: building npy-pkg config files running build_py running build_ext customize MSVCCompiler customize MSVCCompiler using build_ext customize GnuFCompiler Could not locate executable g77 Could not locate executable f77 customize IntelVisualFCompiler Could not locate executable ifort Could not locate executable ifl customize AbsoftFCompiler Could not locate executable f90 customize CompaqVisualFCompiler Could not locate executable DF customize IntelItaniumVisualFCompiler Could not locate executable efl customize Gnu95FCompiler Could not locate executable gfortran Could not locate executable f95 customize G95FCompiler Could not locate executable g95 customize IntelEM64VisualFCompiler customize IntelEM64TFCompiler Could not locate executable efort Could not locate executable efc don't know how to compile Fortran code on platform 'nt' warning: build_ext: f77_compiler=None is not available.

building 'calc_kdp_ray_fir' extension error: extension 'calc_kdp_ray_fir' has Fortran sources but no Fortran compiler found

Is there a way to solve the problem? Thank you.

[tjlang]

CSU_RadarTools is not well tested under Windows, so we are in a bit of uncharted territory here. It appears that you are missing a Fortran compiler, so you need to look into how to install one under Windows. Are you working with Anaconda? I have heard that ships with a Fortran compiler for Windows. I don't have Windows here so unfortunately I really can't help much. It's possible @pfhein might know more here.

Once a Fortran compiler is installed, we can figure out if there are other hurdles. If you hit a wall, I can create a branch that doesn't use Fortran, and you can install that. The KDP routines will be much slower, however. Like several minutes versus 1 second to process a volume slower.

[pfhein]

@morstan I have not tested this on Windows, but I see that it has been done. Check out https://scivision.co/f2py-running-fortran-code-in-python-on-windows/ Let me know if it works or if you still have problems. I will check things out on a Win7 laptop and let you know what I find.

[morstan]

@pfhein Actually i have tried that solution before, but failed with the minGW setup, with the error message of 'Cannot download repository.txt[0]' comes out during installation. But then i have followed advice from @tjlang, i found a package installation of MinGW in Anaconda and installed successfully. I tried to run setup.py again, this time the setup seems to successfully compiled the fortran code but another error comes out as below:

D:\Python\CSU_RadarTools-master>python setup.py install running install running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src build_src building extension "calc_kdp_ray_fir" sources f2py options: [] adding 'build\src.win-amd64-3.4\fortranobject.c' to sources. adding 'build\src.win-amd64-3.4' to include_dirs. adding 'build\src.win-amd64-3.4\calc_kdp_ray_fir-f2pywrappers.f' to sources. build_src: building npy-pkg config files running build_py running build_ext customize MSVCCompiler customize MSVCCompiler using build_ext customize GnuFCompiler Could not locate executable g77 Could not locate executable f77 customize IntelVisualFCompiler Could not locate executable ifort Could not locate executable ifl customize AbsoftFCompiler Could not locate executable f90 customize CompaqVisualFCompiler Could not locate executable DF customize IntelItaniumVisualFCompiler Could not locate executable efl customize Gnu95FCompiler Found executable C:\Users\cwlok\AppData\Local\Continuum\Anaconda3\Scripts\gfortran.bat Found executable C:\Users\cwlok\AppData\Local\Continuum\Anaconda3\Scripts\gfortran.bat customize Gnu95FCompiler customize Gnu95FCompiler using build_ext building 'calc_kdp_ray_fir' extension compiling C sources C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\Bin\amd64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ibuild\src.win-amd64- .4 -IC:\Users\cwlok\AppData\Local\Continuum\Anaconda3\lib\site-packages\numpy\core\include -IC:\Users\cwlok\AppData\Local\Continuum Anaconda3\include -IC:\Users\cwlok\AppData\Local\Continuum\Anaconda3\include /Tcbuild\src.win-amd64-3.4\calc_kdp_ray_firmodule.c /F build\temp.win-amd64-3.4\Release\build\src.win-amd64-3.4\calc_kdp_ray_firmodule.obj Found executable C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\Bin\amd64\cl.exe C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\Bin\amd64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ibuild\src.win-amd64- .4 -IC:\Users\cwlok\AppData\Local\Continuum\Anaconda3\lib\site-packages\numpy\core\include -IC:\Users\cwlok\AppData\Local\Continuum Anaconda3\include -IC:\Users\cwlok\AppData\Local\Continuum\Anaconda3\include /Tcbuild\src.win-amd64-3.4\fortranobject.c /Fobuild\te p.win-amd64-3.4\Release\build\src.win-amd64-3.4\fortranobject.obj compiling Fortran sources Fortran f77 compiler: C:\Users\cwlok\AppData\Local\Continuum\Anaconda3\Scripts\gfortran.bat -Wall -g -ffixed-form -fno-second-under core -O2 -march=x86-64 -DMS_WIN64 -mtune=generic -msse2 Fortran f90 compiler: C:\Users\cwlok\AppData\Local\Continuum\Anaconda3\Scripts\gfortran.bat -Wall -g -fno-second-underscore -O2 -ma ch=x86-64 -DMS_WIN64 -mtune=generic -msse2 Fortran fix compiler: C:\Users\cwlok\AppData\Local\Continuum\Anaconda3\Scripts\gfortran.bat -Wall -g -ffixed-form -fno-second-under core -Wall -g -fno-second-underscore -O2 -march=x86-64 -DMS_WIN64 -mtune=generic -msse2 compile options: '-Ibuild\src.win-amd64-3.4 -IC:\Users\cwlok\AppData\Local\Continuum\Anaconda3\lib\site-packages\numpy\core\include -IC:\Users\cwlok\AppData\Local\Continuum\Anaconda3\include -IC:\Users\cwlok\AppData\Local\Continuum\Anaconda3\include -c' gfortran.bat:f77: csu_radartools/calc_kdp_ray_fir.f gfortran.bat:f77: build\src.win-amd64-3.4\calc_kdp_ray_fir-f2pywrappers.f C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\Bin\amd64\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:c:\users\cwlok\appd ta\local\continuum\anaconda3\mingw\lib\gcc\x86_64-w64-mingw32\4.7.0 /LIBPATH:c:\users\cwlok\appdata\local\continuum\anaconda3\mingw x86_64-w64-mingw32\lib /LIBPATH:C:\Users\cwlok\AppData\Local\Continuum\Anaconda3\libs /LIBPATH:C:\Users\cwlok\AppData\Local\Continu m\Anaconda3\PCbuild\amd64 /LIBPATH:C:\Users\cwlok\AppData\Local\Continuum\Anaconda3\libs /LIBPATH:C:\Users\cwlok\AppData\Local\Cont nuum\Anaconda3\PCbuild\amd64 /EXPORT:PyInit_calc_kdp_ray_fir build\temp.win-amd64-3.4\Release\build\src.win-amd64-3.4\calc_kdpray irmodule.obj build\temp.win-amd64-3.4\Release\build\src.win-amd64-3.4\fortranobject.obj build\temp.win-amd64-3.4\Release\csu_radart ols\calc_kdp_ray_fir.o build\temp.win-amd64-3.4\Release\build\src.win-amd64-3.4\calc_kdp_ray_fir-f2pywrappers.o /OUT:build\lib.win- md64-3.4\calc_kdp_ray_fir.pyd /IMPLIB:build\temp.win-amd64-3.4\Release\build\src.win-amd64-3.4\calc_kdp_ray_fir.lib /MANIFESTFILE:b ild\temp.win-amd64-3.4\Release\build\src.win-amd64-3.4\calc_kdp_ray_fir.pyd.manifest Found executable C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\Bin\amd64\link.exe calc_kdp_ray_firmodule.obj : warning LNK4197: export 'PyInit_calc_kdp_ray_fir' specified multiple times; using first specification Creating library build\temp.win-amd64-3.4\Release\build\src.win-amd64-3.4\calc_kdp_ray_fir.lib and object build\temp.win-amd64-3 4\Release\build\src.win-amd64-3.4\calc_kdp_ray_fir.exp calc_kdp_ray_fir.o : error LNK2019: unresolved external symbol ___chkstk_ms referenced in function calc_kdp_rayfir build\lib.win-amd64-3.4\calc_kdp_ray_fir.pyd : fatal error LNK1120: 1 unresolved externals error: Command "C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\Bin\amd64\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:c:\ sers\cwlok\appdata\local\continuum\anaconda3\mingw\lib\gcc\x86_64-w64-mingw32\4.7.0 /LIBPATH:c:\users\cwlok\appdata\local\continuum anaconda3\mingw\x86_64-w64-mingw32\lib /LIBPATH:C:\Users\cwlok\AppData\Local\Continuum\Anaconda3\libs /LIBPATH:C:\Users\cwlok\AppDa a\Local\Continuum\Anaconda3\PCbuild\amd64 /LIBPATH:C:\Users\cwlok\AppData\Local\Continuum\Anaconda3\libs /LIBPATH:C:\Users\cwlok\Ap Data\Local\Continuum\Anaconda3\PCbuild\amd64 /EXPORT:PyInit_calc_kdp_ray_fir build\temp.win-amd64-3.4\Release\build\src.win-amd64-3 4\calc_kdp_ray_firmodule.obj build\temp.win-amd64-3.4\Release\build\src.win-amd64-3.4\fortranobject.obj build\temp.win-amd64-3.4\Re ease\csu_radartools\calc_kdp_ray_fir.o build\temp.win-amd64-3.4\Release\build\src.win-amd64-3.4\calc_kdp_ray_fir-f2pywrappers.o /OU :build\lib.win-amd64-3.4\calc_kdp_ray_fir.pyd /IMPLIB:build\temp.win-amd64-3.4\Release\build\src.win-amd64-3.4\calc_kdp_ray_fir.lib /MANIFESTFILE:build\temp.win-amd64-3.4\Release\build\src.win-amd64-3.4\calc_kdp_ray_fir.pyd.manifest" failed with exit status 1120

I'm really not good at dealing with these kind of issues, I am using Anacaonda3, python 3.4.3, VS2010 on Windows 7, please help, thank you.

[tjlang]

OK, maybe at some point we can figure out the Windows install issue with Fortran. But for now I created a branch with all the Fortran calls disabled. Expect csu_kdp to take MUCH longer to run.

https://github.com/CSU-Radarmet/CSU_RadarTools/tree/no_fortran

[tjlang]

I should point out that I cannot officially support that branch, so it will not be updated as often as we update the main branch.

[pfhein]

@morstan - Yes I found the same error. Frustrating that the MinGW installer is broken. I manually installed MinGW but have been unable to get it to work with the CSU_RadarTools install. You are farther with the install than I am. My install does not find the compiler though gfortran is found on the command line. I will look into your way since the compiler is found and it appears you have a library problem.

[pfhein]

@morstan - I am having trouble getting it to work. For me it is not recognizing the compiler because I have some updated stuff. From the internet I found some patches that I would need to apply to numpy to get it to recognize the mingw compiler. If I were to apply the patches there is no guarantee that that solve my problems. Likely I would get to the point where you are. So I think @tjlang has the best solution for now. Here are the patches I think would move me closer http://stackoverflow.com/questions/34135280/valueerror-unknown-ms-compiler-version-1900

[morstan]

@tjlang Many thanks to your great effort for helping me, now I am able to install successfully. Since I am using it on research purpose, I don't mind spending a few more minutes as long as it can be run properly.

By the way, for the blended rainfall, would you consider to include the rainfall-attenuation(R(A)) relationship also? As many research suggest diameter relationship with rainfall and attenuation are in the same order, thus can be much more accurately estimate rainfall.

[morstan]

@pfhein Thanks for your help too, actually it is a problem to deal with two compilers. One is the visual studio to deal with C extensions anf minGW for the fortran one. I think I have solve the fortran one but not the VS part, I will try other solutions and see if we can come with a conclusion to solve the problem, thank you.

[tjlang]

@morstan If you have a preferred R(A) relationship, please point me to the reference. Maybe the best thing to do is open a new issue requesting this enhancement, with the information listed. I can't promise a timeline, but that way it's a public, documented request that myself or others can address.

[bdolan44]

@morstan As @tjlang said, we can add a R(A) relationship (I'm assuming here you are referring to something like Ryzhkov et al. 2014?) to CSU-radartools as a method for calculating rain rates. However, we have not done significant testing of this relationship compared to other estimators or for use in the 'blended algorithm' (which selects the best rainfall estimator based on the presence of ice and various thresholds on the variables). So at this point we do not have plans to add it into the 'blended algorithm' framework.

[morstan]

@tjlang ,@bdolan44, I don't have a specific reference to refer to, but the paper you have mentioned from Ryzhkov et al. 2014 about 'Potential Utilization of Specific Attenuation for Rainfall Estimation, Mitigation of Partial Beam Blockage, and Radar Networking' should be the one related to, i don't have access to the paper so i can't read it and judge.

I have learnt the R(A) relationship in a radar course which was given by Dr.Dusan Zrnic from the NSSL of Norman last year, showing us some R(A) relationships with different radar bands and recommendations of choosing the best rainfall estimator for S-band in pure rain.(see below)

R(A) relationships Wavelengths λ = 10 to 11 cm (S band H polarization): R = 4120 (Ah)1.03

λ ~ 5.5 cm (C band V polarization): R = 393 (Av)0.93

λ ~  3.2 cm (X band V polarization): 
 R = 51.1 (Av)0.81

  A in dB km-1;  R in mm h-1 

---

S band in Pure Rain If ΦDP < 1o -------------- take R(Z) If ΦDP ≥ 1o and Z < 48 – 52 dBZ? take R(A)
If ΦDP ≥ 1o and Z ≥ 48 – 52 dBZ? take R(KDP)

*We have not determined the best strategy for C and X band yet. Generally the thresholds on Z and ΦDP should be lower at shorter wavelengths.

X-band: KDP can be used exclusively or in combination with R(Av)

Av = specific attenuation at Vertical polarization.

[tjlang]

@morstan You are encouraged to try the latest update to the master branch, which enables the use of Cython instead of f2py to compile the KDP performance improvement code. I am hoping this allows greater platform support (i.e., Windows).

[drhirsch]

Expanding upon what I wrote on the f2py for Windows page, in general for Windows users and such programs, to save a great deal of time and effort they should consider any one of:

1. using Windows Subsystem for Linux
2. using a virtual machine with Ubuntu
3. Cygwin

[tjlang]

Thanks, @scienceopen, for these tips. Hopefully, between that information and the enabled Cython option, folks should have fewer problems with Windows installs.

[tjlang]

Closing this issue for now, given lack of activity and Cython updates that should have improved Windows 7.