search

OpenFAST / openfast

Main repository for the NREL-supported OpenFAST whole-turbine and FAST.Farm wind farm simulation codes.
http://openfast.readthedocs.io
Apache License 2.0
692 stars 458 forks source link

Some regression tests failed on windows #742

Closed Yubo-Huang closed 3 years ago

Yubo-Huang commented 3 years ago

Hello, everyone

I have successfully compiled Openfast on Windows via:

  1. Visual Studio 2019
  2. Inter Fortran Complier 2021

But in regression tests, I have met the following problems:

executing AWT_YFix_WSt                           PASS
executing AWT_WSt_StartUp_HighSpShutDown         PASS
executing AWT_YFree_WSt                          PASS
executing AWT_YFree_WTurb                        PASS
executing AWT_WSt_StartUpShutDown                PASS
executing AOC_WSt                                PASS
executing AOC_YFree_WTurb                        PASS
executing AOC_YFix_WSt                           PASS
executing UAE_Dnwind_YRamp_WSt                   PASS
executing WP_VSP_WTurb_PitchFail                 PASS
executing WP_VSP_WTurb                           PASS
executing SWRT_YFree_VS_EDG01                    PASS
executing SWRT_YFree_VS_EDC01                    PASS
executing SWRT_YFree_VS_WTurb                    PASS
executing 5MW_Land_DLL_WTurb                     PASS
executing 5MW_OC3Mnpl_DLL_WTurb_WavesIrr         FAIL
executing 5MW_OC3Trpd_DLL_WSt_WavesReg           PASS
executing 5MW_OC4Jckt_DLL_WTurb_WavesIrr_MGrowth FAIL
executing 5MW_ITIBarge_DLL_WTurb_WavesIrr        FAIL
executing 5MW_TLP_DLL_WTurb_WavesIrr_WavesMulti  PASS
executing 5MW_OC3Spar_DLL_WTurb_WavesIrr         PASS
executing 5MW_OC4Semi_WSt_WavesWN                PASS
executing 5MW_OC4Jckt_ExtPtfm                    PASS
executing 5MW_Land_BD_DLL_WTurb                  FAIL
executing IEA_LB_RWT-AeroAcoustics               PASS
executing HelicalWake_OLAF                       PASS
executing EllipticalWing_OLAF                    Traceback (most recent call last):
  File "C:\code\openfast\reg_tests\executeOpenfastRegressionCase.py", line 163, in <module>
    performance = pass_fail.calculateNorms(testData, baselineData)
  File "C:\code\openfast\reg_tests\.\lib\pass_fail.py", line 68, in calculateNorms
    relative_norm = calculate_max_norm_over_range(test_data, baseline_data)
  File "C:\code\openfast\reg_tests\.\lib\pass_fail.py", line 56, in calculate_max_norm_over_range
    diff = abs(test_data - baseline_data)
ValueError: operands could not be broadcast together with shapes (6,413) (11,215)
FAIL
executing WP_Stationary_Linear                   PASS
executing 5MW_Land_BD_Linear                     PASS
executing 5MW_OC4Semi_Linear                     PASS
executing Ideal_Beam_Fixed_Free_Linear           PASS
executing Ideal_Beam_Free_Free_Linear            PASS

I guess the failure of EllipticalWing_OLAF is caused by wrong versions of Python (3.9.5) and Numpy (1.20.2). And therefore I want to inqure the versions of Python and NumPy you used in regression tests. Additionally, what are reasons for other '5MW failures'?

andrew-platt commented 3 years ago

Which version of OpenFAST are you using?

Yubo-Huang commented 3 years ago

Which version of OpenFAST are you using?

The version I used is v2.5.0, which is the last version.

ebranlard commented 3 years ago

The failing of the EllipticalWing case is due to the fact that this test case has now different signals and number of time steps, and we have not updated these baselines for windows. I believe Andy is on it. I'm not sure about the other cases failing

rafmudaf commented 3 years ago

The Windows baselines are generated using a binary compiled with Intel oneAPI 2021 and CMake/NMake. It is possible that the compile flags in the Visual Studio project in vs-build are slightly different than the default CMake flags, so I'm not surprised that some results would be different. I suggest generating the error plots (run the Python script again with the added flags -n -p to avoid running the cases but generate the error plots on the results) and inspecting for any major differences. You can also run individual cases with the -case <case name> flag.

Yubo-Huang commented 3 years ago

The failing of the EllipticalWing case is due to the fact that this test case has now different signals and number of time steps, and we have not updated these baselines for windows. I believe Andy is on it. I'm not sure about the other cases failing

Thanks very much for your reply. So, if I use the old version of OpenFAST, can I avoid this problem?

ebranlard commented 3 years ago

You can indeed roll back to the previous commit (from april 27) making sure you also get the r-test that corresponds to that commit (updating the submodules as well). Until the new baselines are run, you can also just try to copy paste the linux .outb and.out from the elliptical wing test directory to the windows folder.

Yubo-Huang commented 3 years ago

You can indeed roll back to the previous commit (from april 27) making sure you also get the r-test that corresponds to that commit (updating the submodules as well). Until the new baselines are run, you can also just try to copy paste the linux .outb and.out from the elliptical wing test directory to the windows folder.

Thank you. I will try it :)

Yubo-Huang commented 3 years ago

The Windows baselines are generated using a binary compiled with Intel oneAPI 2021 and CMake/NMake. It is possible that the compile flags in the Visual Studio project in vs-build are slightly different than the default CMake flags, so I'm not surprised that some results would be different. I suggest generating the error plots (run the Python script again with the added flags -n -p to avoid running the cases but generate the error plots on the results) and inspecting for any major differences. You can also run individual cases with the -case <case name> flag.

Hi, Ebranlard, Thanks very much for your suggestion. I have executed it and the results are:

PS C:\code\openfast\reg_tests> python .\manualRegressionTest.py ..\build\bin\openfast_x64_Double.exe Windows Inter 1e-5 -n -p
executing AWT_YFix_WSt                           PASS
executing AWT_WSt_StartUp_HighSpShutDown         PASS
executing AWT_YFree_WSt                          PASS
executing AWT_YFree_WTurb                        PASS
executing AWT_WSt_StartUpShutDown                PASS
executing AOC_WSt                                PASS
executing AOC_YFree_WTurb                        PASS
executing AOC_YFix_WSt                           PASS
executing UAE_Dnwind_YRamp_WSt                   Traceback (most recent call last):
  File "C:\code\openfast\reg_tests\executeOpenfastRegressionCase.py", line 181, in <module>
    finalizePlotDirectory(localOutFile, testInfo["attribute_names"], caseName)
  File "C:\code\openfast\reg_tests\.\lib\errorPlotting.py", line 185, in finalizePlotDirectory
    with open(_path, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '..\\build\\reg_tests\\glue-codes\\openfast\\UAE_Dnwind_YRamp_WSt\\plots\\Time_div.txt'
FAIL
executing UAE_Upwind_Rigid_WRamp_PwrCurve        PASS
executing WP_VSP_WTurb_PitchFail                 PASS
executing WP_VSP_ECD                             PASS
executing WP_VSP_WTurb                           PASS
executing SWRT_YFree_VS_EDG01                    PASS
executing SWRT_YFree_VS_EDC01                    PASS
executing SWRT_YFree_VS_WTurb                    Traceback (most recent call last):
  File "C:\code\openfast\reg_tests\executeOpenfastRegressionCase.py", line 181, in <module>
    finalizePlotDirectory(localOutFile, testInfo["attribute_names"], caseName)
  File "C:\code\openfast\reg_tests\.\lib\errorPlotting.py", line 185, in finalizePlotDirectory
    with open(_path, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '..\\build\\reg_tests\\glue-codes\\openfast\\SWRT_YFree_VS_WTurb\\plots\\Time_div.txt'
FAIL
executing 5MW_Land_DLL_WTurb                     PASS
executing 5MW_OC3Mnpl_DLL_WTurb_WavesIrr         Traceback (most recent call last):
  File "C:\code\openfast\reg_tests\executeOpenfastRegressionCase.py", line 181, in <module>
    finalizePlotDirectory(localOutFile, testInfo["attribute_names"], caseName)
  File "C:\code\openfast\reg_tests\.\lib\errorPlotting.py", line 185, in finalizePlotDirectory
    with open(_path, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '..\\build\\reg_tests\\glue-codes\\openfast\\5MW_OC3Mnpl_DLL_WTurb_WavesIrr\\plots\\Time_div.txt'
FAIL
executing 5MW_OC3Trpd_DLL_WSt_WavesReg           Traceback (most recent call last):
  File "C:\code\openfast\reg_tests\executeOpenfastRegressionCase.py", line 181, in <module>
    finalizePlotDirectory(localOutFile, testInfo["attribute_names"], caseName)
  File "C:\code\openfast\reg_tests\.\lib\errorPlotting.py", line 185, in finalizePlotDirectory
    with open(_path, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '..\\build\\reg_tests\\glue-codes\\openfast\\5MW_OC3Trpd_DLL_WSt_WavesReg\\plots\\Time_div.txt'
FAIL
executing 5MW_OC4Jckt_DLL_WTurb_WavesIrr_MGrowth Traceback (most recent call last):
  File "C:\code\openfast\reg_tests\executeOpenfastRegressionCase.py", line 181, in <module>
    finalizePlotDirectory(localOutFile, testInfo["attribute_names"], caseName)
  File "C:\code\openfast\reg_tests\.\lib\errorPlotting.py", line 185, in finalizePlotDirectory
    with open(_path, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '..\\build\\reg_tests\\glue-codes\\openfast\\5MW_OC4Jckt_DLL_WTurb_WavesIrr_MGrowth\\plots\\Time_div.txt'
FAIL
executing 5MW_ITIBarge_DLL_WTurb_WavesIrr        Traceback (most recent call last):
  File "C:\code\openfast\reg_tests\executeOpenfastRegressionCase.py", line 181, in <module>
    finalizePlotDirectory(localOutFile, testInfo["attribute_names"], caseName)
  File "C:\code\openfast\reg_tests\.\lib\errorPlotting.py", line 185, in finalizePlotDirectory
    with open(_path, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '..\\build\\reg_tests\\glue-codes\\openfast\\5MW_ITIBarge_DLL_WTurb_WavesIrr\\plots\\Time_div.txt'
FAIL
executing 5MW_TLP_DLL_WTurb_WavesIrr_WavesMulti  Traceback (most recent call last):
  File "C:\code\openfast\reg_tests\executeOpenfastRegressionCase.py", line 181, in <module>
    finalizePlotDirectory(localOutFile, testInfo["attribute_names"], caseName)
  File "C:\code\openfast\reg_tests\.\lib\errorPlotting.py", line 185, in finalizePlotDirectory
    with open(_path, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '..\\build\\reg_tests\\glue-codes\\openfast\\5MW_TLP_DLL_WTurb_WavesIrr_WavesMulti\\plots\\Time_div.txt'
FAIL
executing 5MW_OC3Spar_DLL_WTurb_WavesIrr         Traceback (most recent call last):
  File "C:\code\openfast\reg_tests\executeOpenfastRegressionCase.py", line 181, in <module>
    finalizePlotDirectory(localOutFile, testInfo["attribute_names"], caseName)
  File "C:\code\openfast\reg_tests\.\lib\errorPlotting.py", line 185, in finalizePlotDirectory
    with open(_path, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '..\\build\\reg_tests\\glue-codes\\openfast\\5MW_OC3Spar_DLL_WTurb_WavesIrr\\plots\\Time_div.txt'
FAIL
executing 5MW_OC4Semi_WSt_WavesWN                Traceback (most recent call last):
  File "C:\code\openfast\reg_tests\executeOpenfastRegressionCase.py", line 181, in <module>
    finalizePlotDirectory(localOutFile, testInfo["attribute_names"], caseName)
  File "C:\code\openfast\reg_tests\.\lib\errorPlotting.py", line 185, in finalizePlotDirectory
    with open(_path, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '..\\build\\reg_tests\\glue-codes\\openfast\\5MW_OC4Semi_WSt_WavesWN\\plots\\Time_div.txt'
FAIL
executing 5MW_OC4Jckt_ExtPtfm                    PASS
executing 5MW_Land_BD_DLL_WTurb                  Traceback (most recent call last):
  File "C:\code\openfast\reg_tests\executeOpenfastRegressionCase.py", line 181, in <module>
    finalizePlotDirectory(localOutFile, testInfo["attribute_names"], caseName)
  File "C:\code\openfast\reg_tests\.\lib\errorPlotting.py", line 185, in finalizePlotDirectory
    with open(_path, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '..\\build\\reg_tests\\glue-codes\\openfast\\5MW_Land_BD_DLL_WTurb\\plots\\Time_div.txt'
FAIL
executing IEA_LB_RWT-AeroAcoustics               PASS
executing HelicalWake_OLAF                       PASS
executing EllipticalWing_OLAF                    Traceback (most recent call last):
  File "C:\code\openfast\reg_tests\executeOpenfastRegressionCase.py", line 163, in <module>
    performance = pass_fail.calculateNorms(testData, baselineData)
  File "C:\code\openfast\reg_tests\.\lib\pass_fail.py", line 68, in calculateNorms
    relative_norm = calculate_max_norm_over_range(test_data, baseline_data)
  File "C:\code\openfast\reg_tests\.\lib\pass_fail.py", line 56, in calculate_max_norm_over_range
    diff = abs(test_data - baseline_data)
ValueError: operands could not be broadcast together with shapes (6,413) (11,215)
FAIL
executing WP_Stationary_Linear                   PASS
executing 5MW_Land_BD_Linear                     PASS
executing 5MW_OC4Semi_Linear                     PASS
executing Ideal_Beam_Fixed_Free_Linear           FAIL
executing Ideal_Beam_Free_Free_Linear            FAIL

Regression test execution completed with these results:
AWT_YFix_WSt                           PASS
AWT_WSt_StartUp_HighSpShutDown         PASS
AWT_YFree_WSt                          PASS
AWT_YFree_WTurb                        PASS
AWT_WSt_StartUpShutDown                PASS
AOC_WSt                                PASS
AOC_YFree_WTurb                        PASS
AOC_YFix_WSt                           PASS
UAE_Dnwind_YRamp_WSt                   FAIL
UAE_Upwind_Rigid_WRamp_PwrCurve        PASS
WP_VSP_WTurb_PitchFail                 PASS
WP_VSP_ECD                             PASS
WP_VSP_WTurb                           PASS
SWRT_YFree_VS_EDG01                    PASS
SWRT_YFree_VS_EDC01                    PASS
SWRT_YFree_VS_WTurb                    FAIL
5MW_Land_DLL_WTurb                     PASS
5MW_OC3Mnpl_DLL_WTurb_WavesIrr         FAIL
5MW_OC3Trpd_DLL_WSt_WavesReg           FAIL
5MW_OC4Jckt_DLL_WTurb_WavesIrr_MGrowth FAIL
5MW_ITIBarge_DLL_WTurb_WavesIrr        FAIL
5MW_TLP_DLL_WTurb_WavesIrr_WavesMulti  FAIL
5MW_OC3Spar_DLL_WTurb_WavesIrr         FAIL
5MW_OC4Semi_WSt_WavesWN                FAIL
5MW_OC4Jckt_ExtPtfm                    PASS
5MW_Land_BD_DLL_WTurb                  FAIL
IEA_LB_RWT-AeroAcoustics               PASS
HelicalWake_OLAF                       PASS
EllipticalWing_OLAF                    FAIL
WP_Stationary_Linear                   PASS
5MW_Land_BD_Linear                     PASS
5MW_OC4Semi_Linear                     PASS
Ideal_Beam_Fixed_Free_Linear           FAIL
Ideal_Beam_Free_Free_Linear            FAIL
Total PASSING tests - 21
Total FAILING tests - 13

I used the exe. file generated last time but the failure number of regression tests in this time is different with the last time. It is very confusing.

Due to I should use OpenFast recently and our project requires to change some source codes, so I have to compile and test it. Could you recommend me a stable version of OpenFast on Windows (currently, I use the latest version)?

rafmudaf commented 3 years ago

This is because you haven't installed the Python package to generate the plots. @ebranlard does pip work for Windows? If so, @LuckyYubo could do pip install Bokeh==1.4.

Yubo-Huang commented 3 years ago

This is because you haven't installed the Python package to generate the plots. @ebranlard does pip work for Windows? If so, @LuckyYubo could do pip install Bokeh==1.4.

I have installed the Python package Bokeh on Windows. And when I run the command:

python manualRegressionTest.py ..\build\bin\openfast_x64_Double.exe Windows Intel 1e-5 -n -p

the result is same with the first time I test and there is no plot generated.

rafmudaf commented 3 years ago

@LuckyYubo

Yubo-Huang commented 3 years ago

@LuckyYubo

  • When you recompiled with the commit @ebranlard suggested, did you rerun the test cases? My suggestion to use the -p -n flag is so that you can generate the plots after you have already run the test cases. If you recompiled but didn't create new local results, you should do that first. To do so, run the Python script without -p -n just as you did initially.
  • As for the failing plot generation, if Bokeh is installed correctly then I'm not sure of the problem. @ebranlard have you noticed anything similar on Windows in the past?
  • In your initial post you mention a possible incompatibility between your Python and Numpy versions. This test system is not very sensitive to versions of Python and Numpy, so you should be ok. In fact, I use similar versions of both.

Yes, after I have run the test cases, I used the -p -n flag to generate the failing plot but I failed and I am sure I have successfully installed the Bokeh package:

PS C:\code\openfast\reg_tests> pip list
Package         Version
--------------- -------
bokeh           1.4.0
cycler          0.10.0
Jinja2          2.11.3
kiwisolver      1.3.1
MarkupSafe      1.1.1
matplotlib      3.4.1
numpy           1.20.2
packaging       20.9
Pillow          8.2.0
pip             21.1.1
pyparsing       2.4.7
python-dateutil 2.8.1
PyYAML          5.4.1
setuptools      56.0.0
six             1.15.0
tornado         6.1
wheel           0.36.2

Today, I have compiled many old versions but I met more fails in regression tests. Eventually, I tried the rc-v3.0.0 branch and the result is:

Regression test execution completed with these results:
AWT_YFix_WSt                           PASS
AWT_WSt_StartUp_HighSpShutDown         PASS
AWT_YFree_WSt                          PASS
AWT_YFree_WTurb                        PASS
AWT_WSt_StartUpShutDown                PASS
AOC_WSt                                PASS
AOC_YFree_WTurb                        PASS
AOC_YFix_WSt                           PASS
UAE_Dnwind_YRamp_WSt                   PASS
UAE_Upwind_Rigid_WRamp_PwrCurve        PASS
WP_VSP_WTurb_PitchFail                 PASS
WP_VSP_ECD                             PASS
WP_VSP_WTurb                           PASS
SWRT_YFree_VS_EDG01                    PASS
SWRT_YFree_VS_EDC01                    PASS
SWRT_YFree_VS_WTurb                    PASS
5MW_Land_DLL_WTurb                     PASS
5MW_OC3Mnpl_DLL_WTurb_WavesIrr         FAIL
5MW_OC3Trpd_DLL_WSt_WavesReg           PASS
5MW_OC4Jckt_DLL_WTurb_WavesIrr_MGrowth FAIL
5MW_ITIBarge_DLL_WTurb_WavesIrr        FAIL
5MW_TLP_DLL_WTurb_WavesIrr_WavesMulti  PASS
5MW_OC3Spar_DLL_WTurb_WavesIrr         PASS
5MW_OC4Semi_WSt_WavesWN                PASS
5MW_OC4Jckt_ExtPtfm                    PASS
IEA_LB_RWT-AeroAcoustics               PASS
HelicalWake_OLAF                       PASS
EllipticalWing_OLAF                    PASS
WP_Stationary_Linear                   PASS
5MW_Land_BD_Linear                     PASS
5MW_OC4Semi_Linear                     PASS
Ideal_Beam_Fixed_Free_Linear           PASS
Ideal_Beam_Free_Free_Linear            PASS
Total PASSING tests - 30
Total FAILING tests - 4

So, currently, only the case of 5MW can not pass the regression test. @ebranlard could you tell me what versions of OpenFAST and Visual Studio you used which can pass the regression tests on Windows?

ebranlard commented 3 years ago

I've actually never run the regression tests on Windows :) It's @rafmudaf that runs them and as he mentioned he is using a binary file that is compiled slightly differently. As a side note, these tests are mostly here if you are planning to do some developments and check that the results do not change. If you are planning to do some developments, it's good if they all pass, otherwise it's probably not too important. You can also update the baselines for the failing cases based on your results, and then develop locally based on those.

salitaliano commented 3 years ago

Hi folks I also got some fails, but I cannot do the plots to check the fails. I guess same problem of @LuckyYubo !!!

Yubo-Huang commented 3 years ago

I've actually never run the regression tests on Windows :) It's @rafmudaf that runs them and as he mentioned he is using a binary file that is compiled slightly differently. As a side note, these tests are mostly here if you are planning to do some developments and check that the results do not change. If you are planning to do some developments, it's good if they all pass, otherwise it's probably not too important. You can also update the baselines for the failing cases based on your results, and then develop locally based on those.

Got it! Thanks very much for your help. Actually, I want to change the drive mechanism of wind turbines: from mechanical drive to hydraulic drive. So is it important that failing to pass some tests?

bjonkman commented 3 years ago

I had problems generating the regression test plots on Windows at one point, too. I recall having to use -p=true and -n=true (and possibly moving them earlier in the command, though I'm not sure about that). I haven't had those issues on my current Windows setup, though, so I can't test the exact syntax anymore.

rafmudaf commented 3 years ago

@bjonkman This particular problem is not related to the syntax of the script since the error is in actually generating the plots. One step in the process is to generate temporary txt files that are ultimately deleted. In this case, the script is attempting to read the temp files, so the script is correctly configured to generate plots. Also, FWIW you can use the -h flag to get the syntax of the script.

@LuckyYubo After reading https://github.com/OpenFAST/openfast/issues/742#issuecomment-833765848, I wonder if the failing test cases in your original post successfully completed the simulation. It sounds like your OpenFAST and r-test commits may have been out of sync. With your latest attempt, did you try to generate plots?

Yubo-Huang commented 3 years ago

@bjonkman This particular problem is not related to the syntax of the script since the error is in actually generating the plots. One step in the process is to generate temporary txt files that are ultimately deleted. In this case, the script is attempting to read the temp files, so the script is correctly configured to generate plots. Also, FWIW you can use the -h flag to get the syntax of the script.

@LuckyYubo After reading #742 (comment), I wonder if the failing test cases in your original post successfully completed the simulation. It sounds like your OpenFAST and r-test commits may have been out of sync. With your latest attempt, did you try to generate plots?

@rafmudaf In my latest attempt, I still tried to generate plots but I failed again. Currently, I am reading the user document to simulate by using OpenFast I compiled. In the near future, If I have any result, I will tell you.

@bjonkman Have you pass all regression tests on Windows? If you have, could you tell me the version of OpenFast you used?

bjonkman commented 3 years ago

@rafmudaf , I have had these python scripts just ignore the flags, which is what the behavior sounded like to me. The help prompts did not actually help when I had issues plotting.... after some debugging inside the python scripts, I could tell that it wasn't parsing the flags as expected/documented. When I reordered them and explicitly set them to true, I finally got it to work. Since then I have a new system, and I thankfully can't reproduce the problem any more. But, there was definitely some strange parsing related to the scripts and/or my python installation on Windows. Yes, I know it sounds strange; I spent a lot of time frustrated with it.

@LuckyYubo, I'm not sure I've ever had all of the regression tests pass on Windows at once. I typically use the double-precision Visual Studio build (not cmake), and 1e-5 tolerance. The plots are important to see why cases failed. On failing cases, I often go to the subdirectories and look at the .log files there to see that they actually ran to completion (and that I'm not using old data if I forgot to delete all of the files there before running the regression tests). Sometimes I will run individual failing cases again with the -v flag since that should tell you if something occurred to make the simulation fail to run as expected.

If you still can't get the python version to plot, you can try running a simple matlab script I created to run the files and plot the results: https://github.com/OpenFAST/r-test/blob/929dfd572af5e4a22c41e4b332ded0190e599a21/runRegressionTests.m. You'll need to have the files from https://github.com/OpenFAST/matlab-toolbox; also make sure that you update the paths at the top of runRegressionTests.m as necessary.

Yubo-Huang commented 3 years ago

@rafmudaf , I have had these python scripts just ignore the flags, which is what the behavior sounded like to me. The help prompts did not actually help when I had issues plotting.... after some debugging inside the python scripts, I could tell that it wasn't parsing the flags as expected/documented. When I reordered them and explicitly set them to true, I finally got it to work. Since then I have a new system, and I thankfully can't reproduce the problem any more. But, there was definitely some strange parsing related to the scripts and/or my python installation on Windows. Yes, I know it sounds strange; I spent a lot of time frustrated with it.

@LuckyYubo, I'm not sure I've ever had all of the regression tests pass on Windows at once. I typically use the double-precision Visual Studio build (not cmake), and 1e-5 tolerance. The plots are important to see why cases failed. On failing cases, I often go to the subdirectories and look at the .log files there to see that they actually ran to completion (and that I'm not using old data if I forgot to delete all of the files there before running the regression tests). Sometimes I will run individual failing cases again with the -v flag since that should tell you if something occurred to make the simulation fail to run as expected.

If you still can't get the python version to plot, you can try running a simple matlab script I created to run the files and plot the results: https://github.com/OpenFAST/r-test/blob/929dfd572af5e4a22c41e4b332ded0190e599a21/runRegressionTests.m. You'll need to have the files from https://github.com/OpenFAST/matlab-toolbox; also make sure that you update the paths at the top of runRegressionTests.m as necessary.

@bjonkman Thanks very much for your experience and suggestion. I will try it in the near future. Recently, I have spent much time to pass the regression tests since I should do some developments and I do not know what will happen if I can not pass all regression tests.

rafmudaf commented 3 years ago

@LuckyYubo and I debugged his issue together. While we were not able to reproduce the specific problem around generating the plots, we were able to get his system aligned so that the expected plots are found.

MHanKama commented 2 years ago

Dear ALL,

I just downloaded and installed the OpenFast (Windows computer user) based on OpenFAST Documentation Release v3.0.0, and I had received some failed test issues related to regression test execution as follows:

  1. 5MW_OC3Mnpl_DLL_WTurb_WaveIrr
  2. 5MW_OC4Jckt_DLL_WTurb_WaveIrr_MGrowth
  3. 5MW_ITIBarge_DLL_WTurb_WaveIrr
  4. 5MW_Land_BD_DLL_WTurb

I noticed that I have a similar issue with @LuckyYubo commented on May 7, 2021. However, based on the discussion above, @rafmudaf had debugged his issue. Thus, how I can solve this issue so that I can execute all regression tests successfully?

FYI, I've compiled and built OpenFAST with Visual Studio 2022.

Kindly advise.

rafmudaf commented 2 years ago

Hi @MHanKama try plotting the differences as mentioned in https://github.com/OpenFAST/openfast/issues/742#issuecomment-833092439. Using the results of that script, you can evaluate whether these differences are problematic. If you have not changed the source code, it's likely that your differences are very small.

MHanKama commented 2 years ago

Hi @rafmudaf , after I ran the command python manualRegressionTest.py ..\build\bin\openfast_x64_Double.exe Windows Intel 1e-5 -n -p, I received the following error on the failed tests, respectively:

File "E:\HA0003053\openfast\reg_tests\executeOpenfastRegressionCase.py", line 181, in finalizePlotDirectory(localOutFile, testInfo["attribute_names"], caseName) File "E:\HA0003053\openfast\reg_tests\lib\errorPlotting.py", line 185, in finalizePlotDirectory with open(_path, 'r') as f: FileNotFoundError: [Errno 2] No such file or directory: '..\build\reg_tests\glue-codes\openfast\5MW_Land_BD_DLL_WTurb\plots\Time_div.txt' FAIL

Is it because there's a file/directory missing? Or did I miss some step regression procedure?

rafmudaf commented 2 years ago

@MHanKama Try adding an additional -v flag to get verbose output. Do you have the Bokeh package installed in your Python environment? You can check with python -c "import bokeh"

xiaoyangzhan commented 2 years ago 
(base) C:\Open\FAST\openfast-3.0.0\reg_tests>python manualRegressionTest.py ..\build\bin\openfast_x64_Double.exe Windows Intel 1e-5 -p -n -v -case 5MW_OC3Mnpl_DLL_WTurb_WavesIrr
executing 5MW_OC3Mnpl_DLL_WTurb_WavesIrr -- Using gold standard files with machine-compiler type windows-intel
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Error generating plots: unexpected attribute 'legend_label' to Line, possible attributes are js_event_callbacks, js_property_callbacks, line_alpha, line_cap, line_color, line_dash, line_dash_offset, line_join, line_width, name, subscribed_events, tags, x or y
Traceback (most recent call last):
  File "executeOpenfastRegressionCase.py", line 181, in <module>
    finalizePlotDirectory(localOutFile, testInfo["attribute_names"], caseName)
  File ".\lib\errorPlotting.py", line 185, in finalizePlotDirectory
    with open(_path, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '..\\build\\reg_tests\\glue-codes\\openfast\\5MW_OC3Mnpl_DLL_WTurb_WavesIrr\\plots\\Time_div.txt'
FAIL

Regression test execution completed with these results:
5MW_OC3Mnpl_DLL_WTurb_WavesIrr FAIL
Total PASSING tests - 0
Total FAILING tests - 1
xiaoyangzhan commented 2 years ago

@rafmudaf I also met this problem,this result is what python manualRegressionTest.py ..\build\bin\openfast_x64_Double.exe Windows Intel 1e-5 -p -n -v -case 5MW_OC3Mnpl_DLL_WTurb_WavesIrr. hope your reply best regards

rafmudaf commented 2 years ago

@MHanKama I missed your comment when you posted it, are you still stuck on the same problem?

@xiaoyangzhan can you verify that you have the correct version of Bokeh (see https://github.com/OpenFAST/openfast/issues/742#issuecomment-833079051).

Do you need to run the regression tests? Did you compile OpenFAST yourself? Are you changing the code? If not, the best path forward is to download the precompiled binaries from the GitHub Releases page.

ZPY12 commented 2 years ago

Hello, everyone

I have successfully compiled on Windows via:Openfast But in regression tests, I have met the following problems:

image

I guess the failure of is caused by wrong versions of bokeh, but I don't known how to do.

rafmudaf commented 2 years ago

Yes, the required version of Bokeh is now at least 2.4. If you already have it installed, upgrade with pip install --upgrade Bokeh.

ZPY12 commented 2 years ago

thanks you can answer my question. But I finished installing Bokeh-2.4.3. In regression testing, I had the same problem.