search

SNL-WaterPower / WecOptTool-MATLAB

WEC Design Optimization Toolbox
GNU General Public License v3.0
12 stars 9 forks source link

CI unit testing for MATLAB #97

Closed ryancoe closed 3 years ago

ryancoe commented 4 years ago

Let's discuss the cost/benefit of setting up CI with the unit tests as has been done for MHKit.

@zmorrell-sand - Can you lay out the process (or point us to something that does)?

zmorrell-sand commented 4 years ago

@ryancoe I will send you the full setup document that I wrote up for Kelley on WecSim. Basically, the way that the server is currently set up, it will send a build status to GitHub to display on the top of the front page of the repository. There are difficulties with setting up email notifications since the server is behind Sandia's firewall, but it at least indicates whether the build is failing so someone can run the tests and fix them.

ryancoe commented 4 years ago

@zmorrell-sand - Thanks for the info. Would you be able to execute the solution you've outlined? Also, did you look at using MATLAB Online to run tests (I don't see anything about this from some quick Googles)?

H0R5E commented 4 years ago

Its a nice idea but I don't think there is any kind of RESTful API for MATLAB online, so that wouldn't work. We also ready need (at least) 2 machine configurations, one without the PCT and one with (ideally mirrored in Windows and Linux = 4 virtual machines).

zmorrell-sand commented 4 years ago

@ryancoe If I recall from my discussion with Kelley or Rebecca on using MATLAB Online for WecSim or MHKiT, PNNL had issues with some of the terms of use. I am not sure exactly what those issues were, but my understanding was that the issue arose from some of the legal wording. That being said, it may be a good option for this project, but it would probably require some more investigation into the Terms of Use.

As for the solution I have outlined, it should be easy enough to get set up, so I will try to get progress made on that today. I am assuming that the unit tests we would want to run would be the runTests.m file, correct?

H0R5E commented 4 years ago

@zmorrell-sand, just hypothetically, can you tell me how this could be done with MATLAB Online? Can you start instances remotely? I couldn't find any documentation about it but I'd really like to know if its possible, if you can help me?

zmorrell-sand commented 4 years ago

@H0R5E I have never actually done any running with MATLAB Online. My past experience with MATLAB Online got as far as finding it as an alternative option for CI, suggesting it, and finding out about the potential legal issues. Because it wasn't an option at the time I never looked into it further

H0R5E commented 4 years ago

OK, no problem, just thought id pick your brains because its new to me. Regarding the tests, yes its runTests. Does it need to generate particular output? You can see which plugins are installed at the moment but we could run in different configurations if different output formats would be useful. Also, what do you think about running an additional instance without the parallel toolbox?

zmorrell-sand commented 4 years ago

From what I have seen after a little bit of searching, using Jenkins seems to be the suggested method with MATLAB . The guy who wrote this is a senior developer for MATLAB and he seems to write most of the articles I have seen on setting up CI with MATLAB

zmorrell-sand commented 4 years ago

The easiest thing based on what I have worked with in the past is having a test suite set up, but I think this fits the bill. I will let you know if I run into any weird issues that would necessitate any modification of the test file.

zmorrell-sand commented 4 years ago

@kmruehl @H0R5E @ryancoe I have been having some difficulty getting the WecOptTool setup on the Jenkins server, and the issue seems to be due to the installation of MATLAB not having the Optimization Toolbox. I am currently trying to install the other toolboxes that are available with the license file that it is using, but I am not sure that the license has the Optimization toolbox. Fortunately, this also brought a bug to light since the dependency checker throws an exception if the Optimization_Toolbox is not installed on the machine. I cannot currently take a screenshot of the error message, since MATLAB is in the process of installing the additional toolboxes, but if I recall correctly, it was an issue in line 63 of dependency checker. It claims that one of the variables being compared does not exist. The toolbox update seems like it will take some time, so I will try to update you when they are finished installing, but it is possible that the current license file doesn't have the correct toolbox.

zmorrell-sand commented 4 years ago

All of the toolboxes with this license have been installed, and it seems that the Optimization Toolbox is not on the license file. Also, I figured out what the bug was, there was a typo in a variable name. I have pushed the fix to my fork and submitted a pull request.

H0R5E commented 4 years ago

I figured out what the bug was, there was a typo in a variable name. I have pushed the fix to my fork and submitted a pull request.

@zmorrell-sand, good spot. For some reason I don't see your pull request (was it against SNL-WaterPower/WecOptTool?), but I have found the typo and I will add it to #113. The edge cases for this tool are really hard to test, but I am also surprised that the static analysis did not pick up the undefined variable.

H0R5E commented 4 years ago

All of the toolboxes with this license have been installed, and it seems that the Optimization Toolbox is not on the license file.

So I guess we need to purchase toolbox licenses for this setup. Is it not possible to use Sandia's site license?

EDIT: Another possibility is that you have a fixed number of toolbox licenses available (at Sandia) and that they are all checked out. In this case, @ryancoe, it might be worth purchasing a license for WecOptTool, specifically to do this work.

ryancoe commented 4 years ago

@zmorrell-sand - Is your existing CI machine using a site license or stand-alone?

kmruehl commented 4 years ago

@ryancoe @zmorrell-sand @H0R5E

The Jenkins server is using my stand-alone license, not the Sandia CEE site license. The server cannot use the CEE license since the Virtual Machine (VM) is on the Sandia Open Network (SON). FYI, My stand-alone license is expiring, and I plan to renew it, but also buy a separate license for Jenkins on the VM. When I purchase the second license for Jenkins I can add the Optimization Toolbox. Is that the only required package?

kmruehl commented 4 years ago

@ryancoe

@zmorrell-sand - Thanks for the info. Would you be able to execute the solution you've outlined? Also, did you look at using MATLAB Online to run tests (I don't see anything about this from some quick Googles)?

We looked into this, it was not a viable solution ~6 months ago, but things change

H0R5E commented 3 years ago

Closed with #199