search

idaholab / raven

RAVEN is a flexible and multi-purpose probabilistic risk analysis, validation and uncertainty quantification, parameter optimization, model reduction and data knowledge-discovering framework.
https://raven.inl.gov/
Apache License 2.0
212 stars 131 forks source link

[UNDER-DISCUSSION] SMT vs. PyDOE #2294

Open wangcj05 opened 3 months ago

wangcj05 commented 3 months ago

Under Discussion Topic

Summary of the topic to be discussed with the development team

Currently both SMT and PyDOE are used inside RAVEN to generate random samples. SMT is used by BayesianOptimizer to perform LHS sampling, while PyDOE is used by Samplers to perform Factorial Design, ResponseSurfaceDesign etc. In addition SMT is not available for MAC. We should consider to use just one of them. Possible solution is to remove both SMT and PyDOE, and use PyDOE3 directly (https://pydoe3.readthedocs.io/en/stable/index.html). The current issue with SMT is that we can not run BayesianOptimizer using MAC and we can not generate the corresponding manual also in MAC system. While PyDOE is currently used as contrib library for raven.

Describe the solution you'd like to be implemented A clear and concise description of what you want to happen.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

For Change Control Board: Issue Review

This review should occur before any development is performed as a response to this issue.

For Change Control Board: Issue Closure

This review should occur when the issue is imminently going to be closed.

wangcj05 commented 3 months ago

Hi @joshua-cogliati-inl @PaulTalbot-INL @mandd @dylanjm @Jimmy-INL, please let me know what do you think.

PaulTalbot-INL commented 3 months ago

Hm, that's interesting. If PyDOE can do the same LHS sampling that BO needs, we should probably pivot to using that. How did this not show up in the regression tests? Or did it?

dylanjm commented 3 months ago

@wangcj05 Would switching to PYDOE3 allow us to install from conda and not rely on lugging around another contrib library? I think ideally we use a consistent library for all sampling tasks and reduce number of contrib libraries.

wangcj05 commented 3 months ago

Internally, SMT is using PyDOE3 to perform LHS sampling, so for me there is no need to use SMT. In addition, we are using PyDOE as contrib, and I do not remember the specific reason for that. For me, I do not think we need SMT, but there is probably some minor work to do when switching PyDOE to PyDOE3. Regarding the regression test, PyDOE and LHS are used in different entities, we do test them, and there is no conflict. @PaulTalbot-INL @dylanjm

wangcj05 commented 3 months ago

PyDOE3 can be installed by both conda and pip. It seems to me it is very possible for use to reduce the contrib library and at the same time to use consistent library. @dylanjm

PaulTalbot-INL commented 3 months ago

Do we have any idea what we added to pyDOE for our needs? As I recall, we only used it in the Design of Experiment sampler, which barely sees the light of day as far as I know.

joshua-cogliati-inl commented 3 months ago

I don't recall having to do major changes to pyDOE, just things like adding a few docstrings (to make pylint happy) and making some imports relative (so we didn't have to mangle the PYTHONPATH for pyDOE) ab269846be6f8bc13adce562f80f84bcbb497f9d and 3b253877bde67832265c5f5414bba22b45b26272

PaulTalbot-INL commented 3 months ago

Hm. Why doe we have pyDOE as a contrib rather than a conda/pip dependency? Did I miss this detail?

joshua-cogliati-inl commented 3 months ago

I am not sure at this point. It might be worth making a quick pull request that deletes the directory and adds it as a pip/conda dependency and seeing what happens.

alfoa commented 3 months ago

Internally, SMT is using PyDOE3 to perform LHS sampling, so for me there is no need to use SMT. In addition, we are using PyDOE as contrib, and I do not remember the specific reason for that. For me, I do not think we need SMT, but there is probably some minor work to do when switching PyDOE to PyDOE3. Regarding the regression test, PyDOE and LHS are used in different entities, we do test them, and there is no conflict. @PaulTalbot-INL @dylanjm

Hi guys. The reason why PyDOE was added as a contrib folder 10 years ago was that it was not available as pip/conda package at that time. I used pyDOE (from conda) in the recent past. The API did not basically change since a decade

PaulTalbot-INL commented 3 months ago

Thanks, @alfoa !

joshua-cogliati-inl commented 2 months ago

I created a pull request to use pyDOE3 directly. #2298

I don't have time at the moment to switch the smt stuff to pyDOE, but the above pull request is ready to review.