Clean up of psuade files

[fsoepyan]

When I run OUU, the iterations are no longer recorded in the Flowsheet table.

[eslickj]

I think this must be related to the client program that send the simulations back to FOQUS over a socket connection. It should be like the one that we use for ALAMO adaptive sampling, which I fixed. Can you confirm that the samples all run correctly and the only problem is that they are not recorded in the flowsheet run data.

[fsoepyan]

• OUU still runs
• But when I checked the Flowsheet Table, it is blank (the OUU runs are not recorded)

[ou3llnl]

I think this has to do with OUU not able to advance due to issue #101. When I can fully debug what is going on and fix it, this should no longer be an issue. Some progress has been made, but it is still ongoing.

[ou3llnl]

@fsoepyan : Try this with psuade 1.7.12 and let me know. I think it should work now.

[fsoepyan]

• I downloaded psuade 1.7.12 from GitHub
• I made sure the psuade directory under the "Settings" tab refers to the correct folder
• When I ran OUU, I got this error message:

• I closed the error message
• I checked the "Flowsheet Results" table, and the OUU runs are not recorded

[ou3llnl]

Ok. That's an error I was not anticipating. At least you are getting plot points now, which I assume you were not getting before. This may be a PSUADE issue. It's funny though that you were able to get through 360 iterations before this occurred.

[fsoepyan]

Actually, I managed to get the plot points (in the bottom right corner) with the previous version of psuade.

[ou3llnl]

@fsoepyan : What is the method the flowsheet is using to calculate the errors? Is this a standard turbine node? Or is this an attempt to use the surrogate capability that Josh Morgan and John Eslick have been adding to flowsheet?

If you were to run a standard turbine/turbine lite problem, does it work?

[fsoepyan]

Answering your first question:

• I did not use the new surrogate capability
• I used "Plugin" and the ALAMO surrogate model

[fsoepyan]

Answering your 2nd question:

• I ran OUU using a turbine lite model
• I got the same error message that I showed earlier, and the OUU runs were not recorded in the Flowsheet table
• I'm also having the same problem as that mentioned in Issue #131

[ksbeattie]

@ou3llnl says that this is waiting on a new version of psuade?

[ou3llnl]

@fsoepyan : We found the error from foqusPSUADEClient.py is actually a problem with Python setup in Windows, which causes calls to Python scripts from console to omit command line arguments. Follow the steps in the following links to fix your environment.

https://eli.thegreenplace.net/2010/12/14/problem-passing-arguments-to-python-scripts-on-windows https://stackoverflow.com/questions/2640971/windows-is-not-passing-command-line-arguments-to-python-programs-executed-from-t

[fsoepyan]

• I went to the links Jeremy (@ou3llnl) recommended.
• The %* symbol is the key to resolving the issue.
• Can anyone please tell me where I need to insert this symbol in the foqusPSUADEClient.py code?

[sotorrio1]

@fsoepyan , you actually need to add that to your registry keys. In Windows go to Start button and search for "regedit". Open the regedit.exe and follow the instructions in the first link Jeremy sent (https://eli.thegreenplace.net/2010/12/14/problem-passing-arguments-to-python-scripts-on-windows). I used to have the same issue and now it's fixed. Let me know if you need help with this

[fsoepyan]

• I went to the Windows Start button and searched for "regedit"
• Registry Editor is opened
• I looked for: HKEY_CLASSES_ROOT\py_auto_file\shell\open\command
• I double-clicked "(Default)"
• I changed the "Value data" to: "C:\Users\fsoepyan\AppData\Local\Continuum\anaconda2\python.exe" "%1" %*
• I ran OUU
• The results still are not recorded in the Flowsheet table

[sotorrio1]

Did you change both registry keys? HKEY_CLASSES_ROOT\Applications\python26.exe\shell\open\command HKEY_CLASSES_ROOT\py_auto_file\shell\open\command

The results are not recorded but, do you get the error message anymore?

[fsoepyan]

• I changed both registry keys
• I still get the error message

[sotorrio1]

odd... @fsoepyan what happens when you run the simple example from the link in the command line? Does it work?

[fsoepyan]

I ran the example problem from the link, and it works:

[sotorrio1]

@fsoepyan so you still get this error?

[fsoepyan]

No.

[sotorrio1]

ah ok, what error do you get then?

[fsoepyan]

• I used to get the error message that I showed in my June 13 comment, which is the same error message that you showed in your July 30 comment
• But when I ran OUU today with 2 different files, I no longer get this error message

[sotorrio1]

got it. So the only issue now is that the runs are not recorded?

[fsoepyan]

Yes.

[ou3llnl]

@fsoepyan: I just tried it and I get runs recorded in the Flowsheet run table. I loaded Rosenbrock_no_vectors, went to OUU, set Rosenbrock.x1 to Z1, and then "Run OUU". When done, I went to flowsheet and hit the "Results and Filtering" table and runs show up. Do you get any kind of errors or warnings in the console?

[eslickj]

There is a chance this is the same as issue #251. @fsoepyan can you try again now that that has been fixed? That's only if the samples disappear after saving and loading. If this is still a problem can send log messages from session where the problem occurs.

[fsoepyan]

• I tried running OUU with the latest pull requests (#256 and #257)
• The OUU runs are still not being recorded in the Flowsheet table
• However, the OUU runs are successful
• Here is the message from "Log Viewer":

2018-08-09 15:25:57,506 - DEBUG - foqus.foqus_lib.framework.graph.graph - Running flowsheet(s) locally 2018-08-09 15:25:57,553 - DEBUG - foqus.foqus_lib.framework.sim.turbineConfiguration - Starting simulation consumer... 2018-08-09 15:25:57,553 - DEBUG - foqus.foqus_lib.framework.graph.node - Generated Job JSON: [ { "ConsumerId": "{CA99B34B-A19A-4213-B437-BD4F6FB7078D}", "Initialize": false, "Input": { "B2.alpha": 5.0, "B2.beta": 5.0, "B2.d1": 1.0, "B2.d2": 1.0, "B2.d3": 1.0, "B2.d4": 1.0, "B2.delta": 5.0, "B2.omega1": 1.0, "B2.omega2": 1.0, "B2.omega3": 1.0, "B2.omega4": 1.0, "MinStepSize": 0.001, "RunMode": "Steady Optimization", "Script": "", "Snapshot": "", "TimeSeries": [ 0.0 ], "TimeUnits": "Hours", "homotopy": 0, "printlevel": 0 }, "Reset": false, "Simulation": "ouu_toy2", "Visible": false } ] 2018-08-09 15:25:57,773 - DEBUG - foqus.foqus_lib.framework.graph.node - Started Job: 1845 Turbin SID: 1bd40982-2e91-4925-b99c-fa86269a0216 2018-08-09 15:26:02,046 - INFO - foqus.foqus_lib.framework.sim.turbineConfiguration - Job 1845 Started Running 2018-08-09 15:26:02,046 - DEBUG - foqus.foqus_lib.framework.graph.node - Job finished successfully: 1845 2018-08-09 15:26:02,046 - INFO - foqus.foqus_lib.framework.graph.node - Job 1845 Finished Successfully 2018-08-09 15:26:02,046 - DEBUG - foqus.foqus_lib.framework.graph.node - Outputs: status = 0 B2.z = 2.85705080757 B2.x4 = 4.875 B2.x3 = -3.25 B2.x2 = -2.5 B2.x1 = -1.0

[ksbeattie]

@eslickj will follow up with @fsoepyan on this.

[fsoepyan]

• I tested the OUU tab using the same example Aspen Plus file from issue #126
• I have 3 decision variables, 2 uncertain (UQ) variables, and I ran OUU using 50 samples
• The OUU results are saved in the Flowsheet table with "listener" as the set name
• According to the OUU tab, there are 3150 iterations
• But the Flowsheet table only shows 1300 data points
• Can anyone please let me know why these numbers are different?

[ksbeattie]

@sotorrio1 is blocked from being able to reproduce this possibly because of #147 @fsoepyan is now mostly concerned about the difference described above @ou3llnl will follow up with Charles Tong on if this is an issue in psuade.

[ou3llnl]

Per Charles, the only reason he can think of for why # of iterations is greater than number of data points is that PSUADE can detect duplicate runs and not repeat calculations that have already been performed. However, we need to find out if that really is the case here.

[ksbeattie]

@sotorrio1 can you now reproduce this? And maybe look into if psuade skipping dup calculations explains the discrepancy that @fsoepyan mentions?

[sotorrio1]

I tested OUU with one of Byron's problems "MEA_Column_02_15_2018.foqus":

• X1, X2, X3 and X4 --> Decision Variables (Z1)
• X5, X6, X7, X8, X9, X10, X11, X12, X13, X14, X15, X16, X17 --> Uncertainty Variables (Z4)
• Objective Function: surrogate.negCO2
• Generate new sample for Z4, Latin Hypercube, Sample Size = 25
• Run OUU

OUU finishes successfully with 1,975 iterations and when I go to the flowsheet and check the runs, there are 1,975 runs recorded in the flowsheet.

What I've noticed is that if I clear all the data in flowsheet and run the same problem again, OUU runs successfully but the runs are not longer recorded in the flowsheet. I've also noticed that the files generated by PSUADE of the format 'psuadeEval.out.xxx' (being xxx the iteration) are not getting deleted in the working directory. If the user doesn't clear the working directory psuade uses those files (you can see the WARNINGS in the psuadelog) and the runs don't get recorded in the flowsheet. I can check with Charles next week (he's on vacation this week), because those psuade generated files are supposed to get deleted when psuade generates the file for the next iteration.

[fsoepyan]

• Before running OUU, I went to my FOQUS working directory, and deleted all the files with names that start with "psuadeEval"
• After running OUU, I checked the Flowsheet table, and the OUU runs are recorded
• I think this is an acceptable solution, and I believe we can close this issue

[ksbeattie]

@sotorrio1 believes this might be an issue with psuade not cleaning up old files. It seems to work from the command line, but not within FOQUS. @sotorrio1 will follow up on this.

In the meantime, some clean-up of the working dir should be done by FOQUS before launching psuade.

[ksbeattie]

@sotorrio1 is continuing to work on this with Charles Tong

[ksbeattie]

@sotorrio1 will continue to look into this.

[sotorrio1]

PR #339 should take care of cleaning up the psuade files in the working directory

[ksbeattie]

@fsoepyan will test this now that PR #339 has been merged.

[fsoepyan]

I have tested OUU in FOQUS, and the old "psuadeEval.out" files are automatically deleted at the beginning of each OUU run.

The PR works. Thank you!