Convergence

[meneal]

So this one is sort of a question mark at this point. Not totally sure whether we're going to actually need to do this or not, but it should be reasonably straightforward to add this in if we need it. If you want to take a look at how it's done, you can see it in these places:

• Here in section B under performance measures. Pretty interesting paper anyway if you want to check it out.
• here in Menzies' group's storm project.

If we do add this in, one question is really where to put it. We can put it into the end of the run shell script and run it with the hypervolume/spread, or we can just have it run in the state at the end of a run. There's a stub for it on line 114 of state.py.

[ghost]

I pushed a script for convergence. It looks in the obtained PFs folder and for each file finds the matching true PF file and calculates the convergence for each. Right now it just prints out the list of values for each model, but I can make it create an output file instead with the values.

I'm wondering if I should append to this file in case we do multiple runs of the same model or if we should just overwrite the file each time.

[meneal]

Not totally sure yet about the file. Are you printing out the name of the optimizer and the model along with the convergence figure? Thinking about it a bit it might be best if it output to a file and just appended each time. Maybe with a timestamp?

On Fri, Dec 4, 2015 at 8:29 PM, Joseph Sankar notifications@github.com wrote:

I pushed a script for convergence. It looks in the obtained PFs folder and for each file finds the matching true PF file and calculates the convergence for each. Right now it just prints out the list of values for each model, but I can make it create an output file instead with the values.

I'm wondering if I should append to this file in case we do multiple runs of the same model or if we should just overwrite the file each time.

— Reply to this email directly or view it on GitHub https://github.com/LambdaConglomerate/x9115lam/issues/21#issuecomment-162124208 .

[ghost]

Not right now, just the number. So you want the model name, the number, and the timestamp in one output file?

[meneal]

Yeah in one output file. Don't worry about the timestamp. I thought about it and that probably doesn't make sense. We'll probably end up purging the output file from time to time.

On Fri, Dec 4, 2015 at 8:39 PM, Joseph Sankar notifications@github.com wrote:

Not right now, just the number. So you want the model name, the number, and the timestamp in one output file?

— Reply to this email directly or view it on GitHub https://github.com/LambdaConglomerate/x9115lam/issues/21#issuecomment-162124942 .

[ghost]

I ran test.py but it doesn't look like it's writing to the obtained PF file. I see that it creates the file but it's empty. Any reason why that may be happening?

[meneal]

It's because there's nothing currently being written to the frontier in the state. I'm working on that right now actually. I've run into some weirdness with retries, for some reason I'm getting a crash if I run more than 1 retry.

On Fri, Dec 4, 2015 at 8:56 PM, Joseph Sankar notifications@github.com wrote:

I ran test.py but it doesn't look like it's writing to the obtained PF file. I see that it creates the file but it's empty. Any reason why that may be happening?

— Reply to this email directly or view it on GitHub https://github.com/LambdaConglomerate/x9115lam/issues/21#issuecomment-162125932 .

[ghost]

I pushed the changes requested. Running convergence.py now outputs to a file. Each line in the file corresponds to an obtained PF and includes the model name and the convergence value. Right now the file is overwritten every time the script is run.

Let me know if you see any issues or want to make changes.

[meneal]

Something weird. I pulled this and tried running it with some obtained fronts and I'm not getting any change to convergence.txt. Any idea why that might happen?

[ghost]

Hmmm, I ran it again with different models and the file updated. I noticed that all the output was all on one line though, so I just pushed a fix for that.

Maybe you have a write lock on the file? Try deleting the file and running the script again.

[meneal]

I just ran it again after deleting the txt file and nothing happens... What file is it supposed to be working on? Does it just use whatever is in Spread?

DESKTOP-CN1FQA2:Convergence meneal$ python convergence.py DESKTOP-CN1FQA2:Convergence meneal$ ls convergence.py

[meneal]

damnit! It's a Windoze issue. This line convergence_file = open(".\convergence.txt", 'w') creates a dotfile on OSX or linux. DESKTOP-CN1FQA2:Convergence meneal$ ls -la total 16 drwxr-xr-x 4 meneal staff 136 Dec 5 13:56 . drwxr-xr-x 7 meneal staff 238 Nov 29 17:38 .. -rw-r--r-- 1 meneal staff 19 Dec 5 13:56 .\convergence.txt -rw-r--r-- 1 meneal staff 2198 Dec 5 13:52 convergence.py

Do you need that ".\" for this to work on your machine? I would think that it would work without that.

[ghost]

I'll try when I get home later. I'm thinking it should work though. On Dec 5, 2015 2:03 PM, "Matthew Neal" notifications@github.com wrote:

damnit! It's a Windoze issue. This line convergence_file = open(".\convergence.txt", 'w') creates a dotfile on OSX or linux. DESKTOP-CN1FQA2:Convergence meneal$ ls -la total 16 drwxr-xr-x 4 meneal staff 136 Dec 5 13:56 . drwxr-xr-x 7 meneal staff 238 Nov 29 17:38 .. -rw-r--r-- 1 meneal staff 19 Dec 5 13:56 .\convergence.txt -rw-r--r-- 1 meneal staff 2198 Dec 5 13:52 convergence.py

Do you need that ".\" for this to work on your machine? I would think that it would work without that.

— Reply to this email directly or view it on GitHub https://github.com/LambdaConglomerate/x9115lam/issues/21#issuecomment-162237390 .

[ghost]

Removing the "./" works on my machine, so I pushed the changes. Let me know if it works.