Overall Rig Running Setup

[meneal]

I've been working on this all morning. Basically what I've done is setup something where we can run as much as possible automatically. There are a few things that prevent automation here's the list on that:

• Since we need to log each optimizer separately we cant run multiple optimizers in one run.
• Since Hypervolume can't work on mixed 2-D and 3-D fronts, we need to run them separately.

So I have automated the running as far as possible within those constraints using the shell script and a bit of python shell command stuff. Here's how it works:

• Run.sh takes two arguments -x and the name of the output file. So you can run like this: sh run.sh outfile.txt
• When the script runs it basically will run what's in test.py and then output its log to out/<>. It also outputs all of the frontiers that it got to a folder called obtained in each of the metric setups.
• The shell script will then run each of the metric rigs against all of the frontiers and output to one file in metrics/out/<>
• The shell script will delete the file in both the metrics folder and the out folder locations if it already exists.

The idea is as follows:

• test.py now has two lists that will run against the metrics, 2d and 3d. Some of the models just won't work with the metrics for some reason. It's a bug with convergence, but it's not worth fixing at the 11th hour.
• The idea is we run each optimizer against all of the 2D and 3D models using the following command: sh run.sh -x outfile where outfile is Optimizer_(2D or 3D)_K#.txt where # is the number of changes in a run. An example is "AdaptiveGlobalwithProbs_2D_K100.txt" for probs with 2d front and 100 runs for each retry.

You'll need to manually change test.py to set the number of retries and number of changes and also manually set the 2D and 3D fronts.

My thought is that we pare back to 3 optimizers, the base AdaptiveGlobalPSO, AdaptiveGlobalPSOwithProbs, and AdaptiveGlobalPSOV2. We run each for 500, 1000, and 5000 changes with 30 retries on each. Problem is that there are six of those runs in total.

If we want to finish we'll need to start running them now, I'm not totally sure whether we can finish at all if we run graphs too. But I'm going to test out 500 2D and 3D on Adaptive global now. I have set the constriction factor up with phi_1=3.8 and phi_2=2.2 which gives us a constriction factor of .27, but keeps the ratio the same between phi_1 and phi_2.

I'm going to test without the tracked particle part though since that should be fairly time consuming. it might work to just use the regular grapher though. I'll probably try both. Either way I'm running now and as soon as it finishes I'll push. One bug to note, if for some reason we only end up with one member of a final front, the Spread metric will crash. Just find the file in obtained that had that one member front, delete it and then run python SpreadMill.py >> ../out/<<whatever the filename was>>

[meneal]

I also just created a graph dump in google drive. We can just dump off all of the graphs in there so we don't need to keep them all in the repo. After a run just upload the graphs to the folder for the optimizer and the appropriate K number. We'll run out of space if we don't do that, not only that the grapher isn't set up to keep track of anything except for the model name and the optimizer name, so we need to keep the number of changes separate.

[ghost]

Are you running on AWS? If the runs are going to take a while I can run them, although I'm not sure if I can get the shell script to run on Windows.

[meneal]

I'm running on my local machine for now. Check it out and let me know if you can run it on your local machine if not we can certainly set you up a vm somewhere.

[meneal]

Just added pics from 2D Adaptive Global K500. Pushing metrics, and outfiles.

[aisobran]

Just pushed PSOv2. Also added optimizer name to grapher.

I'll run 2D Probs 500

[meneal]

I'm running 3D AdaptiveGlobal with K500

[aisobran]

Pushed up k100 Probs, stating k100 probs 3d

If you guys pull I added unique names for grapher so everything in grapher will be prepended to the name ala "PSOProbsK100.0..."

I added this to PSO and PSOv2.py

Now we don't have to worry about the google drive. I don't think we'll hit the 200 MB limit, we're at 80 mb right now for the entire repo.

[aisobran]

Starting on k500_probs_2d

[ghost]

Finished k500_probs_2d with PSOv2. I uploaded the pics to the folder on google drive. Is there anything else I should add, like the metrics?

[aisobran]

I would rename it k500_v2_2d.txt before adding is you used PSOv2, probs is for PSOprobs

[meneal]

AdaptiveGlobalPSO_3D_K500 finally complete. putting images into the google drive for now. Will fix later. Running AdaptiveGlobalPSO_2D_Tracked now. Basically turning on tracking for 2d and then using 1 retry and 500 changes.

[ghost]

Pushed PSOV2 2D with 500 changes. I renamed the text files so there weren't any conflicts.

[aisobran]

Pushed Probs_2D_500

Moving on the Probs_3D_500

[meneal]

tracked run for both 2d and 3d complete. Running adaptiveGlobal for 3D K5000 1 retry.

[meneal]

Might crash for these runs if for some reason we don't end up with any diversity. Hypervolume doesn't like it when there is only one point in the frontier.

[meneal]

5K 3d complete. running 5K 2d.

[ghost]

PSOV2 3D K500 complete.

[ghost]

Pushed PSOV2 2D K5000. Will push PSOV2 3D K5000 tomorrow.

[aisobran]

Starting 2d 5K Probs, I though we weren't doing these?

[meneal]

Just the 1 retry 5k. Was what I did.

On Tuesday, December 8, 2015, Alexander Sobran notifications@github.com wrote:

Starting 2d 5K Probs, I though we weren't doing these?

— Reply to this email directly or view it on GitHub https://github.com/LambdaConglomerate/x9115lam/issues/40#issuecomment-162864913 .

[aisobran]

5k 2d done and pushed, completing 5k 3d Probs

[aisobran]

Push 5k 3d Probs.

Moving on to 60 particles, 100 changes, 30 retries, Probs

[ghost]

Running PSOV2, 3D, 1 retry, 5000 changes.