Error in packaging for real_galaxy run

[pgee2000]

I am getting the following throw from builder.py when I run the following:

great3sims.run("real_galaxy", gal_dir="sample", truth_dir="truth", experiments=["real_galaxy"], obs_type="ground", shear_type=["constant"])

The error appears during the creation of a tar file at the end of the script. Not sure if the file in question is not being created, or if I am running the real_galaxy option incorrectly. I just updated to the latest commit, d4... just to be sure.

-----------------------------------------------------------------
Packaging data for real_galaxy / ground / constant
Traceback (most recent call last):
  File "real.py", line 2, in <module>
    great3sims.run("real_galaxy", gal_dir="sample", truth_dir="truth", experiments=["real_galaxy"], obs_type="ground", shear_type=["constant"])
  File "/sandbox/lsstshared/pgee/mylsst9/great3-public/great3sims/__init__.py", line 242, in run
    builder.packageTruth(subfield_min, subfield_max)
  File "/sandbox/lsstshared/pgee/mylsst9/great3-public/great3sims/builder.py", line 1409, in packageTruth
    shutil.copy2(infile, outfile)
  File "/sandbox/lsstshared/v10_0/Linux64/anaconda/2.1.0/lib/python2.7/shutil.py", line 130, in copy2
    copyfile(src, dst)
  File "/sandbox/lsstshared/v10_0/Linux64/anaconda/2.1.0/lib/python2.7/shutil.py", line 82, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: '/sandbox/lsstshared/pgee/mylsst9/great3-public/great3sims/real_galaxy/real_galaxy/ground/constant/star_test_images.fits'
---------------------------------------------------------------------

[rmandelb]

Hi Perry - you are doing nothing wrong. I confirm that I can reproduce this issue on my computer, and here's an explanation for why this happens at all:

When generating the GREAT3 images, we initially used the great3sims.run() command for our tests of the full image generation process. But when we wanted to mass produce images, we would use great3sims.run() just to generate parameters, catalogs, and config files. Then we'd go off and separately run galsim on the config files to generate all images, and then we'd use great3sims.run() to package them up. At some point late in that process, we realized that we needed these star test images to eyeball the PSFs and make sure they were sane. So great3sims.run() generates the appropriate yaml files for to make those test images, but I was relying on the fact that I was going off and separately running galsim on all generated yaml files. There is no step in great3sims.run() that will make the star test images, so it fails when you try to use great3sims.run() to make the images and package them up.

In the short term, you can fix this by pulling the latest version of the master branch of this repo. I just pushed a single commit that first checks for the existence of the star test images before trying to package them up, so if they aren't there, it will quietly ignore the current lines 1409-1410.

Alternatively, you can use great3sims.run() for the early steps (generating parameters, catalogs, config files), then generate the sims using the galsim executable on all the .yaml files, then do the packaging steps with great3sims.run(). This way you'll get the star test images if you want them.

[rmandelb]

Sorry for the extra post, I meant to say that I have confirmed that this fixes the problem on my systeme. If you find this works for you, please feel free to go ahead and close this issue (but I'll leave it open in case you have any more questions / comments).

[pgee2000]

Thanks, Rachel. I did figure out that everything I needed was actually being generated.

It was just confusing at first that a canned experiment failed.

I will pull and test it tomorrow.

On Tue, Apr 21, 2015 at 6:03 PM, Rachel Mandelbaum <notifications@github.com

wrote:

Hi Perry - you are doing nothing wrong. I confirm that I can reproduce this issue on my computer, and here's an explanation for why this happens at all:

When generating the GREAT3 images, we initially used the great3sims.run() command for our tests of the full image generation process. But when we wanted to mass produce images, we would use great3sims.run() just to generate parameters, catalogs, and config files. Then we'd go off and separately run galsim on the config files to generate all images, and then we'd use great3sims.run() to package them up. At some point late in that process, we realized that we needed these star test images to eyeball the PSFs and make sure they were sane. So great3sims.run() generates the appropriate yaml files for to make those test images, but I was relying on the fact that I was going off and separately running galsim on all generated yaml files. There is no step in great3sims.run() that will make the star test images, so it fails when you try to use great3sims.run() to make the images and package them up.

In the short term, you can fix this by pulling the latest version of the master branch of this repo. I just pushed a single commit that first checks for the existence of the star test images before trying to package them up, so if they aren't there, it will quietly ignore the current lines 1409-1410.

Alternatively, you can use great3sims.run() for the early steps (generating parameters, catalogs, config files), then generate the sims using the galsim executable on all the .yaml files, then do the packaging steps with great3sims.run(). This way you'll get the star test images if you want them.

— Reply to this email directly or view it on GitHub https://github.com/barnabytprowe/great3-public/issues/26#issuecomment-94988532 .