Rerunning the simulations

[kuntzer]

Hi there,

I'm trying to reproduce parts of the Great3 simulations. I am contemplating the tests/test_run.py file. The settings for this file on the github repository is to run all branches. I decided to restrict myself to CSC. However, the scripts seems to ignore this because it still needs the COSMOS files. So that's my first question: how to make sure we use only what we need?

I decided to download everything while setting up test_run.py to CSC only. Now the script goes a little bit further. However, it stumbles when executing generateSubfieldParameters() in great3sims/galaxies.py. Actually when in the block of code between lines 312 to 317. I get the following error:

bash-4.1$ python test_run.py
Generating metaparameters for control / space / constant
Regenerating metaparameters for all subfields, not just the chosen subset
Generating catalogs for control / space / constant
  subfield 0Traceback (most recent call last):
  File "test_run.py", line 113, in <module>
    subfield_min=subfield_min, subfield_max=subfield_max, **kwargs)
  File "../great3sims/__init__.py", line 188, in run
    builder.writeSubfieldCatalog(subfield_index)
  File "../great3sims/builder.py", line 382, in writeSubfieldCatalog
    effective_seeing)
  File "../great3sims/galaxies.py", line 318, in generateCatalog
    self.peak_image_pixel_count[self.peak_image_pixel_count == 0.] = 1.e-4
  File "/usr/lib64/python2.6/site-packages/pyfits/core.py", line 5507, in __setitem__
    self.array.field(indx)[self.row] = value
ValueError: field named peak_image_pixel_count not found

which comes from

mask_catalog = pyfits.getdata(os.path.join(self.gal_dir, self.rgc_mask_file)
[...]
self.peak_image_pixel_count = mask_catalog['peak_image_pixel_count']

where the fits array to load is called real_galaxy_mask_info.fits which does exist in the real galaxies data, but doesn't seem to contain this peak_image_pixel_count field. Again a real galaxy issue. How can I get rid of this error ?

Cheers Thibault

[rmandelb]

Hi Thibault -

However, the scripts seems to ignore this because it still needs the COSMOS files. So that's my first question: how to make sure we use only what we need?

Since we wanted to use the same galaxy selection criteria for both control and real_galaxy, the files for COSMOS selection are needed for all branches. So this is a good feature that you're seeing, not a bug! You do not need the huge COSMOS image files, however, just the catalogs for selection.

As for the second error, that's strange. I will have to check if I put an old version of the file into the code package - let me check and get back to you...

[kuntzer]

Hi Rachel. Thanks for the quick answer. If there's anything I can do to help let me know,

[rmandelb]

If I use the files in that code package and run the script using them, it runs just fine. And indeed I notice the line with the error is not the one where you try to extract a field from the catalog, but rather the line just after, where you try to modify the extracted field using an assignment operator. So that field is in the catalog (because if it were not, the code would have died on the line before), but for some reason after you extract and save the field, you're not able to modify it. Can you please tell me what version of pyfits and numpy you are using?

[rmandelb]

Sorry, I meant "in that package of galaxy selection files", not "in that code package".

[rmandelb]

...and let me request one more thing. Immediately before the line where it is failing (line 318 in galaxies.py) can you please insert a line that says print type(mask_catalog['peak_image_pixel_count']) and let me know what happens when you run with this line included?

[kuntzer]

Hi Rachel,

I'm running on Python 2.6.6 with numpy 1.8.1 and pyfits 2.3.1. The line print type(mask_catalog['peak_image_pixel_count']) returns <class 'pyfits.core.FITS_record'>.

As I'm running this on a server with sometimes weird behaviours. Thus I rerun it on my laptop and indeed it's working just fine. I don't get where the difference is. I'll talk to the guy in charge of the cluster. I'll get back to you with the feedback from the engineer.

[kuntzer]

I'd bet its pyfits actually. I just looked at the version I have on my laptop: 3.2.2 ! Let me update the pyfits package I use on the cluster.

[rmandelb]

Yes, it's definitely the old pyfits. Quite a long time ago, the behavior of pyfits FITSrec was changed so that the assignment we're doing from the catalog in 317 would give a numpy array. This is why we're able to do the assignment in line 318. But in older pyfits versions the assignment in 317 gives a pyfits.FITSrec which can't be used like a numpy array, so that's why line 318 fails. That pyfits version really is quite old and i cannot guarantee that you won't have more issues like this even if you come up with a way to sidestep this particular one, so updating pyfits is probably the best way forward. Sorry... please keep me posted about whether it starts working after you update.

[kuntzer]

Okay, it works perfectly. Cheers for the help!