Corr2 assertion error when running presubmission script

[msimet]

I'm opening this issue after some email discussion with @HironaoMiyatake and Yuki Okura. When running the presubmission script, they're getting a corr2 assertion error (reproduced below), presumably due to a problem with the presubmission script or the data. I don't know the inner workings of corr2 well enough to know how to diagnose this problem, so I thought we would ping @rmjarvis for some help as to possible causes.

The error:

Running presubmission.py on file /data1/GREAT3/control/ground/variable/results/GREAT3.dat 
  Using variable-shear analysis
  Column containing galaxy ids is 0 (default)
  Column containing g1 is 1 (default)
  Column containing g2 is 2 (default)
  Column containing weight is 3
Reading file 1/1 ...
All files read.
Checking for correct galaxy IDs... (this may take a while)
Computing summary statistics for field 0 ...
Error - Assert _w != 0. failed
on line 80 in file Cell.cpp
Traceback (most recent call last):
  File "presubmission.py", line 710, in <module>
    generate_submission_files(args)
  File "presubmission.py", line 580, in generate_submission_files
    corr2=args.corr2)
  File "presubmission.py", line 338, in print_variable_summary
    nbins=nbins, xy_units='degrees', sep_units='degrees', corr2_executable=corr2)
  File "presubmission.py", line 281, in run_corr2
    results = readfile(m2file)
  File "presubmission.py", line 193, in readfile
    with open(filename) as f:
IOError: [Errno 2] No such file or directory: '/tmp/tmps_cVkw_temp.m2'

So there is an assertion error (Assert _w != 0. failed) from corr2, and no output file is written. The file, then, does not exist, and the presubmission script fails because it expects to read that output file back in.

(As a general question: would users prefer a more useful error in presubmission.py? I don't mind writing one, but I'm also not sure we want to make a change to the script this close to the challenge end date for something this minor.)

[rmandelb]

That assertion means that the total weight over all galaxies in the file is zero. So the input file has zero weights, and perhaps they mean to be running in no-weight mode? Or maybe they mean the weights to be something different?

[rmandelb]

As for the question of a more useful error, I think that there are a few classes of things that can go wrong when the input is weird, so the best we could do would be something like “Corr2 failed. Please check inputs for nonzero weights and other issues like that.” Or did you have something else in mind?

[msimet]

Yes, I just meant "File not found--this usually means there was a problem with corr2" or something, which is probably more helpful than an IOError in this case.

[msimet]

Just to check (since this is a problem with weights): when you indicated which column holds the weights information with the -w switch, did you use zero-based numbering for the columns or 1-based numbering? The script automatically assumes zero-based, which is the Python default.

[rmjarvis]

Error - Assert _w != 0. failed on line 80 in file Cell.cpp

This was actually a bug in the corr2 code. It was supposed to skip objects with w = 0, but it wasn't in v2.6. This has been fixed on the trunk version, but I haven't released a new version with that fix yet. (Probably soon - there is another bug that Vinu pointed out that I was to fix first though.) Anyway, they can use the trunk version for now until v2.7 is released.

svn checkout http://mjarvis.googlecode.com/svn/corr/trunk corr_trunk

[rmandelb]

Oh... for some reason I thought it already did that!

[rmandelb]

(So, sorry for my wrong explanation of what was going wrong!)

[yuki-okura]

I succeeded in processing by corr2 v2.7. Thank you very much!

[rmandelb]

Closing #14.