missing default 'else' in step1.py get_best and get_longest

[GoogleCodeExporter]

What steps will reproduce the problem?
1. replace v2.mean_comb with a file that has all 'valid' monthly average data 
replaced with the 
station's mean for that month, then execute "tool/run.py -s 1"

What is the expected output? What do you see instead?
Error message says:
  File "/Users/anthony/Documents/Climate/ccc-gistemp_testing/code/step1.py", line 365, in 
get_best
    return longest_rec, longest_id
UnboundLocalError: local variable 'longest_rec' referenced before assignment

What version of the product are you using? On what operating system?
Version 0.3.0 on MacOS 10.6.2

Please provide any additional information below.
This subroutine has 'if this, elseif that', but no final 'else' statement. I am 
testing something, and 
have hit on a case that needs the 'else'.

Fixed by 'if longest ==0' then set all the values that might be returned. This 
sets them for the 
first case, then they may or may not get overridden by later cases, depending 
on if the new ones 
are longer.

get_longest has a similar error. 

Also, in 'sigma' if the code is fed all equal values, and sigma = 0, the 
machine error can produce 
a negative number, resulting in sqrt(negative), which causes an error.

Original issue reported on code.google.com by awh142...@gmail.com on 12 Feb 2010 at 4:48

[GoogleCodeExporter]

I am investigating this, but so far have failed to reproduce the problem.

Can you attach an example of a v2.mean_comb that fails?  From what you've said, 
it should be possible to show 
this with a v2.mean_comb file that has records for just one station (as long as 
that station has duplicates).  So 
you can cut down the v2.mean_comb file to a section where the first 11 
characters (the station identifier) are all 
the same.

Original comment by d...@ravenbrook.com on 12 Feb 2010 at 8:14

[GoogleCodeExporter]

There is a data dependent bug, similar to the one you describe.

(assuming "rank > best" fires; in other words all records have "UNKNOWN" source)
Since longest is initialised to 0, to assign longest_rec it is sufficient for 
there to be one record where length > 
0.  length is the number of years for which a record has an annual anomaly.  An 
annual anomaly is computed 
when at least 3 of the 4 seasons have an anomaly, and a season has an anomaly 
when at least 2 of its 3 
months have an anomaly (any month with valid data will have an anomaly for this 
computation).

Basically there can be a problem when all years in all the records (for a given 
station) have insufficient data.  
Specifically there can be a problem when:
all records are of UNKNOWN source, and;
all years fail to yield an anomaly.

For example, this can happen if a station never has data for July to December.

Following is my (entire, because I trimmed it) prepared v2.mean_comb which 
causes a failure:

$ cat work/v2.mean_comb 
1016035500001967  100  112  129  143  181  193-9999-9999-9999-9999-9999-9999
1016035500001968  100  123  130  160  181  207-9999-9999-9999-9999-9999-9999
1016035500001969  100  114  134  149  182  194-9999-9999-9999-9999-9999-9999
1016035500001970  100  110  122  138  165  211-9999-9999-9999-9999-9999-9999
1016035500001971  100  104  105  155  183  202-9999-9999-9999-9999-9999-9999
1016035500001972  100  128  141-9999  160  204-9999-9999-9999-9999-9999-9999
1016035500001973  100  101  106  136  183  216-9999-9999-9999-9999-9999-9999
1016035500001974  100  118  137  134  181  216-9999-9999-9999-9999-9999-9999
1016035500001975  100  109  119  136  167  192-9999-9999-9999-9999-9999-9999
1016035500001976  100  108  118  145-9999-9999-9999-9999-9999-9999-9999-9999
1016035500001977  100  129  133  148  177  203-9999-9999-9999-9999-9999-9999
1016035500001978  100  134-9999  138  169  206-9999-9999-9999-9999-9999-9999
1016035500001979  100  128  127  130-9999  213-9999-9999-9999-9999-9999-9999
1016035500001980  100  116  124  130  163  204-9999-9999-9999-9999-9999-9999
1016035500001981  100  109  139  158  177  209-9999-9999-9999-9999-9999-9999
1016035500001982  100  123  133  148  181  229-9999-9999-9999-9999-9999-9999
1016035500001983  100  109  133  163  163  216-9999-9999-9999-9999-9999-9999
1016035500001984  100  104  125  150-9999  205-9999-9999-9999-9999-9999-9999
1016035500001985  100  143-9999  163  179  219-9999-9999-9999-9999-9999-9999
1016035500001986-9999  122  134  154  194  209-9999-9999-9999-9999-9999-9999
1016035500001987  100  128  130  162  173  214-9999-9999-9999-9999-9999-9999
1016035500001988  100  119  141  159  189  218-9999-9999-9999-9999-9999-9999
1016035500001989  100  125  150  159  180  214-9999-9999-9999-9999-9999-9999
1016035500001990-9999-9999  144  154  187  219-9999-9999-9999-9999-9999-9999
1016035500001991-9999  112  151  140  159  211-9999-9999-9999-9999-9999-9999
1016035500011880  100  111  124  152  171  207-9999-9999-9999-9999-9999-9999
1016035500011931-9999   97-9999-9999  185  239-9999-9999-9999-9999-9999-9999
1016035500011932  100   98-9999  142  184-9999-9999-9999-9999-9999-9999-9999
1016035500011933-9999-9999-9999-9999  187-9999-9999-9999-9999-9999-9999-9999
1016035500011934  100   97  119  155  176-9999-9999-9999-9999-9999-9999-9999
1016035500011935  100  109  122  150-9999  215-9999-9999-9999-9999-9999-9999
1016035500011936  100  138  136  155-9999  195-9999-9999-9999-9999-9999-9999
1016035500011937  100  125  145  147-9999  220-9999-9999-9999-9999-9999-9999
1016035500011938  100   90  110  132  168  216-9999-9999-9999-9999-9999-9999
1016035500011939  100  118  116  147  155  201-9999-9999-9999-9999-9999-9999
1016035500011940  100  130  143  147  174  200-9999-9999-9999-9999-9999-9999
1016035500011941  100-9999  140  157-9999-9999-9999-9999-9999-9999-9999-9999
1016035500011966-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
1016035500011967  100-9999  129-9999-9999  193-9999-9999-9999-9999-9999-9999
1016035500011968  100  123  130  160  181  207-9999-9999-9999-9999-9999-9999
1016035500011969  100  114  134  149  182  194-9999-9999-9999-9999-9999-9999
1016035500011970  100  110  122  138  165  211-9999-9999-9999-9999-9999-9999
1016035500011971  100  104  105  155  183  202-9999-9999-9999-9999-9999-9999
1016035500011972  100  128  141-9999  160  204-9999-9999-9999-9999-9999-9999
1016035500011973  100  101  106  136  183  216-9999-9999-9999-9999-9999-9999
1016035500011974  100  118  137  137  181-9999-9999-9999-9999-9999-9999-9999
1016035500021987  100  128  130  162  173  214-9999-9999-9999-9999-9999-9999
1016035500021988  100  119  141  159  189  218-9999-9999-9999-9999-9999-9999
1016035500021989  100  125  150  159  180  214-9999-9999-9999-9999-9999-9999
1016035500021990-9999-9999  144  154  187  219-9999-9999-9999-9999-9999-9999
1016035500021991-9999  112  151  140  159  211-9999-9999-9999-9999-9999-9999
1016035500021992  100  117-9999  150  178  209-9999-9999-9999-9999-9999-9999
1016035500021993-9999  111  129  155  191  219-9999-9999-9999-9999-9999-9999
1016035500021994-9999  132  143  145  195  216-9999-9999-9999-9999-9999-9999
1016035500021995  100  141  135-9999  196  216-9999-9999-9999-9999-9999-9999
1016035500021996-9999-9999  144  168  184  215-9999-9999-9999-9999-9999-9999
1016035500021997  100  133  136  158  198  234-9999-9999-9999-9999-9999-9999
1016035500021998  100  132  146  170  187  237-9999-9999-9999-9999-9999-9999
1016035500021999  100  112  148  163  208  234-9999-9999-9999-9999-9999-9999
1016035500022000  100  130  145-9999-9999  225-9999-9999-9999-9999-9999-9999
1016035500022001  100  120  179-9999  190  237-9999-9999-9999-9999-9999-9999
1016035500022002  100  152  153  164  194  232-9999-9999-9999-9999-9999-9999
1016035500022003  100  115-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
1016035500022004-9999-9999-9999-9999-9999  217-9999-9999-9999-9999-9999-9999
1016035500022005  100-9999  133  164-9999  237-9999-9999-9999-9999-9999-9999
1016035500022006  100  120  149  182  210  231-9999-9999-9999-9999-9999-9999
1016035500022007  100  145  139  172  202  230-9999-9999-9999-9999-9999-9999
1016035500022008  100  133  140  175  198  225-9999-9999-9999-9999-9999-9999
1016035500022009  100-9999-9999  156  206  230-9999-9999-9999-9999-9999-9999
$ python tool/run.py -s1
====> STEP 1  ====
Traceback (most recent call last):
  File "tool/run.py", line 217, in <module>
    sys.exit(main(options, args))
  File "tool/run.py", line 201, in main
    for _ in data:
  File "/Users/drj/ccc-gistemp/tool/giss_io.py", line 630, in step1_output
    for thing in data:
  File "/Users/drj/ccc-gistemp/code/step1.py", line 741, in step1
    for record in alter_discont(without_strange):
  File "/Users/drj/ccc-gistemp/code/step1.py", line 712, in alter_discont
    for record in data:
  File "/Users/drj/ccc-gistemp/code/step1.py", line 664, in drop_strange
    for record in data:
  File "/Users/drj/ccc-gistemp/code/step1.py", line 623, in do_combine
    for record in record_set:
  File "/Users/drj/ccc-gistemp/code/step1.py", line 425, in adjust_helena
    for record in stream:
  File "/Users/drj/ccc-gistemp/code/step1.py", line 632, in do_combine
    record, rec_id = select_func(record_dict)
  File "/Users/drj/ccc-gistemp/code/step1.py", line 336, in get_best
    return longest_rec, longest_id
UnboundLocalError: local variable 'longest_rec' referenced before assignment

With real data this is extremely unlikely to happen (and we know it never does 
with the input files we use of 
course).

Of course I will fix it in this case (to throw away all records for that 
station, I guess).

Please can you show me your v2.mean_comb file?

Original comment by d...@ravenbrook.com on 12 Feb 2010 at 8:38

• Changed state: Accepted

[GoogleCodeExporter]

(updating my proposed "fix") When there are no years for which there is a valid 
anomaly, I think the right course 
of action is to pass the records through uncombined, not to drop them.

Original comment by d...@ravenbrook.com on 12 Feb 2010 at 8:46

[GoogleCodeExporter]

The sigma thing I have moved to Issue 43.

Original comment by d...@ravenbrook.com on 12 Feb 2010 at 8:46