Unhandled error in ``create_ini()``

[jameshiebert]

This request: http://tools.pacificclimate.org/dataportal/data/pcds/agg/?from-date=1870%2F01%2F01&to-date=2017%2F07%2F31&input-polygon=&input-var=&network-name=FRBC&input-freq=&data-format=ascii&cliptodate=cliptodate&download-timeseries=Timeseries

leads to this unhandled exception:

[2017-07-26 11:18:12 +0000] [27692] [ERROR] Error handling request
Traceback (most recent call last):
  File "/home/websites/tools.pacificclimate.org/data_portal-2.4.2/env/local/lib/python2.7/site-packages/gunicorn/workers/async.py", line 112, in handle_request
    for item in respiter:
  File "/home/websites/tools.pacificclimate.org/data_portal-2.4.2/env/local/lib/python2.7/site-packages/pdp/error.py", line 62, in __call__
    start_response(status, response_headers, sys.exc_info())
  File "/home/websites/tools.pacificclimate.org/data_portal-2.4.2/env/local/lib/python2.7/site-packages/gunicorn/http/wsgi.py", line 241, in start_response
    reraise(exc_info[0], exc_info[1], exc_info[2])
  File "/home/websites/tools.pacificclimate.org/data_portal-2.4.2/env/local/lib/python2.7/site-packages/pdp/error.py", line 56, in __call__
    for block in response_iter:
  File "/home/websites/tools.pacificclimate.org/data_portal-2.4.2/env/local/lib/python2.7/site-packages/ga_wsgi_client/__init__.py", line 41, in __call__
    for chunk in app_iter:
  File "/home/websites/tools.pacificclimate.org/data_portal-2.4.2/env/local/lib/python2.7/site-packages/pdp_util/agg.py", line 79, in ziperator
    for name, responder in responders:
  File "/home/websites/tools.pacificclimate.org/data_portal-2.4.2/env/local/lib/python2.7/site-packages/pdp_util/agg.py", line 166, in get_pcds_responders
    yield (name, handler(newenv, null_start_response))
  File "/home/websites/tools.pacificclimate.org/data_portal-2.4.2/env/local/lib/python2.7/site-packages/pydap/handlers/pcic/__init__.py", line 70, in __call__
    s = self.create_ini(sesh, net_name, native_id)
  File "/home/websites/tools.pacificclimate.org/data_portal-2.4.2/env/local/lib/python2.7/site-packages/pydap/handlers/pcic/__init__.py", line 144, in create_ini
    ''' % locals()
TypeError: a float is required

Is it possible that stations without location (either x/y or elevation) metadata are being selected and not guarded against?

[jameshiebert]

So we check lat/lon and substitute nan if they come back NULL, but we don't do the same for elevation. Highly likely that this is the problem.

[jameshiebert]

@corviday Would you be able to take on verification that this is the problem and writing a fix for the issue? And keep @faronium in the loop? If my assessment is correct, it should be a one-liner.

[corviday]

FRBC network stations have NULL elevations, supporting @jameshiebert 's assessment of the issue.

crmp=> SELECT meta_network.network_name, meta_station.station_id, meta_history.station_name, meta_history.lon, meta_history.lat, meta_history.elev
FROM crmp.meta_history, crmp.meta_network, crmp.meta_station
WHERE meta_history.station_id = meta_station.station_id AND meta_network.network_id = meta_station.network_id AND meta_network.network_id = 16;

 network_name | station_id |            station_name            |    lon    |   lat    | elev 
--------------+------------+------------------------------------+-----------+----------+------
 FRBC         |       3124 | BERRYMAN POINT CLIMATE             | -125.6706 | 49.14833 |     
 FRBC         |       3122 | MARION CREEK NEAR UCLUELET         | -125.3175 | 49.20583 |     
 FRBC         |       3112 | ELAHO RIVER NEAR THE MOUTH         | -123.4361 | 50.11944 |     
 FRBC         |       3126 | CLANNINICK CREEK AT HEADWATERS     | -127.3956 | 50.10222 |     
 FRBC         |       3131 | PUGH CREEK NEAR NAHWITTI LAKE      | -127.8831 |   50.735 |     
 FRBC         |       3118 | COTTONWOOD CREEK HEADWATERS        |   -124.25 | 48.93389 |     
 FRBC         |       3117 | HARRIS CREEK NEAR LAKE COWICHAN    | -124.2261 | 48.71833 |     
 FRBC         |       3125 | CYPRE RIVER NEAR GOLD RIVER        | -125.8683 | 49.36972 |     
 FRBC         |       3123 | SAND RIVER CLIMATE                 | -125.4932 | 49.14794 |     
 FRBC         |       3128 | BENSON RIVER ABOVE BENSON LAKE     |   -127.25 |    50.38 |     
 FRBC         |       3115 | MCALLISTER CREEK AT THOMPSON SOUND | -126.0096 | 50.79689 |     
 FRBC         |       3127 | MCKELVIE CREEK ABOVE INTAKE        | -126.6333 | 49.93333 |     
 FRBC         |       3129 | ZEBALLOS RIVER HEADWATERS          |   -126.83 |    50.14 |     
 FRBC         |       3132 | SIMPSON CREEK NEAR KOPRINO HARBOUR | -127.8431 | 50.51194 |     
 FRBC         |       3120 | PORT RENFREW CLIMATE               | -124.4205 | 48.55502 |     
 FRBC         |       3119 | WALBRAN CREEK IN CANYON            |    -124.6 |    48.64 |     
 FRBC         |       3113 | ICY CREEK NEAR THE MOUTH           | -125.6715 | 51.22761 |     
 FRBC         |       3114 | KIPPAN CREEK NEAR THE MOUTH        | -126.3584 | 51.06831 |     
 FRBC         |       3116 | RENFREW CREEK NEAR PORT RENFREW    | -124.2919 | 48.63694 |     
 FRBC         |       3130 | KLASKISH RIVER                     | -127.6833 | 50.30833 |     
 FRBC         |       3133 | CAMP CREEK NEAR THE MOUTH          | -120.7364 | 52.78389 |     
 FRBC         |       3121 | TOFINO CREEK NEAR THE MOUTH        | -125.5806 | 49.24944 |     
(22 rows)

There are about a hundred other stations on various networks with defined longitude and latitude but NULL elevation.

[jameshiebert]

Turns out that this fix actually fails on PostgreSQL databases:

env/local/lib/python2.7/site-packages/pydap/handlers/pcic/__init__.py:109: in create_ini
    lat, lon = (sesh.scalar(geom.y), sesh.scalar(geom.x)) if geom else (float('nan'), float('nan'))
env/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py:538: in __bool__
    raise TypeError("Boolean value of this clause is not defined")
E   TypeError: Boolean value of this clause is not defined

We should convert the tests in the package to use testing.postgresql and get the fix right.