search

DFO-Ocean-Navigator / Ocean-Data-Map-Project

The Ocean Navigator is an online tool that is used to help visualise scientific research data. a users guide is available at https://dfo-ocean-navigator.github.io/Ocean-Navigator-Manual/ and the tool is live at
http://navigator.oceansdata.ca
GNU General Public License v3.0
49 stars 20 forks source link

API Stability: Virtual Mooring Plot #673

Closed jamesprayner closed 4 years ago

jamesprayner commented 4 years ago

Describe the bug Requesting a Virtual Mooring plot doesn't return an image, instead it returns: IndexError: index 0 is out of bounds for axis 0 with size 0

full stack:

Traceback (most recent call last):
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1985, in wsgi_app
    response = self.handle_exception(e)
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1540, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/buildadm/Ocean-Data-Map-Project/routes/api_v1_0.py", line 549, in plot_v1_0
    img, mime, filename = plotter.run()
  File "/home/buildadm/Ocean-Data-Map-Project/plotting/plotter.py", line 58, in run
    _ = self.prepare_plot()
  File "/home/buildadm/Ocean-Data-Map-Project/plotting/plotter.py", line 51, in prepare_plot
    self.load_data()
  File "/home/buildadm/Ocean-Data-Map-Project/plotting/timeseries.py", line 112, in load_data
    starttime_idx = dataset.timestamp_to_time_index(self.starttime)
  File "/home/buildadm/Ocean-Data-Map-Project/data/netcdf_data.py", line 115, in timestamp_to_time_index
    return result if result.shape[0] > 1 else result[0]
IndexError: index 0 is out of bounds for axis 0 with size 0

To Reproduce Steps to reproduce the behavior: request the following query -

/api/v1.0/plot/?query=%7B%22colormap%22%3A%22default%22%2C%22dataset%22%3A%22giops_day%22%2C%22depth%22%3A0%2C%22endtime%22%3A2214388800%2C%22names%22%3A%5B%2243.9612%2C%20-39.7609%22%5D%2C%22plotTitle%22%3A%22%22%2C%22quantum%22%3A%22day%22%2C%22scale%22%3A%22-5%2C30%2Cauto%22%2C%22showmap%22%3Atrue%2C%22starttime%22%3A2214388776%2C%22station%22%3A%5B%5B43.96119063892027%2C-39.76089477539063%2Cnull%5D%5D%2C%22type%22%3A%22timeseries%22%2C%22variable%22%3A%22votemper%22%7D&format=json

Expected behavior Time series plot should be generated and sent to the user.

nsoontie commented 4 years ago

I think I remember a proposed 'fix' to this issue related to rounding the timestamp but this didn't address the underlying issue: why are the timestamps not matching?

htmlboss commented 4 years ago

This crash happens further down the line if the above crash doesn't happen.

[2020-03-05 12:47:03 +0000] [9410] [ERROR] Error handling request /api/v1.0/plot/?query=%7B%22colormap%22%3A%22default%22%2C%22dataset%22%3A%22giops_day%22%2C%22depth%22%3A0%2C%22endtime%22%3A2214518400%2C%22names%22%3A%5B%2248.2247%2C%20-48.9902%22%5D%2C%22plotTitle%22%3A%22%22%2C%22quantum%22%3A%22day%22%2C%22scale%22%3A%22-5%2C30%2Cauto%22%2C%22showmap%22%3Atrue%2C%22starttime%22%3A2214172800%2C%22station%22%3A%5B%5B48.22467264956521%2C-48.9902338385582%2Cnull%5D%5D%2C%22type%22%3A%22timeseries%22%2C%22variable%22%3A%22votemper%22%7D&format=json
Traceback (most recent call last):
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1985, in wsgi_app
    response = self.handle_exception(e)
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1540, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/buildadm/Ocean-Data-Map-Project/routes/api_v1_0.py", line 549, in plot_v1_0
    img, mime, filename = plotter.run()
  File "/home/buildadm/Ocean-Data-Map-Project/plotting/plotter.py", line 65, in run
    return self.plot()
  File "/home/buildadm/Ocean-Data-Map-Project/plotting/timeseries.py", line 356, in plot
    datenum, self.data[:, 0, :].transpose(), '-', figure=fig)
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/matplotlib/pyplot.py", line 2799, in plot_date
    data} if data is not None else {}), **kwargs)
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/matplotlib/__init__.py", line 1589, in inner
    return func(ax, *map(sanitize_sequence, args), **kwargs)
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/matplotlib/axes/_axes.py", line 1743, in plot_date
    ret = self.plot(x, y, fmt, **kwargs)
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/matplotlib/axes/_axes.py", line 1666, in plot
    lines = [*self._get_lines(*args, data=data, **kwargs)]
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/matplotlib/axes/_base.py", line 225, in __call__
    yield from self._plot_args(this, kwargs)
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/matplotlib/axes/_base.py", line 391, in _plot_args
    x, y = self._xy_from_xy(x, y)
  File "/home/buildadm/miniconda/3/ubuntu/16.04/amd64/envs/navigator/lib/python3.6/site-packages/matplotlib/axes/_base.py", line 270, in _xy_from_xy
    "have shapes {} and {}".format(x.shape, y.shape))
ValueError: x and y must have same first dimension, but have shapes (0,) and (2, 1)
htmlboss commented 4 years ago

Scratch that...if I hit the endpoint I linked just above on the python debugger, I get the image just fine:

image

jamesprayner commented 4 years ago

IndexError: index 0 is out of bounds for axis 0 with size 0 Is caused by selecting a NaN value as start or end date, which is chosen by default when you choose the virtual mooring tab (frontend issue).

ValueError: x and y must have same first dimension, but have shapes (0,) and (2, 1) Is more interesting because we can only replicate it in testing environments. The shape of datenum does not match self.data. Still not sure why it works on production but not locally.

htmlboss commented 4 years ago

I've got a tentative fix on my system. I'll work on cleaning it up and making a PR. Several things had to be changed.

htmlboss commented 4 years ago

To keep things running smoothly I've opted to wait until #641 is merged before trying to merge this fix.

htmlboss commented 4 years ago

Still need to play with the bottom detection...