AttributeError: 'Quantile' object has no attribute 'loss_name'

[dmitra79]

Description

I get this error trying to run a simple evaluation as described in: https://ts.gluon.ai/tutorials/forecasting/extended_tutorial.html

Error message or code output

(Paste the complete error message, including stack trace, or the undesired output that the above snippet produces.)

RemoteTraceback                           Traceback (most recent call last)
RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/datascience/conda/derv0/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/home/datascience/conda/derv0/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/home/datascience/conda/derv0/lib/python3.7/site-packages/gluonts/evaluation/_base.py", line 55, in worker_function
    return evaluator.get_metrics_per_ts(ts, forecast)
  File "/home/datascience/conda/derv0/lib/python3.7/site-packages/gluonts/evaluation/_base.py", line 398, in get_metrics_per_ts
    pred_target, forecast_quantile, quantile.value
AttributeError: 'Quantile' object has no attribute 'loss_name'
"""

The above exception was the direct cause of the following exception:

AttributeError                            Traceback (most recent call last)
<ipython-input-33-428f18e1ed3b> in <module>
      1 evaluator = Evaluator(quantiles=[0.1, 0.5, 0.9])
----> 2 agg_metrics, item_metrics = evaluator(iter(tss), iter(forecasts), num_series=len(tds.test))
      3 print(json.dumps(agg_metrics, indent=4))
      4 item_metrics.head()

~/conda/derv0/lib/python3.7/site-packages/gluonts/evaluation/_base.py in __call__(self, ts_iterator, fcst_iterator, num_series)
    220                     func=partial(worker_function, self),
    221                     iterable=iter(it),
--> 222                     chunksize=self.chunk_size,
    223                 )
    224                 mp_pool.close()

~/conda/derv0/lib/python3.7/multiprocessing/pool.py in map(self, func, iterable, chunksize)
    266         in a list that is returned.
    267         '''
--> 268         return self._map_async(func, iterable, mapstar, chunksize).get()
    269 
    270     def starmap(self, func, iterable, chunksize=None):

~/conda/derv0/lib/python3.7/multiprocessing/pool.py in get(self, timeout)
    655             return self._value
    656         else:
--> 657             raise self._value
    658 
    659     def _set(self, i, obj):

~/conda/derv0/lib/python3.7/multiprocessing/pool.py in worker()
    119         job, i, func, args, kwds = task
    120         try:
--> 121             result = (True, func(*args, **kwds))
    122         except Exception as e:
    123             if wrap_exception and func is not _helper_reraises_exception:

~/conda/derv0/lib/python3.7/multiprocessing/pool.py in mapstar()
     42 
     43 def mapstar(args):
---> 44     return list(map(*args))
     45 
     46 def starmapstar(args):

~/conda/derv0/lib/python3.7/site-packages/gluonts/evaluation/_base.py in worker_function()
     53 def worker_function(evaluator: "Evaluator", inp: tuple):
     54     ts, forecast = inp
---> 55     return evaluator.get_metrics_per_ts(ts, forecast)
     56 
     57 

~/conda/derv0/lib/python3.7/site-packages/gluonts/evaluation/_base.py in get_metrics_per_ts()
    396 
    397             metrics[quantile.loss_name] = quantile_loss(
--> 398                 pred_target, forecast_quantile, quantile.value
    399             )
    400             metrics[quantile.coverage_name] = coverage(

AttributeError: 'Quantile' object has no attribute 'loss_name'

Environment

• Operating system: Linux
• Python version: 3.7.10
• GluonTS version: 0.9.4
• MXNet version: 1.7.0.post1

[dmitra79]

I am aware of https://github.com/awslabs/gluon-ts/issues/1570 but it doesn't provide a solution

[jaheba]

Thanks for raising the issue!

What's weird is that we define loss_name as a property here:

https://github.com/awslabs/gluon-ts/blob/42311c3a78260ad05d27f2deb0edfa1efe6c5b78/src/gluonts/model/forecast.py#L218-L224

Maybe it has to do with multiprocessing. Can you try rerun the code with Evaluator(num_workers=None, ...)?

[dmitra79]

@jaheba Thanks! It ran fine with num_workers=None. That is weird, also somewhat annoying if one can't use multiple CPUs for evaluation.

[jaheba]

Thanks for confirming that it works that way.

The issue seems to be related to using notebooks and how they are setup. On my local machine the notebook runs fine, so we would need to investigate what causes the failure.

[jaheba]

I've tried some more, but still can't replicate the issue.

My guess is that something went missing during pickling/un-pickling. Quantile is defined as a NamedTuple and the attributes appear to work -- value is accessed before loss_name without issues.

Maybe there is some optimisation for named-tuples at work, or there can be an entirely different reason.

We could try and use a dataclass instead to check whether that changes anything.

@dmitra79 Can you tell a bit more about how you run the code? Is it a notebook and is there anything "special" about yours setup?

[dmitra79]

It is a notebook, running in a cloud session. There may be some package incompatibilities in the environment (I am limited in what packages/versions I have access to in that session), and some proprietory packages. I am using conda for package management. I realize this makes it impossible to replicate exactly.

[jaheba]

I wonder if #2047 would fix this.