BUG: vocab parameter does not exist for Regression Models

abhijitramesh commented 2 years ago

Describe the bug

Learner for regression models does not have the vocab parameter as it is generally a list of labels. While tracking experiments with neptune-ai for fastai/tsai the experiment fails as the vocab parameter is expected by neptune callback.

This should be an optional parameter since fastai and tsai by default exclude _vocab for regression models.

Reproduction

from tsai.all import *
import neptune.new as neptune
from neptune.new.integrations.fastai import NeptuneCallback

def main():
    neptune_run = neptune.init()

    dsid = 'AppliancesEnergy' 
    X, y, splits = get_regression_data(dsid, split_data=False)
    X.shape, y.shape, y[:10]

    tfms  = [None, [TSRegression()]]
    batch_tfms = TSStandardize(by_sample=True, by_var=True)
    dls = get_ts_dls(X, y, splits=splits, tfms=tfms, batch_tfms=batch_tfms, bs=128)

    learn = ts_learner(dls, InceptionTime, metrics=[mae, rmse], cbs=[NeptuneCallback(neptune_run, 'experiment')])
    learn.fit_one_cycle(50, 1e-2)

if __name__ == '__main__':
    main()

This code is taken from https://timeseriesai.github.io/tsai/tutorials.html#Time-series-regression with modification to include neptune tracking.

Run the above snippet.

Expected behavior

The training starts and neptune logs everything normally.

Traceback

For the above snippet

Partial Traceback:

AttributeError: 'InceptionTime' object has no attribute '_vocab'

Complete Traceback:

Info (NVML): NVML Shared Library Not Found. GPU usage metrics may not be reported. For more information, see https://docs.neptune.ai/you-should-know/what-can-you-log-and-display#hardware-consumption
119it [00:03, 39.47it/s]
66it [00:01, 43.77it/s]
epoch     train_loss  valid_loss  mae       _rmse     time
Traceback (most recent call last):
  File "/Users/abhijitramesh/Development/DeepLearning/example.py", line 20, in <module>
    main()
  File "/Users/abhijitramesh/Development/DeepLearning/example.py", line 16, in main
    learn.fit_one_cycle(50, 1e-2)
  File "/Users/abhijitramesh/Development/DeepLearning/Theta-Tech-ai/Semler-Consulting-local/Semler-Consulting-local/Semler-Consulting/integration/qf_plus_trained/env/lib/python3.9/site-packages/fastai/callback/schedule.py", line 116, in fit_one_cycle
    self.fit(n_epoch, cbs=ParamScheduler(scheds)+L(cbs), reset_opt=reset_opt, wd=wd)
  File "/Users/abhijitramesh/Development/DeepLearning/Theta-Tech-ai/Semler-Consulting-local/Semler-Consulting-local/Semler-Consulting/integration/qf_plus_trained/env/lib/python3.9/site-packages/fastai/learner.py", line 221, in fit
    self._with_events(self._do_fit, 'fit', CancelFitException, self._end_cleanup)
  File "/Users/abhijitramesh/Development/DeepLearning/Theta-Tech-ai/Semler-Consulting-local/Semler-Consulting-local/Semler-Consulting/integration/qf_plus_trained/env/lib/python3.9/site-packages/fastai/learner.py", line 163, in _with_events
    try: self(f'before_{event_type}');  f()
  File "/Users/abhijitramesh/Development/DeepLearning/Theta-Tech-ai/Semler-Consulting-local/Semler-Consulting-local/Semler-Consulting/integration/qf_plus_trained/env/lib/python3.9/site-packages/fastai/learner.py", line 141, in __call__
    def __call__(self, event_name): L(event_name).map(self._call_one)
  File "/Users/abhijitramesh/Development/DeepLearning/Theta-Tech-ai/Semler-Consulting-local/Semler-Consulting-local/Semler-Consulting/integration/qf_plus_trained/env/lib/python3.9/site-packages/fastcore/foundation.py", line 155, in map
    def map(self, f, *args, gen=False, **kwargs): return self._new(map_ex(self, f, *args, gen=gen, **kwargs))
  File "/Users/abhijitramesh/Development/DeepLearning/Theta-Tech-ai/Semler-Consulting-local/Semler-Consulting-local/Semler-Consulting/integration/qf_plus_trained/env/lib/python3.9/site-packages/fastcore/basics.py", line 698, in map_ex
    return list(res)
  File "/Users/abhijitramesh/Development/DeepLearning/Theta-Tech-ai/Semler-Consulting-local/Semler-Consulting-local/Semler-Consulting/integration/qf_plus_trained/env/lib/python3.9/site-packages/fastcore/basics.py", line 683, in __call__
    return self.func(*fargs, **kwargs)
  File "/Users/abhijitramesh/Development/DeepLearning/Theta-Tech-ai/Semler-Consulting-local/Semler-Consulting-local/Semler-Consulting/integration/qf_plus_trained/env/lib/python3.9/site-packages/fastai/learner.py", line 145, in _call_one
    for cb in self.cbs.sorted('order'): cb(event_name)
  File "/Users/abhijitramesh/Development/DeepLearning/Theta-Tech-ai/Semler-Consulting-local/Semler-Consulting-local/Semler-Consulting/integration/qf_plus_trained/env/lib/python3.9/site-packages/fastai/callback/core.py", line 45, in __call__
    if self.run and _run: res = getattr(self, event_name, noop)()
  File "/Users/abhijitramesh/Development/DeepLearning/Theta-Tech-ai/Semler-Consulting-local/Semler-Consulting-local/Semler-Consulting/integration/qf_plus_trained/env/lib/python3.9/site-packages/neptune_fastai/impl/__init__.py", line 163, in before_fit
    self._log_model_configuration()
  File "/Users/abhijitramesh/Development/DeepLearning/Theta-Tech-ai/Semler-Consulting-local/Semler-Consulting-local/Semler-Consulting/integration/qf_plus_trained/env/lib/python3.9/site-packages/neptune_fastai/impl/__init__.py", line 139, in _log_model_configuration
    'details': self._vocab,
  File "/Users/abhijitramesh/Development/DeepLearning/Theta-Tech-ai/Semler-Consulting-local/Semler-Consulting-local/Semler-Consulting/integration/qf_plus_trained/env/lib/python3.9/site-packages/fastcore/basics.py", line 389, in __getattr__
    if attr is not None: return getattr(attr,k)
  File "/Users/abhijitramesh/Development/DeepLearning/Theta-Tech-ai/Semler-Consulting-local/Semler-Consulting-local/Semler-Consulting/integration/qf_plus_trained/env/lib/python3.9/site-packages/fastcore/basics.py", line 389, in __getattr__
    if attr is not None: return getattr(attr,k)
  File "/Users/abhijitramesh/Development/DeepLearning/Theta-Tech-ai/Semler-Consulting-local/Semler-Consulting-local/Semler-Consulting/integration/qf_plus_trained/env/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'InceptionTime' object has no attribute '_vocab'
Shutting down background jobs, please wait a moment...
Done!
Waiting for the remaining 86 operations to synchronize with Neptune. Do not kill this process.
All 86 operations synced, thanks for waiting!

Environment

The output of pip list:

Package                Version
---------------------- ---------
alembic                1.7.5
appnope                0.1.2
asttokens              2.0.5
attrs                  21.4.0
autopage               0.4.0
backcall               0.2.0
black                  21.12b0
blis                   0.7.5
boto3                  1.18.18
botocore               1.21.65
bravado                11.0.3
bravado-core           5.17.0
catalogue              2.0.6
certifi                2021.10.8
charset-normalizer     2.0.10
click                  8.0.3
cliff                  3.10.0
cloudpickle            2.0.0
cmaes                  0.8.2
cmd2                   2.3.3
colorlog               6.6.0
cycler                 0.11.0
cymem                  2.0.6
dask                   2022.1.0
debugpy                1.5.1
decorator              5.1.1
Deprecated             1.2.13
distributed            2022.1.0
entrypoints            0.3
et-xmlfile             1.1.0
executing              0.8.2
fastai                 2.5.3
fastcore               1.3.27
fastdownload           0.0.5
fastprogress           1.0.0
fonttools              4.28.5
fsspec                 2022.1.0
future                 0.18.2
gitdb                  4.0.9
GitPython              3.1.26
greenlet               1.1.2
HeapDict               1.0.1
idna                   3.3
imbalanced-learn       0.9.0
ipykernel              6.7.0
ipython                8.0.0
ipython-genutils       0.2.0
jedi                   0.18.1
Jinja2                 3.0.3
jmespath               0.10.0
joblib                 1.1.0
jsonpointer            2.2
jsonref                0.2
jsonschema             3.2.0
jupyter-client         7.1.1
jupyter-core           4.9.1
kiwisolver             1.3.2
langcodes              3.3.0
llvmlite               0.36.0
locket                 0.2.1
Mako                   1.1.6
MarkupSafe             2.0.1
matplotlib             3.5.1
matplotlib-inline      0.1.3
matrixprofile          1.1.10
monotonic              1.6
msgpack                1.0.3
murmurhash             1.0.6
mypy-extensions        0.4.3
nbformat               5.1.3
neptune-client         0.14.2
neptune-fastai         0.10.0
neptune-optuna         0.9.13
nest-asyncio           1.5.4
numba                  0.53.1
numpy                  1.19.3
oauthlib               3.1.1
openpyxl               3.0.7
optuna                 2.9.1
packaging              21.3
pandas                 1.3.5
parso                  0.8.3
partd                  1.2.0
pathspec               0.9.0
pathy                  0.6.1
patsy                  0.5.2
pbr                    5.8.0
pexpect                4.8.0
pickleshare            0.7.5
Pillow                 9.0.0
pip                    21.1.1
platformdirs           2.4.1
plotly                 5.5.0
preshed                3.0.6
prettytable            3.0.0
prompt-toolkit         3.0.24
protobuf               3.11.2
psutil                 5.9.0
psycopg2-binary        2.9.1
ptyprocess             0.7.0
pure-eval              0.2.1
pydantic               1.8.2
Pygments               2.11.2
PyJWT                  2.3.0
pyparsing              3.0.6
pyperclip              1.8.2
pyrsistent             0.18.1
python-dateutil        2.8.2
pytorch-ranger         0.1.1
pyts                   0.12.0
pytz                   2021.3
PyWavelets             1.2.0
PyYAML                 6.0
pyzmq                  22.3.0
requests               2.27.1
requests-oauthlib      1.3.0
rfc3987                1.3.8
s3transfer             0.5.0
scikit-learn           1.0.2
scipy                  1.7.0
seaborn                0.11.2
setuptools             56.0.0
simplejson             3.17.6
six                    1.16.0
sklearn                0.0
sktime                 0.9.0
smart-open             5.2.1
smmap                  5.0.0
sortedcontainers       2.4.0
spacy                  3.2.1
spacy-legacy           3.0.8
spacy-loggers          1.0.1
SQLAlchemy             1.4.29
srsly                  2.4.2
stack-data             0.1.4
statsmodels            0.12.1
stevedore              3.5.0
strict-rfc3339         0.7
stumpy                 1.10.2
swagger-spec-validator 2.7.4
tblib                  1.7.0
tenacity               8.0.1
thinc                  8.0.13
threadpoolctl          3.0.0
tomli                  1.2.3
toolz                  0.11.2
torch                  1.9.1
torch-optimizer        0.3.0
torchvision            0.10.1
tornado                6.1
tqdm                   4.62.3
traitlets              5.1.1
tsai                   0.2.24
tsfresh                0.18.0
typer                  0.4.0
typing-extensions      4.0.1
urllib3                1.26.8
wasabi                 0.9.0
wcwidth                0.2.5
webcolors              1.11.1
websocket-client       1.2.3
wheel                  0.37.1
wrapt                  1.13.3
zict                   2.0.0

The operating system you're using: macOS 12.0.1 The output of python --version: Python 3.9.5

Additional context

I found hotfix's for the problem but its not ideal:

Remove vocab from here: https://github.com/neptune-ai/neptune-fastai/blob/b8e0f3c308170861806c7e23423d76685f71f22e/neptune_fastai/impl/__init__.py#L249

This is not ideal we need vocab for classification.

In the training script (above) set learn._vocab=[]

These fixes are not ideal, vocab should be optional parameter in the init.py for the neptune_fastai implementation.

slawekslex commented 2 years ago

I've run into the same bug. I use dls.vocab='' as a workaround. But this should be fixed not to make assumptions like that.

Blaizzy commented 2 years ago

Hi @abhijitramesh and @slawekslex,

Prince Canuma here, a Data Scientist at Neptune.ai,

Thank you very much for spotting this bug and providing such detailed feedback,

Indeed, vocab is should be optional in such cases, I will submit this feedback to the engineering team and get back to you,

In the meantime, I think the workarounds that you guys came up with work fine although far from ideal:

Remove vocab from the implementation: neptune-fastai/neptune_fastai/impl/__init__.py
Set some value for dls.vocab = '' or dls.vocab = []

Blaizzy commented 2 years ago

On the other hand, is this the only issue? Is there anything else I can help you with?

abhijitramesh commented 2 years ago

On the other hand, is this the only issue? Is there anything else I can help you with?

Thanks, @Blaizzy Nope, That is it for now. I will let you know if something comes up.

neptune-ai / neptune-fastai