Torch not compiled with CUDA enabled when doing inference on CPU

[mattmcclean]

Am getting the following error when attempting to do inference on a fastai2 model using a CPU only server. The error is the following: AssertionError: Torch not compiled with CUDA enabled

The version of PyTorch is 1.3.1+cpu and fastai2 version 0.0.11

The stacktrace error is the following:

Torch not compiled with CUDA enabled: AssertionError
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 31, in lambda_handler
    pred_class,pred_idx,outputs = learn.predict(bytes)
  File "/mnt/test/fastai2/lib/python3.6/dist-packages/fastai2/learner.py", line 215, in predict
    inp,preds,_,dec_preds = self.get_preds(dl=dl, with_input=True, with_decoded=True)
  File "/mnt/test/fastai2/lib/python3.6/dist-packages/fastai2/learner.py", line 202, in get_preds
    self(event.begin_epoch if inner else _before_epoch)
  File "/mnt/test/fastai2/lib/python3.6/dist-packages/fastai2/learner.py", line 108, in __call__
    def __call__(self, event_name): L(event_name).map(self._call_one)
  File "/mnt/test/fastai2/lib/python3.6/dist-packages/fastcore/foundation.py", line 362, in map
    return self._new(map(g, self))
  File "/mnt/test/fastai2/lib/python3.6/dist-packages/fastcore/foundation.py", line 315, in _new
    def _new(self, items, *args, **kwargs): return type(self)(items, *args, use_list=None, **kwargs)
  File "/mnt/test/fastai2/lib/python3.6/dist-packages/fastcore/foundation.py", line 41, in __call__
    res = super().__call__(*((x,) + args), **kwargs)
  File "/mnt/test/fastai2/lib/python3.6/dist-packages/fastcore/foundation.py", line 306, in __init__
    items = list(items) if use_list else _listify(items)
  File "/mnt/test/fastai2/lib/python3.6/dist-packages/fastcore/foundation.py", line 242, in _listify
    if is_iter(o): return list(o)
  File "/mnt/test/fastai2/lib/python3.6/dist-packages/fastcore/foundation.py", line 208, in __call__
    return self.fn(*fargs, **kwargs)
  File "/mnt/test/fastai2/lib/python3.6/dist-packages/fastai2/learner.py", line 111, in _call_one
    [cb(event_name) for cb in sort_by_run(self.cbs)]
  File "/mnt/test/fastai2/lib/python3.6/dist-packages/fastai2/learner.py", line 111, in <listcomp>
    [cb(event_name) for cb in sort_by_run(self.cbs)]
  File "/mnt/test/fastai2/lib/python3.6/dist-packages/fastai2/callback/core.py", line 23, in __call__
    if self.run and _run: getattr(self, event_name, noop)()
  File "/mnt/test/fastai2/lib/python3.6/dist-packages/fastai2/callback/core.py", line 43, in begin_fit
    self.model.to(self.dls.device)
  File "/mnt/test/fastai2/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 426, in to
    return self._apply(convert)
  File "/mnt/test/fastai2/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 202, in _apply
    module._apply(fn)
  File "/mnt/test/fastai2/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 202, in _apply
    module._apply(fn)
  File "/mnt/test/fastai2/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 224, in _apply
    param_applied = fn(param)
  File "/mnt/test/fastai2/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 424, in convert
    return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
  File "/mnt/test/fastai2/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 192, in _lazy_init
    _check_driver()
  File "/mnt/test/fastai2/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 95, in _check_driver
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

The sample code I am running is the following:

import os
import json
import string
import time
import io

from fastai2.vision.all import *

defaults.device = torch.device('cpu')
learn = torch.load('/mnt/test/models/export.pkl', map_location=torch.device('cpu'))

def lambda_handler(event, context):
    print(json.dumps(event, indent=4))
    body = json.loads(event['body'])
    url = body['url']
    print(f'Getting image from URL: {url}')
    r = requests.get(url)
    print("Retrieved image from URL. Loading into memory")
    bytes = io.BytesIO(r.content).read()
    print("Doing prediction")
    pred_class,pred_idx,outputs = learn.predict(bytes)
    print(f'Predicted class is {pred_class}')
    print(f'pred_idx: {pred_idx}')
    print(f'Outputs: {outputs}')
    result = { "pred_class": pred_class }
    return {
        'statusCode': 200,
        'body': json.dumps(result) 
    }

[mattmcclean]

The problematic line of code is the following. I wonder if the problem is because the callback class named TrainEvalCallback is set to be default (see code snippet). Do I need to remove this callback class from the learner to do inference only ?

[sgugger]

You should use the function load_learner with cpu=True instead of torch.load. Here your learn.dls have serialized their device attribute, which is one GPU because you trained in an env with cuda enabled I guess, so it tries to put the model on the same device to match. load_learner will fix that problem for you.