search

john-hewitt / structural-probes

Codebase for testing whether hidden states of neural networks encode discrete structures.
Other
381 stars 77 forks source link

TypeError: can't convert CUDA tensor to numpy #4

Closed Hyperparticle closed 5 years ago

Hyperparticle commented 5 years ago

When running with CUDA and generating tables, the labels are trying to be implicitly converted to numpy using scipy. I can see that there is already a .detach().cpu().numpy() in regimen.py, but for some reason it appears that the labels are being converted back to CUDA. Calling .cpu() before handing off the labels to scipy fixes it for me.

Full trace below.

Traceback (most recent call last):
  File "structural-probes/run_experiment.py", line 242, in <module>
    execute_experiment(yaml_args, train_probe=cli_args.train_probe, report_results=cli_args.report_results)
  File "structural-probes/run_experiment.py", line 182, in execute_experiment
    run_report_results(args, expt_probe, expt_dataset, expt_model, expt_loss, expt_reporter, expt_regimen)
  File "structural-probes/run_experiment.py", line 143, in run_report_results
    reporter(dev_predictions, dev_dataloader, 'dev')
  File "/home/hyper/Documents/repos/structural-probes/structural-probes/reporter.py", line 50, in __call__
    , dataloader, split_name)
  File "/home/hyper/Documents/repos/structural-probes/structural-probes/reporter.py", line 143, in report_image_examples
    ax = sns.heatmap(label)
  File "/home/hyper/Documents/anaconda3/envs/probe/lib/python3.7/site-packages/seaborn/matrix.py", line 517, in heatmap
    yticklabels, mask)
  File "/home/hyper/Documents/anaconda3/envs/probe/lib/python3.7/site-packages/seaborn/matrix.py", line 109, in __init__
    plot_data = np.asarray(data)
  File "/home/hyper/Documents/anaconda3/envs/probe/lib/python3.7/site-packages/numpy/core/numeric.py", line 538, in asarray
    return array(a, dtype, copy=False, order=order)
  File "/home/hyper/Documents/anaconda3/envs/probe/lib/python3.7/site-packages/torch/tensor.py", line 450, in __array__
    return self.numpy()
TypeError: can't convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
john-hewitt commented 5 years ago

Thanks for the find! Could you note your pytorch, numpy, and scipy versions? I mostly tested this code on (regrettably) old installs of each of these, so this'll help me debug / make a fix that works for a lot of people's versions.

Hyperparticle commented 5 years ago
howardhsu commented 5 years ago

Got the same problem on PyTorch 1.0.1, change back to 0.4.1(0.4.1.post2) works.

john-hewitt commented 5 years ago

Should be fixed in PR #5 ; check it out if you'd like; I've tested it on old + new versions on parse-distance and parse-depth tasks. If all is well / no comments, I'll merge this weekend.