search

thundergolfer / reasoning-about-entailment-tensorflow

:school: Tensorflow implementation of "Reasoning About Entailment with Neural Attention"
MIT License
10 stars 5 forks source link

Problem in getting the training to work on Python 3 with GPU #2

Closed thundergolfer closed 7 years ago

thundergolfer commented 7 years ago

On the AWS GPU Instance, installing and attempting to execute training results in the following error:

TypeError: only integer scalar arrays can be converted to a scalar index

This is an error originating in the LSTM_with_attention network class.

Also, currently training with a GPU on the work AWS GPU instance has hit a roadblock in that my CUDA and CuDNN local installs aren't compatible with the Tensorflow installs I'm trying to use.

thundergolfer commented 7 years ago

Update after consulting with Soon-ee:

So it turns out that as I suspected the GPU Tensorflow installs at work are compiled against older CUDA and CuDNN. It would be pretty painful to change those versions, but it is possible apparently to compile Python 3 GPU Tensorflow against those older versions without using sudo on the GPU instance.

This is one option. There are two others I see:

  1. Wait for OPS to upgrade the drivers so that the team can move to CUDA 8.0 (and hopefully CuDNN 5.1+).
  2. Wait for AWS credentials and then change this repo's code to be 2/3 compatible.
thundergolfer commented 7 years ago

Closing this and going down this route:

  1. Wait for AWS credentials and then change this repo's code to be 2/3 compatible.

It will be too much of a pain to get GPUs working for Python 3.