search

deepchem / deepchem

Democratizing Deep-Learning for Drug Discovery, Quantum Chemistry, Materials Science and Biology
https://deepchem.io/
MIT License
5.42k stars 1.67k forks source link

TypeError: Expected binary or unicode string, got test_atom_features #355

Closed SteppenW0lf closed 7 years ago

SteppenW0lf commented 7 years ago

Hi!

I've tested some scripts from low_data folder (trying to reproduce results), precisely sider_attn_one_fold.py & sider_attn_one_fold_no_train.py and they both crashed with the same error. I also changed some lines in dataset.py in low_data folder and in all scripts removed debug=True. Here is a log of one of the scripts: python ./sider_attn_one_fold.py Using gpu device 0: GeForce GTX 1080 (CNMeM is disabled, cuDNN 5105) About to featurize SIDER dataset. SIDER tasks: ['Hepatobiliary disorders', 'Metabolism and nutrition disorders', 'Product issues', 'Eye disorders', 'Investigations', 'Musculoskeletal and connective tissue disorders', 'Gastrointestinal disorders', 'Social circumstances', 'Immune system disorders', 'Reproductive system and breast disorders', 'Neoplasms benign, malignant and unspecified (incl cysts and polyps)', 'General disorders and administration site conditions', 'Endocrine disorders', 'Surgical and medical procedures', 'Vascular disorders', 'Blood and lymphatic system disorders', 'Skin and subcutaneous tissue disorders', 'Congenital, familial and genetic disorders', 'Infections and infestations', 'Respiratory, thoracic and mediastinal disorders', 'Psychiatric disorders', 'Renal and urinary disorders', 'Pregnancy, puerperium and perinatal conditions', 'Ear and labyrinth disorders', 'Cardiac disorders', 'Nervous system disorders', 'Injury, poisoning and procedural complications'] 27 tasks in total Loading raw samples now. shard_size: 8192 About to start loading CSV from /home/steppenwolf/workspace/drugstructure/deepchem/examples/low_data/../sider/sider.csv.gz Loading shard 1 of size 8192. Featurizing sample 0 Featurizing sample 1000 TIMING: featurizing shard 0 took 2.681 s TIMING: dataset construction took 4.549 s True Loading dataset from disk. 1427 datapoints in SIDER dataset About to transform data TIMING: dataset construction took 2.882 s True Loading dataset from disk. Assigning extra tasks to last fold due to uneven split Traceback (most recent call last): File "./sider_attn_one_fold_no_train.py", line 49, in support_model.add(dc.nn.GraphConv(64, activation='relu')) File "/usr/local/lib/python2.7/dist-packages/deepchem/models/tf_keras_models/graph_models.py", line 100, in add self.test = layer([self.test] + self.test_graph_topology.topology) File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 569, in call self.add_inbound_node(inbound_layers, node_indices, tensor_indices) File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 632, in add_inbound_node Node.create_node(self, inbound_layers, node_indices, tensor_indices) File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 170, in create_node output_tensors = to_list(outbound_layer.call(input_tensors, mask=input_masks)) File "/usr/local/lib/python2.7/dist-packages/deepchem/models/tf_keras_models/keras_layers.py", line 298, in call self.min_deg, self.W_list, self.b_list)
File "/usr/local/lib/python2.7/dist-packages/deepchem/models/tf_keras_models/keras_layers.py", line 73, in graph_conv deg_summed = sum_neigh(atoms, deg_adj_lists, max_deg) File "/usr/local/lib/python2.7/dist-packages/deepchem/models/tf_keras_models/keras_layers.py", line 35, in sum_neigh gathered_atoms = tf.gather(atoms, deg_adj_lists[deg-1]) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 1293, in gather validate_indices=validate_indices, name=name) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 493, in apply_op raise err TypeError: Expected binary or unicode string, got test_atom_features

Could someone help me to cope with this problem?

rbharath commented 7 years ago

I tried running sider_attn_one_fold.py. The latest refactorings have slightly changed APIs, so I had to make the following changes:

With these changes, I'm able to train the model. Here's the output I get on a CPU node:

(deepchem) [rbharath@sh-5-36 /scratch/users/rbharath/deepchem/examples/low_data]$ python sider_attn_one_fold.py
I tensorflow/stream_executor/dso_loader.cc:130] successfully opened CUDA library libcublas.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:130] successfully opened CUDA library libcudnn.so.5.1 locally
I tensorflow/stream_executor/dso_loader.cc:130] successfully opened CUDA library libcufft.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:121] Couldn't open CUDA library libcuda.so.1. LD_LIBRARY_PATH: /opt/rh/python27/root/usr/lib64:/share/sw/free/boost/1.60.0/lib:/share/sw/free/swig/3.0.8/lib:/share/sw/free/cuDNN/v5.1/lib64:/usr/lib64/nvidia:/share/sw/free/cuda/8.0/extras/Debugger/lib64:/share/sw/free/cuda/8.0/extras/CUPTI/lib64:/share/sw/free/cuda/8.0/lib64:/share/sw/free/libjpeg-turbo/1.4.2/lib
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:165] hostname: sh-5-36.local
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:189] libcuda reported version is: Not found: was unable to find libcuda.so DSO loaded into this program
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:193] kernel reported version is: Permission denied: could not open driver version path for reading: /proc/driver/nvidia/version
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1092] LD_LIBRARY_PATH: /opt/rh/python27/root/usr/lib64:/share/sw/free/boost/1.60.0/lib:/share/sw/free/swig/3.0.8/lib:/share/sw/free/cuDNN/v5.1/lib64:/usr/lib64/nvidia:/share/sw/free/cuda/8.0/extras/Debugger/lib64:/share/sw/free/cuda/8.0/extras/CUPTI/lib64:/share/sw/free/cuda/8.0/lib64:/share/sw/free/libjpeg-turbo/1.4.2/lib
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1093] failed to find libcuda.so on this system: Failed precondition: could not dlopen DSO: libcuda.so.1; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
I tensorflow/stream_executor/dso_loader.cc:130] successfully opened CUDA library libcurand.so.8.0 locally
Using TensorFlow backend.
About to featurize SIDER dataset.
SIDER tasks: ['Hepatobiliary disorders', 'Metabolism and nutrition disorders', 'Product issues', 'Eye disorders', 'Investigations', 'Musculoskeletal and connective tissue disorders', 'Gastrointestinal disorders', 'Social circumstances', 'Immune system disorders', 'Reproductive system and breast disorders', 'Neoplasms benign, malignant and unspecified (incl cysts and polyps)', 'General disorders and administration site conditions', 'Endocrine disorders', 'Surgical and medical procedures', 'Vascular disorders', 'Blood and lymphatic system disorders', 'Skin and subcutaneous tissue disorders', 'Congenital, familial and genetic disorders', 'Infections and infestations', 'Respiratory, thoracic and mediastinal disorders', 'Psychiatric disorders', 'Renal and urinary disorders', 'Pregnancy, puerperium and perinatal conditions', 'Ear and labyrinth disorders', 'Cardiac disorders', 'Nervous system disorders', 'Injury, poisoning and procedural complications']
27 tasks in total
Loading raw samples now.
shard_size: 8192
About to start loading CSV from /scratch/users/rbharath/deepchem/examples/low_data/../sider/sider.csv.gz
Loading shard 1 of size 8192.
Featurizing sample 0
Featurizing sample 1000
TIMING: featurizing shard 0 took 5.908 s
TIMING: dataset construction took 10.813 s True
Loading dataset from disk.
1427 datapoints in SIDER dataset
About to transform data
TIMING: dataset construction took 6.611 s True
Loading dataset from disk.
Assigning extra tasks to last fold due to uneven split
E tensorflow/stream_executor/cuda/cuda_driver.cc:509] failed call to cuInit: CUDA_ERROR_NO_DEVICE
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:145] kernel driver does not appear to be running on this host (sh-5-36.local): /proc/driver/nvidia/version does not exist
/share/sw/free/tensorflow/0.12.1/tensorflow/python/ops/gradients_impl.py:91: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
WARNING:tensorflow:From /scratch/users/rbharath/deepchem/deepchem/models/tf_keras_models/support_classifier.py:61 in __init__.: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use `tf.global_variables_initializer` instead.
FIT
Sampling support 0
Sampling support 50
Sampling support 100
Sampling support 150
Sampling support 200
Sampling support 250
Sampling support 300
Sampling support 350
Sampling support 400
Sampling support 450
Sampling support 500
Sampling support 550
Sampling support 600
Sampling support 650
Sampling support 700
Sampling support 750
Sampling support 800
Sampling support 850
Sampling support 900
Sampling support 950
.... (Repeats for a bit)
Constructing EpisodeGenerator took 4.15765500069 seconds
Epoch 0, Sample 0 from task 9
/share/sw/free/tensorflow/0.12.1/numpy/core/_methods.py:59: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)
        mean loss is nan
Epoch 0, Sample 50 from task 11
        mean loss is 77.4445
Epoch 0, Sample 100 from task 3
        mean loss is 76.8856
Epoch 0, Sample 150 from task 12
        mean loss is 79.2981
Epoch 0, Sample 200 from task 19
        mean loss is 70.4257

I'm in the middle of refactoring this code, and I'll make a PR soon that fixes the bugs I've noted below. In the meantime, maybe try cloning a fresh copy, making the changes I noted above, and try running again?

rbharath commented 7 years ago

I just merged #360 which has fixed the bugs noted above. I'm going to close this issue. Please feel free to open a new issue if you're still running into bugs.