solubility_test_new_message.ipynb ValueError: All input arrays (x) should have the same number of samples.

[muu4649]

Hi! I met an error in solubility_test_new_message.ipynb.

---Error section ----- with warnings.catch_warnings(): warnings.simplefilter("ignore")

hist = model.fit_generator(train_generator,validation_data=test_generator,epochs=50, verbose=2)

---

----Error output---- Epoch 1/50

ValueError Traceback (most recent call last)

in 2 warnings.simplefilter("ignore") 3 ----> 4 hist = model.fit_generator(train_generator,validation_data=test_generator,epochs=50, verbose=1) /anaconda3/lib/python3.6/site-packages/keras/legacy/interfaces.py in wrapper(*args, **kwargs) 89 warnings.warn('Update your `' + object_name + '` call to the ' + 90 'Keras 2 API: ' + signature, stacklevel=2) ---> 91 return func(*args, **kwargs) 92 wrapper._original_function = func 93 return wrapper /anaconda3/lib/python3.6/site-packages/keras/engine/training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch) 1416 use_multiprocessing=use_multiprocessing, 1417 shuffle=shuffle, -> 1418 initial_epoch=initial_epoch) 1419 1420 @interfaces.legacy_generator_methods_support /anaconda3/lib/python3.6/site-packages/keras/engine/training_generator.py in fit_generator(model, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch) 215 outs = model.train_on_batch(x, y, 216 sample_weight=sample_weight, --> 217 class_weight=class_weight) 218 219 outs = to_list(outs) /anaconda3/lib/python3.6/site-packages/keras/engine/training.py in train_on_batch(self, x, y, sample_weight, class_weight) 1209 x, y, 1210 sample_weight=sample_weight, -> 1211 class_weight=class_weight) 1212 if self._uses_dynamic_learning_phase(): 1213 ins = x + y + sample_weights + [1.] /anaconda3/lib/python3.6/site-packages/nfp/models/models.py in _standardize_user_data(self, *args, **kwargs) 10 def _standardize_user_data(self, *args, **kwargs): 11 kwargs['check_array_lengths'] = False ---> 12 return super(GraphModel, self)._standardize_user_data(*args, **kwargs) /anaconda3/lib/python3.6/site-packages/keras/engine/training.py in _standardize_user_data(self, x, y, sample_weight, class_weight, check_array_lengths, batch_size) 802 ] 803 # Check that all arrays have the same length. --> 804 check_array_length_consistency(x, y, sample_weights) 805 if self._is_graph_network: 806 # Additional checks to avoid users mistakenly /anaconda3/lib/python3.6/site-packages/keras/engine/training_utils.py in check_array_length_consistency(inputs, targets, weights) 226 raise ValueError('All input arrays (x) should have ' 227 'the same number of samples. Got array shapes: ' + --> 228 str([x.shape for x in inputs])) 229 if len(set_y) > 1: 230 raise ValueError('All target arrays (y) should have ' ValueError: All input arrays (x) should have the same number of samples. Got array shapes: [(657, 1), (657, 1), (1350, 1), (1350, 2)] ----

[pstjohn]

I'm guessing this is related to https://github.com/keras-team/keras/pull/11548. We had to submit a PR upstream to keras to get the GraphModel functionality to work correctly. Until that gets merged into a release, you'll need to install keras from the github master. That usually works via

1. git clone (the keras repo)
2. cd keras
3. python setup.py develop (inside the keras directory)

[muu4649]

Thank you for your response. So, Do I have to do "pip install git+git://github.com/keras-team/keras.git --upgrade --no-deps"?

[pstjohn]

that would probably work, yeah.

[pstjohn]

@muu4649, did that fix the issue? I'd be happy to put that command in the README if it did

[muu4649]

Thank you for your response.

[jgmeyerucsd]

when I use keras 2.2.4 as given in the example notebook print - I get the error referenced here. When I upgrade to keras 2.3.1 I get the error (with tf 1.10, 1.12, or 1.14): module 'tensorflow.python.keras.backend' has no attribute 'get_graph' after trying to import keras.

I would love to try this but having trouble getting the correct group of package versions. Would it be possible to get the exact versions of everything that will work together?

[pstjohn]

I'm able to run the test suite using keras 2.3.1 and tensorflow 1.14. they're also working on travis, you can see the build config here: https://travis-ci.org/NREL/nfp/builds/597295956

I'm guessing that your tensorflow install for 1.14 is broken somehow? maybe try in a fresh environment by just running pip install nfp.

[jgmeyerucsd]

yes i tried pip install nfp and it doesn't come with numpy, seaborn, or rdkit - after installing those the notebook still doesn't run.

It looks like the test suite doesnt run the referenced example notebook, right? Could you please test if your pip install nfpenvironment works with that notebook?

[pstjohn]

sure I can try it. You could also clone the repo and run the pytest suite to verify the tests run successfully on your environment. edit: the notebook seems to work on my installation at least.

[jgmeyerucsd]

I cloned the repo, ran pip install -r requirements.txt, and tried running pytest but i get an error that theano isnt installed. I'm sorry it must be some environment problem this is over my head. Thanks for your help anyway.

[pstjohn]

lol, no worries. The theano message is likely coming from keras complaining that it can't find a backend -- here we want it to be using tensorflow. What python distribution are you using? Platform?

One caveat I'll mention is that a lot of this code is based on tensorflow 1.x and the keras package, both of which are somewhat deprecated with the tf 2.x release. Building models with the tf.keras module looks a lot different, and was something I was playing around with in the tf2 branch. https://github.com/NREL/nfp/blob/tf2/examples/tf2_tests.ipynb.

[jgmeyerucsd]

I use anaconda 3, so before install i made an empty conda environment with python 3.6 and then source activate nfp pip install -r requirements.txt Should I be using conda env create -f .travis.yml ?

pip and conda don't seem to play nice together - is this the problem?

[jgmeyerucsd]

It definitely seems like it doesn't recognize tensorflow - even when I manually install using the conda scripts inside .travis.yml and run pytest tests/ it complains: Using Theano backend. ImportError while loading conftest '/home/jgmeyer2/NREL/nfp/tests/conftest.py'. tests/conftest.py:3: in from nfp.preprocessing import SmilesPreprocessor, MolPreprocessor, GraphSequence nfp/init.py:1: in from .layers import nfp/layers/init.py:1: in from .layers import nfp/layers/layers.py:1: in from keras.engine import Layer ../../anaconda3/envs/test/lib/python3.6/site-packages/keras/init.py:3: in from . import utils ../../anaconda3/envs/test/lib/python3.6/site-packages/keras/utils/init.py:6: in from . import conv_utils ../../anaconda3/envs/test/lib/python3.6/site-packages/keras/utils/conv_utils.py:9: in from .. import backend as K ../../anaconda3/envs/test/lib/python3.6/site-packages/keras/backend/init.py:1: in from .load_backend import epsilon ../../anaconda3/envs/test/lib/python3.6/site-packages/keras/backend/load_backend.py:87: in from .theano_backend import ../../anaconda3/envs/test/lib/python3.6/site-packages/keras/backend/theano_backend.py:7: in import theano ../../.local/lib/python3.6/site-packages/theano/init.py:88: in from theano.configdefaults import config ../../.local/lib/python3.6/site-packages/theano/configdefaults.py:137: in in_c_key=False) ../../.local/lib/python3.6/site-packages/theano/configparser.py:287: in AddConfigVar configparam.get(root, type(root), delete_key=True) ../../.local/lib/python3.6/site-packages/theano/configparser.py:335: in get self.set(cls, val_str) ../../.local/lib/python3.6/site-packages/theano/configparser.py:346: in set self.val = self.filter(val) ../../.local/lib/python3.6/site-packages/theano/configdefaults.py:116: in filter 'You are tring to use the old GPU back-end. ' E ValueError: You are tring to use the old GPU back-end. It was removed from Theano. Use device=cuda now. See https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29 for more information.

[pstjohn]

You can see how I install things with conda in the .travis.yml script, but no, you wouldn't run conda env create -f .travis.yml.

You essentially want to recreate this pattern: https://github.com/NREL/nfp/blob/18c4402b08da261546e2863a9d3961745433083a/.travis.yml#L15-L23

Looks like I install tensorflow from conda-forge; but you could likely remove that from the conda install line and let the requirements.txt file install it.

[jgmeyerucsd]

Yes, that is exactly what I did to get the error I posted above in my linux boot.

I booted into windows to try and ran the same code and got it to work, but it did not come with all the requirements needed to run the notebook using only those lines. I needed to add ipykernel to configure the env for jupyter, matplotlib and seaborn for the second imports, and it didn't recongnize the first rdkit so that needed to be reinstalled.

Now to get it on the GPUs...

Thanks for your help.

[pstjohn]

for gpus you typically install tensorflow-gpu with pip, just make sure you get a pre 2.0 build (and match the CUDA / CUDNN binaries)

[WardLT]

How are things going on the TF2 integration? I'm working on using your tf2 branch with 1.15.0 in hopes of being able to use tf.Estimator

[pstjohn]

Ah, good question, I actually haven't been working on it recently. I've got a bit more experience with tf 2.x nowadays, so I'd probably do it a bit differently than the tf2 branch. You mean using tensorflow 1.15.0?

What's the motivation for using the tf.Estimator? As opposed to a tf.keras model in tf 2.0.

[WardLT]

Yes, I did mean using Tensorflow 1.15.0.

I'm looking to run on a system that relaties on compiling a tensorflow model from a tf.Estimator class. Given that it's possible convert from a tf.keras model to an Estimator, using either is fine with me.

My other consideration for 1.15 is I'm unsure if horovod works will with 2.0 yet. I'm still tinkering around to figure out if switching fully to TF2 will work for me in the short term.

[WardLT]

I'll probably build out from the notebook example you've put together and will issue a PR if I get nfp in a form that works with TF2

[pstjohn]

Sounds good. I'm using horovod with tf2 in another project; there are a few tensorflow 2 examples in https://github.com/horovod/horovod/tree/master/examples. One issue with the current tf2 implementation is that the keras hdf5 saves don't work if you subclass tf.keras.model; it would likely be better to just define new layers and still build the model using the functional syntax.