strongio / keras-bert

A simple technique to integrate BERT from tf hub to keras
258 stars 108 forks source link

Unable to run initialise training #18

Closed jordanparker6 closed 5 years ago

jordanparker6 commented 5 years ago

I am having the following issue...

The model compiles and prints the following output. However, on model.fit() nothing happens, despite verbose mode being turned on.

When i look at my hardware utilisation, my GPU has memory allocated to the process however utilisation is 0-2%. On my CPU, only one core is getting worked by the process at 100% utilisation.

To test my tensorflow-gpu install, I ran the CNN example on tensorflow and got 20% GPU utilisation.

I don't think it is a preprocessing bottleneck as I load my training data into memory.

Thanks.

Code: ` bert_path = "https://tfhub.dev/google/bert_uncased_L-12_H-768_A-12/1" max_seq_length = 256

corpus = MyDocs("datasets/bbc/raw", bert_path, max_seq_length)

ids = []
masks = []
segment_ids = []
for id, mask, segment, label in corpus:
    ids.append(id)
    masks.append(masks)
    segment_ids.append(segment)
X = [ids, masks, segment_ids]

labels = corpus.labels
label_encoder = OneHotEncoder()
y = label_encoder.fit_transform(np.array(labels).reshape(-1, 1)).todense()

print('Building model...')
model = build_model(bert_path, max_seq_length)

print('Training model...')
history = model.fit(X, y,
                    validation_split=0.2,
                    epochs=1,
                    batch_size=1,
                    verbose=2,
                    use_multiprocessing=True)`

Output:

Building model... W0709 21:57:53.871020 140194145126208 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor W0709 21:57:53.922768 140194145126208 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where

Training model...