search

NVIDIA / DIGITS

Deep Learning GPU Training System
https://developer.nvidia.com/digits
BSD 3-Clause "New" or "Revised" License
4.12k stars 1.38k forks source link

Train Tensorflow Model task failed with error code -9 #2191

Open Snapple49 opened 5 years ago

Snapple49 commented 5 years ago

problem

I was recently at a NVIDIA course where we got an introcution to Deep Learning, and I really like DIGITS and I would like to explore it a bit more. So I wanted to try to run it in our local kubernetes cluster which has 10 GTX 1080 Ti cards, and I'm running into a few issues, mainly I cannot run training either on Tensorflow or Caffe (I'm not a ML guy but I prefer TF due to it being standard in our group). I sort of managed to run the mnist example, but only using DIGITS container build 17.10, and now I wanted to try something more custom. Currently I'm hosting a pod of the nvcr.io/nvidia/digits:19.09-tensorflow image but I tried caffe as well, with similar results.

I'm seeing a lot of warnings in the embedded examples as well, it seems a lot of things are deprecated and it does seem like not a lot has been changed in 2 years looking through the repository. I would really love to be able to utilize DIGITS though, it is an awesome tool!

details platform: kubernetes version: 1.14.6 host os: ubuntu 16.04.4 NVIDIA drivers nvidia-smi: NVIDIA-SMI 418.87.00 Driver Version: 418.87.00 CUDA Version: 10.1 docker image: nvcr.io/nvidia/digits:19.09-tensorflow os in docker image: Ubuntu 18.04.2 docker version 17.03.2-ce nvidia-container-runtime verson:

nvidia-container-runtime --version
runc version 1.0.0-rc2
commit: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe-dirty
spec: 1.0.0-rc2-dev

Not sure what else to dig up, looking at this I'm thinking it coooould be the nvidia container runtime, but as mentioned Tensorflow complains about a lot of stuff being deprecated. Any help is appreciated!

job specifics image size 256x256 (dataset from kaggle, creating dataset works fine) only change from default was 5 epochs, more details from log:

Task subprocess args: "/usr/bin/python /opt/digits/digits/tools/tensorflow/main.py --network=network.py --epoch=5 --networkDirectory=/workspace/jobs/20190927-193801-dcf2 --save=/workspace/jobs/20190927-193801-dcf2 --snapshotPrefix=snapshot --snapshotInterval=1.0 --lr_base_rate=0.01 --lr_policy=step --mean=/workspace/jobs/20190927-192950-64cf/mean.binaryproto --labels_list=/workspace/jobs/20190927-192950-64cf/labels.txt --train_db=/workspace/jobs/20190927-192950-64cf/train_db --validation_db=/workspace/jobs/20190927-192950-64cf/val_db --lr_gamma=0.1 --lr_stepvalues=33.0 --shuffle=1 --subtractMean=image --optimization=sgd --validation_interval=1 --log_runtime_stats_per_step=0 --augFlip=none --augHSVh=0 --augHSVs=0 --augHSVv=0"

Network.py:

# Preferred settings for this model is:
# Base Learning Rate = 0.001
# Crop Size = 224

from model import Tower
from utils import model_property
import tensorflow as tf
import tensorflow.contrib.slim as slim
import utils as digits

class UserModel(Tower):

    @model_property
    def inference(self):
        x = tf.reshape(self.x, shape=[-1, self.input_shape[0], self.input_shape[1], self.input_shape[2]])
        with slim.arg_scope([slim.conv2d, slim.fully_connected],
                            weights_initializer=tf.contrib.layers.xavier_initializer(),
                            weights_regularizer=slim.l2_regularizer(1e-6)):
            model = slim.conv2d(x, 96, [11, 11], 4, padding='VALID', scope='conv1')
            model = slim.max_pool2d(model, [3, 3], 2, scope='pool1')
            model = slim.conv2d(model, 256, [5, 5], 1, scope='conv2')
            model = slim.max_pool2d(model, [3, 3], 2, scope='pool2')
            model = slim.conv2d(model, 384, [3, 3], 1, scope='conv3')
            model = slim.conv2d(model, 384, [3, 3], 1, scope='conv4')
            model = slim.conv2d(model, 256, [3, 3], 1, scope='conv5')
            model = slim.max_pool2d(model, [3, 3], 2, scope='pool5')
            model = slim.flatten(model)
            model = slim.fully_connected(model, 4096, activation_fn=None, scope='fc1')
            model = slim.dropout(model, 0.5, is_training=self.is_training, scope='do1')
            model = slim.fully_connected(model, 4096, activation_fn=None, scope='fc2')
            model = slim.dropout(model, 0.5, is_training=self.is_training, scope='do2')
            model = slim.fully_connected(model, self.nclasses, activation_fn=None, scope='fc3')
        return model

    @model_property
    def loss(self):
        model = self.inference
        loss = digits.classification_loss(model, self.y)
        accuracy = digits.classification_accuracy(model, self.y)
        self.summaries.append(tf.summary.scalar(accuracy.op.name, accuracy))
        return loss

logs

Logs from submitting the job to last message after failing job:

2019-09-27 19:38:03 [20190927-193801-dcf2] [INFO ] Task subprocess args: "/usr/bin/python /opt/digits/digits/tools/tensorflow/main.py --network=network.py --epoch=5 --networkDirectory=/workspace/jobs/20190927-193801-dcf2 --save=/workspace/jobs/20190927-193801-dcf2 --snapshotPrefix=snapshot --snapshotInterval=1.0 --lr_base_rate=0.01 --lr_policy=step --mean=/workspace/jobs/20190927-192950-64cf/mean.binaryproto --labels_list=/workspace/jobs/20190927-192950-64cf/labels.txt --train_db=/workspace/jobs/20190927-192950-64cf/train_db --validation_db=/workspace/jobs/20190927-192950-64cf/val_db --lr_gamma=0.1 --lr_stepvalues=33.0 --shuffle=1 --subtractMean=image --optimization=sgd --validation_interval=1 --log_runtime_stats_per_step=0 --augFlip=none --augHSVh=0 --augHSVs=0 --augHSVv=0"

I0927 19:38:03.321882 140597179560544 task.py:213] Task subprocess args: "/usr/bin/python /opt/digits/digits/tools/tensorflow/main.py --network=network.py --epoch=5 --networkDirectory=/workspace/jobs/20190927-193801-dcf2 --save=/workspace/jobs/20190927-193801-dcf2 --snapshotPrefix=snapshot --snapshotInterval=1.0 --lr_base_rate=0.01 --lr_policy=step --mean=/workspace/jobs/20190927-192950-64cf/mean.binaryproto --labels_list=/workspace/jobs/20190927-192950-64cf/labels.txt --train_db=/workspace/jobs/20190927-192950-64cf/train_db --validation_db=/workspace/jobs/20190927-192950-64cf/val_db --lr_gamma=0.1 --lr_stepvalues=33.0 --shuffle=1 --subtractMean=image --optimization=sgd --validation_interval=1 --log_runtime_stats_per_step=0 --augFlip=none --augHSVh=0 --augHSVs=0 --augHSVv=0"

2019-09-27 19:38:03 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:03.745362: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1

W0927 19:38:03.773541 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:03.745362: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1

2019-09-27 19:38:05 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: I0927 19:38:05.819633 140116389373760 main.py:425] Train batch size is 16 and validation batch size is 16

W0927 19:38:05.905564 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: I0927 19:38:05.819633 140116389373760 main.py:425] Train batch size is 16 and validation batch size is 16

2019-09-27 19:38:05 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: I0927 19:38:05.819813 140116389373760 main.py:429] Training epochs to be completed for each validation : 1

W0927 19:38:05.906068 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: I0927 19:38:05.819813 140116389373760 main.py:429] Training epochs to be completed for each validation : 1

2019-09-27 19:38:05 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: I0927 19:38:05.820301 140116389373760 main.py:433] Training epochs to be completed before taking a snapshot : 1.0

W0927 19:38:05.906439 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: I0927 19:38:05.820301 140116389373760 main.py:433] Training epochs to be completed before taking a snapshot : 1.0

2019-09-27 19:38:05 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: I0927 19:38:05.820651 140116389373760 main.py:437] Model weights will be saved as snapshot_<EPOCH>_Model.ckpt

W0927 19:38:05.906802 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: I0927 19:38:05.820651 140116389373760 main.py:437] Model weights will be saved as snapshot_<EPOCH>_Model.ckpt

2019-09-27 19:38:05 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: I0927 19:38:05.820871 140116389373760 main.py:450] Loading mean tensor from /workspace/jobs/20190927-192950-64cf/mean.binaryproto file

W0927 19:38:05.907147 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: I0927 19:38:05.820871 140116389373760 main.py:450] Loading mean tensor from /workspace/jobs/20190927-192950-64cf/mean.binaryproto file

2019-09-27 19:38:05 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: I0927 19:38:05.865480 140116389373760 main.py:456] Loading label definitions from /workspace/jobs/20190927-192950-64cf/labels.txt file

W0927 19:38:05.907476 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: I0927 19:38:05.865480 140116389373760 main.py:456] Loading label definitions from /workspace/jobs/20190927-192950-64cf/labels.txt file

2019-09-27 19:38:05 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: I0927 19:38:05.865655 140116389373760 main.py:462] Found 2 classes

W0927 19:38:05.907784 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: I0927 19:38:05.865655 140116389373760 main.py:462] Found 2 classes

W0927 19:38:06.009073 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: I0927 19:38:05.995457 140116389373760 tf_data.py:221] Found 5416 images in db /workspace/jobs/20190927-192950-64cf/train_db

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: I0927 19:38:05.995457 140116389373760 tf_data.py:221] Found 5416 images in db /workspace/jobs/20190927-192950-64cf/train_db

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:06.009669 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensor_slices(string_tensor).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs)`. If `shuffle=False`, omit the `.shuffle(...)`.

W0927 19:38:06.010023 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensor_slices(string_tensor).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs)`. If `shuffle=False`, omit the `.shuffle(...)`.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: W0927 19:38:05.996030 140116389373760 deprecation.py:323] From /opt/digits/digits/tools/tensorflow/tf_data.py:472: string_input_producer (from tensorflow.python.training.input) is deprecated and will be removed in a future version.

W0927 19:38:06.010365 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: W0927 19:38:05.996030 140116389373760 deprecation.py:323] From /opt/digits/digits/tools/tensorflow/tf_data.py:472: string_input_producer (from tensorflow.python.training.input) is deprecated and will be removed in a future version.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:06.010723 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensor_slices(string_tensor).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs)`. If `shuffle=False`, omit the `.shuffle(...)`.

W0927 19:38:06.011027 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensor_slices(string_tensor).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs)`. If `shuffle=False`, omit the `.shuffle(...)`.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:06.061865 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensor_slices(input_tensor).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs)`. If `shuffle=False`, omit the `.shuffle(...)`.

W0927 19:38:06.062249 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensor_slices(input_tensor).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs)`. If `shuffle=False`, omit the `.shuffle(...)`.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: W0927 19:38:06.031169 140116389373760 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/input.py:278: input_producer (from tensorflow.python.training.input) is deprecated and will be removed in a future version.

W0927 19:38:06.062603 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: W0927 19:38:06.031169 140116389373760 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/input.py:278: input_producer (from tensorflow.python.training.input) is deprecated and will be removed in a future version.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:06.062947 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensor_slices(input_tensor).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs)`. If `shuffle=False`, omit the `.shuffle(...)`.

W0927 19:38:06.063240 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensor_slices(input_tensor).shuffle(tf.shape(input_tensor, out_type=tf.int64)[0]).repeat(num_epochs)`. If `shuffle=False`, omit the `.shuffle(...)`.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:06.063683 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensors(tensor).repeat(num_epochs)`.

W0927 19:38:06.063982 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensors(tensor).repeat(num_epochs)`.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: W0927 19:38:06.032020 140116389373760 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/input.py:190: limit_epochs (from tensorflow.python.training.input) is deprecated and will be removed in a future version.

W0927 19:38:06.064273 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: W0927 19:38:06.032020 140116389373760 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/input.py:190: limit_epochs (from tensorflow.python.training.input) is deprecated and will be removed in a future version.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:06.064568 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensors(tensor).repeat(num_epochs)`.

W0927 19:38:06.064855 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.from_tensors(tensor).repeat(num_epochs)`.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:06.065270 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Prefer Dataset.range instead.

W0927 19:38:06.065560 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Prefer Dataset.range instead.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: W0927 19:38:06.035306 140116389373760 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/input.py:113: count_up_to (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.

W0927 19:38:06.065850 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: W0927 19:38:06.035306 140116389373760 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/input.py:113: count_up_to (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:06.066134 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Prefer Dataset.range instead.

W0927 19:38:06.066406 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Prefer Dataset.range instead.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:06.066852 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Prefer Dataset.range instead.

W0927 19:38:06.067118 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Prefer Dataset.range instead.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: W0927 19:38:06.035453 140116389373760 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py:2322: count_up_to (from tensorflow.python.ops.state_ops) is deprecated and will be removed in a future version.

W0927 19:38:06.067378 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: W0927 19:38:06.035453 140116389373760 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py:2322: count_up_to (from tensorflow.python.ops.state_ops) is deprecated and will be removed in a future version.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:06.067642 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Prefer Dataset.range instead.

W0927 19:38:06.067893 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Prefer Dataset.range instead.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:06.068263 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: To construct input pipelines, use the `tf.data` module.

W0927 19:38:06.068528 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: To construct input pipelines, use the `tf.data` module.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: W0927 19:38:06.038276 140116389373760 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/input.py:199: __init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.

W0927 19:38:06.068794 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: W0927 19:38:06.038276 140116389373760 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/input.py:199: __init__ (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:06.069062 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: To construct input pipelines, use the `tf.data` module.

W0927 19:38:06.069329 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: To construct input pipelines, use the `tf.data` module.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:06.069700 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: To construct input pipelines, use the `tf.data` module.

W0927 19:38:06.069962 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: To construct input pipelines, use the `tf.data` module.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: W0927 19:38:06.039386 140116389373760 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/input.py:199: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.

W0927 19:38:06.070219 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: W0927 19:38:06.039386 140116389373760 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/input.py:199: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:06.070478 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: To construct input pipelines, use the `tf.data` module.

W0927 19:38:06.070760 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: To construct input pipelines, use the `tf.data` module.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:06.071424 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: tf.py_func is deprecated in TF V2. Instead, there are two

W0927 19:38:06.071897 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: tf.py_func is deprecated in TF V2. Instead, there are two

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: options available in V2.

W0927 19:38:06.072227 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: options available in V2.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: - tf.py_function takes a python function which manipulates tf eager

W0927 19:38:06.072547 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: - tf.py_function takes a python function which manipulates tf eager

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: tensors instead of numpy arrays. It's easy to convert a tf eager tensor to

W0927 19:38:06.072848 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: tensors instead of numpy arrays. It's easy to convert a tf eager tensor to

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: an ndarray (just call tensor.numpy()) but having access to eager tensors

W0927 19:38:06.073157 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: an ndarray (just call tensor.numpy()) but having access to eager tensors

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: means `tf.py_function`s can use accelerators such as GPUs as well as

W0927 19:38:06.073462 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: means `tf.py_function`s can use accelerators such as GPUs as well as

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: being differentiable using a gradient tape.

W0927 19:38:06.073765 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: being differentiable using a gradient tape.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: - tf.numpy_function maintains the semantics of the deprecated tf.py_func

W0927 19:38:06.074062 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: - tf.numpy_function maintains the semantics of the deprecated tf.py_func

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: (it is not differentiable, and manipulates numpy arrays). It drops the

W0927 19:38:06.074354 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: (it is not differentiable, and manipulates numpy arrays). It drops the

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: stateful argument making all functions stateful.

W0927 19:38:06.074678 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: stateful argument making all functions stateful.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: W0927 19:38:06.043739 140116389373760 deprecation.py:323] From /opt/digits/digits/tools/tensorflow/tf_data.py:547: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.

W0927 19:38:06.125327 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: W0927 19:38:06.043739 140116389373760 deprecation.py:323] From /opt/digits/digits/tools/tensorflow/tf_data.py:547: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:06.125744 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: tf.py_func is deprecated in TF V2. Instead, there are two

W0927 19:38:06.126051 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: tf.py_func is deprecated in TF V2. Instead, there are two

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: options available in V2.

W0927 19:38:06.126352 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: options available in V2.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: - tf.py_function takes a python function which manipulates tf eager

W0927 19:38:06.126682 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: - tf.py_function takes a python function which manipulates tf eager

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: tensors instead of numpy arrays. It's easy to convert a tf eager tensor to

W0927 19:38:06.126966 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: tensors instead of numpy arrays. It's easy to convert a tf eager tensor to

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: an ndarray (just call tensor.numpy()) but having access to eager tensors

W0927 19:38:06.127243 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: an ndarray (just call tensor.numpy()) but having access to eager tensors

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: means `tf.py_function`s can use accelerators such as GPUs as well as

W0927 19:38:06.127531 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: means `tf.py_function`s can use accelerators such as GPUs as well as

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: being differentiable using a gradient tape.

W0927 19:38:06.127805 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: being differentiable using a gradient tape.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: - tf.numpy_function maintains the semantics of the deprecated tf.py_func

W0927 19:38:06.128091 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: - tf.numpy_function maintains the semantics of the deprecated tf.py_func

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: (it is not differentiable, and manipulates numpy arrays). It drops the

W0927 19:38:06.128364 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: (it is not differentiable, and manipulates numpy arrays). It drops the

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: stateful argument making all functions stateful.

W0927 19:38:06.128633 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: stateful argument making all functions stateful.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:06.179543 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Use `tf.cast` instead.

W0927 19:38:06.180011 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Use `tf.cast` instead.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: W0927 19:38:06.046334 140116389373760 deprecation.py:323] From /opt/digits/digits/tools/tensorflow/tf_data.py:274: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.

W0927 19:38:06.180298 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: W0927 19:38:06.046334 140116389373760 deprecation.py:323] From /opt/digits/digits/tools/tensorflow/tf_data.py:274: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:06.180577 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Use `tf.cast` instead.

W0927 19:38:06.180838 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Use `tf.cast` instead.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:06.181229 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.batch(batch_size)` (or `padded_batch(...)` if `dynamic_pad=True`).

W0927 19:38:06.181505 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.batch(batch_size)` (or `padded_batch(...)` if `dynamic_pad=True`).

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: W0927 19:38:06.049055 140116389373760 deprecation.py:323] From /opt/digits/digits/tools/tensorflow/tf_data.py:410: batch (from tensorflow.python.training.input) is deprecated and will be removed in a future version.

W0927 19:38:06.181839 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: W0927 19:38:06.049055 140116389373760 deprecation.py:323] From /opt/digits/digits/tools/tensorflow/tf_data.py:410: batch (from tensorflow.python.training.input) is deprecated and will be removed in a future version.

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:06.182107 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.batch(batch_size)` (or `padded_batch(...)` if `dynamic_pad=True`).

W0927 19:38:06.182368 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Queue-based input pipelines have been replaced by `tf.data`. Use `tf.data.Dataset.batch(batch_size)` (or `padded_batch(...)` if `dynamic_pad=True`).

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.070614: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199950000 Hz

W0927 19:38:06.182688 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.070614: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2199950000 Hz

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.073639: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5615a16948b0 executing computations on platform Host. Devices:

W0927 19:38:06.186625 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.073639: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5615a16948b0 executing computations on platform Host. Devices:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.073671: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>

W0927 19:38:06.187032 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.073671: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.077832: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1

W0927 19:38:06.187335 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.077832: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1

W0927 19:38:06.691137 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.654802: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5615a3d3b730 executing computations on platform CUDA. Devices:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.654802: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5615a3d3b730 executing computations on platform CUDA. Devices:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.654851: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce GTX 1080 Ti, Compute Capability 6.1

W0927 19:38:06.691665 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.654851: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce GTX 1080 Ti, Compute Capability 6.1

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.654859: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): GeForce GTX 1080 Ti, Compute Capability 6.1

W0927 19:38:06.692025 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.654859: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (1): GeForce GTX 1080 Ti, Compute Capability 6.1

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.665230: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:

W0927 19:38:06.692348 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.665230: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582

W0927 19:38:06.692661 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: pciBusID: 0000:0e:00.0

W0927 19:38:06.692962 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: pciBusID: 0000:0e:00.0

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.672769: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:

W0927 19:38:06.693264 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.672769: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582

W0927 19:38:06.693561 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: pciBusID: 0000:0f:00.0

W0927 19:38:06.693856 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: pciBusID: 0000:0f:00.0

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.672826: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1

W0927 19:38:06.694154 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.672826: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.676462: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10

W0927 19:38:06.694454 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.676462: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.679495: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10

W0927 19:38:06.694814 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.679495: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.680043: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10

W0927 19:38:06.695116 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.680043: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.683681: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10

W0927 19:38:06.695425 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.683681: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.685720: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10

W0927 19:38:06.695733 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.685720: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.693149: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7

W0927 19:38:06.696048 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.693149: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.717250: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1

W0927 19:38:06.746706 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.717250: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1

2019-09-27 19:38:06 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.717300: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1

W0927 19:38:06.747087 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:06.717300: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1

2019-09-27 19:38:07 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:07.803751: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:

W0927 19:38:07.908510 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:07.803751: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:

2019-09-27 19:38:07 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:07.803787: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1

W0927 19:38:07.908934 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:07.803787: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1

2019-09-27 19:38:07 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:07.803795: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y

W0927 19:38:07.909193 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:07.803795: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y

2019-09-27 19:38:07 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:07.803800: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N

W0927 19:38:07.909436 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:07.803800: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N

2019-09-27 19:38:07 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:07.833233: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:0 with 10283 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0e:00.0, compute capability: 6.1)

W0927 19:38:07.909702 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:07.833233: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:0 with 10283 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0e:00.0, compute capability: 6.1)

2019-09-27 19:38:07 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:07.841609: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:1 with 10411 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:0f:00.0, compute capability: 6.1)

W0927 19:38:07.909945 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:07.841609: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:1 with 10411 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:0f:00.0, compute capability: 6.1)

2019-09-27 19:38:07 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: W0927 19:38:07.853491 140116389373760 deprecation_wrapper.py:119] From /opt/digits/digits/tools/tensorflow/model.py:156: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0927 19:38:07.960506 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: W0927 19:38:07.853491 140116389373760 deprecation_wrapper.py:119] From /opt/digits/digits/tools/tensorflow/model.py:156: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

2019-09-27 19:38:08 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:08.363987 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:08 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Use keras.layers.flatten instead.

W0927 19:38:08.364383 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Use keras.layers.flatten instead.

2019-09-27 19:38:08 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: W0927 19:38:08.316798 140116389373760 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/layers.py:1637: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.

W0927 19:38:08.364715 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: W0927 19:38:08.316798 140116389373760 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/layers.py:1637: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.

2019-09-27 19:38:08 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:08.365031 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:08 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Use keras.layers.flatten instead.

W0927 19:38:08.365331 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Use keras.layers.flatten instead.

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: W0927 19:38:09.128760 140116389373760 deprecation_wrapper.py:119] From /opt/digits/digits/tools/tensorflow/model.py:160: The name tf.get_variable_scope is deprecated. Please use tf.compat.v1.get_variable_scope instead.

W0927 19:38:09.221976 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: W0927 19:38:09.128760 140116389373760 deprecation_wrapper.py:119] From /opt/digits/digits/tools/tensorflow/model.py:160: The name tf.get_variable_scope is deprecated. Please use tf.compat.v1.get_variable_scope instead.

W0927 19:38:09.323100 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: W0927 19:38:09.136358 140116389373760 deprecation_wrapper.py:119] From /opt/digits/digits/tools/tensorflow/model.py:300: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: W0927 19:38:09.136358 140116389373760 deprecation_wrapper.py:119] From /opt/digits/digits/tools/tensorflow/model.py:300: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: W0927 19:38:09.136517 140116389373760 deprecation_wrapper.py:119] From /opt/digits/digits/tools/tensorflow/model.py:300: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.

W0927 19:38:09.424246 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: W0927 19:38:09.136517 140116389373760 deprecation_wrapper.py:119] From /opt/digits/digits/tools/tensorflow/model.py:300: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: W0927 19:38:09.136657 140116389373760 deprecation_wrapper.py:119] From /opt/digits/digits/tools/tensorflow/model.py:168: The name tf.add_to_collection is deprecated. Please use tf.compat.v1.add_to_collection instead.

W0927 19:38:09.525564 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: W0927 19:38:09.136657 140116389373760 deprecation_wrapper.py:119] From /opt/digits/digits/tools/tensorflow/model.py:168: The name tf.add_to_collection is deprecated. Please use tf.compat.v1.add_to_collection instead.

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: W0927 19:38:09.137783 140116389373760 deprecation_wrapper.py:119] From /opt/digits/digits/tools/tensorflow/model.py:176: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

W0927 19:38:09.626677 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: W0927 19:38:09.137783 140116389373760 deprecation_wrapper.py:119] From /opt/digits/digits/tools/tensorflow/model.py:176: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: I0927 19:38:09.139045 140116389373760 model.py:265] Optimizer:sgd

W0927 19:38:09.677603 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: I0927 19:38:09.139045 140116389373760 model.py:265] Optimizer:sgd

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: W0927 19:38:09.139211 140116389373760 deprecation_wrapper.py:119] From /opt/digits/digits/tools/tensorflow/model.py:267: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

W0927 19:38:09.728379 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: W0927 19:38:09.139211 140116389373760 deprecation_wrapper.py:119] From /opt/digits/digits/tools/tensorflow/model.py:267: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: I0927 19:38:09.608761 140116389373760 tf_data.py:221] Found 1806 images in db /workspace/jobs/20190927-192950-64cf/val_db

W0927 19:38:09.778973 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: I0927 19:38:09.608761 140116389373760 tf_data.py:221] Found 1806 images in db /workspace/jobs/20190927-192950-64cf/val_db

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.651412: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:

W0927 19:38:09.779223 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.651412: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582

W0927 19:38:09.779424 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: pciBusID: 0000:0e:00.0

W0927 19:38:09.779649 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: pciBusID: 0000:0e:00.0

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.655684: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:

W0927 19:38:09.779855 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.655684: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582

W0927 19:38:09.780064 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: pciBusID: 0000:0f:00.0

W0927 19:38:09.780268 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: pciBusID: 0000:0f:00.0

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.655714: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1

W0927 19:38:09.780467 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.655714: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.655762: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10

W0927 19:38:09.780663 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.655762: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.655778: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10

W0927 19:38:09.780868 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.655778: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.655792: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10

W0927 19:38:09.781069 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.655792: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.655816: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10

W0927 19:38:09.781261 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.655816: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.655835: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10

W0927 19:38:09.781426 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.655835: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.655850: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7

W0927 19:38:09.781610 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.655850: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.672596: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1

W0927 19:38:09.781796 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.672596: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.672686: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:

W0927 19:38:09.781981 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.672686: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.672694: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1

W0927 19:38:09.782145 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.672694: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.672700: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y

W0927 19:38:09.782306 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.672700: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.672704: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N

W0927 19:38:09.782469 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.672704: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.690711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:0 with 10283 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0e:00.0, compute capability: 6.1)

W0927 19:38:09.782646 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.690711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:0 with 10283 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0e:00.0, compute capability: 6.1)

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.695934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:1 with 10411 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:0f:00.0, compute capability: 6.1)

W0927 19:38:09.782859 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.695934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/device:GPU:1 with 10411 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:0f:00.0, compute capability: 6.1)

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.860220: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582

W0927 19:38:09.883517 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.860220: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:

W0927 19:38:09.883759 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: pciBusID: 0000:0e:00.0

W0927 19:38:09.883966 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: pciBusID: 0000:0e:00.0

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.864605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:

W0927 19:38:09.884160 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.864605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582

W0927 19:38:09.884380 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: pciBusID: 0000:0f:00.0

W0927 19:38:09.884586 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: pciBusID: 0000:0f:00.0

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.864643: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1

W0927 19:38:09.884799 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.864643: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.864677: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10

W0927 19:38:09.884998 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.864677: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.864691: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10

W0927 19:38:09.885205 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.864691: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.864704: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10

W0927 19:38:09.885406 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.864704: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.864716: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10

W0927 19:38:09.885607 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.864716: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.864729: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10

W0927 19:38:09.885788 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.864729: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.864742: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7

W0927 19:38:09.885987 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.864742: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.880904: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1

W0927 19:38:09.886182 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.880904: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0, 1

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.880975: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:

W0927 19:38:09.886374 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.880975: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.880982: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1

W0927 19:38:09.886594 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.880982: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 1

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.880987: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y

W0927 19:38:09.886794 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.880987: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N Y

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.880991: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N

W0927 19:38:09.886995 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.880991: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1:   Y N

W0927 19:38:09.983083 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.895784: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10283 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0e:00.0, compute capability: 6.1)

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.895784: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10283 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0e:00.0, compute capability: 6.1)

2019-09-27 19:38:09 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.901669: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10411 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:0f:00.0, compute capability: 6.1)

W0927 19:38:09.983329 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:09.901669: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 10411 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:0f:00.0, compute capability: 6.1)

2019-09-27 19:38:10 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: W0927 19:38:09.905936 140116389373760 deprecation_wrapper.py:119] From /opt/digits/digits/tools/tensorflow/main.py:548: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0927 19:38:10.034274 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: W0927 19:38:09.905936 140116389373760 deprecation_wrapper.py:119] From /opt/digits/digits/tools/tensorflow/main.py:548: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

2019-09-27 19:38:10 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:10.056024: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.

W0927 19:38:10.084955 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:10.056024: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.

2019-09-27 19:38:12 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: W0927 19:38:12.469755 140116389373760 deprecation_wrapper.py:119] From /opt/digits/digits/tools/tensorflow/main.py:559: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

W0927 19:38:12.538279 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: W0927 19:38:12.469755 140116389373760 deprecation_wrapper.py:119] From /opt/digits/digits/tools/tensorflow/main.py:559: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

2019-09-27 19:38:12 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: I0927 19:38:12.765419 140116389373760 model.py:203] Starting queue runners (val)

W0927 19:38:12.791570 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: I0927 19:38:12.765419 140116389373760 model.py:203] Starting queue runners (val)

2019-09-27 19:38:12 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:12.792138 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:12 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: To construct input pipelines, use the `tf.data` module.

W0927 19:38:12.792433 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: To construct input pipelines, use the `tf.data` module.

2019-09-27 19:38:12 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: W0927 19:38:12.766309 140116389373760 deprecation.py:323] From /opt/digits/digits/tools/tensorflow/model.py:212: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.

W0927 19:38:12.792709 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: W0927 19:38:12.766309 140116389373760 deprecation.py:323] From /opt/digits/digits/tools/tensorflow/model.py:212: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.

2019-09-27 19:38:12 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: Instructions for updating:

W0927 19:38:12.792974 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: Instructions for updating:

2019-09-27 19:38:12 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: To construct input pipelines, use the `tf.data` module.

W0927 19:38:12.793224 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: To construct input pipelines, use the `tf.data` module.

2019-09-27 19:38:12 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: I0927 19:38:12.785586 140116389373760 model.py:213] Queue runners started (val)

W0927 19:38:12.793592 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: I0927 19:38:12.785586 140116389373760 model.py:213] Queue runners started (val)

2019-09-27 19:38:12 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: W0927 19:38:12.785973 140116389373760 deprecation_wrapper.py:119] From /opt/digits/digits/tools/tensorflow/model.py:245: The name tf.summary.merge is deprecated. Please use tf.compat.v1.summary.merge instead.

W0927 19:38:12.844299 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: W0927 19:38:12.785973 140116389373760 deprecation_wrapper.py:119] From /opt/digits/digits/tools/tensorflow/model.py:245: The name tf.summary.merge is deprecated. Please use tf.compat.v1.summary.merge instead.

2019-09-27 19:38:13 [20190927-193801-dcf2] [WARNING] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:13.393811: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10

W0927 19:38:13.397892 140597179560544 task.py:242] Train Tensorflow Model unrecognized output: 2019-09-27 19:38:13.393811: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10

Traceback (most recent call last):

  File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run

  File "/opt/digits/digits/model/tasks/train.py", line 220, in hw_socketio_updater

    nvml_info = device_query.get_nvml_info(index)

  File "/opt/digits/digits/device_query.py", line 252, in get_nvml_info

    raise RuntimeError('nvmlInit() failed with error #%s' % rc)

RuntimeError: nvmlInit() failed with error #999

2019-09-27T19:38:21Z <Greenlet at 0x7fdf54f1b578: <bound method TensorflowTrainTask.hw_socketio_updater of <digits.model.tasks.tensorflow_train.TensorflowTrainTask object at 0x7fdf54a29790>>(['0', '1'])> failed with RuntimeError

2019-09-27 19:38:21 [20190927-193801-dcf2] [ERROR] Train Tensorflow Model task failed with error code -9

E0927 19:38:21.842375 140597179560544 task.py:261] Train Tensorflow Model task failed with error code -9
Snapple49 commented 5 years ago

So after running some more tests, I figured out that part of the problem was me not giving enough ram to the pod. I guess trying to run AlexNet on 10k images with 2GB of ram was rather ambitious... However, even with about 200 images it crashes unless I give the pods about 8GB ram, is this normal behaviour? This was again just my custom test, I did not yet test the included mnist example with more ram, I'll report back on that tomorrow.