Open AndreyKudryavtsev opened 5 years ago
Team, anyone can please take a look on that or share their experience running the stack with single GPU? Thank you.
This is probably too late, but for anyone else that might run into the same problem, you could lower the batch_size=128
argument in the 01_compute_input_embeddings.py
.
Hopefully, I'm not facing any simple mistake here. I wanted to practise creating my own sounds and followed online instructions. I'm running Core-i7 desktop with GTX1080. However it fails on the first script, and looks memory is not enough. I set up # of GPUs in json file. Not sure if I missed anything else. Please, support. { "pads": { "NW": ["resopad"], "NE": ["flute"], "SW": ["rhodes"], "SE": ["tubularbells"] },
}
[root@localhost workdir]# python ./01_compute_input_embeddings.py /usr/lib64/python2.7/site-packages/scipy/sparse/lil.py:19: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import _csparsetools /usr/lib64/python2.7/site-packages/scipy/sparse/csgraph/init.py:165: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._shortest_path import shortest_path, floyd_warshall, dijkstra,\ /usr/lib64/python2.7/site-packages/scipy/sparse/csgraph/_validation.py:5: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._tools import csgraph_to_dense, csgraph_from_dense,\ /usr/lib64/python2.7/site-packages/scipy/sparse/csgraph/init.py:167: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._traversal import breadth_first_order, depth_first_order, \ /usr/lib64/python2.7/site-packages/scipy/sparse/csgraph/init.py:169: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._min_spanning_tree import minimum_spanning_tree /usr/lib64/python2.7/site-packages/scipy/sparse/csgraph/init.py:170: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._reordering import reverse_cuthill_mckee, maximum_bipartite_matching, \ /usr/lib64/python2.7/site-packages/scipy/linalg/basic.py:17: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._solve_toeplitz import levinson /usr/lib64/python2.7/site-packages/scipy/linalg/init.py:207: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._decomp_update import /usr/lib64/python2.7/site-packages/scipy/special/init.py:640: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._ufuncs import /usr/lib64/python2.7/site-packages/scipy/special/_ellip_harm.py:7: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._ellip_harm_2 import _ellipsoid, _ellipsoid_norm /usr/lib64/python2.7/site-packages/scipy/interpolate/_bsplines.py:10: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import _bspl /usr/lib64/python2.7/site-packages/scipy/spatial/init.py:95: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .ckdtree import /usr/lib64/python2.7/site-packages/scipy/spatial/init.py:96: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .qhull import /usr/lib64/python2.7/site-packages/scipy/spatial/_spherical_voronoi.py:18: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import _voronoi /usr/lib64/python2.7/site-packages/scipy/spatial/distance.py:122: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import _hausdorff /usr/lib64/python2.7/site-packages/scipy/ndimage/measurements.py:36: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import _ni_label /usr/lib64/python2.7/site-packages/pandas/_libs/init.py:4: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .tslib import iNaT, NaT, Timestamp, Timedelta, OutOfBoundsDatetime /usr/lib64/python2.7/site-packages/pandas/init.py:26: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from pandas._libs import (hashtable as _hashtable, /usr/lib64/python2.7/site-packages/pandas/core/dtypes/common.py:6: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from pandas._libs import algos, lib /usr/lib64/python2.7/site-packages/pandas/core/util/hashing.py:7: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from pandas._libs import hashing, tslib /usr/lib64/python2.7/site-packages/pandas/core/indexes/base.py:7: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from pandas._libs import (lib, index as libindex, tslib as libts, /usr/lib64/python2.7/site-packages/pandas/tseries/offsets.py:21: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 import pandas._libs.tslibs.offsets as liboffsets /usr/lib64/python2.7/site-packages/pandas/core/ops.py:16: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from pandas._libs import algos as libalgos, ops as libops /usr/lib64/python2.7/site-packages/pandas/core/indexes/interval.py:32: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from pandas._libs.interval import ( /usr/lib64/python2.7/site-packages/pandas/core/internals.py:14: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from pandas._libs import internals as libinternals /usr/lib64/python2.7/site-packages/pandas/core/sparse/array.py:33: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 import pandas._libs.sparse as splib /usr/lib64/python2.7/site-packages/pandas/core/window.py:36: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 import pandas._libs.window as _window /usr/lib64/python2.7/site-packages/pandas/core/groupby/groupby.py:68: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from pandas._libs import (lib, reduction, /usr/lib64/python2.7/site-packages/pandas/core/reshape/reshape.py:30: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from pandas._libs import algos as _algos, reshape as _reshape /usr/lib64/python2.7/site-packages/pandas/io/parsers.py:45: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 import pandas._libs.parsers as parsers /usr/lib64/python2.7/site-packages/pandas/io/pytables.py:50: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from pandas._libs import algos, lib, writers as libwriters /usr/lib64/python2.7/site-packages/scipy/signal/_max_len_seq.py:8: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._max_len_seq_inner import _max_len_seq_inner /usr/lib64/python2.7/site-packages/scipy/signal/_upfirdn.py:36: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._upfirdn_apply import _output_len, _apply /usr/lib64/python2.7/site-packages/scipy/optimize/_trlib/init.py:1: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._trlib import TRLIBQuadraticSubproblem /usr/lib64/python2.7/site-packages/scipy/optimize/_numdiff.py:10: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._group_columns import group_dense, group_sparse /usr/lib64/python2.7/site-packages/scipy/signal/spectral.py:10: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._spectral import _lombscargle /usr/lib64/python2.7/site-packages/scipy/stats/_continuous_distns.py:18: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import _stats /usr/lib64/python2.7/site-packages/scipy/signal/_peak_finding.py:13: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._peak_finding_utils import (_argmaxima1d, _select_by_peak_distance, /usr/lib64/python2.7/site-packages/sklearn/utils/init.py:9: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .murmurhash import murmurhash3_32 /usr/lib64/python2.7/site-packages/sklearn/utils/extmath.py:24: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._logistic_sigmoid import _log_logistic_sigmoid /usr/lib64/python2.7/site-packages/sklearn/utils/extmath.py:26: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .sparsefuncs_fast import csr_row_norms /usr/lib64/python2.7/site-packages/sklearn/metrics/cluster/supervised.py:23: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .expected_mutual_info_fast import expected_mutual_information /usr/lib64/python2.7/site-packages/sklearn/metrics/pairwise.py:30: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .pairwise_fast import _chi2_kernel_fast, _sparse_manhattan /usr/lib64/python2.7/site-packages/sklearn/linear_model/base.py:35: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ..utils.seq_dataset import ArrayDataset, CSRDataset /usr/lib64/python2.7/site-packages/sklearn/linear_model/least_angle.py:23: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ..utils import arrayfuncs, as_float_array, check_X_y, deprecated /usr/lib64/python2.7/site-packages/sklearn/utils/random.py:10: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._random import sample_without_replacement /usr/lib64/python2.7/site-packages/sklearn/linear_model/coordinate_descent.py:29: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import cd_fast /usr/lib64/python2.7/site-packages/sklearn/linear_model/init.py:22: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .sgd_fast import Hinge, Log, ModifiedHuber, SquaredLoss, Huber /usr/lib64/python2.7/site-packages/sklearn/linear_model/sag.py:12: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .sag_fast import sag /usr/lib64/python2.7/site-packages/sklearn/svm/base.py:8: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import libsvm, liblinear /usr/lib64/python2.7/site-packages/sklearn/svm/base.py:9: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import libsvm_sparse /usr/lib64/python2.7/site-packages/sklearn/neighbors/init.py:6: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .ball_tree import BallTree /usr/lib64/python2.7/site-packages/sklearn/neighbors/init.py:7: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .kd_tree import KDTree /usr/lib64/python2.7/site-packages/sklearn/decomposition/online_lda.py:28: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._online_lda import (mean_change, _dirichlet_expectation_1d, /usr/lib64/python2.7/site-packages/sklearn/utils/graph.py:15: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .graph_shortest_path import graph_shortest_path # noqa /usr/lib64/python2.7/site-packages/sklearn/isotonic.py:12: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._isotonic import _inplace_contiguous_isotonic_regression, _make_unique /usr/lib64/python2.7/site-packages/sklearn/manifold/t_sne.py:24: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import _utils /usr/lib64/python2.7/site-packages/sklearn/manifold/t_sne.py:25: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import _barnes_hut_tsne /usr/lib64/python2.7/site-packages/sklearn/tree/tree.py:40: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._criterion import Criterion /usr/lib64/python2.7/site-packages/sklearn/cluster/kmeans.py:35: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import _k_means /usr/lib64/python2.7/site-packages/sklearn/cluster/kmeans.py:36: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._k_means_elkan import k_means_elkan /usr/lib64/python2.7/site-packages/sklearn/cluster/hierarchical.py:23: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import hierarchical /usr/lib64/python2.7/site-packages/sklearn/cluster/dbscan.py:19: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._dbscan_inner import dbscan_inner /usr/lib64/python2.7/site-packages/sklearn/feature_extraction/hashing.py:10: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import _hashing /usr/lib64/python2.7/site-packages/scipy/io/matlab/mio4.py:18: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .mio_utils import squeeze_element, chars_to_strings /usr/lib64/python2.7/site-packages/scipy/io/matlab/mio5.py:98: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .mio5_utils import VarReader5 WARNING:tensorflow:From /usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/masked.py:116: init (from tensorflow.python.ops.init_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.initializers.variance_scaling instead with distribution=uniform to get equivalent behavior. INFO:tensorflow:Will restore from checkpoint: /root/magenta/magenta/models/nsynth/wavenet-ckpt/model.ckpt-200000 INFO:tensorflow:Will load Wavs from /mnt/ssd/open-nsynth-super-master/audio/workdir/audio_input. INFO:tensorflow:Will save embeddings to /mnt/ssd/open-nsynth-super-master/audio/workdir/embeddings_input. INFO:tensorflow:On file number 0 (batch 1). 2018-09-25 09:04:29.767892: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA 2018-09-25 09:04:29.914347: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1392] Found device 0 with properties: name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.7715 pciBusID: 0000:65:00.0 totalMemory: 7.93GiB freeMemory: 7.61GiB 2018-09-25 09:04:29.914394: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0 2018-09-25 09:04:30.144628: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix: 2018-09-25 09:04:30.144675: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958] 0 2018-09-25 09:04:30.144685: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: N 2018-09-25 09:04:30.144858: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7343 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:65:00.0, compute capability: 6.1) INFO:tensorflow:Restoring parameters from /root/magenta/magenta/models/nsynth/wavenet-ckpt/model.ckpt-200000 2018-09-25 09:04:36.170063: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.95GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2018-09-25 09:04:46.411265: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.95GiB. Current allocation summary follows. 2018-09-25 09:04:46.411396: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (256): Total Chunks: 6, Chunks in use: 5. 1.5KiB allocated for chunks. 1.2KiB in use in bin. 80B client-requested in use in bin. 2018-09-25 09:04:46.411475: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (512): Total Chunks: 62, Chunks in use: 61. 31.0KiB allocated for chunks. 30.5KiB in use in bin. 30.5KiB client-requested in use in bin. 2018-09-25 09:04:46.411519: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1024): Total Chunks: 36, Chunks in use: 36. 37.2KiB allocated for chunks. 37.2KiB in use in bin. 36.5KiB client-requested in use in bin. 2018-09-25 09:04:46.411560: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2048): Total Chunks: 37, Chunks in use: 31. 76.0KiB allocated for chunks. 63.5KiB in use in bin. 62.0KiB client-requested in use in bin. 2018-09-25 09:04:46.411609: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4096): Total Chunks: 62, Chunks in use: 61. 252.5KiB allocated for chunks. 248.5KiB in use in bin. 246.0KiB client-requested in use in bin. 2018-09-25 09:04:46.411653: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8192): Total Chunks: 1, Chunks in use: 1. 8.0KiB allocated for chunks. 8.0KiB in use in bin. 8.0KiB client-requested in use in bin. 2018-09-25 09:04:46.411701: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16384): Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin. 2018-09-25 09:04:46.411746: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (32768): Total Chunks: 1, Chunks in use: 0. 58.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-09-25 09:04:46.411785: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (65536): Total Chunks: 60, Chunks in use: 60. 3.86MiB allocated for chunks. 3.86MiB in use in bin. 3.75MiB client-requested in use in bin. 2018-09-25 09:04:46.411850: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (131072): Total Chunks: 33, Chunks in use: 30. 6.12MiB allocated for chunks. 5.68MiB in use in bin. 5.62MiB client-requested in use in bin. 2018-09-25 09:04:46.411891: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (262144): Total Chunks: 3, Chunks in use: 3. 888.0KiB allocated for chunks. 888.0KiB in use in bin. 704.0KiB client-requested in use in bin. 2018-09-25 09:04:46.411939: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (524288): Total Chunks: 35, Chunks in use: 31. 19.74MiB allocated for chunks. 17.00MiB in use in bin. 15.50MiB client-requested in use in bin. 2018-09-25 09:04:46.411985: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1048576): Total Chunks: 33, Chunks in use: 30. 33.29MiB allocated for chunks. 30.29MiB in use in bin. 30.00MiB client-requested in use in bin. 2018-09-25 09:04:46.412031: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2097152): Total Chunks: 1, Chunks in use: 0. 3.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-09-25 09:04:46.412076: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4194304): Total Chunks: 30, Chunks in use: 28. 181.91MiB allocated for chunks. 169.91MiB in use in bin. 168.00MiB client-requested in use in bin. 2018-09-25 09:04:46.412125: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8388608): Total Chunks: 2, Chunks in use: 2. 17.75MiB allocated for chunks. 17.75MiB in use in bin. 12.00MiB client-requested in use in bin. 2018-09-25 09:04:46.412170: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-09-25 09:04:46.412227: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-09-25 09:04:46.412271: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-09-25 09:04:46.412316: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (134217728): Total Chunks: 1, Chunks in use: 0. 244.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-09-25 09:04:46.412361: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (268435456): Total Chunks: 4, Chunks in use: 3. 6.67GiB allocated for chunks. 5.95GiB in use in bin. 5.86GiB client-requested in use in bin. 2018-09-25 09:04:46.412406: I tensorflow/core/common_runtime/bfc_allocator.cc:646] Bin for 1.95GiB was 256.00MiB, Chunk State: 2018-09-25 09:04:46.412448: I tensorflow/core/common_runtime/bfc_allocator.cc:652] Size: 736.38MiB | Requested Size: 15.62MiB | in_use: 0, prev: Size: 1.95GiB | Requested Size: 1.95GiB | in_use: 1 2018-09-25 09:04:46.412491: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe55a000000 of size 2097152000 2018-09-25 09:04:46.412522: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe5d7000000 of size 772151552 2018-09-25 09:04:46.412553: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe6b6000000 of size 2147483648 2018-09-25 09:04:46.412586: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe73c000000 of size 2147483648 2018-09-25 09:04:46.412617: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7bc000000 of size 6291456 2018-09-25 09:04:46.412647: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7bc600000 of size 6291456 2018-09-25 09:04:46.412676: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7bcc00000 of size 255852544 2018-09-25 09:04:46.412706: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cc000000 of size 1048576 2018-09-25 09:04:46.412737: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cc100000 of size 1048576 2018-09-25 09:04:46.412766: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cc200000 of size 1048576 2018-09-25 09:04:46.412796: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cc300000 of size 1048576 2018-09-25 09:04:46.412835: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cc400000 of size 1048576 2018-09-25 09:04:46.412866: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cc500000 of size 1048576 2018-09-25 09:04:46.412896: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cc600000 of size 1048576 2018-09-25 09:04:46.412927: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cc700000 of size 1048576 2018-09-25 09:04:46.412956: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7cc800000 of size 1048576 2018-09-25 09:04:46.412986: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cc900000 of size 1048576 2018-09-25 09:04:46.413017: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cca00000 of size 196608 2018-09-25 09:04:46.413049: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7cca30000 of size 851968 2018-09-25 09:04:46.413079: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ccb00000 of size 1048576 2018-09-25 09:04:46.413109: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ccc00000 of size 6291456 2018-09-25 09:04:46.413138: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cd200000 of size 6291456 2018-09-25 09:04:46.413171: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cd800000 of size 6291456 2018-09-25 09:04:46.413202: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cde00000 of size 6291456 2018-09-25 09:04:46.413230: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ce400000 of size 6291456 2018-09-25 09:04:46.413261: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cea00000 of size 6291456 2018-09-25 09:04:46.413289: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cf000000 of size 6291456 2018-09-25 09:04:46.413319: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cf600000 of size 6291456 2018-09-25 09:04:46.413347: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cfc00000 of size 6291456 2018-09-25 09:04:46.413377: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7d0200000 of size 6291456 2018-09-25 09:04:46.413408: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d0800000 of size 2048 2018-09-25 09:04:46.413440: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d0800800 of size 4096 2018-09-25 09:04:46.413474: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d0801800 of size 2048 2018-09-25 09:04:46.413503: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d0802000 of size 4096 2018-09-25 09:04:46.413532: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d0803000 of size 249856 2018-09-25 09:04:46.413561: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d0840000 of size 6291456 2018-09-25 09:04:46.413592: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d0e40000 of size 65536 2018-09-25 09:04:46.413621: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d0e50000 of size 196608 2018-09-25 09:04:46.413650: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d0e80000 of size 1048576 2018-09-25 09:04:46.413681: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d0f80000 of size 1048576 2018-09-25 09:04:46.413709: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1080000 of size 1048576 2018-09-25 09:04:46.413738: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1180000 of size 1048576 2018-09-25 09:04:46.413767: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1280000 of size 2048 2018-09-25 09:04:46.413797: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1280800 of size 1046528 2018-09-25 09:04:46.413832: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7d1380000 of size 1048576 2018-09-25 09:04:46.413862: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1480000 of size 1048576 2018-09-25 09:04:46.413892: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1580000 of size 1048576 2018-09-25 09:04:46.413922: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1680000 of size 2048 2018-09-25 09:04:46.413951: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1680800 of size 4096 2018-09-25 09:04:46.413983: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7d1681800 of size 59392 2018-09-25 09:04:46.414012: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1690000 of size 524288 2018-09-25 09:04:46.414041: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1710000 of size 524288 2018-09-25 09:04:46.414070: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1790000 of size 1048576 2018-09-25 09:04:46.414100: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1890000 of size 524288 2018-09-25 09:04:46.414128: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1910000 of size 524288 2018-09-25 09:04:46.414158: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1990000 of size 1048576 2018-09-25 09:04:46.414187: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1a90000 of size 1048576 2018-09-25 09:04:46.414218: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1b90000 of size 1048576 2018-09-25 09:04:46.414246: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7d1c90000 of size 3145728 2018-09-25 09:04:46.414275: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1f90000 of size 1048576 2018-09-25 09:04:46.414305: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2090000 of size 1048576 2018-09-25 09:04:46.414334: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2190000 of size 65536 2018-09-25 09:04:46.414363: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d21a0000 of size 6291456 2018-09-25 09:04:46.414394: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d27a0000 of size 1024 2018-09-25 09:04:46.414422: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d27a0400 of size 524288 2018-09-25 09:04:46.414451: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2820400 of size 512 2018-09-25 09:04:46.414480: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2820600 of size 512 2018-09-25 09:04:46.414511: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2820800 of size 8192 2018-09-25 09:04:46.414540: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2822800 of size 196608 2018-09-25 09:04:46.414570: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2852800 of size 319488 2018-09-25 09:04:46.414599: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d28a0800 of size 256 2018-09-25 09:04:46.414628: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d28a0900 of size 256 2018-09-25 09:04:46.414658: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d28a0a00 of size 512 2018-09-25 09:04:46.414687: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d28a0c00 of size 524288 2018-09-25 09:04:46.414717: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2920c00 of size 1024 2018-09-25 09:04:46.414747: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2921000 of size 524288 2018-09-25 09:04:46.414776: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d29a1000 of size 65536 2018-09-25 09:04:46.414804: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d29b1000 of size 512 2018-09-25 09:04:46.414844: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d29b1200 of size 512 2018-09-25 09:04:46.414873: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d29b1400 of size 524288 2018-09-25 09:04:46.414902: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2a31400 of size 1024 2018-09-25 09:04:46.414932: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2a31800 of size 196608 2018-09-25 09:04:46.414962: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2a61800 of size 327680 2018-09-25 09:04:46.414992: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7d2ab1800 of size 524288 2018-09-25 09:04:46.415022: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2b31800 of size 512 2018-09-25 09:04:46.415052: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2b31a00 of size 512 2018-09-25 09:04:46.415080: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2b31c00 of size 512 2018-09-25 09:04:46.415110: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2b31e00 of size 512 2018-09-25 09:04:46.415138: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2b32000 of size 524288 2018-09-25 09:04:46.415167: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2bb2000 of size 1024 2018-09-25 09:04:46.415197: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2bb2400 of size 524288 2018-09-25 09:04:46.415228: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2c32400 of size 1024 2018-09-25 09:04:46.415256: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2c32800 of size 524288 2018-09-25 09:04:46.415286: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2cb2800 of size 1024 2018-09-25 09:04:46.415315: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2cb2c00 of size 524288 2018-09-25 09:04:46.415343: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2d32c00 of size 524288 2018-09-25 09:04:46.415373: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2db2c00 of size 1024 2018-09-25 09:04:46.415402: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2db3000 of size 1024 2018-09-25 09:04:46.415433: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2db3400 of size 524288 2018-09-25 09:04:46.415462: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2e33400 of size 1024 2018-09-25 09:04:46.415492: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2e33800 of size 512 2018-09-25 09:04:46.415520: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2e33a00 of size 524288 2018-09-25 09:04:46.415548: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7d2eb3a00 of size 525312 2018-09-25 09:04:46.415577: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2f33e00 of size 4096 2018-09-25 09:04:46.415607: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2f34e00 of size 2048 2018-09-25 09:04:46.415637: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2f35600 of size 3072 2018-09-25 09:04:46.415666: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2f36200 of size 196864 2018-09-25 09:04:46.415694: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2f66300 of size 512 2018-09-25 09:04:46.415724: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2f66500 of size 196608 2018-09-25 09:04:46.415753: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2f96500 of size 512 2018-09-25 09:04:46.415784: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2f96700 of size 197120 2018-09-25 09:04:46.415819: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2fc6900 of size 196608 2018-09-25 09:04:46.415843: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2ff6900 of size 512 2018-09-25 09:04:46.415868: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2ff6b00 of size 65536 2018-09-25 09:04:46.415897: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7d3006b00 of size 131584 2018-09-25 09:04:46.415927: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3026d00 of size 65536 2018-09-25 09:04:46.415957: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3036d00 of size 4096 2018-09-25 09:04:46.415986: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3037d00 of size 196608 2018-09-25 09:04:46.416015: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3067d00 of size 65536 2018-09-25 09:04:46.416045: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3077d00 of size 65536 2018-09-25 09:04:46.416070: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3087d00 of size 65536 2018-09-25 09:04:46.416099: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3097d00 of size 196608 2018-09-25 09:04:46.416130: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d30c7d00 of size 196608 2018-09-25 09:04:46.416159: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d30f7d00 of size 196608 2018-09-25 09:04:46.416189: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3127d00 of size 196608 2018-09-25 09:04:46.416218: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3157d00 of size 196608 2018-09-25 09:04:46.416247: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3187d00 of size 196608 2018-09-25 09:04:46.416276: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d31b7d00 of size 196608 2018-09-25 09:04:46.416305: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d31e7d00 of size 512 2018-09-25 09:04:46.416335: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d31e7f00 of size 512 2018-09-25 09:04:46.416364: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d31e8100 of size 196608 2018-09-25 09:04:46.416394: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3218100 of size 6291456 2018-09-25 09:04:46.416424: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3818100 of size 512 2018-09-25 09:04:46.416453: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3818300 of size 8289536 2018-09-25 09:04:46.416483: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d4000000 of size 65536 2018-09-25 09:04:46.416513: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d4010000 of size 6291456 2018-09-25 09:04:46.416542: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d4610000 of size 65536 2018-09-25 09:04:46.416571: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d4620000 of size 65536 2018-09-25 09:04:46.416602: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d4630000 of size 6291456 2018-09-25 09:04:46.416631: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d4c30000 of size 65536 2018-09-25 09:04:46.416661: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d4c40000 of size 6291456 2018-09-25 09:04:46.416690: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d5240000 of size 6291456 2018-09-25 09:04:46.416719: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d5840000 of size 6291456 2018-09-25 09:04:46.416748: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d5e40000 of size 6291456 2018-09-25 09:04:46.416777: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d6440000 of size 6291456 2018-09-25 09:04:46.416815: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d6a40000 of size 6291456 2018-09-25 09:04:46.416849: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7d7040000 of size 6291456 2018-09-25 09:04:46.416876: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d7640000 of size 10223616 2018-09-25 09:04:46.416906: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d8000000 of size 6291456 2018-09-25 09:04:46.416935: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d8600000 of size 6291456 2018-09-25 09:04:46.416964: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d8c00000 of size 6291456 2018-09-25 09:04:46.416993: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d9200000 of size 6291456 2018-09-25 09:04:46.417024: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d9800000 of size 8388608 2018-09-25 09:04:46.417054: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da000000 of size 65536 2018-09-25 09:04:46.417084: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da010000 of size 65536 2018-09-25 09:04:46.417114: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da020000 of size 65536 2018-09-25 09:04:46.417144: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da030000 of size 65536 2018-09-25 09:04:46.417173: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da040000 of size 65536 2018-09-25 09:04:46.417202: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da050000 of size 65536 2018-09-25 09:04:46.417231: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da060000 of size 65536 2018-09-25 09:04:46.417261: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da070000 of size 65536 2018-09-25 09:04:46.417291: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da080000 of size 65536 2018-09-25 09:04:46.417320: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da090000 of size 2048 2018-09-25 09:04:46.417349: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da090800 of size 2048 2018-09-25 09:04:46.417379: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da091000 of size 4096 2018-09-25 09:04:46.417408: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da092000 of size 196608 2018-09-25 09:04:46.417438: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da0c2000 of size 256 2018-09-25 09:04:46.417467: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7da0c2100 of size 256 2018-09-25 09:04:46.417497: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da0c2200 of size 196608 2018-09-25 09:04:46.417526: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da0f2200 of size 512 2018-09-25 09:04:46.417556: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da0f2400 of size 196608 2018-09-25 09:04:46.417586: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da122400 of size 512 2018-09-25 09:04:46.417615: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da122600 of size 196608 2018-09-25 09:04:46.417645: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da152600 of size 512 2018-09-25 09:04:46.417675: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da152800 of size 196608 2018-09-25 09:04:46.417704: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da182800 of size 512 2018-09-25 09:04:46.417733: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da182a00 of size 196608 2018-09-25 09:04:46.417763: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da1b2a00 of size 197120 2018-09-25 09:04:46.417792: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da1e2c00 of size 512 2018-09-25 09:04:46.417827: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da1e2e00 of size 65536 2018-09-25 09:04:46.417857: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7da1f2e00 of size 131072 2018-09-25 09:04:46.417886: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da212e00 of size 512 2018-09-25 09:04:46.417915: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da213000 of size 197120 2018-09-25 09:04:46.417944: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da243200 of size 512 2018-09-25 09:04:46.417974: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da243400 of size 65536 2018-09-25 09:04:46.418004: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da253400 of size 65536 2018-09-25 09:04:46.418033: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da263400 of size 65536 2018-09-25 09:04:46.418063: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da273400 of size 65536 2018-09-25 09:04:46.418092: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da283400 of size 196608 2018-09-25 09:04:46.418121: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da2b3400 of size 196608 2018-09-25 09:04:46.418150: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7da2e3400 of size 198144 2018-09-25 09:04:46.418179: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da313a00 of size 196608 2018-09-25 09:04:46.418208: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da343a00 of size 196608 2018-09-25 09:04:46.418238: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da373a00 of size 512 2018-09-25 09:04:46.418268: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da373c00 of size 262144 2018-09-25 09:04:46.418297: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da3b3c00 of size 4096 2018-09-25 09:04:46.418326: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da3b4c00 of size 1048576 2018-09-25 09:04:46.418356: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da4b4c00 of size 1048576 2018-09-25 09:04:46.418385: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da5b4c00 of size 1048576 2018-09-25 09:04:46.418415: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da6b4c00 of size 1356800 2018-09-25 09:04:46.418444: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da800000 of size 65536 2018-09-25 09:04:46.418473: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da810000 of size 6291456 2018-09-25 09:04:46.418502: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7dae10000 of size 1048576 2018-09-25 09:04:46.418532: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7daf10000 of size 1048576 2018-09-25 09:04:46.418561: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db010000 of size 1048576 2018-09-25 09:04:46.418590: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7db110000 of size 1048576 2018-09-25 09:04:46.418620: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db210000 of size 1048576 2018-09-25 09:04:46.418650: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db310000 of size 526336 2018-09-25 09:04:46.418679: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db390800 of size 1024 2018-09-25 09:04:46.418709: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db390c00 of size 525312 2018-09-25 09:04:46.418739: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db411000 of size 524288 2018-09-25 09:04:46.418769: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db491000 of size 1024 2018-09-25 09:04:46.418799: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db491400 of size 512 2018-09-25 09:04:46.418839: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db491600 of size 524288 2018-09-25 09:04:46.418870: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db511600 of size 1024 2018-09-25 09:04:46.418900: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db511a00 of size 524288 2018-09-25 09:04:46.418929: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db591a00 of size 512 2018-09-25 09:04:46.418959: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7db591c00 of size 512 2018-09-25 09:04:46.418988: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db591e00 of size 524288 2018-09-25 09:04:46.419018: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db611e00 of size 524288 2018-09-25 09:04:46.419048: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db691e00 of size 1024 2018-09-25 09:04:46.419078: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db692200 of size 1024 2018-09-25 09:04:46.419107: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db692600 of size 524288 2018-09-25 09:04:46.419137: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db712600 of size 512 2018-09-25 09:04:46.419166: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db712800 of size 512 2018-09-25 09:04:46.419196: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7db712a00 of size 972288 2018-09-25 09:04:46.419225: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb800000 of size 65536 2018-09-25 09:04:46.419255: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb810000 of size 65536 2018-09-25 09:04:46.419284: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb820000 of size 65536 2018-09-25 09:04:46.419313: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb830000 of size 1024 2018-09-25 09:04:46.419343: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb830400 of size 65536 2018-09-25 09:04:46.419373: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb840400 of size 65536 2018-09-25 09:04:46.419402: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb850400 of size 65536 2018-09-25 09:04:46.419432: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb860400 of size 130048 2018-09-25 09:04:46.419461: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb880000 of size 1024 2018-09-25 09:04:46.419491: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb880400 of size 512 2018-09-25 09:04:46.419520: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb880600 of size 512 2018-09-25 09:04:46.419550: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb880800 of size 1024 2018-09-25 09:04:46.419579: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb880c00 of size 512 2018-09-25 09:04:46.419609: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb880e00 of size 512 2018-09-25 09:04:46.419638: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb881000 of size 524288 2018-09-25 09:04:46.419668: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb901000 of size 1044480 2018-09-25 09:04:46.419701: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eba00000 of size 524800 2018-09-25 09:04:46.419732: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eba80200 of size 65536 2018-09-25 09:04:46.419762: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eba90200 of size 65536 2018-09-25 09:04:46.419791: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebaa0200 of size 65536 2018-09-25 09:04:46.419826: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebab0200 of size 4096 2018-09-25 09:04:46.419857: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebab1200 of size 16384 2018-09-25 09:04:46.419887: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebab5200 of size 2048 2018-09-25 09:04:46.419916: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebab5a00 of size 2048 2018-09-25 09:04:46.419945: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebab6200 of size 2048 2018-09-25 09:04:46.419974: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebab6a00 of size 2048 2018-09-25 09:04:46.420003: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebab7200 of size 1024 2018-09-25 09:04:46.420033: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebab7600 of size 1024 2018-09-25 09:04:46.420062: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebab7a00 of size 2048 2018-09-25 09:04:46.420091: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebab8200 of size 4096 2018-09-25 09:04:46.420120: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebab9200 of size 4096 2018-09-25 09:04:46.420149: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebaba200 of size 512 2018-09-25 09:04:46.420179: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebaba400 of size 512 2018-09-25 09:04:46.420208: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebaba600 of size 512 2018-09-25 09:04:46.420238: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebaba800 of size 256 2018-09-25 09:04:46.420267: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebaba900 of size 256 2018-09-25 09:04:46.420296: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebabaa00 of size 512 2018-09-25 09:04:46.420325: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebabac00 of size 512 2018-09-25 09:04:46.420355: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebabae00 of size 512 2018-09-25 09:04:46.420384: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebabb000 of size 512 2018-09-25 09:04:46.420413: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebabb200 of size 4096 2018-09-25 09:04:46.420443: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebabc200 of size 4096 2018-09-25 09:04:46.420472: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebabd200 of size 4096 2018-09-25 09:04:46.420502: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebabe200 of size 4096 2018-09-25 09:04:46.420531: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebabf200 of size 2048 2018-09-25 09:04:46.420561: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebabfa00 of size 1024 2018-09-25 09:04:46.420590: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebabfe00 of size 512 2018-09-25 09:04:46.420619: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebac0000 of size 512 2018-09-25 09:04:46.420648: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebac0200 of size 65536 2018-09-25 09:04:46.420677: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebad0200 of size 65536 2018-09-25 09:04:46.420706: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebae0200 of size 65536 2018-09-25 09:04:46.420735: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebaf0200 of size 65536 2018-09-25 09:04:46.420765: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb00200 of size 1024 2018-09-25 09:04:46.420794: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb00600 of size 512 2018-09-25 09:04:46.420828: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb00800 of size 512 2018-09-25 09:04:46.420858: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb00a00 of size 65536 2018-09-25 09:04:46.420887: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb10a00 of size 65536 2018-09-25 09:04:46.420916: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7ebb20a00 of size 4096 2018-09-25 09:04:46.420945: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb21a00 of size 1024 2018-09-25 09:04:46.420974: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb21e00 of size 2048 2018-09-25 09:04:46.421003: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb22600 of size 1024 2018-09-25 09:04:46.421032: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb22a00 of size 1024 2018-09-25 09:04:46.421061: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb22e00 of size 2048 2018-09-25 09:04:46.421090: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb23600 of size 2048 2018-09-25 09:04:46.421119: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7ebb23e00 of size 2048 2018-09-25 09:04:46.421150: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb24600 of size 2048 2018-09-25 09:04:46.421181: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb24e00 of size 2048 2018-09-25 09:04:46.421212: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb25600 of size 4096 2018-09-25 09:04:46.421242: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb26600 of size 2048 2018-09-25 09:04:46.421273: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb26e00 of size 1024 2018-09-25 09:04:46.421304: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb27200 of size 1024 2018-09-25 09:04:46.421335: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb27600 of size 2048 2018-09-25 09:04:46.421365: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7ebb27e00 of size 2048 2018-09-25 09:04:46.421397: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb28600 of size 2560 2018-09-25 09:04:46.421426: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb29000 of size 2048 2018-09-25 09:04:46.421457: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb29800 of size 2048 2018-09-25 09:04:46.421487: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2a000 of size 2048 2018-09-25 09:04:46.421517: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7ebb2a800 of size 2048 2018-09-25 09:04:46.421547: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2b000 of size 1024 2018-09-25 09:04:46.421578: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2b400 of size 1024 2018-09-25 09:04:46.421607: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2b800 of size 2048 2018-09-25 09:04:46.421638: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7ebb2c000 of size 2048 2018-09-25 09:04:46.421684: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2c800 of size 2048 2018-09-25 09:04:46.421705: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2d000 of size 2048 2018-09-25 09:04:46.421722: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2d800 of size 2048 2018-09-25 09:04:46.421744: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2e000 of size 512 2018-09-25 09:04:46.421761: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2e200 of size 1536 2018-09-25 09:04:46.421782: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7ebb2e800 of size 2048 2018-09-25 09:04:46.421800: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2f000 of size 512 2018-09-25 09:04:46.421821: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2f200 of size 512 2018-09-25 09:04:46.421843: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2f400 of size 1024 2018-09-25 09:04:46.421860: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2f800 of size 2048 2018-09-25 09:04:46.421910: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7ebb30000 of size 2560 2018-09-25 09:04:46.421938: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb30a00 of size 65536 2018-09-25 09:04:46.421959: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb40a00 of size 65536 2018-09-25 09:04:46.421976: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb50a00 of size 4096 2018-09-25 09:04:46.421998: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb51a00 of size 4096 2018-09-25 09:04:46.422014: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb52a00 of size 4096 2018-09-25 09:04:46.422037: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb53a00 of size 4096 2018-09-25 09:04:46.422053: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb54a00 of size 4096 2018-09-25 09:04:46.422074: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb55a00 of size 4096 2018-09-25 09:04:46.422095: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb56a00 of size 4096 2018-09-25 09:04:46.422116: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb57a00 of size 4096 2018-09-25 09:04:46.422137: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb58a00 of size 4096 2018-09-25 09:04:46.422158: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb59a00 of size 4096 2018-09-25 09:04:46.422175: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb5aa00 of size 4096 2018-09-25 09:04:46.422197: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb5ba00 of size 4096 2018-09-25 09:04:46.422214: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb5ca00 of size 4096 2018-09-25 09:04:46.422235: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb5da00 of size 4096 2018-09-25 09:04:46.422252: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb5ea00 of size 4096 2018-09-25 09:04:46.422273: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb5fa00 of size 4096 2018-09-25 09:04:46.422290: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb60a00 of size 65536 2018-09-25 09:04:46.422311: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb70a00 of size 71680 2018-09-25 09:04:46.422328: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb82200 of size 512 2018-09-25 09:04:46.422353: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb82400 of size 512 2018-09-25 09:04:46.422375: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb82600 of size 512 2018-09-25 09:04:46.422398: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb82800 of size 512 2018-09-25 09:04:46.422419: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb82a00 of size 524288 2018-09-25 09:04:46.422440: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc02a00 of size 1024 2018-09-25 09:04:46.422461: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc02e00 of size 512 2018-09-25 09:04:46.422482: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc03000 of size 512 2018-09-25 09:04:46.422499: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc03200 of size 512 2018-09-25 09:04:46.422520: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc03400 of size 1024 2018-09-25 09:04:46.422537: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc03800 of size 512 2018-09-25 09:04:46.422557: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc03a00 of size 2048 2018-09-25 09:04:46.422575: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc04200 of size 6144 2018-09-25 09:04:46.422596: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc05a00 of size 512 2018-09-25 09:04:46.422613: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc05c00 of size 512 2018-09-25 09:04:46.422633: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc05e00 of size 512 2018-09-25 09:04:46.422650: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc06000 of size 1024 2018-09-25 09:04:46.422670: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc06400 of size 512 2018-09-25 09:04:46.422687: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc06600 of size 512 2018-09-25 09:04:46.422708: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc06800 of size 512 2018-09-25 09:04:46.422725: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc06a00 of size 5632 2018-09-25 09:04:46.422746: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc08000 of size 4096 2018-09-25 09:04:46.422763: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc09000 of size 4096 2018-09-25 09:04:46.422784: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc0a000 of size 4096 2018-09-25 09:04:46.422801: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc0b000 of size 4096 2018-09-25 09:04:46.422883: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc0c000 of size 4096 2018-09-25 09:04:46.422904: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc0d000 of size 4096 2018-09-25 09:04:46.422921: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc0e000 of size 4096 2018-09-25 09:04:46.422942: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc0f000 of size 1536 2018-09-25 09:04:46.422959: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc0f600 of size 4096 2018-09-25 09:04:46.422980: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc10600 of size 4096 2018-09-25 09:04:46.423001: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc11600 of size 5120 2018-09-25 09:04:46.423021: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc12a00 of size 1024 2018-09-25 09:04:46.423039: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc12e00 of size 65536 2018-09-25 09:04:46.423060: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc22e00 of size 524288 2018-09-25 09:04:46.423076: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebca2e00 of size 65536 2018-09-25 09:04:46.423097: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebcb2e00 of size 524288 2018-09-25 09:04:46.423114: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd32e00 of size 65536 2018-09-25 09:04:46.423136: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd42e00 of size 4096 2018-09-25 09:04:46.423153: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd43e00 of size 65536 2018-09-25 09:04:46.423174: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd53e00 of size 65536 2018-09-25 09:04:46.423194: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd63e00 of size 4096 2018-09-25 09:04:46.423216: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd64e00 of size 65536 2018-09-25 09:04:46.423233: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd74e00 of size 65536 2018-09-25 09:04:46.423254: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd84e00 of size 65536 2018-09-25 09:04:46.423271: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd94e00 of size 4096 2018-09-25 09:04:46.423287: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd95e00 of size 4096 2018-09-25 09:04:46.423311: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd96e00 of size 4096 2018-09-25 09:04:46.423334: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd97e00 of size 4096 2018-09-25 09:04:46.423355: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd98e00 of size 4096 2018-09-25 09:04:46.423374: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd99e00 of size 4096 2018-09-25 09:04:46.423394: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd9ae00 of size 4096 2018-09-25 09:04:46.423415: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd9be00 of size 4096 2018-09-25 09:04:46.423432: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd9ce00 of size 4096 2018-09-25 09:04:46.423453: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd9de00 of size 4096 2018-09-25 09:04:46.423474: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd9ee00 of size 4096 2018-09-25 09:04:46.423491: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd9fe00 of size 4096 2018-09-25 09:04:46.423512: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebda0e00 of size 4096 2018-09-25 09:04:46.423532: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebda1e00 of size 4096 2018-09-25 09:04:46.423553: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebda2e00 of size 4096 2018-09-25 09:04:46.423570: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebda3e00 of size 4096 2018-09-25 09:04:46.423591: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebda4e00 of size 65536 2018-09-25 09:04:46.423608: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebdb4e00 of size 65536 2018-09-25 09:04:46.423629: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebdc4e00 of size 65536 2018-09-25 09:04:46.423649: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebdd4e00 of size 65536 2018-09-25 09:04:46.423670: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebde4e00 of size 111104 2018-09-25 09:04:46.423688: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe85f600000 of size 1280 2018-09-25 09:04:46.423709: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe85f600500 of size 512 2018-09-25 09:04:46.423726: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe85f600700 of size 1046784 2018-09-25 09:04:46.423747: I tensorflow/core/common_runtime/bfc_allocator.cc:671] Summary of in-use Chunks by size: 2018-09-25 09:04:46.423768: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 5 Chunks of size 256 totalling 1.2KiB 2018-09-25 09:04:46.423789: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 61 Chunks of size 512 totalling 30.5KiB 2018-09-25 09:04:46.423817: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 33 Chunks of size 1024 totalling 33.0KiB 2018-09-25 09:04:46.423839: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1280 totalling 1.2KiB 2018-09-25 09:04:46.423861: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 1536 totalling 3.0KiB 2018-09-25 09:04:46.423879: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 29 Chunks of size 2048 totalling 58.0KiB 2018-09-25 09:04:46.423902: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 2560 totalling 2.5KiB 2018-09-25 09:04:46.423924: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 3072 totalling 3.0KiB 2018-09-25 09:04:46.423947: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 58 Chunks of size 4096 totalling 232.0KiB 2018-09-25 09:04:46.423968: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 5120 totalling 5.0KiB 2018-09-25 09:04:46.423987: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 5632 totalling 5.5KiB 2018-09-25 09:04:46.424007: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 6144 totalling 6.0KiB 2018-09-25 09:04:46.424029: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 8192 totalling 8.0KiB 2018-09-25 09:04:46.424047: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 16384 totalling 16.0KiB 2018-09-25 09:04:46.424072: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 57 Chunks of size 65536 totalling 3.56MiB 2018-09-25 09:04:46.424095: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 71680 totalling 70.0KiB 2018-09-25 09:04:46.424119: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 111104 totalling 108.5KiB 2018-09-25 09:04:46.424141: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 130048 totalling 127.0KiB 2018-09-25 09:04:46.424163: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 25 Chunks of size 196608 totalling 4.69MiB 2018-09-25 09:04:46.424181: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 196864 totalling 192.2KiB 2018-09-25 09:04:46.424203: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 3 Chunks of size 197120 totalling 577.5KiB 2018-09-25 09:04:46.424224: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 249856 totalling 244.0KiB 2018-09-25 09:04:46.424244: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 262144 totalling 256.0KiB 2018-09-25 09:04:46.424265: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 319488 totalling 312.0KiB 2018-09-25 09:04:46.424286: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 327680 totalling 320.0KiB 2018-09-25 09:04:46.424308: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 25 Chunks of size 524288 totalling 12.50MiB 2018-09-25 09:04:46.424334: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 524800 totalling 512.5KiB 2018-09-25 09:04:46.424355: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 525312 totalling 513.0KiB 2018-09-25 09:04:46.424377: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 526336 totalling 514.0KiB 2018-09-25 09:04:46.424395: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1044480 totalling 1020.0KiB 2018-09-25 09:04:46.424420: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1046528 totalling 1022.0KiB 2018-09-25 09:04:46.424442: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1046784 totalling 1022.2KiB 2018-09-25 09:04:46.424465: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 29 Chunks of size 1048576 totalling 29.00MiB 2018-09-25 09:04:46.424488: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1356800 totalling 1.29MiB 2018-09-25 09:04:46.424512: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 27 Chunks of size 6291456 totalling 162.00MiB 2018-09-25 09:04:46.424532: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 8289536 totalling 7.91MiB 2018-09-25 09:04:46.424555: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 8388608 totalling 8.00MiB 2018-09-25 09:04:46.424578: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 10223616 totalling 9.75MiB 2018-09-25 09:04:46.424600: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 2097152000 totalling 1.95GiB 2018-09-25 09:04:46.424623: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 2147483648 totalling 4.00GiB 2018-09-25 09:04:46.424645: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Sum Total of in-use chunks: 6.19GiB 2018-09-25 09:04:46.424667: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats: Limit: 7700093338 InUse: 6649803008 MaxInUse: 6650327296 NumAllocs: 783 MaxAllocSize: 2147483648
2018-09-25 09:04:46.424764: W tensorflow/core/common_runtime/bfcallocator.cc:279] ****____*****__**** 2018-09-25 09:04:46.424806: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at conv_ops.cc:693 : Resource exhausted: OOM when allocating tensor with shape[128,128,1,32000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc INFO:tensorflow:Unexpected error happened: OOM when allocating tensor with shape[128,128,1,32000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: Conv2D_3 = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Conv2D_3-0-TransposeNHWCToNCHW-LayoutOptimizer, ae_dilatedconv_2/W/read)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Caused by op u'Conv2D_3', defined at: File "/usr/bin/nsynth_save_embeddings", line 11, in
sys.exit(console_entry_point())
File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/nsynth_save_embeddings.py", line 124, in console_entry_point
tf.app.run(main)
File "/usr/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/nsynth_save_embeddings.py", line 103, in main
encoding = encode(wav_data, checkpoint_path, sample_length=sample_length)
File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/fastgen.py", line 109, in encode
net = load_nsynth(batch_size=batch_size, sample_length=sample_length)
File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/fastgen.py", line 65, in load_nsynth
graph = config.build({"wav": x}, is_training=False)
File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/h512_bo16.py", line 233, in build
name='aedilatedconv%d' % (num_layer + 1))
File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/masked.py", line 154, in conv1d
y = tf.nn.conv2d(x_4d, weights, strides, padding=padding)
File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 956, in conv2d
data_format=data_format, dilations=dilations, name=name)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3414, in create_op
op_def=op_def)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1740, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[128,128,1,32000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: Conv2D_3 = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Conv2D_3-0-TransposeNHWCToNCHW-LayoutOptimizer, ae_dilatedconv_2/W/read)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
. Traceback (most recent call last): File "/usr/bin/nsynth_save_embeddings", line 11, in
sys.exit(console_entry_point())
File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/nsynth_save_embeddings.py", line 124, in console_entry_point
tf.app.run(main)
File "/usr/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/nsynth_save_embeddings.py", line 103, in main
encoding = encode(wav_data, checkpoint_path, sample_length=sample_length)
File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/fastgen.py", line 112, in encode
encodings = sess.run(net["encoding"], feed_dict={net["X"]: wav_data})
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[128,128,1,32000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: Conv2D_3 = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Conv2D_3-0-TransposeNHWCToNCHW-LayoutOptimizer, ae_dilatedconv_2/W/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Caused by op u'Conv2D_3', defined at: File "/usr/bin/nsynth_save_embeddings", line 11, in
sys.exit(console_entry_point())
File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/nsynth_save_embeddings.py", line 124, in console_entry_point
tf.app.run(main)
File "/usr/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/nsynth_save_embeddings.py", line 103, in main
encoding = encode(wav_data, checkpoint_path, sample_length=sample_length)
File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/fastgen.py", line 109, in encode
net = load_nsynth(batch_size=batch_size, sample_length=sample_length)
File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/fastgen.py", line 65, in load_nsynth
graph = config.build({"wav": x}, is_training=False)
File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/h512_bo16.py", line 233, in build
name='aedilatedconv%d' % (num_layer + 1))
File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/masked.py", line 154, in conv1d
y = tf.nn.conv2d(x_4d, weights, strides, padding=padding)
File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 956, in conv2d
data_format=data_format, dilations=dilations, name=name)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3414, in create_op
op_def=op_def)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1740, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[128,128,1,32000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: Conv2D_3 = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Conv2D_3-0-TransposeNHWCToNCHW-LayoutOptimizer, ae_dilatedconv_2/W/read)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.