search

googlecreativelab / open-nsynth-super

Open NSynth Super is an experimental physical interface for the NSynth algorithm
Apache License 2.0
2.42k stars 270 forks source link

Out of memory running 01_compute_input_embeddings #61

Open AndreyKudryavtsev opened 5 years ago

AndreyKudryavtsev commented 5 years ago

Hopefully, I'm not facing any simple mistake here. I wanted to practise creating my own sounds and followed online instructions. I'm running Core-i7 desktop with GTX1080. However it fails on the first script, and looks memory is not enough. I set up # of GPUs in json file. Not sure if I missed anything else. Please, support. { "pads": { "NW": ["resopad"], "NE": ["flute"], "SW": ["rhodes"], "SE": ["tubularbells"] },

"magenta_dir":  "~/magenta",
"pitches":      [24, 28, 32, 36, 40, 44, 48, 52, 56, 60, 64, 68, 72, 76, 80, 84],
"resolution":   9,
"final_length": 60000,
"gpus":         1

}

[root@localhost workdir]# python ./01_compute_input_embeddings.py /usr/lib64/python2.7/site-packages/scipy/sparse/lil.py:19: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import _csparsetools /usr/lib64/python2.7/site-packages/scipy/sparse/csgraph/init.py:165: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._shortest_path import shortest_path, floyd_warshall, dijkstra,\ /usr/lib64/python2.7/site-packages/scipy/sparse/csgraph/_validation.py:5: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._tools import csgraph_to_dense, csgraph_from_dense,\ /usr/lib64/python2.7/site-packages/scipy/sparse/csgraph/init.py:167: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._traversal import breadth_first_order, depth_first_order, \ /usr/lib64/python2.7/site-packages/scipy/sparse/csgraph/init.py:169: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._min_spanning_tree import minimum_spanning_tree /usr/lib64/python2.7/site-packages/scipy/sparse/csgraph/init.py:170: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._reordering import reverse_cuthill_mckee, maximum_bipartite_matching, \ /usr/lib64/python2.7/site-packages/scipy/linalg/basic.py:17: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._solve_toeplitz import levinson /usr/lib64/python2.7/site-packages/scipy/linalg/init.py:207: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._decomp_update import /usr/lib64/python2.7/site-packages/scipy/special/init.py:640: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._ufuncs import /usr/lib64/python2.7/site-packages/scipy/special/_ellip_harm.py:7: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._ellip_harm_2 import _ellipsoid, _ellipsoid_norm /usr/lib64/python2.7/site-packages/scipy/interpolate/_bsplines.py:10: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import _bspl /usr/lib64/python2.7/site-packages/scipy/spatial/init.py:95: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .ckdtree import /usr/lib64/python2.7/site-packages/scipy/spatial/init.py:96: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .qhull import /usr/lib64/python2.7/site-packages/scipy/spatial/_spherical_voronoi.py:18: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import _voronoi /usr/lib64/python2.7/site-packages/scipy/spatial/distance.py:122: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import _hausdorff /usr/lib64/python2.7/site-packages/scipy/ndimage/measurements.py:36: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import _ni_label /usr/lib64/python2.7/site-packages/pandas/_libs/init.py:4: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .tslib import iNaT, NaT, Timestamp, Timedelta, OutOfBoundsDatetime /usr/lib64/python2.7/site-packages/pandas/init.py:26: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from pandas._libs import (hashtable as _hashtable, /usr/lib64/python2.7/site-packages/pandas/core/dtypes/common.py:6: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from pandas._libs import algos, lib /usr/lib64/python2.7/site-packages/pandas/core/util/hashing.py:7: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from pandas._libs import hashing, tslib /usr/lib64/python2.7/site-packages/pandas/core/indexes/base.py:7: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from pandas._libs import (lib, index as libindex, tslib as libts, /usr/lib64/python2.7/site-packages/pandas/tseries/offsets.py:21: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 import pandas._libs.tslibs.offsets as liboffsets /usr/lib64/python2.7/site-packages/pandas/core/ops.py:16: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from pandas._libs import algos as libalgos, ops as libops /usr/lib64/python2.7/site-packages/pandas/core/indexes/interval.py:32: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from pandas._libs.interval import ( /usr/lib64/python2.7/site-packages/pandas/core/internals.py:14: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from pandas._libs import internals as libinternals /usr/lib64/python2.7/site-packages/pandas/core/sparse/array.py:33: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 import pandas._libs.sparse as splib /usr/lib64/python2.7/site-packages/pandas/core/window.py:36: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 import pandas._libs.window as _window /usr/lib64/python2.7/site-packages/pandas/core/groupby/groupby.py:68: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from pandas._libs import (lib, reduction, /usr/lib64/python2.7/site-packages/pandas/core/reshape/reshape.py:30: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from pandas._libs import algos as _algos, reshape as _reshape /usr/lib64/python2.7/site-packages/pandas/io/parsers.py:45: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 import pandas._libs.parsers as parsers /usr/lib64/python2.7/site-packages/pandas/io/pytables.py:50: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from pandas._libs import algos, lib, writers as libwriters /usr/lib64/python2.7/site-packages/scipy/signal/_max_len_seq.py:8: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._max_len_seq_inner import _max_len_seq_inner /usr/lib64/python2.7/site-packages/scipy/signal/_upfirdn.py:36: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._upfirdn_apply import _output_len, _apply /usr/lib64/python2.7/site-packages/scipy/optimize/_trlib/init.py:1: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._trlib import TRLIBQuadraticSubproblem /usr/lib64/python2.7/site-packages/scipy/optimize/_numdiff.py:10: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._group_columns import group_dense, group_sparse /usr/lib64/python2.7/site-packages/scipy/signal/spectral.py:10: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._spectral import _lombscargle /usr/lib64/python2.7/site-packages/scipy/stats/_continuous_distns.py:18: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import _stats /usr/lib64/python2.7/site-packages/scipy/signal/_peak_finding.py:13: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._peak_finding_utils import (_argmaxima1d, _select_by_peak_distance, /usr/lib64/python2.7/site-packages/sklearn/utils/init.py:9: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .murmurhash import murmurhash3_32 /usr/lib64/python2.7/site-packages/sklearn/utils/extmath.py:24: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._logistic_sigmoid import _log_logistic_sigmoid /usr/lib64/python2.7/site-packages/sklearn/utils/extmath.py:26: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .sparsefuncs_fast import csr_row_norms /usr/lib64/python2.7/site-packages/sklearn/metrics/cluster/supervised.py:23: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .expected_mutual_info_fast import expected_mutual_information /usr/lib64/python2.7/site-packages/sklearn/metrics/pairwise.py:30: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .pairwise_fast import _chi2_kernel_fast, _sparse_manhattan /usr/lib64/python2.7/site-packages/sklearn/linear_model/base.py:35: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ..utils.seq_dataset import ArrayDataset, CSRDataset /usr/lib64/python2.7/site-packages/sklearn/linear_model/least_angle.py:23: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ..utils import arrayfuncs, as_float_array, check_X_y, deprecated /usr/lib64/python2.7/site-packages/sklearn/utils/random.py:10: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._random import sample_without_replacement /usr/lib64/python2.7/site-packages/sklearn/linear_model/coordinate_descent.py:29: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import cd_fast /usr/lib64/python2.7/site-packages/sklearn/linear_model/init.py:22: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .sgd_fast import Hinge, Log, ModifiedHuber, SquaredLoss, Huber /usr/lib64/python2.7/site-packages/sklearn/linear_model/sag.py:12: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .sag_fast import sag /usr/lib64/python2.7/site-packages/sklearn/svm/base.py:8: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import libsvm, liblinear /usr/lib64/python2.7/site-packages/sklearn/svm/base.py:9: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import libsvm_sparse /usr/lib64/python2.7/site-packages/sklearn/neighbors/init.py:6: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .ball_tree import BallTree /usr/lib64/python2.7/site-packages/sklearn/neighbors/init.py:7: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .kd_tree import KDTree /usr/lib64/python2.7/site-packages/sklearn/decomposition/online_lda.py:28: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._online_lda import (mean_change, _dirichlet_expectation_1d, /usr/lib64/python2.7/site-packages/sklearn/utils/graph.py:15: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .graph_shortest_path import graph_shortest_path # noqa /usr/lib64/python2.7/site-packages/sklearn/isotonic.py:12: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._isotonic import _inplace_contiguous_isotonic_regression, _make_unique /usr/lib64/python2.7/site-packages/sklearn/manifold/t_sne.py:24: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import _utils /usr/lib64/python2.7/site-packages/sklearn/manifold/t_sne.py:25: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import _barnes_hut_tsne /usr/lib64/python2.7/site-packages/sklearn/tree/tree.py:40: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._criterion import Criterion /usr/lib64/python2.7/site-packages/sklearn/cluster/kmeans.py:35: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import _k_means /usr/lib64/python2.7/site-packages/sklearn/cluster/kmeans.py:36: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._k_means_elkan import k_means_elkan /usr/lib64/python2.7/site-packages/sklearn/cluster/hierarchical.py:23: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import hierarchical /usr/lib64/python2.7/site-packages/sklearn/cluster/dbscan.py:19: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from ._dbscan_inner import dbscan_inner /usr/lib64/python2.7/site-packages/sklearn/feature_extraction/hashing.py:10: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from . import _hashing /usr/lib64/python2.7/site-packages/scipy/io/matlab/mio4.py:18: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .mio_utils import squeeze_element, chars_to_strings /usr/lib64/python2.7/site-packages/scipy/io/matlab/mio5.py:98: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88 from .mio5_utils import VarReader5 WARNING:tensorflow:From /usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/masked.py:116: init (from tensorflow.python.ops.init_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.initializers.variance_scaling instead with distribution=uniform to get equivalent behavior. INFO:tensorflow:Will restore from checkpoint: /root/magenta/magenta/models/nsynth/wavenet-ckpt/model.ckpt-200000 INFO:tensorflow:Will load Wavs from /mnt/ssd/open-nsynth-super-master/audio/workdir/audio_input. INFO:tensorflow:Will save embeddings to /mnt/ssd/open-nsynth-super-master/audio/workdir/embeddings_input. INFO:tensorflow:On file number 0 (batch 1). 2018-09-25 09:04:29.767892: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA 2018-09-25 09:04:29.914347: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1392] Found device 0 with properties: name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.7715 pciBusID: 0000:65:00.0 totalMemory: 7.93GiB freeMemory: 7.61GiB 2018-09-25 09:04:29.914394: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0 2018-09-25 09:04:30.144628: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix: 2018-09-25 09:04:30.144675: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958] 0 2018-09-25 09:04:30.144685: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: N 2018-09-25 09:04:30.144858: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7343 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:65:00.0, compute capability: 6.1) INFO:tensorflow:Restoring parameters from /root/magenta/magenta/models/nsynth/wavenet-ckpt/model.ckpt-200000 2018-09-25 09:04:36.170063: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.95GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2018-09-25 09:04:46.411265: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.95GiB. Current allocation summary follows. 2018-09-25 09:04:46.411396: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (256): Total Chunks: 6, Chunks in use: 5. 1.5KiB allocated for chunks. 1.2KiB in use in bin. 80B client-requested in use in bin. 2018-09-25 09:04:46.411475: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (512): Total Chunks: 62, Chunks in use: 61. 31.0KiB allocated for chunks. 30.5KiB in use in bin. 30.5KiB client-requested in use in bin. 2018-09-25 09:04:46.411519: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1024): Total Chunks: 36, Chunks in use: 36. 37.2KiB allocated for chunks. 37.2KiB in use in bin. 36.5KiB client-requested in use in bin. 2018-09-25 09:04:46.411560: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2048): Total Chunks: 37, Chunks in use: 31. 76.0KiB allocated for chunks. 63.5KiB in use in bin. 62.0KiB client-requested in use in bin. 2018-09-25 09:04:46.411609: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4096): Total Chunks: 62, Chunks in use: 61. 252.5KiB allocated for chunks. 248.5KiB in use in bin. 246.0KiB client-requested in use in bin. 2018-09-25 09:04:46.411653: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8192): Total Chunks: 1, Chunks in use: 1. 8.0KiB allocated for chunks. 8.0KiB in use in bin. 8.0KiB client-requested in use in bin. 2018-09-25 09:04:46.411701: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16384): Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin. 2018-09-25 09:04:46.411746: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (32768): Total Chunks: 1, Chunks in use: 0. 58.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-09-25 09:04:46.411785: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (65536): Total Chunks: 60, Chunks in use: 60. 3.86MiB allocated for chunks. 3.86MiB in use in bin. 3.75MiB client-requested in use in bin. 2018-09-25 09:04:46.411850: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (131072): Total Chunks: 33, Chunks in use: 30. 6.12MiB allocated for chunks. 5.68MiB in use in bin. 5.62MiB client-requested in use in bin. 2018-09-25 09:04:46.411891: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (262144): Total Chunks: 3, Chunks in use: 3. 888.0KiB allocated for chunks. 888.0KiB in use in bin. 704.0KiB client-requested in use in bin. 2018-09-25 09:04:46.411939: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (524288): Total Chunks: 35, Chunks in use: 31. 19.74MiB allocated for chunks. 17.00MiB in use in bin. 15.50MiB client-requested in use in bin. 2018-09-25 09:04:46.411985: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1048576): Total Chunks: 33, Chunks in use: 30. 33.29MiB allocated for chunks. 30.29MiB in use in bin. 30.00MiB client-requested in use in bin. 2018-09-25 09:04:46.412031: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2097152): Total Chunks: 1, Chunks in use: 0. 3.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-09-25 09:04:46.412076: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4194304): Total Chunks: 30, Chunks in use: 28. 181.91MiB allocated for chunks. 169.91MiB in use in bin. 168.00MiB client-requested in use in bin. 2018-09-25 09:04:46.412125: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8388608): Total Chunks: 2, Chunks in use: 2. 17.75MiB allocated for chunks. 17.75MiB in use in bin. 12.00MiB client-requested in use in bin. 2018-09-25 09:04:46.412170: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-09-25 09:04:46.412227: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-09-25 09:04:46.412271: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-09-25 09:04:46.412316: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (134217728): Total Chunks: 1, Chunks in use: 0. 244.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2018-09-25 09:04:46.412361: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (268435456): Total Chunks: 4, Chunks in use: 3. 6.67GiB allocated for chunks. 5.95GiB in use in bin. 5.86GiB client-requested in use in bin. 2018-09-25 09:04:46.412406: I tensorflow/core/common_runtime/bfc_allocator.cc:646] Bin for 1.95GiB was 256.00MiB, Chunk State: 2018-09-25 09:04:46.412448: I tensorflow/core/common_runtime/bfc_allocator.cc:652] Size: 736.38MiB | Requested Size: 15.62MiB | in_use: 0, prev: Size: 1.95GiB | Requested Size: 1.95GiB | in_use: 1 2018-09-25 09:04:46.412491: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe55a000000 of size 2097152000 2018-09-25 09:04:46.412522: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe5d7000000 of size 772151552 2018-09-25 09:04:46.412553: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe6b6000000 of size 2147483648 2018-09-25 09:04:46.412586: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe73c000000 of size 2147483648 2018-09-25 09:04:46.412617: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7bc000000 of size 6291456 2018-09-25 09:04:46.412647: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7bc600000 of size 6291456 2018-09-25 09:04:46.412676: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7bcc00000 of size 255852544 2018-09-25 09:04:46.412706: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cc000000 of size 1048576 2018-09-25 09:04:46.412737: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cc100000 of size 1048576 2018-09-25 09:04:46.412766: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cc200000 of size 1048576 2018-09-25 09:04:46.412796: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cc300000 of size 1048576 2018-09-25 09:04:46.412835: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cc400000 of size 1048576 2018-09-25 09:04:46.412866: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cc500000 of size 1048576 2018-09-25 09:04:46.412896: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cc600000 of size 1048576 2018-09-25 09:04:46.412927: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cc700000 of size 1048576 2018-09-25 09:04:46.412956: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7cc800000 of size 1048576 2018-09-25 09:04:46.412986: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cc900000 of size 1048576 2018-09-25 09:04:46.413017: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cca00000 of size 196608 2018-09-25 09:04:46.413049: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7cca30000 of size 851968 2018-09-25 09:04:46.413079: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ccb00000 of size 1048576 2018-09-25 09:04:46.413109: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ccc00000 of size 6291456 2018-09-25 09:04:46.413138: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cd200000 of size 6291456 2018-09-25 09:04:46.413171: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cd800000 of size 6291456 2018-09-25 09:04:46.413202: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cde00000 of size 6291456 2018-09-25 09:04:46.413230: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ce400000 of size 6291456 2018-09-25 09:04:46.413261: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cea00000 of size 6291456 2018-09-25 09:04:46.413289: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cf000000 of size 6291456 2018-09-25 09:04:46.413319: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cf600000 of size 6291456 2018-09-25 09:04:46.413347: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7cfc00000 of size 6291456 2018-09-25 09:04:46.413377: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7d0200000 of size 6291456 2018-09-25 09:04:46.413408: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d0800000 of size 2048 2018-09-25 09:04:46.413440: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d0800800 of size 4096 2018-09-25 09:04:46.413474: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d0801800 of size 2048 2018-09-25 09:04:46.413503: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d0802000 of size 4096 2018-09-25 09:04:46.413532: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d0803000 of size 249856 2018-09-25 09:04:46.413561: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d0840000 of size 6291456 2018-09-25 09:04:46.413592: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d0e40000 of size 65536 2018-09-25 09:04:46.413621: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d0e50000 of size 196608 2018-09-25 09:04:46.413650: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d0e80000 of size 1048576 2018-09-25 09:04:46.413681: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d0f80000 of size 1048576 2018-09-25 09:04:46.413709: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1080000 of size 1048576 2018-09-25 09:04:46.413738: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1180000 of size 1048576 2018-09-25 09:04:46.413767: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1280000 of size 2048 2018-09-25 09:04:46.413797: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1280800 of size 1046528 2018-09-25 09:04:46.413832: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7d1380000 of size 1048576 2018-09-25 09:04:46.413862: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1480000 of size 1048576 2018-09-25 09:04:46.413892: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1580000 of size 1048576 2018-09-25 09:04:46.413922: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1680000 of size 2048 2018-09-25 09:04:46.413951: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1680800 of size 4096 2018-09-25 09:04:46.413983: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7d1681800 of size 59392 2018-09-25 09:04:46.414012: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1690000 of size 524288 2018-09-25 09:04:46.414041: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1710000 of size 524288 2018-09-25 09:04:46.414070: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1790000 of size 1048576 2018-09-25 09:04:46.414100: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1890000 of size 524288 2018-09-25 09:04:46.414128: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1910000 of size 524288 2018-09-25 09:04:46.414158: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1990000 of size 1048576 2018-09-25 09:04:46.414187: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1a90000 of size 1048576 2018-09-25 09:04:46.414218: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1b90000 of size 1048576 2018-09-25 09:04:46.414246: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7d1c90000 of size 3145728 2018-09-25 09:04:46.414275: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d1f90000 of size 1048576 2018-09-25 09:04:46.414305: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2090000 of size 1048576 2018-09-25 09:04:46.414334: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2190000 of size 65536 2018-09-25 09:04:46.414363: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d21a0000 of size 6291456 2018-09-25 09:04:46.414394: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d27a0000 of size 1024 2018-09-25 09:04:46.414422: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d27a0400 of size 524288 2018-09-25 09:04:46.414451: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2820400 of size 512 2018-09-25 09:04:46.414480: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2820600 of size 512 2018-09-25 09:04:46.414511: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2820800 of size 8192 2018-09-25 09:04:46.414540: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2822800 of size 196608 2018-09-25 09:04:46.414570: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2852800 of size 319488 2018-09-25 09:04:46.414599: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d28a0800 of size 256 2018-09-25 09:04:46.414628: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d28a0900 of size 256 2018-09-25 09:04:46.414658: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d28a0a00 of size 512 2018-09-25 09:04:46.414687: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d28a0c00 of size 524288 2018-09-25 09:04:46.414717: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2920c00 of size 1024 2018-09-25 09:04:46.414747: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2921000 of size 524288 2018-09-25 09:04:46.414776: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d29a1000 of size 65536 2018-09-25 09:04:46.414804: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d29b1000 of size 512 2018-09-25 09:04:46.414844: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d29b1200 of size 512 2018-09-25 09:04:46.414873: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d29b1400 of size 524288 2018-09-25 09:04:46.414902: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2a31400 of size 1024 2018-09-25 09:04:46.414932: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2a31800 of size 196608 2018-09-25 09:04:46.414962: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2a61800 of size 327680 2018-09-25 09:04:46.414992: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7d2ab1800 of size 524288 2018-09-25 09:04:46.415022: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2b31800 of size 512 2018-09-25 09:04:46.415052: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2b31a00 of size 512 2018-09-25 09:04:46.415080: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2b31c00 of size 512 2018-09-25 09:04:46.415110: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2b31e00 of size 512 2018-09-25 09:04:46.415138: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2b32000 of size 524288 2018-09-25 09:04:46.415167: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2bb2000 of size 1024 2018-09-25 09:04:46.415197: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2bb2400 of size 524288 2018-09-25 09:04:46.415228: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2c32400 of size 1024 2018-09-25 09:04:46.415256: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2c32800 of size 524288 2018-09-25 09:04:46.415286: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2cb2800 of size 1024 2018-09-25 09:04:46.415315: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2cb2c00 of size 524288 2018-09-25 09:04:46.415343: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2d32c00 of size 524288 2018-09-25 09:04:46.415373: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2db2c00 of size 1024 2018-09-25 09:04:46.415402: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2db3000 of size 1024 2018-09-25 09:04:46.415433: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2db3400 of size 524288 2018-09-25 09:04:46.415462: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2e33400 of size 1024 2018-09-25 09:04:46.415492: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2e33800 of size 512 2018-09-25 09:04:46.415520: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2e33a00 of size 524288 2018-09-25 09:04:46.415548: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7d2eb3a00 of size 525312 2018-09-25 09:04:46.415577: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2f33e00 of size 4096 2018-09-25 09:04:46.415607: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2f34e00 of size 2048 2018-09-25 09:04:46.415637: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2f35600 of size 3072 2018-09-25 09:04:46.415666: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2f36200 of size 196864 2018-09-25 09:04:46.415694: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2f66300 of size 512 2018-09-25 09:04:46.415724: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2f66500 of size 196608 2018-09-25 09:04:46.415753: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2f96500 of size 512 2018-09-25 09:04:46.415784: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2f96700 of size 197120 2018-09-25 09:04:46.415819: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2fc6900 of size 196608 2018-09-25 09:04:46.415843: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2ff6900 of size 512 2018-09-25 09:04:46.415868: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d2ff6b00 of size 65536 2018-09-25 09:04:46.415897: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7d3006b00 of size 131584 2018-09-25 09:04:46.415927: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3026d00 of size 65536 2018-09-25 09:04:46.415957: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3036d00 of size 4096 2018-09-25 09:04:46.415986: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3037d00 of size 196608 2018-09-25 09:04:46.416015: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3067d00 of size 65536 2018-09-25 09:04:46.416045: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3077d00 of size 65536 2018-09-25 09:04:46.416070: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3087d00 of size 65536 2018-09-25 09:04:46.416099: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3097d00 of size 196608 2018-09-25 09:04:46.416130: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d30c7d00 of size 196608 2018-09-25 09:04:46.416159: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d30f7d00 of size 196608 2018-09-25 09:04:46.416189: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3127d00 of size 196608 2018-09-25 09:04:46.416218: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3157d00 of size 196608 2018-09-25 09:04:46.416247: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3187d00 of size 196608 2018-09-25 09:04:46.416276: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d31b7d00 of size 196608 2018-09-25 09:04:46.416305: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d31e7d00 of size 512 2018-09-25 09:04:46.416335: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d31e7f00 of size 512 2018-09-25 09:04:46.416364: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d31e8100 of size 196608 2018-09-25 09:04:46.416394: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3218100 of size 6291456 2018-09-25 09:04:46.416424: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3818100 of size 512 2018-09-25 09:04:46.416453: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d3818300 of size 8289536 2018-09-25 09:04:46.416483: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d4000000 of size 65536 2018-09-25 09:04:46.416513: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d4010000 of size 6291456 2018-09-25 09:04:46.416542: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d4610000 of size 65536 2018-09-25 09:04:46.416571: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d4620000 of size 65536 2018-09-25 09:04:46.416602: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d4630000 of size 6291456 2018-09-25 09:04:46.416631: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d4c30000 of size 65536 2018-09-25 09:04:46.416661: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d4c40000 of size 6291456 2018-09-25 09:04:46.416690: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d5240000 of size 6291456 2018-09-25 09:04:46.416719: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d5840000 of size 6291456 2018-09-25 09:04:46.416748: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d5e40000 of size 6291456 2018-09-25 09:04:46.416777: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d6440000 of size 6291456 2018-09-25 09:04:46.416815: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d6a40000 of size 6291456 2018-09-25 09:04:46.416849: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7d7040000 of size 6291456 2018-09-25 09:04:46.416876: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d7640000 of size 10223616 2018-09-25 09:04:46.416906: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d8000000 of size 6291456 2018-09-25 09:04:46.416935: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d8600000 of size 6291456 2018-09-25 09:04:46.416964: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d8c00000 of size 6291456 2018-09-25 09:04:46.416993: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d9200000 of size 6291456 2018-09-25 09:04:46.417024: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7d9800000 of size 8388608 2018-09-25 09:04:46.417054: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da000000 of size 65536 2018-09-25 09:04:46.417084: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da010000 of size 65536 2018-09-25 09:04:46.417114: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da020000 of size 65536 2018-09-25 09:04:46.417144: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da030000 of size 65536 2018-09-25 09:04:46.417173: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da040000 of size 65536 2018-09-25 09:04:46.417202: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da050000 of size 65536 2018-09-25 09:04:46.417231: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da060000 of size 65536 2018-09-25 09:04:46.417261: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da070000 of size 65536 2018-09-25 09:04:46.417291: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da080000 of size 65536 2018-09-25 09:04:46.417320: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da090000 of size 2048 2018-09-25 09:04:46.417349: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da090800 of size 2048 2018-09-25 09:04:46.417379: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da091000 of size 4096 2018-09-25 09:04:46.417408: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da092000 of size 196608 2018-09-25 09:04:46.417438: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da0c2000 of size 256 2018-09-25 09:04:46.417467: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7da0c2100 of size 256 2018-09-25 09:04:46.417497: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da0c2200 of size 196608 2018-09-25 09:04:46.417526: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da0f2200 of size 512 2018-09-25 09:04:46.417556: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da0f2400 of size 196608 2018-09-25 09:04:46.417586: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da122400 of size 512 2018-09-25 09:04:46.417615: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da122600 of size 196608 2018-09-25 09:04:46.417645: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da152600 of size 512 2018-09-25 09:04:46.417675: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da152800 of size 196608 2018-09-25 09:04:46.417704: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da182800 of size 512 2018-09-25 09:04:46.417733: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da182a00 of size 196608 2018-09-25 09:04:46.417763: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da1b2a00 of size 197120 2018-09-25 09:04:46.417792: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da1e2c00 of size 512 2018-09-25 09:04:46.417827: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da1e2e00 of size 65536 2018-09-25 09:04:46.417857: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7da1f2e00 of size 131072 2018-09-25 09:04:46.417886: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da212e00 of size 512 2018-09-25 09:04:46.417915: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da213000 of size 197120 2018-09-25 09:04:46.417944: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da243200 of size 512 2018-09-25 09:04:46.417974: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da243400 of size 65536 2018-09-25 09:04:46.418004: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da253400 of size 65536 2018-09-25 09:04:46.418033: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da263400 of size 65536 2018-09-25 09:04:46.418063: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da273400 of size 65536 2018-09-25 09:04:46.418092: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da283400 of size 196608 2018-09-25 09:04:46.418121: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da2b3400 of size 196608 2018-09-25 09:04:46.418150: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7da2e3400 of size 198144 2018-09-25 09:04:46.418179: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da313a00 of size 196608 2018-09-25 09:04:46.418208: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da343a00 of size 196608 2018-09-25 09:04:46.418238: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da373a00 of size 512 2018-09-25 09:04:46.418268: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da373c00 of size 262144 2018-09-25 09:04:46.418297: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da3b3c00 of size 4096 2018-09-25 09:04:46.418326: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da3b4c00 of size 1048576 2018-09-25 09:04:46.418356: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da4b4c00 of size 1048576 2018-09-25 09:04:46.418385: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da5b4c00 of size 1048576 2018-09-25 09:04:46.418415: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da6b4c00 of size 1356800 2018-09-25 09:04:46.418444: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da800000 of size 65536 2018-09-25 09:04:46.418473: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7da810000 of size 6291456 2018-09-25 09:04:46.418502: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7dae10000 of size 1048576 2018-09-25 09:04:46.418532: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7daf10000 of size 1048576 2018-09-25 09:04:46.418561: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db010000 of size 1048576 2018-09-25 09:04:46.418590: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7db110000 of size 1048576 2018-09-25 09:04:46.418620: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db210000 of size 1048576 2018-09-25 09:04:46.418650: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db310000 of size 526336 2018-09-25 09:04:46.418679: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db390800 of size 1024 2018-09-25 09:04:46.418709: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db390c00 of size 525312 2018-09-25 09:04:46.418739: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db411000 of size 524288 2018-09-25 09:04:46.418769: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db491000 of size 1024 2018-09-25 09:04:46.418799: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db491400 of size 512 2018-09-25 09:04:46.418839: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db491600 of size 524288 2018-09-25 09:04:46.418870: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db511600 of size 1024 2018-09-25 09:04:46.418900: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db511a00 of size 524288 2018-09-25 09:04:46.418929: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db591a00 of size 512 2018-09-25 09:04:46.418959: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7db591c00 of size 512 2018-09-25 09:04:46.418988: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db591e00 of size 524288 2018-09-25 09:04:46.419018: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db611e00 of size 524288 2018-09-25 09:04:46.419048: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db691e00 of size 1024 2018-09-25 09:04:46.419078: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db692200 of size 1024 2018-09-25 09:04:46.419107: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db692600 of size 524288 2018-09-25 09:04:46.419137: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db712600 of size 512 2018-09-25 09:04:46.419166: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7db712800 of size 512 2018-09-25 09:04:46.419196: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7db712a00 of size 972288 2018-09-25 09:04:46.419225: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb800000 of size 65536 2018-09-25 09:04:46.419255: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb810000 of size 65536 2018-09-25 09:04:46.419284: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb820000 of size 65536 2018-09-25 09:04:46.419313: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb830000 of size 1024 2018-09-25 09:04:46.419343: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb830400 of size 65536 2018-09-25 09:04:46.419373: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb840400 of size 65536 2018-09-25 09:04:46.419402: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb850400 of size 65536 2018-09-25 09:04:46.419432: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb860400 of size 130048 2018-09-25 09:04:46.419461: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb880000 of size 1024 2018-09-25 09:04:46.419491: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb880400 of size 512 2018-09-25 09:04:46.419520: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb880600 of size 512 2018-09-25 09:04:46.419550: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb880800 of size 1024 2018-09-25 09:04:46.419579: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb880c00 of size 512 2018-09-25 09:04:46.419609: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb880e00 of size 512 2018-09-25 09:04:46.419638: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb881000 of size 524288 2018-09-25 09:04:46.419668: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eb901000 of size 1044480 2018-09-25 09:04:46.419701: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eba00000 of size 524800 2018-09-25 09:04:46.419732: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eba80200 of size 65536 2018-09-25 09:04:46.419762: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7eba90200 of size 65536 2018-09-25 09:04:46.419791: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebaa0200 of size 65536 2018-09-25 09:04:46.419826: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebab0200 of size 4096 2018-09-25 09:04:46.419857: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebab1200 of size 16384 2018-09-25 09:04:46.419887: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebab5200 of size 2048 2018-09-25 09:04:46.419916: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebab5a00 of size 2048 2018-09-25 09:04:46.419945: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebab6200 of size 2048 2018-09-25 09:04:46.419974: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebab6a00 of size 2048 2018-09-25 09:04:46.420003: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebab7200 of size 1024 2018-09-25 09:04:46.420033: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebab7600 of size 1024 2018-09-25 09:04:46.420062: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebab7a00 of size 2048 2018-09-25 09:04:46.420091: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebab8200 of size 4096 2018-09-25 09:04:46.420120: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebab9200 of size 4096 2018-09-25 09:04:46.420149: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebaba200 of size 512 2018-09-25 09:04:46.420179: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebaba400 of size 512 2018-09-25 09:04:46.420208: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebaba600 of size 512 2018-09-25 09:04:46.420238: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebaba800 of size 256 2018-09-25 09:04:46.420267: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebaba900 of size 256 2018-09-25 09:04:46.420296: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebabaa00 of size 512 2018-09-25 09:04:46.420325: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebabac00 of size 512 2018-09-25 09:04:46.420355: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebabae00 of size 512 2018-09-25 09:04:46.420384: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebabb000 of size 512 2018-09-25 09:04:46.420413: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebabb200 of size 4096 2018-09-25 09:04:46.420443: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebabc200 of size 4096 2018-09-25 09:04:46.420472: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebabd200 of size 4096 2018-09-25 09:04:46.420502: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebabe200 of size 4096 2018-09-25 09:04:46.420531: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebabf200 of size 2048 2018-09-25 09:04:46.420561: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebabfa00 of size 1024 2018-09-25 09:04:46.420590: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebabfe00 of size 512 2018-09-25 09:04:46.420619: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebac0000 of size 512 2018-09-25 09:04:46.420648: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebac0200 of size 65536 2018-09-25 09:04:46.420677: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebad0200 of size 65536 2018-09-25 09:04:46.420706: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebae0200 of size 65536 2018-09-25 09:04:46.420735: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebaf0200 of size 65536 2018-09-25 09:04:46.420765: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb00200 of size 1024 2018-09-25 09:04:46.420794: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb00600 of size 512 2018-09-25 09:04:46.420828: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb00800 of size 512 2018-09-25 09:04:46.420858: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb00a00 of size 65536 2018-09-25 09:04:46.420887: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb10a00 of size 65536 2018-09-25 09:04:46.420916: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7ebb20a00 of size 4096 2018-09-25 09:04:46.420945: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb21a00 of size 1024 2018-09-25 09:04:46.420974: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb21e00 of size 2048 2018-09-25 09:04:46.421003: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb22600 of size 1024 2018-09-25 09:04:46.421032: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb22a00 of size 1024 2018-09-25 09:04:46.421061: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb22e00 of size 2048 2018-09-25 09:04:46.421090: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb23600 of size 2048 2018-09-25 09:04:46.421119: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7ebb23e00 of size 2048 2018-09-25 09:04:46.421150: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb24600 of size 2048 2018-09-25 09:04:46.421181: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb24e00 of size 2048 2018-09-25 09:04:46.421212: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb25600 of size 4096 2018-09-25 09:04:46.421242: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb26600 of size 2048 2018-09-25 09:04:46.421273: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb26e00 of size 1024 2018-09-25 09:04:46.421304: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb27200 of size 1024 2018-09-25 09:04:46.421335: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb27600 of size 2048 2018-09-25 09:04:46.421365: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7ebb27e00 of size 2048 2018-09-25 09:04:46.421397: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb28600 of size 2560 2018-09-25 09:04:46.421426: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb29000 of size 2048 2018-09-25 09:04:46.421457: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb29800 of size 2048 2018-09-25 09:04:46.421487: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2a000 of size 2048 2018-09-25 09:04:46.421517: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7ebb2a800 of size 2048 2018-09-25 09:04:46.421547: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2b000 of size 1024 2018-09-25 09:04:46.421578: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2b400 of size 1024 2018-09-25 09:04:46.421607: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2b800 of size 2048 2018-09-25 09:04:46.421638: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7ebb2c000 of size 2048 2018-09-25 09:04:46.421684: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2c800 of size 2048 2018-09-25 09:04:46.421705: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2d000 of size 2048 2018-09-25 09:04:46.421722: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2d800 of size 2048 2018-09-25 09:04:46.421744: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2e000 of size 512 2018-09-25 09:04:46.421761: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2e200 of size 1536 2018-09-25 09:04:46.421782: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7ebb2e800 of size 2048 2018-09-25 09:04:46.421800: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2f000 of size 512 2018-09-25 09:04:46.421821: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2f200 of size 512 2018-09-25 09:04:46.421843: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2f400 of size 1024 2018-09-25 09:04:46.421860: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb2f800 of size 2048 2018-09-25 09:04:46.421910: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe7ebb30000 of size 2560 2018-09-25 09:04:46.421938: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb30a00 of size 65536 2018-09-25 09:04:46.421959: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb40a00 of size 65536 2018-09-25 09:04:46.421976: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb50a00 of size 4096 2018-09-25 09:04:46.421998: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb51a00 of size 4096 2018-09-25 09:04:46.422014: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb52a00 of size 4096 2018-09-25 09:04:46.422037: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb53a00 of size 4096 2018-09-25 09:04:46.422053: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb54a00 of size 4096 2018-09-25 09:04:46.422074: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb55a00 of size 4096 2018-09-25 09:04:46.422095: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb56a00 of size 4096 2018-09-25 09:04:46.422116: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb57a00 of size 4096 2018-09-25 09:04:46.422137: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb58a00 of size 4096 2018-09-25 09:04:46.422158: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb59a00 of size 4096 2018-09-25 09:04:46.422175: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb5aa00 of size 4096 2018-09-25 09:04:46.422197: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb5ba00 of size 4096 2018-09-25 09:04:46.422214: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb5ca00 of size 4096 2018-09-25 09:04:46.422235: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb5da00 of size 4096 2018-09-25 09:04:46.422252: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb5ea00 of size 4096 2018-09-25 09:04:46.422273: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb5fa00 of size 4096 2018-09-25 09:04:46.422290: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb60a00 of size 65536 2018-09-25 09:04:46.422311: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb70a00 of size 71680 2018-09-25 09:04:46.422328: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb82200 of size 512 2018-09-25 09:04:46.422353: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb82400 of size 512 2018-09-25 09:04:46.422375: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb82600 of size 512 2018-09-25 09:04:46.422398: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb82800 of size 512 2018-09-25 09:04:46.422419: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebb82a00 of size 524288 2018-09-25 09:04:46.422440: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc02a00 of size 1024 2018-09-25 09:04:46.422461: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc02e00 of size 512 2018-09-25 09:04:46.422482: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc03000 of size 512 2018-09-25 09:04:46.422499: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc03200 of size 512 2018-09-25 09:04:46.422520: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc03400 of size 1024 2018-09-25 09:04:46.422537: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc03800 of size 512 2018-09-25 09:04:46.422557: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc03a00 of size 2048 2018-09-25 09:04:46.422575: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc04200 of size 6144 2018-09-25 09:04:46.422596: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc05a00 of size 512 2018-09-25 09:04:46.422613: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc05c00 of size 512 2018-09-25 09:04:46.422633: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc05e00 of size 512 2018-09-25 09:04:46.422650: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc06000 of size 1024 2018-09-25 09:04:46.422670: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc06400 of size 512 2018-09-25 09:04:46.422687: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc06600 of size 512 2018-09-25 09:04:46.422708: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc06800 of size 512 2018-09-25 09:04:46.422725: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc06a00 of size 5632 2018-09-25 09:04:46.422746: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc08000 of size 4096 2018-09-25 09:04:46.422763: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc09000 of size 4096 2018-09-25 09:04:46.422784: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc0a000 of size 4096 2018-09-25 09:04:46.422801: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc0b000 of size 4096 2018-09-25 09:04:46.422883: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc0c000 of size 4096 2018-09-25 09:04:46.422904: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc0d000 of size 4096 2018-09-25 09:04:46.422921: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc0e000 of size 4096 2018-09-25 09:04:46.422942: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc0f000 of size 1536 2018-09-25 09:04:46.422959: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc0f600 of size 4096 2018-09-25 09:04:46.422980: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc10600 of size 4096 2018-09-25 09:04:46.423001: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc11600 of size 5120 2018-09-25 09:04:46.423021: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc12a00 of size 1024 2018-09-25 09:04:46.423039: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc12e00 of size 65536 2018-09-25 09:04:46.423060: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebc22e00 of size 524288 2018-09-25 09:04:46.423076: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebca2e00 of size 65536 2018-09-25 09:04:46.423097: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebcb2e00 of size 524288 2018-09-25 09:04:46.423114: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd32e00 of size 65536 2018-09-25 09:04:46.423136: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd42e00 of size 4096 2018-09-25 09:04:46.423153: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd43e00 of size 65536 2018-09-25 09:04:46.423174: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd53e00 of size 65536 2018-09-25 09:04:46.423194: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd63e00 of size 4096 2018-09-25 09:04:46.423216: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd64e00 of size 65536 2018-09-25 09:04:46.423233: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd74e00 of size 65536 2018-09-25 09:04:46.423254: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd84e00 of size 65536 2018-09-25 09:04:46.423271: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd94e00 of size 4096 2018-09-25 09:04:46.423287: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd95e00 of size 4096 2018-09-25 09:04:46.423311: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd96e00 of size 4096 2018-09-25 09:04:46.423334: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd97e00 of size 4096 2018-09-25 09:04:46.423355: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd98e00 of size 4096 2018-09-25 09:04:46.423374: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd99e00 of size 4096 2018-09-25 09:04:46.423394: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd9ae00 of size 4096 2018-09-25 09:04:46.423415: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd9be00 of size 4096 2018-09-25 09:04:46.423432: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd9ce00 of size 4096 2018-09-25 09:04:46.423453: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd9de00 of size 4096 2018-09-25 09:04:46.423474: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd9ee00 of size 4096 2018-09-25 09:04:46.423491: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebd9fe00 of size 4096 2018-09-25 09:04:46.423512: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebda0e00 of size 4096 2018-09-25 09:04:46.423532: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebda1e00 of size 4096 2018-09-25 09:04:46.423553: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebda2e00 of size 4096 2018-09-25 09:04:46.423570: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebda3e00 of size 4096 2018-09-25 09:04:46.423591: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebda4e00 of size 65536 2018-09-25 09:04:46.423608: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebdb4e00 of size 65536 2018-09-25 09:04:46.423629: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebdc4e00 of size 65536 2018-09-25 09:04:46.423649: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebdd4e00 of size 65536 2018-09-25 09:04:46.423670: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe7ebde4e00 of size 111104 2018-09-25 09:04:46.423688: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe85f600000 of size 1280 2018-09-25 09:04:46.423709: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe85f600500 of size 512 2018-09-25 09:04:46.423726: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe85f600700 of size 1046784 2018-09-25 09:04:46.423747: I tensorflow/core/common_runtime/bfc_allocator.cc:671] Summary of in-use Chunks by size: 2018-09-25 09:04:46.423768: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 5 Chunks of size 256 totalling 1.2KiB 2018-09-25 09:04:46.423789: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 61 Chunks of size 512 totalling 30.5KiB 2018-09-25 09:04:46.423817: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 33 Chunks of size 1024 totalling 33.0KiB 2018-09-25 09:04:46.423839: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1280 totalling 1.2KiB 2018-09-25 09:04:46.423861: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 1536 totalling 3.0KiB 2018-09-25 09:04:46.423879: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 29 Chunks of size 2048 totalling 58.0KiB 2018-09-25 09:04:46.423902: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 2560 totalling 2.5KiB 2018-09-25 09:04:46.423924: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 3072 totalling 3.0KiB 2018-09-25 09:04:46.423947: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 58 Chunks of size 4096 totalling 232.0KiB 2018-09-25 09:04:46.423968: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 5120 totalling 5.0KiB 2018-09-25 09:04:46.423987: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 5632 totalling 5.5KiB 2018-09-25 09:04:46.424007: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 6144 totalling 6.0KiB 2018-09-25 09:04:46.424029: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 8192 totalling 8.0KiB 2018-09-25 09:04:46.424047: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 16384 totalling 16.0KiB 2018-09-25 09:04:46.424072: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 57 Chunks of size 65536 totalling 3.56MiB 2018-09-25 09:04:46.424095: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 71680 totalling 70.0KiB 2018-09-25 09:04:46.424119: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 111104 totalling 108.5KiB 2018-09-25 09:04:46.424141: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 130048 totalling 127.0KiB 2018-09-25 09:04:46.424163: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 25 Chunks of size 196608 totalling 4.69MiB 2018-09-25 09:04:46.424181: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 196864 totalling 192.2KiB 2018-09-25 09:04:46.424203: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 3 Chunks of size 197120 totalling 577.5KiB 2018-09-25 09:04:46.424224: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 249856 totalling 244.0KiB 2018-09-25 09:04:46.424244: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 262144 totalling 256.0KiB 2018-09-25 09:04:46.424265: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 319488 totalling 312.0KiB 2018-09-25 09:04:46.424286: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 327680 totalling 320.0KiB 2018-09-25 09:04:46.424308: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 25 Chunks of size 524288 totalling 12.50MiB 2018-09-25 09:04:46.424334: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 524800 totalling 512.5KiB 2018-09-25 09:04:46.424355: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 525312 totalling 513.0KiB 2018-09-25 09:04:46.424377: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 526336 totalling 514.0KiB 2018-09-25 09:04:46.424395: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1044480 totalling 1020.0KiB 2018-09-25 09:04:46.424420: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1046528 totalling 1022.0KiB 2018-09-25 09:04:46.424442: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1046784 totalling 1022.2KiB 2018-09-25 09:04:46.424465: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 29 Chunks of size 1048576 totalling 29.00MiB 2018-09-25 09:04:46.424488: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1356800 totalling 1.29MiB 2018-09-25 09:04:46.424512: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 27 Chunks of size 6291456 totalling 162.00MiB 2018-09-25 09:04:46.424532: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 8289536 totalling 7.91MiB 2018-09-25 09:04:46.424555: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 8388608 totalling 8.00MiB 2018-09-25 09:04:46.424578: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 10223616 totalling 9.75MiB 2018-09-25 09:04:46.424600: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 2097152000 totalling 1.95GiB 2018-09-25 09:04:46.424623: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 2147483648 totalling 4.00GiB 2018-09-25 09:04:46.424645: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Sum Total of in-use chunks: 6.19GiB 2018-09-25 09:04:46.424667: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats: Limit: 7700093338 InUse: 6649803008 MaxInUse: 6650327296 NumAllocs: 783 MaxAllocSize: 2147483648

2018-09-25 09:04:46.424764: W tensorflow/core/common_runtime/bfcallocator.cc:279] ****____*****__**** 2018-09-25 09:04:46.424806: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at conv_ops.cc:693 : Resource exhausted: OOM when allocating tensor with shape[128,128,1,32000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc INFO:tensorflow:Unexpected error happened: OOM when allocating tensor with shape[128,128,1,32000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: Conv2D_3 = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Conv2D_3-0-TransposeNHWCToNCHW-LayoutOptimizer, ae_dilatedconv_2/W/read)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op u'Conv2D_3', defined at: File "/usr/bin/nsynth_save_embeddings", line 11, in sys.exit(console_entry_point()) File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/nsynth_save_embeddings.py", line 124, in console_entry_point tf.app.run(main) File "/usr/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run _sys.exit(main(argv)) File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/nsynth_save_embeddings.py", line 103, in main encoding = encode(wav_data, checkpoint_path, sample_length=sample_length) File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/fastgen.py", line 109, in encode net = load_nsynth(batch_size=batch_size, sample_length=sample_length) File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/fastgen.py", line 65, in load_nsynth graph = config.build({"wav": x}, is_training=False) File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/h512_bo16.py", line 233, in build name='aedilatedconv%d' % (num_layer + 1)) File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/masked.py", line 154, in conv1d y = tf.nn.conv2d(x_4d, weights, strides, padding=padding) File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 956, in conv2d data_format=data_format, dilations=dilations, name=name) File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3414, in create_op op_def=op_def) File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1740, in init self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[128,128,1,32000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: Conv2D_3 = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Conv2D_3-0-TransposeNHWCToNCHW-LayoutOptimizer, ae_dilatedconv_2/W/read)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

. Traceback (most recent call last): File "/usr/bin/nsynth_save_embeddings", line 11, in sys.exit(console_entry_point()) File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/nsynth_save_embeddings.py", line 124, in console_entry_point tf.app.run(main) File "/usr/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run _sys.exit(main(argv)) File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/nsynth_save_embeddings.py", line 103, in main encoding = encode(wav_data, checkpoint_path, sample_length=sample_length) File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/fastgen.py", line 112, in encode encodings = sess.run(net["encoding"], feed_dict={net["X"]: wav_data}) File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 900, in run run_metadata_ptr) File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1135, in _run feed_dict_tensor, options, run_metadata) File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run run_metadata) File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[128,128,1,32000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: Conv2D_3 = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Conv2D_3-0-TransposeNHWCToNCHW-LayoutOptimizer, ae_dilatedconv_2/W/read)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op u'Conv2D_3', defined at: File "/usr/bin/nsynth_save_embeddings", line 11, in sys.exit(console_entry_point()) File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/nsynth_save_embeddings.py", line 124, in console_entry_point tf.app.run(main) File "/usr/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run _sys.exit(main(argv)) File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/nsynth_save_embeddings.py", line 103, in main encoding = encode(wav_data, checkpoint_path, sample_length=sample_length) File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/fastgen.py", line 109, in encode net = load_nsynth(batch_size=batch_size, sample_length=sample_length) File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/fastgen.py", line 65, in load_nsynth graph = config.build({"wav": x}, is_training=False) File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/h512_bo16.py", line 233, in build name='aedilatedconv%d' % (num_layer + 1)) File "/usr/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/masked.py", line 154, in conv1d y = tf.nn.conv2d(x_4d, weights, strides, padding=padding) File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 956, in conv2d data_format=data_format, dilations=dilations, name=name) File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3414, in create_op op_def=op_def) File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1740, in init self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[128,128,1,32000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: Conv2D_3 = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Conv2D_3-0-TransposeNHWCToNCHW-LayoutOptimizer, ae_dilatedconv_2/W/read)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

AndreyKudryavtsev commented 5 years ago

Team, anyone can please take a look on that or share their experience running the stack with single GPU? Thank you.

where-is-brett commented 2 years ago

This is probably too late, but for anyone else that might run into the same problem, you could lower the batch_size=128 argument in the 01_compute_input_embeddings.py.