error after latest git pull (main branch)

[ciphercharly]

getting this error while launching a training script which was previously working (with the same configuration file):

Traceback (most recent call last):
  File "demos/paired_MMIV/demo_train.py", line 29, in <module>
    log_dir=log_dir,
  File "/home/charlie/3DREG-tests/DeepReg/deepreg/train.py", line 94, in train
    max_epochs=max_epochs,
  File "/home/charlie/3DREG-tests/DeepReg/deepreg/train.py", line 45, in build_config
    config = config_parser.load_configs(config_path)
  File "/home/charlie/3DREG-tests/DeepReg/deepreg/parser.py", line 41, in load_configs
    config_sanity_check(config)
  File "/home/charlie/3DREG-tests/DeepReg/deepreg/parser.py", line 90, in config_sanity_check
    if config["train"]["method"] == "conditional":
KeyError: 'method'

the script is

from deepreg.train import train
import sys

######## TRAINING ########

gpu = "2,3"
gpu_allow_growth = True
log_dir = "paired_logs_train"

config_path = [
    r"demos/paired_MMIV/paired_MMIV_train.yaml",
    r"demos/paired_MMIV/paired_MMIV.yaml",
]

if len(sys.argv) != 2:
    print('using training log dir ', log_dir)
else:
    log_dir = "paired_logs_train_"+sys.argv[1]
    print('using training log dir ', log_dir)

#ckpt_path = "logs/paired_logs_train_10000epochs_gmi_unet_lw1rw1_learnrate1e-5/save/weights-epoch10000.ckpt" # restart from saved model
ckpt_path = ""

train(
    gpu=gpu,
    config_path=config_path,
    gpu_allow_growth=gpu_allow_growth,
    ckpt_path=ckpt_path,
    log_dir=log_dir,
)

the config file I use

train:
  model:
    method: "ddf" # ddf / dvf / conditional
    backbone: "unet" # options include "local", "unet" and "global" - use "global" when method=="affine"
    local:
      num_channel_initial: 4 # 4
      extract_levels: [0, 1, 2, 3] # [0, 1, 2, 3]
    unet:
      num_channel_initial: 16
      depth: 3 # original 3
      pooling: true
      concat_skip: true

  loss:
    dissimilarity:
      image:
        name: "gmi"  # "lncc" (local normalised cross correlation), "ssd" (sum of squared distance) and "gmi" (differentiable global mutual information loss via Parzen windowing method)
        weight: 1.0
      label:
        weight: 0.0
        name: "multi_scale"
        multi_scale:
          loss_type: "dice" # options include "dice", "cross-entropy", "mean-squared", "generalised_dice" and "jaccard"
          loss_scales: [0, 1, 2, 4, 8, 16]
    regularization:
      weight: 1.0
      energy_type: "gradient-l2" # "bending" # options include "bending", "gradient-l1" and "gradient-l2"

  preprocess:
    batch_size: 2 #original 2
    shuffle_buffer_num_batch: 1

  optimizer:
    name: "adam"
    adam:
      learning_rate: 1.0e-5 #1.0e-2 bad
    sgd:
      learning_rate: 1.0e-4
      momentum: 0.9
    rms:
      learning_rate: 1.0e-4
      momentum: 0.9

  epochs: 10000
  save_period: 500

[NMontanaBrown]

Hi @ciphercharly , this is not a problem with your code but a problem with backcompatibility of our code.

This is related to issue #525 and issue #567 #568.

We updated DeepReg with PR 568 to use a Registry type class to allow for flexibility in the definition of new losses, architectures... whilst still using the train function. This brought about changes into the config file, which explains why your otherwise functional code is now not working after a pull from main. It also means that from these PRs onwards the code is not backcompatible (as you are experiencing), which is why we are aiming at a release sometime in the next two weeks which should resolve this issue. For the moment there are two hot fixes:

a) modify the config file you are using, which should allow you to proceed as normal - I am writing docs in #525 but am missing one argument to complete (@mathpluscode will be able to advise for how you should modify your config)

b) if you are just working with no modifs on the DeepReg code, you can checkout to the commit before merge of issue #567, which will allow you to work without this bug.

[ciphercharly]

thanks a lot for the answer! so if for now I want to have working code would git checkout 499-release-v010 do the job?

[NMontanaBrown]

@ciphercharly Yes, any verified/tagged release should do.

[ciphercharly]

mmmh, sorry about my ignorance of git, if I do for example git checkout 557-refactoring-models and then pull, I get 'Already up-to-date.' how can I properly revert back?

[NMontanaBrown]

Hi @ciphercharly can you try with the following git commit hash? So, git checkout main git checkout 135d054edc2233a43bd7760cde8ba315c7d97fbd .

If not we can look at modifying your config file 😄

[ciphercharly]

did the above and then git pull origin 135d054edc2233a43bd7760cde8ba315c7d97fbd and I get

From https://github.com/DeepRegNet/DeepReg
 * branch              135d054edc2233a43bd7760cde8ba315c7d97fbd -> FETCH_HEAD
Already up-to-date.

[NMontanaBrown]

Hi @ciphercharly, you shouldn't need to pull - pulling only updates your fork with the main repository fork at the branch. This only tells you the code is up to date with the associated commit.

Have you tried running your code from this commit location?

What is your environment? After you pulled from main which caused the errors, did you pip re-install?

[ciphercharly]

yes I re-run the code and get the same error, I did not pip re-install no

I am on Linux 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

pip list dump from the env where I use deepreg is

Package                       Version             Location
----------------------------- ------------------- ---------------------------------
absl-py                       0.9.0
alabaster                     0.7.12
appdirs                       1.4.4
argon2-cffi                   20.1.0
aspy.refactor-imports         2.1.1
astunparse                    1.6.3
async-generator               1.10
attrs                         20.2.0
Babel                         2.8.0
backcall                      0.2.0
backports.functools-lru-cache 1.6.1
black                         20.8b1
bleach                        3.2.1
blinker                       1.4
brotlipy                      0.7.0
cached-property               1.5.1
cachetools                    4.1.1
certifi                       2020.6.20
cffi                          1.14.2
cfgv                          3.2.0
chardet                       3.0.4
click                         7.1.2
coverage                      5.3
cryptography                  3.1
cycler                        0.10.0
dataclasses                   0.6
decorator                     4.4.2
deepreg                       0.1.0b1             /home/charlie/3DREG-tests/DeepReg
defusedxml                    0.6.0
distlib                       0.3.1
docutils                      0.16
entrypoints                   0.3
filelock                      3.0.12
flake8                        3.8.3
fsleyes                       0.34.2
fsleyes-props                 1.7.0
fsleyes-widgets               0.9.0
fslpy                         3.3.0
fslutils                      0.1
gast                          0.3.3
google-auth                   1.21.1
google-auth-oauthlib          0.4.1
google-pasta                  0.2.0
grpcio                        1.31.0
gviz-api                      1.9.0
h5py                          2.10.0
identify                      1.5.2
idna                          2.10
imagesize                     1.2.0
importlib-metadata            1.7.0
indexed-gzip                  1.3.2
iniconfig                     1.0.1
ipykernel                     5.3.4
ipython                       7.18.1
ipython-genutils              0.2.0
isodate                       0.6.0
isort                         5.5.2
jedi                          0.17.2
Jinja2                        2.11.2
jsonschema                    3.2.0
jupyter-client                6.1.7
jupyter-core                  4.6.3
jupyterlab-pygments           0.1.2
Keras-Preprocessing           1.1.2
kiwisolver                    1.2.0
m2r2                          0.2.5
Markdown                      3.2.2
MarkupSafe                    1.1.1
matplotlib                    3.3.2
mccabe                        0.6.1
mistune                       0.8.4
mkl-fft                       1.1.0
mkl-random                    1.1.1
mkl-service                   2.3.0
more-itertools                8.5.0
mypy-extensions               0.4.3
nbclient                      0.5.0
nbconvert                     6.0.6
nbformat                      5.0.7
nest-asyncio                  1.4.1
nibabel                       3.1.1
nodeenv                       1.5.0
notebook                      6.1.4
numpy                         1.18.5
oauthlib                      3.1.0
olefile                       0.46
opt-einsum                    3.1.0
packaging                     20.4
pandas                        1.1.2
pandocfilters                 1.4.2
parso                         0.7.1
pathspec                      0.8.0
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow                        7.2.0
pip                           20.2.2
pluggy                        0.13.1
pre-commit                    2.7.1
progressbar2                  3.53.1
prometheus-client             0.8.0
prompt-toolkit                3.0.7
protobuf                      3.13.0
ptyprocess                    0.6.0
py                            1.9.0
pyasn1                        0.4.8
pyasn1-modules                0.2.7
pycodestyle                   2.6.0
pycparser                     2.20
pydicom                       2.0.0
pyflakes                      2.2.0
Pygments                      2.7.1
PyJWT                         1.7.1
PyOpenGL                      3.1.5
pyOpenSSL                     19.1.0
pyparsing                     2.4.7
pyrsistent                    0.17.3
PySocks                       1.7.1
pytest                        6.0.2
pytest-cov                    2.10.1
pytest-dependency             0.5.1
python-dateutil               2.8.1
python-utils                  2.4.0
pytz                          2020.1
PyYAML                        5.3.1
pyzmq                         19.0.2
regex                         2020.7.14
requests                      2.24.0
requests-oauthlib             1.3.0
rsa                           4.6
Rtree                         0.9.4
scipy                         1.4.1
seed-isort-config             2.2.0
Send2Trash                    1.5.0
setuptools                    49.6.0.post20200814
simple-http-server            0.3.1
six                           1.15.0
snowballstemmer               2.0.0
Sphinx                        3.2.1
sphinx-notfound-page          0.5
sphinx-rtd-theme              0.5.0
sphinx-tabs                   1.3.0
sphinxcontrib-applehelp       1.0.2
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        1.0.3
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.4
tensorboard                   2.3.0
tensorboard-plugin-profile    2.3.0
tensorboard-plugin-wit        1.6.0
tensorflow                    2.3.1
tensorflow-estimator          2.3.0
termcolor                     1.1.0
terminado                     0.9.1
testfixtures                  6.14.2
testpath                      0.4.4
toml                          0.10.1
tornado                       6.0.4
tqdm                          4.49.0
traitlets                     5.0.4
trimesh                       3.8.10
typed-ast                     1.4.1
typing-extensions             3.7.4.3
urllib3                       1.25.10
virtualenv                    20.0.31
wcwidth                       0.2.5
webencodings                  0.5.1
Werkzeug                      1.0.1
wheel                         0.35.1
wrapt                         1.12.1
wxnatpy                       0.3.2
wxPython                      4.0.4
xnat                          0.3.25
zipp                          3.1.0

[mathpluscode]

Hi @ciphercharly, sorry for the issue, the error is due to a simplification in the config

Your current config is:

train:
  model:
    method: "ddf" # ddf / dvf / conditional
    backbone: "unet" # options include "local", "unet" and "global" - use "global" when method=="affine"
    local:
      num_channel_initial: 4 # 4
      extract_levels: [0, 1, 2, 3] # [0, 1, 2, 3]
    unet:
      num_channel_initial: 16
      depth: 3 # original 3
      pooling: true
      concat_skip: true

Please change to

train:
  method: "ddf" # ddf / dvf / conditional
  backbone:
    name: "unet" # options include "local", "unet" and "global" - use "global" when method=="affine"
    num_channel_initial: 16
    depth: 3 # original 3
    pooling: true
    concat_skip: true

An example is here https://github.com/DeepRegNet/DeepReg/blob/main/demos/grouped_mr_heart/grouped_mr_heart.yaml#L14

[ciphercharly]

thanks! I will try asap, is a similar simplification required for the optimizer part too?

  optimizer:
    name: "adam"
    adam:
      learning_rate: 1.0e-5 #1.0e-2 bad
    sgd:
      learning_rate: 1.0e-4
      momentum: 0.9
    rms:
      learning_rate: 1.0e-4
      momentum: 0.9

to

  optimizer:
    name: "adam"
    learning_rate: 1.0e-5 

?

[mathpluscode]

thanks! I will try asap, is a similar simplification required for the optimizer part too?

  optimizer:
    name: "adam"
    adam:
      learning_rate: 1.0e-5 #1.0e-2 bad
    sgd:
      learning_rate: 1.0e-4
      momentum: 0.9
    rms:
      learning_rate: 1.0e-4
      momentum: 0.9

For the optimizer, in current main it has not been refactored yet. So for now you do not need to change that.

It will be like this in the near future.

[mathpluscode]

We are working on providing a better documentation for this, and sorry for the confusing. We aim to simplify the configs, and enable users to define custom functionalities and use them more easily without changing deepreg code ^^ thx for your understanding!

[ciphercharly]

no problem! on the contrary, it's good from my perspective to be using something that it's being actively developed and improved

[mathpluscode]

@ciphercharly when you got the time to test the proposed config, please give us feedback, if the error disappeared, please close this issue ^^ thx! Oh, btw, happy new year!

[ciphercharly]

happy new year to you too! will do, hopefully on Monday 18th

[ciphercharly]

ok fixed the config file with the above and got a new error, seems like the part related to loss: needs to be simplified too? getting a key error on "image"

Traceback (most recent call last):
  File "demos/paired_MMIV/demo_train.py", line 29, in <module>
    log_dir=log_dir,
  File "/home/charlie/3DREG-tests/DeepReg/deepreg/train.py", line 94, in train
    max_epochs=max_epochs,
  File "/home/charlie/3DREG-tests/DeepReg/deepreg/train.py", line 45, in build_config
    config = config_parser.load_configs(config_path)
  File "/home/charlie/3DREG-tests/DeepReg/deepreg/parser.py", line 41, in load_configs
    config_sanity_check(config)
  File "/home/charlie/3DREG-tests/DeepReg/deepreg/parser.py", line 97, in config_sanity_check
    loss_weight = config["train"]["loss"]["image"]["weight"]
KeyError: 'image'

config file now:

train:
  method: "ddf" # ddf / dvf / conditional
  backbone:
    name: "unet" # options include "local", "unet" and "global" - use "global" when method=="affine"
    num_channel_initial: 16
    depth: 3 # original 3
    pooling: true
    concat_skip: true

  loss:
    dissimilarity:
      image:
        name: "gmi"  # "lncc" (local normalised cross correlation), "ssd" (sum of squared distance) and "gmi" (differentiable global mutual information loss via Parzen windowing method)
        weight: 1.0
      label:
        weight: 0.0
        name: "multi_scale"
        multi_scale:
          loss_type: "dice" # options include "dice", "cross-entropy", "mean-squared", "generalised_dice" and "jaccard"
          loss_scales: [0, 1, 2, 4, 8, 16]
    regularization:
      weight: 1.0
      energy_type: "gradient-l2" # "bending" # options include "bending", "gradient-l1" and "gradient-l2"

  preprocess:
    batch_size: 2 #original 2
    shuffle_buffer_num_batch: 1

  optimizer:
    name: "adam"
    adam:
      learning_rate: 1.0e-5 #1.0e-2 bad
    sgd:
      learning_rate: 1.0e-4
      momentum: 0.9
    rms:
      learning_rate: 1.0e-4
      momentum: 0.9

  epochs: 10000
  save_period: 500

[mathpluscode]

train: method: "ddf" # ddf / dvf / conditional backbone: name: "unet" # options include "local", "unet" and "global" - use "global" when method=="affine" num_channel_initial: 16 depth: 3 # original 3 pooling: true concat_skip: true

loss: dissimilarity: image: name: "gmi" # "lncc" (local normalised cross correlation), "ssd" (sum of squared distance) and "gmi" (differentiable global mutual information loss via Parzen windowing method) weight: 1.0 label: weight: 0.0 name: "multi_scale" multi_scale: loss_type: "dice" # options include "dice", "cross-entropy", "mean-squared", "generalised_dice" and "jaccard" loss_scales: [0, 1, 2, 4, 8, 16] regularization: weight: 1.0 energy_type: "gradient-l2" # "bending" # options include "bending", "gradient-l1" and "gradient-l2"

preprocess: batch_size: 2 #original 2 shuffle_buffer_num_batch: 1

optimizer: name: "adam" adam: learning_rate: 1.0e-5 #1.0e-2 bad sgd: learning_rate: 1.0e-4 momentum: 0.9 rms: learning_rate: 1.0e-4 momentum: 0.9

epochs: 10000 save_period: 500

Please try

train:
  method: "ddf" # ddf / dvf / conditional
  backbone:
    name: "unet" # options include "local", "unet" and "global" - use "global" when method=="affine"
    num_channel_initial: 16
    depth: 3 # original 3
    pooling: true
    concat_skip: true

  loss:
      image:
        name: "gmi"  # "lncc" (local normalised cross correlation), "ssd" (sum of squared distance) and "gmi" (differentiable global mutual information loss via Parzen windowing method)
        weight: 1.0
      label:
        weight: 0.0
        name: "multi_scale"
        multi_scale:
          loss_type: "dice" # options include "dice", "cross-entropy", "mean-squared", "generalised_dice" and "jaccard"
          loss_scales: [0, 1, 2, 4, 8, 16]
    regularization:
      weight: 1.0
      energy_type: "gradient-l2" # "bending" # options include "bending", "gradient-l1" and "gradient-l2"

  preprocess:
    batch_size: 2 #original 2
    shuffle_buffer_num_batch: 1

  optimizer:
    name: "adam"
    adam:
      learning_rate: 1.0e-5 #1.0e-2 bad
    sgd:
      learning_rate: 1.0e-4
      momentum: 0.9
    rms:
      learning_rate: 1.0e-4
      momentum: 0.9

  epochs: 10000
  save_period: 500

which remove the layer of dissimilarity

also, we have now https://deepreg.readthedocs.io/en/latest/docs/configuration.html a page for config docs.

moreover, we will add some auto transformation to improve the back compatibility this week!

[ciphercharly]

ok, the run now starts but I get a new error and I think the problem is indentation, how should the indentation of the regularization block be compared to image and label?

Traceback (most recent call last):
  File "demos/paired_MMIV/demo_train.py", line 29, in <module>
    log_dir=log_dir,
  File "/home/charlie/3DREG-tests/DeepReg/deepreg/train.py", line 94, in train
    max_epochs=max_epochs,
  File "/home/charlie/3DREG-tests/DeepReg/deepreg/train.py", line 45, in build_config
    config = config_parser.load_configs(config_path)
  File "/home/charlie/3DREG-tests/DeepReg/deepreg/parser.py", line 39, in load_configs
    config_i = yaml.load(file, Loader=yaml.FullLoader)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/yaml/composer.py", line 127, in compose_mapping_node
    while not self.check_event(MappingEndEvent):
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/yaml/parser.py", line 439, in parse_block_mapping_key
    "expected <block end>, but found %r" % token.id, token.start_mark)
yaml.parser.ParserError: while parsing a block mapping
  in "demos/paired_MMIV/paired_MMIV_train.yaml", line 2, column 3
expected <block end>, but found '<block mapping start>'
  in "demos/paired_MMIV/paired_MMIV_train.yaml", line 20, column 5

config now:

train:
  method: "ddf" # ddf / dvf / conditional
  backbone:
    name: "unet" # options include "local", "unet" and "global" - use "global" when method=="affine"
    num_channel_initial: 16
    depth: 3 # original 3
    pooling: true
    concat_skip: true

  loss:
      image:
        name: "gmi"  # "lncc" (local normalised cross correlation), "ssd" (sum of squared distance) and "gmi" (differentiable global mutual information loss via Parzen windowing method)
        weight: 1.0
      label:
        weight: 0.0
        name: "multi_scale"
        multi_scale:
          loss_type: "dice" # options include "dice", "cross-entropy", "mean-squared", "generalised_dice" and "jaccard"
          loss_scales: [0, 1, 2, 4, 8, 16]
    regularization:
      weight: 1.0
      energy_type: "gradient-l2" # "bending" # options include "bending", "gradient-l1" and "gradient-l2"

  preprocess:
    batch_size: 2 #original 2
    shuffle_buffer_num_batch: 1

  optimizer:
    name: "adam"
    adam:
      learning_rate: 1.0e-5 #1.0e-2 bad
    sgd:
      learning_rate: 1.0e-4
      momentum: 0.9
    rms:
      learning_rate: 1.0e-4
      momentum: 0.9

  epochs: 10000
  save_period: 500

[mathpluscode]

ok, the run now starts but I get a new error and I think the problem is indentation, how should the indentation of the regularization block be compared to image and label?

Traceback (most recent call last):
  File "demos/paired_MMIV/demo_train.py", line 29, in <module>
    log_dir=log_dir,
  File "/home/charlie/3DREG-tests/DeepReg/deepreg/train.py", line 94, in train
    max_epochs=max_epochs,
  File "/home/charlie/3DREG-tests/DeepReg/deepreg/train.py", line 45, in build_config
    config = config_parser.load_configs(config_path)
  File "/home/charlie/3DREG-tests/DeepReg/deepreg/parser.py", line 39, in load_configs
    config_i = yaml.load(file, Loader=yaml.FullLoader)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/yaml/composer.py", line 127, in compose_mapping_node
    while not self.check_event(MappingEndEvent):
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/yaml/parser.py", line 439, in parse_block_mapping_key
    "expected <block end>, but found %r" % token.id, token.start_mark)
yaml.parser.ParserError: while parsing a block mapping
  in "demos/paired_MMIV/paired_MMIV_train.yaml", line 2, column 3
expected <block end>, but found '<block mapping start>'
  in "demos/paired_MMIV/paired_MMIV_train.yaml", line 20, column 5

config now:

train:
  method: "ddf" # ddf / dvf / conditional
  backbone:
    name: "unet" # options include "local", "unet" and "global" - use "global" when method=="affine"
    num_channel_initial: 16
    depth: 3 # original 3
    pooling: true
    concat_skip: true

  loss:
      image:
        name: "gmi"  # "lncc" (local normalised cross correlation), "ssd" (sum of squared distance) and "gmi" (differentiable global mutual information loss via Parzen windowing method)
        weight: 1.0
      label:
        weight: 0.0
        name: "multi_scale"
        multi_scale:
          loss_type: "dice" # options include "dice", "cross-entropy", "mean-squared", "generalised_dice" and "jaccard"
          loss_scales: [0, 1, 2, 4, 8, 16]
    regularization:
      weight: 1.0
      energy_type: "gradient-l2" # "bending" # options include "bending", "gradient-l1" and "gradient-l2"

  preprocess:
    batch_size: 2 #original 2
    shuffle_buffer_num_batch: 1

  optimizer:
    name: "adam"
    adam:
      learning_rate: 1.0e-5 #1.0e-2 bad
    sgd:
      learning_rate: 1.0e-4
      momentum: 0.9
    rms:
      learning_rate: 1.0e-4
      momentum: 0.9

  epochs: 10000
  save_period: 500

try this, I also corrected the loss definitions

train:
  method: "ddf" # ddf / dvf / conditional
  backbone:
    name: "unet" # options include "local", "unet" and "global" - use "global" when method=="affine"
    num_channel_initial: 16
    depth: 3 # original 3
    pooling: true
    concat_skip: true

  loss:
    image:
      name: "gmi"  # "lncc" (local normalised cross correlation), "ssd" (sum of squared distance) and "gmi" (differentiable global mutual information loss via Parzen windowing method)
      weight: 1.0
    label:
      weight: 0.0
      name: "dice"
      scales: [0, 1, 2, 4, 8, 16]
    regularization:
      weight: 1.0
      name: "gradient"

  preprocess:
    batch_size: 2 #original 2
    shuffle_buffer_num_batch: 1

  optimizer:
    name: "adam"
    adam:
      learning_rate: 1.0e-5 #1.0e-2 bad
    sgd:
      learning_rate: 1.0e-4
      momentum: 0.9
    rms:
      learning_rate: 1.0e-4
      momentum: 0.9

  epochs: 10000
  save_period: 500

[ciphercharly]

ok the run seems to have started correctly, I had to tweak demo_train.py because now it seems like I can't use 2 GPUs with batch size 2 anymore (previously it worked, see below for the error log), but this is the matter for another issue perhaps, for now I'll let it run on 1 GPU thanks a lot for the fix! I'll close this issue then

File "/home/charlie/anaconda3/envs/deepreg/lib/python3.7/site-packages/tensorflow/python/keras/losses.py", line 192, in _get_reduction
    'Please use `tf.keras.losses.Reduction.SUM` or '
ValueError: Please use `tf.keras.losses.Reduction.SUM` or `tf.keras.losses.Reduction.NONE` for loss reduction when losses are used with `tf.distribute.Strategy` outside of the built-in training loops. You can implement `tf.keras.losses.Reduction.SUM_OVER_BATCH_SIZE` using global batch size like:  
with strategy.scope():
    loss_obj = tf.keras.losses.CategoricalCrossentropy(reduction=tf.keras.losses.Reduction.NONE)
....
    loss = tf.reduce_sum(loss_obj(labels, predictions)) * (1. / global_batch_size)

Please see https://www.tensorflow.org/tutorials/distribute/custom_training for more details.

[mathpluscode]

@ciphercharly would you mind pull again the main? I just merged a new branch, which solved this issue already ^^.

Please have a backup of your current working dir in case it doesn't work. Sorry for all the config-related problems, we are treating this as priority this week.

[ciphercharly]

will do! thanks!

[ciphercharly]

tried the pull and with 2 GPUs still getting an error

2021-01-21 08:28:22.403984: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
WARNING:root:The label loss 0.0 is not positive.
2021-01-21 08:28:24.201600: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2021-01-21 08:28:24.254764: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:0e:00.0 name: Tesla V100-DGXS-32GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 31.75GiB deviceMemoryBandwidth: 836.37GiB/s
2021-01-21 08:28:24.255868: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 1 with properties: 
pciBusID: 0000:0f:00.0 name: Tesla V100-DGXS-32GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 31.75GiB deviceMemoryBandwidth: 836.37GiB/s
2021-01-21 08:28:24.255930: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-01-21 08:28:24.258280: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2021-01-21 08:28:24.260237: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2021-01-21 08:28:24.260646: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2021-01-21 08:28:24.262653: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2021-01-21 08:28:24.263794: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2021-01-21 08:28:24.268435: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2021-01-21 08:28:24.274688: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0, 1
2021-01-21 08:28:24.279448: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-01-21 08:28:24.296112: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2198815000 Hz
2021-01-21 08:28:24.298881: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x560874367c60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-01-21 08:28:24.298980: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2021-01-21 08:28:24.600993: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x560876198450 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-01-21 08:28:24.601059: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2021-01-21 08:28:24.601075: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2021-01-21 08:28:24.609998: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:0e:00.0 name: Tesla V100-DGXS-32GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 31.75GiB deviceMemoryBandwidth: 836.37GiB/s
2021-01-21 08:28:24.612014: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 1 with properties: 
pciBusID: 0000:0f:00.0 name: Tesla V100-DGXS-32GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 31.75GiB deviceMemoryBandwidth: 836.37GiB/s
2021-01-21 08:28:24.612095: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-01-21 08:28:24.612147: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2021-01-21 08:28:24.612226: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2021-01-21 08:28:24.612260: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2021-01-21 08:28:24.612291: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2021-01-21 08:28:24.612321: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2021-01-21 08:28:24.612352: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2021-01-21 08:28:24.620496: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0, 1
2021-01-21 08:28:24.620580: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-01-21 08:28:25.747883: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-01-21 08:28:25.747940: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 1 
2021-01-21 08:28:25.747950: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N Y 
2021-01-21 08:28:25.747956: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 1:   Y N 
2021-01-21 08:28:25.753233: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2021-01-21 08:28:25.753306: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30129 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2021-01-21 08:28:25.755071: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2021-01-21 08:28:25.755125: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 5197 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2021-01-21 08:28:35.413981: I tensorflow/core/profiler/lib/profiler_session.cc:164] Profiler session started.
2021-01-21 08:28:35.414073: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1391] Profiler found 2 GPUs
2021-01-21 08:28:35.416598: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcupti.so.10.1
2021-01-21 08:28:35.517085: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1441] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI_ERROR_INSUFFICIENT_PRIVILEGES
2021-01-21 08:29:06.332891: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2021-01-21 08:29:08.020853: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2021-01-21 08:29:10.983029: I tensorflow/stream_executor/cuda/cuda_driver.cc:775] failed to allocate 1.01G (1089187328 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-01-21 08:29:10.986457: I tensorflow/stream_executor/cuda/cuda_driver.cc:775] failed to allocate 934.86M (980268544 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-01-21 08:29:11.000381: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_1_bfc) ran out of memory trying to allocate 232.00MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-01-21 08:29:11.000434: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once.
2021-01-21 08:29:21.115353: W tensorflow/core/common_runtime/bfc_allocator.cc:431] Allocator (GPU_1_bfc) ran out of memory trying to allocate 216.00MiB (rounded to 226492416)requested by op replica_1/RegistrationModel/functional_1/u_net/up_sample_resnet_block/residual3d_block_4/conv3d_block_9/conv3d_26/conv3d_27/Conv3D
Current allocation summary follows.
2021-01-21 08:29:21.115703: I tensorflow/core/common_runtime/bfc_allocator.cc:970] BFCAllocator dump for GPU_1_bfc
2021-01-21 08:29:21.115775: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (256):   Total Chunks: 354, Chunks in use: 354. 88.5KiB allocated for chunks. 88.5KiB in use in bin. 34.2KiB client-requested in use in bin.
2021-01-21 08:29:21.115799: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (512):   Total Chunks: 33, Chunks in use: 33. 16.5KiB allocated for chunks. 16.5KiB in use in bin. 16.5KiB client-requested in use in bin.
2021-01-21 08:29:21.115816: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (1024):  Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2021-01-21 08:29:21.115833: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (2048):  Total Chunks: 3, Chunks in use: 3. 10.5KiB allocated for chunks. 10.5KiB in use in bin. 10.1KiB client-requested in use in bin.
2021-01-21 08:29:21.115850: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (4096):  Total Chunks: 4, Chunks in use: 4. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 19.3KiB client-requested in use in bin.
2021-01-21 08:29:21.115866: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-01-21 08:29:21.115884: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (16384):         Total Chunks: 13, Chunks in use: 13. 351.0KiB allocated for chunks. 351.0KiB in use in bin. 351.0KiB client-requested in use in bin.
2021-01-21 08:29:21.115911: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (32768):         Total Chunks: 9, Chunks in use: 9. 486.0KiB allocated for chunks. 486.0KiB in use in bin. 486.0KiB client-requested in use in bin.
2021-01-21 08:29:21.115920: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (65536):         Total Chunks: 12, Chunks in use: 12. 1.27MiB allocated for chunks. 1.27MiB in use in bin. 1.27MiB client-requested in use in bin.
2021-01-21 08:29:21.115929: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (131072):        Total Chunks: 9, Chunks in use: 9. 1.90MiB allocated for chunks. 1.90MiB in use in bin. 1.90MiB client-requested in use in bin.
2021-01-21 08:29:21.115938: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (262144):        Total Chunks: 12, Chunks in use: 12. 5.10MiB allocated for chunks. 5.10MiB in use in bin. 5.06MiB client-requested in use in bin.
2021-01-21 08:29:21.115946: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (524288):        Total Chunks: 8, Chunks in use: 8. 6.75MiB allocated for chunks. 6.75MiB in use in bin. 6.75MiB client-requested in use in bin.
2021-01-21 08:29:21.115955: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (1048576):       Total Chunks: 5, Chunks in use: 5. 8.33MiB allocated for chunks. 8.33MiB in use in bin. 7.59MiB client-requested in use in bin.
2021-01-21 08:29:21.115965: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (2097152):       Total Chunks: 9, Chunks in use: 8. 26.96MiB allocated for chunks. 23.62MiB in use in bin. 21.94MiB client-requested in use in bin.
2021-01-21 08:29:21.115974: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 6.75MiB allocated for chunks. 6.75MiB in use in bin. 6.75MiB client-requested in use in bin.
2021-01-21 08:29:21.115983: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (8388608):       Total Chunks: 14, Chunks in use: 14. 189.00MiB allocated for chunks. 189.00MiB in use in bin. 189.00MiB client-requested in use in bin.
2021-01-21 08:29:21.116025: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (16777216):      Total Chunks: 5, Chunks in use: 5. 109.62MiB allocated for chunks. 109.62MiB in use in bin. 101.25MiB client-requested in use in bin.
2021-01-21 08:29:21.116037: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (33554432):      Total Chunks: 12, Chunks in use: 11. 634.50MiB allocated for chunks. 580.50MiB in use in bin. 580.50MiB client-requested in use in bin.
2021-01-21 08:29:21.116047: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (67108864):      Total Chunks: 2, Chunks in use: 2. 174.75MiB allocated for chunks. 174.75MiB in use in bin. 92.01MiB client-requested in use in bin.
2021-01-21 08:29:21.116055: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (134217728):     Total Chunks: 11, Chunks in use: 9. 2.15GiB allocated for chunks. 1.81GiB in use in bin. 1.81GiB client-requested in use in bin.
2021-01-21 08:29:21.116064: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (268435456):     Total Chunks: 5, Chunks in use: 5. 1.59GiB allocated for chunks. 1.59GiB in use in bin. 1.45GiB client-requested in use in bin.
2021-01-21 08:29:21.116073: I tensorflow/core/common_runtime/bfc_allocator.cc:993] Bin for 216.00MiB was 128.00MiB, Chunk State: 
2021-01-21 08:29:21.116088: I tensorflow/core/common_runtime/bfc_allocator.cc:999]   Size: 154.62MiB | Requested Size: 222.80MiB | in_use: 0 | bin_num: 19, prev:   Size: 432.00MiB | Requested Size: 432.00MiB | in_use: 1 | bin_num: -1
2021-01-21 08:29:21.116098: I tensorflow/core/common_runtime/bfc_allocator.cc:999]   Size: 193.37MiB | Requested Size: 0B | in_use: 0 | bin_num: 19, prev:   Size: 216.00MiB | Requested Size: 216.00MiB | in_use: 1 | bin_num: -1
2021-01-21 08:29:21.116105: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 882241792
2021-01-21 08:29:21.116117: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e12000000 of size 226492416 next 519
2021-01-21 08:29:21.116124: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e1f800000 of size 226492416 next 516
2021-01-21 08:29:21.116131: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e2d000000 of size 226492416 next 520
2021-01-21 08:29:21.116138: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f1e3a800000 of size 202764544 next 18446744073709551615
2021-01-21 08:29:21.116145: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 2147483648
2021-01-21 08:29:21.116152: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e5c000000 of size 56623104 next 432
2021-01-21 08:29:21.116159: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e5f600000 of size 56623104 next 433
2021-01-21 08:29:21.116166: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e62c00000 of size 56623104 next 440
2021-01-21 08:29:21.116173: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e66200000 of size 14155776 next 441
2021-01-21 08:29:21.116180: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e66f80000 of size 14155776 next 446
2021-01-21 08:29:21.116187: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e67d00000 of size 14155776 next 443
2021-01-21 08:29:21.116193: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e68a80000 of size 14155776 next 451
2021-01-21 08:29:21.116200: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e69800000 of size 14155776 next 448
2021-01-21 08:29:21.116207: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e6a580000 of size 14155776 next 456
2021-01-21 08:29:21.116214: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e6b300000 of size 3538944 next 461
2021-01-21 08:29:21.116221: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e6b660000 of size 3538944 next 458
2021-01-21 08:29:21.116238: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e6b9c0000 of size 3538944 next 466
2021-01-21 08:29:21.116246: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e6bd20000 of size 3538944 next 463
2021-01-21 08:29:21.116254: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e6c080000 of size 17694720 next 468
2021-01-21 08:29:21.116261: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e6d160000 of size 14155776 next 478
2021-01-21 08:29:21.116268: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e6dee0000 of size 14155776 next 471
2021-01-21 08:29:21.116274: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e6ec60000 of size 14155776 next 483
2021-01-21 08:29:21.116281: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e6f9e0000 of size 14155776 next 480
2021-01-21 08:29:21.116288: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e70760000 of size 14155776 next 488
2021-01-21 08:29:21.116295: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e714e0000 of size 56623104 next 485
2021-01-21 08:29:21.116302: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e74ae0000 of size 56623104 next 493
2021-01-21 08:29:21.116309: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e780e0000 of size 56623104 next 497
2021-01-21 08:29:21.116315: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e7b6e0000 of size 56623104 next 494
2021-01-21 08:29:21.116322: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e7ece0000 of size 56623104 next 502
2021-01-21 08:29:21.116329: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e822e0000 of size 325582848 next 499
2021-01-21 08:29:21.116336: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e95960000 of size 56623104 next 503
2021-01-21 08:29:21.116343: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f1e98f60000 of size 56623104 next 508
2021-01-21 08:29:21.116350: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e9c560000 of size 226492416 next 505
2021-01-21 08:29:21.116357: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1ea9d60000 of size 226492416 next 513
2021-01-21 08:29:21.116364: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1eb7560000 of size 452984832 next 510
2021-01-21 08:29:21.116371: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f1ed2560000 of size 162136064 next 18446744073709551615
2021-01-21 08:29:21.116378: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 1073741824
2021-01-21 08:29:21.116385: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2200000000 of size 325582848 next 412
2021-01-21 08:29:21.116392: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2213680000 of size 226492416 next 423
2021-01-21 08:29:21.116399: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2220e80000 of size 226492416 next 420
2021-01-21 08:29:21.116406: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f222e680000 of size 295174144 next 18446744073709551615
2021-01-21 08:29:21.116413: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 536870912
2021-01-21 08:29:21.116420: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2240000000 of size 226492416 next 408
2021-01-21 08:29:21.116427: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f224d800000 of size 310378496 next 18446744073709551615
2021-01-21 08:29:21.116434: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 268435456
2021-01-21 08:29:21.116441: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2290000000 of size 14155776 next 405
2021-01-21 08:29:21.116448: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2290d80000 of size 7077888 next 437
2021-01-21 08:29:21.116463: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2291440000 of size 3538944 next 453
2021-01-21 08:29:21.116470: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a0000 of size 512 next 457
2021-01-21 08:29:21.116477: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a0200 of size 512 next 459
2021-01-21 08:29:21.116484: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a0400 of size 512 next 460
2021-01-21 08:29:21.116491: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a0600 of size 512 next 462
2021-01-21 08:29:21.116498: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a0800 of size 512 next 464
2021-01-21 08:29:21.116504: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a0a00 of size 512 next 465
2021-01-21 08:29:21.116511: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a0c00 of size 512 next 467
2021-01-21 08:29:21.116518: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a0e00 of size 512 next 469
2021-01-21 08:29:21.116525: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1000 of size 512 next 470
2021-01-21 08:29:21.116532: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1200 of size 256 next 472
2021-01-21 08:29:21.116538: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1300 of size 256 next 474
2021-01-21 08:29:21.116545: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1400 of size 256 next 475
2021-01-21 08:29:21.116552: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1500 of size 256 next 473
2021-01-21 08:29:21.116559: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1600 of size 256 next 476
2021-01-21 08:29:21.116566: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1700 of size 256 next 477
2021-01-21 08:29:21.116572: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1800 of size 256 next 479
2021-01-21 08:29:21.116579: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1900 of size 256 next 481
2021-01-21 08:29:21.116586: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1a00 of size 256 next 482
2021-01-21 08:29:21.116593: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1b00 of size 256 next 484
2021-01-21 08:29:21.116600: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1c00 of size 256 next 486
2021-01-21 08:29:21.116606: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1d00 of size 256 next 487
2021-01-21 08:29:21.116613: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1e00 of size 256 next 489
2021-01-21 08:29:21.116620: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1f00 of size 256 next 491
2021-01-21 08:29:21.116627: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2000 of size 256 next 492
2021-01-21 08:29:21.116634: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2100 of size 256 next 490
2021-01-21 08:29:21.116640: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2200 of size 256 next 495
2021-01-21 08:29:21.116647: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2300 of size 256 next 496
2021-01-21 08:29:21.116654: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2400 of size 256 next 498
2021-01-21 08:29:21.116661: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2500 of size 256 next 500
2021-01-21 08:29:21.116668: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2600 of size 256 next 501
2021-01-21 08:29:21.116674: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2700 of size 256 next 504
2021-01-21 08:29:21.116688: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2800 of size 256 next 506
2021-01-21 08:29:21.116696: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2900 of size 256 next 507
2021-01-21 08:29:21.116703: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2a00 of size 256 next 509
2021-01-21 08:29:21.116709: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2b00 of size 256 next 511
2021-01-21 08:29:21.116716: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2c00 of size 256 next 512
2021-01-21 08:29:21.116723: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2d00 of size 256 next 515
2021-01-21 08:29:21.116730: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2e00 of size 256 next 517
2021-01-21 08:29:21.116736: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2f00 of size 256 next 518
2021-01-21 08:29:21.116744: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a3000 of size 27648 next 521
2021-01-21 08:29:21.116751: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f22917a9c00 of size 3499008 next 403
2021-01-21 08:29:21.116758: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2291b00000 of size 28311552 next 406
2021-01-21 08:29:21.116765: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2293600000 of size 28311552 next 409
2021-01-21 08:29:21.116772: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2295100000 of size 14155776 next 414
2021-01-21 08:29:21.116779: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2295e80000 of size 56623104 next 428
2021-01-21 08:29:21.116786: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2299480000 of size 112721920 next 18446744073709551615
2021-01-21 08:29:21.116793: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 134217728
2021-01-21 08:29:21.116801: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22b0000000 of size 42467328 next 371
2021-01-21 08:29:21.116808: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22b2880000 of size 21233664 next 402
2021-01-21 08:29:21.116815: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22b3cc0000 of size 70516736 next 18446744073709551615
2021-01-21 08:29:21.116822: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 134217728
2021-01-21 08:29:21.116830: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22b8000000 of size 134217728 next 18446744073709551615
2021-01-21 08:29:21.116836: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 33554432
2021-01-21 08:29:21.116844: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22ca000000 of size 14155776 next 307
2021-01-21 08:29:21.116851: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22cad80000 of size 19398656 next 18446744073709551615
2021-01-21 08:29:21.116858: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 8388608
2021-01-21 08:29:21.116865: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a800000 of size 512 next 64
2021-01-21 08:29:21.116872: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a800200 of size 512 next 65
2021-01-21 08:29:21.116879: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a800400 of size 512 next 66
2021-01-21 08:29:21.116886: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a800600 of size 512 next 67
2021-01-21 08:29:21.116893: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a800800 of size 884736 next 68
2021-01-21 08:29:21.116900: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a8d8800 of size 256 next 69
2021-01-21 08:29:21.116914: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a8d8900 of size 256 next 70
2021-01-21 08:29:21.116921: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a8d8a00 of size 256 next 71
2021-01-21 08:29:21.116928: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a8d8b00 of size 256 next 72
2021-01-21 08:29:21.116935: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a8d8c00 of size 884736 next 73
2021-01-21 08:29:21.116942: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a9b0c00 of size 256 next 74
2021-01-21 08:29:21.116949: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a9b0d00 of size 256 next 75
2021-01-21 08:29:21.116955: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a9b0e00 of size 256 next 76
2021-01-21 08:29:21.116962: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a9b0f00 of size 256 next 77
2021-01-21 08:29:21.116970: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a9b1000 of size 442368 next 78
2021-01-21 08:29:21.116976: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aa1d000 of size 256 next 79
2021-01-21 08:29:21.116983: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aa1d100 of size 256 next 80
2021-01-21 08:29:21.116990: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aa1d200 of size 256 next 81
2021-01-21 08:29:21.116997: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aa1d300 of size 256 next 82
2021-01-21 08:29:21.117004: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aa1d400 of size 442368 next 83
2021-01-21 08:29:21.117011: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aa89400 of size 256 next 84
2021-01-21 08:29:21.117017: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aa89500 of size 256 next 85
2021-01-21 08:29:21.117024: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aa89600 of size 256 next 86
2021-01-21 08:29:21.117031: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aa89700 of size 256 next 87
2021-01-21 08:29:21.117038: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aa89800 of size 221184 next 88
2021-01-21 08:29:21.117045: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aabf800 of size 256 next 89
2021-01-21 08:29:21.117052: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aabf900 of size 256 next 90
2021-01-21 08:29:21.117059: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aabfa00 of size 256 next 91
2021-01-21 08:29:21.117066: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aabfb00 of size 256 next 92
2021-01-21 08:29:21.117072: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aabfc00 of size 221184 next 93
2021-01-21 08:29:21.117079: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aaf5c00 of size 256 next 94
2021-01-21 08:29:21.117086: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aaf5d00 of size 256 next 95
2021-01-21 08:29:21.117093: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aaf5e00 of size 256 next 96
2021-01-21 08:29:21.117100: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aaf5f00 of size 256 next 97
2021-01-21 08:29:21.117107: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aaf6000 of size 110592 next 98
2021-01-21 08:29:21.117114: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab11000 of size 256 next 99
2021-01-21 08:29:21.117121: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab11100 of size 256 next 100
2021-01-21 08:29:21.117128: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab11200 of size 256 next 101
2021-01-21 08:29:21.117134: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab11300 of size 256 next 102
2021-01-21 08:29:21.117148: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab11400 of size 110592 next 103
2021-01-21 08:29:21.117156: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab2c400 of size 256 next 104
2021-01-21 08:29:21.117163: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab2c500 of size 256 next 105
2021-01-21 08:29:21.117169: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab2c600 of size 256 next 106
2021-01-21 08:29:21.117176: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab2c700 of size 256 next 107
2021-01-21 08:29:21.117235: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab2c800 of size 55296 next 108
2021-01-21 08:29:21.117243: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab3a000 of size 256 next 109
2021-01-21 08:29:21.117250: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab3a100 of size 256 next 110
2021-01-21 08:29:21.117257: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab3a200 of size 256 next 111
2021-01-21 08:29:21.117264: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab3a300 of size 256 next 112
2021-01-21 08:29:21.117270: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab3a400 of size 55296 next 113
2021-01-21 08:29:21.117277: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab47c00 of size 256 next 114
2021-01-21 08:29:21.117284: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab47d00 of size 256 next 115
2021-01-21 08:29:21.117291: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab47e00 of size 256 next 116
2021-01-21 08:29:21.117298: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab47f00 of size 256 next 117
2021-01-21 08:29:21.117305: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab48000 of size 27648 next 118
2021-01-21 08:29:21.117312: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab4ec00 of size 256 next 119
2021-01-21 08:29:21.117318: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab4ed00 of size 256 next 120
2021-01-21 08:29:21.117325: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab4ee00 of size 256 next 121
2021-01-21 08:29:21.117332: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab4ef00 of size 256 next 122
2021-01-21 08:29:21.117339: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab4f000 of size 27648 next 123
2021-01-21 08:29:21.117346: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab55c00 of size 256 next 124
2021-01-21 08:29:21.117352: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab55d00 of size 256 next 125
2021-01-21 08:29:21.117359: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab55e00 of size 256 next 126
2021-01-21 08:29:21.117366: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab55f00 of size 256 next 127
2021-01-21 08:29:21.117373: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab56000 of size 5376 next 128
2021-01-21 08:29:21.117380: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab57500 of size 256 next 129
2021-01-21 08:29:21.117387: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab57600 of size 256 next 131
2021-01-21 08:29:21.117424: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab57700 of size 256 next 130
2021-01-21 08:29:21.117433: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab57800 of size 256 next 132
2021-01-21 08:29:21.117440: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab57900 of size 256 next 133
2021-01-21 08:29:21.117447: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab57a00 of size 256 next 134
2021-01-21 08:29:21.117462: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab57b00 of size 256 next 135
2021-01-21 08:29:21.117470: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab57c00 of size 256 next 136
2021-01-21 08:29:21.117477: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab57d00 of size 256 next 137
2021-01-21 08:29:21.117484: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab57e00 of size 256 next 138
2021-01-21 08:29:21.117491: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab57f00 of size 256 next 139
2021-01-21 08:29:21.117497: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab58000 of size 256 next 140
2021-01-21 08:29:21.117504: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab58100 of size 256 next 141
2021-01-21 08:29:21.117511: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab58200 of size 256 next 142
2021-01-21 08:29:21.117518: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab58300 of size 256 next 143
2021-01-21 08:29:21.117525: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab58400 of size 256 next 144
2021-01-21 08:29:21.117532: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab58500 of size 256 next 145
2021-01-21 08:29:21.117538: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab58600 of size 256 next 146
2021-01-21 08:29:21.117545: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab58700 of size 256 next 147
2021-01-21 08:29:21.117552: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab58800 of size 256 next 148
2021-01-21 08:29:21.117559: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab58900 of size 256 next 149
2021-01-21 08:29:21.117566: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab58a00 of size 3584 next 150
2021-01-21 08:29:21.117573: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab59800 of size 256 next 151
2021-01-21 08:29:21.117580: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab59900 of size 256 next 152
2021-01-21 08:29:21.117587: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab59a00 of size 27648 next 153
2021-01-21 08:29:21.117594: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab60600 of size 256 next 154
2021-01-21 08:29:21.117601: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab60700 of size 256 next 155
2021-01-21 08:29:21.117608: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab60800 of size 27648 next 156
2021-01-21 08:29:21.117615: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab67400 of size 256 next 157
2021-01-21 08:29:21.117622: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab67500 of size 256 next 158
2021-01-21 08:29:21.117629: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab67600 of size 55296 next 159
2021-01-21 08:29:21.117635: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab74e00 of size 256 next 160
2021-01-21 08:29:21.117642: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab74f00 of size 256 next 161
2021-01-21 08:29:21.117649: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab75000 of size 110592 next 162
2021-01-21 08:29:21.117656: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab90000 of size 256 next 163
2021-01-21 08:29:21.117663: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab90100 of size 256 next 164
2021-01-21 08:29:21.117670: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab90200 of size 110592 next 165
2021-01-21 08:29:21.117676: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230abab200 of size 256 next 166
2021-01-21 08:29:21.117690: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230abab300 of size 256 next 167
2021-01-21 08:29:21.117698: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230abab400 of size 221184 next 168
2021-01-21 08:29:21.117705: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230abe1400 of size 256 next 169
2021-01-21 08:29:21.117712: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230abe1500 of size 256 next 170
2021-01-21 08:29:21.117719: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230abe1600 of size 442368 next 171
2021-01-21 08:29:21.117726: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ac4d600 of size 256 next 172
2021-01-21 08:29:21.117733: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ac4d700 of size 256 next 173
2021-01-21 08:29:21.117739: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ac4d800 of size 442368 next 174
2021-01-21 08:29:21.117746: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230acb9800 of size 256 next 175
2021-01-21 08:29:21.117753: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230acb9900 of size 256 next 176
2021-01-21 08:29:21.117760: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230acb9a00 of size 884736 next 177
2021-01-21 08:29:21.117767: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ad91a00 of size 512 next 178
2021-01-21 08:29:21.117774: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ad91c00 of size 512 next 179
2021-01-21 08:29:21.117781: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ad91e00 of size 2548224 next 18446744073709551615
2021-01-21 08:29:21.117789: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 16777216
2021-01-21 08:29:21.117796: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b000000 of size 512 next 181
2021-01-21 08:29:21.117803: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b000200 of size 512 next 182
2021-01-21 08:29:21.117810: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b000400 of size 1769472 next 183
2021-01-21 08:29:21.117817: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1b0400 of size 512 next 184
2021-01-21 08:29:21.117824: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1b0600 of size 512 next 185
2021-01-21 08:29:21.117831: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1b0800 of size 55296 next 186
2021-01-21 08:29:21.117837: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1be000 of size 256 next 187
2021-01-21 08:29:21.117844: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1be100 of size 256 next 188
2021-01-21 08:29:21.117851: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1be200 of size 27648 next 189
2021-01-21 08:29:21.117858: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1c4e00 of size 256 next 190
2021-01-21 08:29:21.117865: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1c4f00 of size 256 next 191
2021-01-21 08:29:21.117872: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1c5000 of size 27648 next 192
2021-01-21 08:29:21.117878: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1cbc00 of size 256 next 193
2021-01-21 08:29:21.117885: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1cbd00 of size 256 next 194
2021-01-21 08:29:21.117892: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1cbe00 of size 55296 next 195
2021-01-21 08:29:21.117899: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1d9600 of size 256 next 196
2021-01-21 08:29:21.117906: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1d9700 of size 256 next 197
2021-01-21 08:29:21.117920: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1d9800 of size 221184 next 198
2021-01-21 08:29:21.117927: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b20f800 of size 256 next 199
2021-01-21 08:29:21.117934: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b20f900 of size 256 next 200
2021-01-21 08:29:21.117941: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b20fa00 of size 110592 next 201
2021-01-21 08:29:21.117948: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b22aa00 of size 256 next 202
2021-01-21 08:29:21.117955: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b22ab00 of size 256 next 203
2021-01-21 08:29:21.117962: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b22ac00 of size 110592 next 204
2021-01-21 08:29:21.117968: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b245c00 of size 256 next 205
2021-01-21 08:29:21.117975: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b245d00 of size 256 next 206
2021-01-21 08:29:21.117982: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b245e00 of size 221184 next 207
2021-01-21 08:29:21.117989: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b27be00 of size 256 next 208
2021-01-21 08:29:21.117996: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b27bf00 of size 256 next 209
2021-01-21 08:29:21.118003: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b27c000 of size 884736 next 210
2021-01-21 08:29:21.118010: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b354000 of size 256 next 211
2021-01-21 08:29:21.118017: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b354100 of size 256 next 212
2021-01-21 08:29:21.118023: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b354200 of size 442368 next 213
2021-01-21 08:29:21.118030: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b3c0200 of size 256 next 214
2021-01-21 08:29:21.118037: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b3c0300 of size 256 next 215
2021-01-21 08:29:21.118044: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b3c0400 of size 442368 next 216
2021-01-21 08:29:21.118051: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b42c400 of size 256 next 217
2021-01-21 08:29:21.118057: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b42c500 of size 256 next 218
2021-01-21 08:29:21.118064: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b42c600 of size 884736 next 219
2021-01-21 08:29:21.118071: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b504600 of size 256 next 220
2021-01-21 08:29:21.118078: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b504700 of size 256 next 221
2021-01-21 08:29:21.118085: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b504800 of size 5376 next 222
2021-01-21 08:29:21.118091: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b505d00 of size 256 next 223
2021-01-21 08:29:21.118098: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b505e00 of size 3584 next 224
2021-01-21 08:29:21.118105: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b506c00 of size 256 next 225
2021-01-21 08:29:21.118112: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b506d00 of size 256 next 226
2021-01-21 08:29:21.118119: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b506e00 of size 27648 next 227
2021-01-21 08:29:21.118125: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b50da00 of size 256 next 228
2021-01-21 08:29:21.118132: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b50db00 of size 256 next 229
2021-01-21 08:29:21.118146: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b50dc00 of size 27648 next 230
2021-01-21 08:29:21.118154: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b514800 of size 256 next 231
2021-01-21 08:29:21.118160: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b514900 of size 256 next 232
2021-01-21 08:29:21.118167: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b514a00 of size 55296 next 233
2021-01-21 08:29:21.118174: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b522200 of size 256 next 234
2021-01-21 08:29:21.118181: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b522300 of size 256 next 235
2021-01-21 08:29:21.118188: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b522400 of size 110592 next 236
2021-01-21 08:29:21.118195: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b53d400 of size 256 next 237
2021-01-21 08:29:21.118202: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b53d500 of size 256 next 238
2021-01-21 08:29:21.118208: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b53d600 of size 110592 next 239
2021-01-21 08:29:21.118215: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b558600 of size 256 next 240
2021-01-21 08:29:21.118222: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b558700 of size 256 next 241
2021-01-21 08:29:21.118229: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b558800 of size 221184 next 242
2021-01-21 08:29:21.118236: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b58e800 of size 256 next 243
2021-01-21 08:29:21.118243: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b58e900 of size 256 next 244
2021-01-21 08:29:21.118249: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b58ea00 of size 442368 next 245
2021-01-21 08:29:21.118256: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b5faa00 of size 256 next 246
2021-01-21 08:29:21.118263: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b5fab00 of size 256 next 247
2021-01-21 08:29:21.118270: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b5fac00 of size 442368 next 248
2021-01-21 08:29:21.118276: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b666c00 of size 256 next 249
2021-01-21 08:29:21.118283: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b666d00 of size 256 next 250
2021-01-21 08:29:21.118290: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b666e00 of size 884736 next 251
2021-01-21 08:29:21.118297: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b73ee00 of size 512 next 252
2021-01-21 08:29:21.118304: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b73f000 of size 512 next 253
2021-01-21 08:29:21.118311: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b73f200 of size 1769472 next 254
2021-01-21 08:29:21.118318: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b8ef200 of size 512 next 255
2021-01-21 08:29:21.118324: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b8ef400 of size 512 next 256
2021-01-21 08:29:21.118331: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b8ef600 of size 1769472 next 257
2021-01-21 08:29:21.118338: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ba9f600 of size 512 next 258
2021-01-21 08:29:21.118345: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ba9f800 of size 512 next 259
2021-01-21 08:29:21.118352: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ba9fa00 of size 55296 next 260
2021-01-21 08:29:21.118359: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230baad200 of size 256 next 261
2021-01-21 08:29:21.118372: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230baad300 of size 256 next 262
2021-01-21 08:29:21.118380: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230baad400 of size 27648 next 263
2021-01-21 08:29:21.118387: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bab4000 of size 256 next 264
2021-01-21 08:29:21.118394: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bab4100 of size 256 next 265
2021-01-21 08:29:21.118401: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bab4200 of size 27648 next 266
2021-01-21 08:29:21.118408: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230babae00 of size 256 next 267
2021-01-21 08:29:21.118414: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230babaf00 of size 256 next 268
2021-01-21 08:29:21.118421: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230babb000 of size 55296 next 269
2021-01-21 08:29:21.118428: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bac8800 of size 256 next 270
2021-01-21 08:29:21.118435: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bac8900 of size 256 next 271
2021-01-21 08:29:21.118442: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bac8a00 of size 221184 next 272
2021-01-21 08:29:21.118449: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bafea00 of size 256 next 273
2021-01-21 08:29:21.118456: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bafeb00 of size 256 next 274
2021-01-21 08:29:21.118463: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bafec00 of size 110592 next 275
2021-01-21 08:29:21.118470: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bb19c00 of size 256 next 276
2021-01-21 08:29:21.118476: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bb19d00 of size 256 next 277
2021-01-21 08:29:21.118483: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bb19e00 of size 110592 next 278
2021-01-21 08:29:21.118490: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bb34e00 of size 256 next 279
2021-01-21 08:29:21.118497: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bb34f00 of size 256 next 280
2021-01-21 08:29:21.118504: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bb35000 of size 221184 next 281
2021-01-21 08:29:21.118510: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bb6b000 of size 256 next 282
2021-01-21 08:29:21.118517: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bb6b100 of size 256 next 283
2021-01-21 08:29:21.118524: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bb6b200 of size 884736 next 284
2021-01-21 08:29:21.118531: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bc43200 of size 256 next 285
2021-01-21 08:29:21.118538: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bc43300 of size 256 next 286
2021-01-21 08:29:21.118545: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bc43400 of size 442368 next 287
2021-01-21 08:29:21.118552: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bcaf400 of size 256 next 288
2021-01-21 08:29:21.118558: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bcaf500 of size 256 next 289
2021-01-21 08:29:21.118565: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bcaf600 of size 442368 next 290
2021-01-21 08:29:21.118572: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bd1b600 of size 256 next 291
2021-01-21 08:29:21.118579: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bd1b700 of size 256 next 292
2021-01-21 08:29:21.118586: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bd1b800 of size 884736 next 293
2021-01-21 08:29:21.118600: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf3800 of size 256 next 294
2021-01-21 08:29:21.118608: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf3900 of size 256 next 295
2021-01-21 08:29:21.118615: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf3a00 of size 5376 next 296
2021-01-21 08:29:21.118622: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf4f00 of size 256 next 297
2021-01-21 08:29:21.118629: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf5000 of size 256 next 298
2021-01-21 08:29:21.118635: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf5100 of size 256 next 299
2021-01-21 08:29:21.118642: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf5200 of size 256 next 300
2021-01-21 08:29:21.118649: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf5300 of size 256 next 301
2021-01-21 08:29:21.118656: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf5400 of size 256 next 302
2021-01-21 08:29:21.118663: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf5500 of size 256 next 303
2021-01-21 08:29:21.118670: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf5600 of size 256 next 304
2021-01-21 08:29:21.118677: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf5700 of size 4352 next 305
2021-01-21 08:29:21.118684: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf6800 of size 256 next 309
2021-01-21 08:29:21.118690: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf6900 of size 256 next 310
2021-01-21 08:29:21.118697: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf6a00 of size 256 next 311
2021-01-21 08:29:21.118704: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf6b00 of size 256 next 312
2021-01-21 08:29:21.118711: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf6c00 of size 256 next 313
2021-01-21 08:29:21.118718: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf6d00 of size 256 next 314
2021-01-21 08:29:21.118725: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf6e00 of size 256 next 315
2021-01-21 08:29:21.118732: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf6f00 of size 256 next 316
2021-01-21 08:29:21.118738: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7000 of size 256 next 317
2021-01-21 08:29:21.118745: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7100 of size 256 next 318
2021-01-21 08:29:21.118752: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7200 of size 256 next 319
2021-01-21 08:29:21.118759: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7300 of size 256 next 320
2021-01-21 08:29:21.118766: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7400 of size 256 next 321
2021-01-21 08:29:21.118773: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7500 of size 256 next 322
2021-01-21 08:29:21.118780: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7600 of size 256 next 323
2021-01-21 08:29:21.118786: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7700 of size 256 next 324
2021-01-21 08:29:21.118793: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7800 of size 256 next 325
2021-01-21 08:29:21.118800: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7900 of size 256 next 326
2021-01-21 08:29:21.118807: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7a00 of size 256 next 327
2021-01-21 08:29:21.118814: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7b00 of size 256 next 328
2021-01-21 08:29:21.118828: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7c00 of size 256 next 329
2021-01-21 08:29:21.118836: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7d00 of size 256 next 330
2021-01-21 08:29:21.118842: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7e00 of size 256 next 331
2021-01-21 08:29:21.118849: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7f00 of size 256 next 332
2021-01-21 08:29:21.118856: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8000 of size 256 next 333
2021-01-21 08:29:21.118863: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8100 of size 256 next 334
2021-01-21 08:29:21.118870: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8200 of size 256 next 335
2021-01-21 08:29:21.118876: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8300 of size 256 next 336
2021-01-21 08:29:21.118884: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8400 of size 256 next 337
2021-01-21 08:29:21.118890: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8500 of size 256 next 338
2021-01-21 08:29:21.118897: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8600 of size 256 next 339
2021-01-21 08:29:21.118904: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8700 of size 256 next 340
2021-01-21 08:29:21.118911: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8800 of size 256 next 341
2021-01-21 08:29:21.118918: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8900 of size 256 next 342
2021-01-21 08:29:21.118933: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8a00 of size 256 next 343
2021-01-21 08:29:21.118943: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8b00 of size 256 next 344
2021-01-21 08:29:21.118950: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8c00 of size 256 next 345
2021-01-21 08:29:21.118957: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8d00 of size 256 next 346
2021-01-21 08:29:21.118963: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8e00 of size 256 next 347
2021-01-21 08:29:21.118970: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8f00 of size 256 next 348
2021-01-21 08:29:21.118977: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9000 of size 256 next 349
2021-01-21 08:29:21.118984: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9100 of size 256 next 350
2021-01-21 08:29:21.118991: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9200 of size 256 next 351
2021-01-21 08:29:21.118998: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9300 of size 256 next 352
2021-01-21 08:29:21.119004: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9400 of size 256 next 353
2021-01-21 08:29:21.119011: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9500 of size 256 next 354
2021-01-21 08:29:21.119018: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9600 of size 256 next 355
2021-01-21 08:29:21.119025: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9700 of size 256 next 356
2021-01-21 08:29:21.119032: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9800 of size 256 next 357
2021-01-21 08:29:21.119039: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9900 of size 256 next 358
2021-01-21 08:29:21.119046: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9a00 of size 256 next 359
2021-01-21 08:29:21.119052: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9b00 of size 256 next 360
2021-01-21 08:29:21.119059: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9c00 of size 256 next 361
2021-01-21 08:29:21.119088: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9d00 of size 256 next 362
2021-01-21 08:29:21.119096: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9e00 of size 256 next 363
2021-01-21 08:29:21.119103: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9f00 of size 256 next 364
2021-01-21 08:29:21.119110: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfa000 of size 256 next 365
2021-01-21 08:29:21.119117: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfa100 of size 256 next 366
2021-01-21 08:29:21.119124: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfa200 of size 256 next 367
2021-01-21 08:29:21.119130: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfa300 of size 256 next 368
2021-01-21 08:29:21.119137: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfa400 of size 256 next 369
2021-01-21 08:29:21.119144: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfa500 of size 256 next 372
2021-01-21 08:29:21.119151: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfa600 of size 256 next 373
2021-01-21 08:29:21.119158: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfa700 of size 256 next 374
2021-01-21 08:29:21.119164: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfa800 of size 256 next 375
2021-01-21 08:29:21.119171: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfa900 of size 256 next 376
2021-01-21 08:29:21.119178: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfaa00 of size 256 next 377
2021-01-21 08:29:21.119185: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfab00 of size 256 next 378
2021-01-21 08:29:21.119192: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfac00 of size 256 next 379
2021-01-21 08:29:21.119198: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfad00 of size 256 next 380
2021-01-21 08:29:21.119205: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfae00 of size 256 next 381
2021-01-21 08:29:21.119212: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfaf00 of size 256 next 382
2021-01-21 08:29:21.119219: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfb000 of size 256 next 383
2021-01-21 08:29:21.119226: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfb100 of size 256 next 384
2021-01-21 08:29:21.119233: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfb200 of size 256 next 385
2021-01-21 08:29:21.119240: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfb300 of size 256 next 386
2021-01-21 08:29:21.119246: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfb400 of size 256 next 387
2021-01-21 08:29:21.119253: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfb500 of size 256 next 388
2021-01-21 08:29:21.119260: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfb600 of size 256 next 389
2021-01-21 08:29:21.119267: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfb700 of size 256 next 390
2021-01-21 08:29:21.119274: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfb800 of size 256 next 391
2021-01-21 08:29:21.119281: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfb900 of size 256 next 392
2021-01-21 08:29:21.119287: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfba00 of size 256 next 393
2021-01-21 08:29:21.119294: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfbb00 of size 256 next 394
2021-01-21 08:29:21.119301: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfbc00 of size 256 next 395
2021-01-21 08:29:21.119319: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfbd00 of size 256 next 396
2021-01-21 08:29:21.119327: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfbe00 of size 256 next 397
2021-01-21 08:29:21.119334: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfbf00 of size 256 next 398
2021-01-21 08:29:21.119341: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfc000 of size 256 next 399
2021-01-21 08:29:21.119347: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfc100 of size 256 next 400
2021-01-21 08:29:21.119354: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfc200 of size 256 next 401
2021-01-21 08:29:21.119361: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfc300 of size 256 next 415
2021-01-21 08:29:21.119368: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfc400 of size 256 next 416
2021-01-21 08:29:21.119375: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfc500 of size 256 next 417
2021-01-21 08:29:21.119381: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfc600 of size 256 next 419
2021-01-21 08:29:21.119388: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfc700 of size 256 next 418
2021-01-21 08:29:21.119395: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfc800 of size 256 next 421
2021-01-21 08:29:21.119402: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfc900 of size 256 next 422
2021-01-21 08:29:21.119408: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfca00 of size 256 next 424
2021-01-21 08:29:21.119415: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfcb00 of size 256 next 426
2021-01-21 08:29:21.119422: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfcc00 of size 256 next 427
2021-01-21 08:29:21.119429: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfcd00 of size 256 next 429
2021-01-21 08:29:21.119436: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfce00 of size 256 next 430
2021-01-21 08:29:21.119442: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfcf00 of size 256 next 431
2021-01-21 08:29:21.119450: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfd000 of size 256 next 410
2021-01-21 08:29:21.119456: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfd100 of size 256 next 413
2021-01-21 08:29:21.119463: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfd200 of size 256 next 434
2021-01-21 08:29:21.119470: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfd300 of size 256 next 435
2021-01-21 08:29:21.119477: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfd400 of size 256 next 436
2021-01-21 08:29:21.119484: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfd500 of size 256 next 438
2021-01-21 08:29:21.119490: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfd600 of size 256 next 439
2021-01-21 08:29:21.119497: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfd700 of size 256 next 442
2021-01-21 08:29:21.119504: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfd800 of size 256 next 444
2021-01-21 08:29:21.119511: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfd900 of size 256 next 445
2021-01-21 08:29:21.119517: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfda00 of size 256 next 447
2021-01-21 08:29:21.119524: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfdb00 of size 256 next 449
2021-01-21 08:29:21.119531: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfdc00 of size 256 next 450
2021-01-21 08:29:21.119545: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfdd00 of size 256 next 452
2021-01-21 08:29:21.119553: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfde00 of size 256 next 454
2021-01-21 08:29:21.119560: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfdf00 of size 256 next 455
2021-01-21 08:29:21.119567: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfe000 of size 2105344 next 18446744073709551615
2021-01-21 08:29:21.119574: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 1048576
2021-01-21 08:29:21.119582: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e00000 of size 256 next 1
2021-01-21 08:29:21.119589: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e00100 of size 1280 next 2
2021-01-21 08:29:21.119596: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e00600 of size 256 next 3
2021-01-21 08:29:21.119603: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e00700 of size 256 next 4
2021-01-21 08:29:21.119610: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e00800 of size 256 next 5
2021-01-21 08:29:21.119617: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e00900 of size 256 next 6
2021-01-21 08:29:21.119623: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e00a00 of size 256 next 7
2021-01-21 08:29:21.119630: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e00b00 of size 3584 next 8
2021-01-21 08:29:21.119637: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e01900 of size 256 next 9
2021-01-21 08:29:21.119644: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e01a00 of size 256 next 10
2021-01-21 08:29:21.119651: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e01b00 of size 256 next 11
2021-01-21 08:29:21.119657: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e01c00 of size 256 next 12
2021-01-21 08:29:21.119664: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e01d00 of size 27648 next 13
2021-01-21 08:29:21.119671: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e08900 of size 256 next 14
2021-01-21 08:29:21.119678: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e08a00 of size 256 next 15
2021-01-21 08:29:21.119685: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e08b00 of size 256 next 16
2021-01-21 08:29:21.119692: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e08c00 of size 256 next 17
2021-01-21 08:29:21.119699: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e08d00 of size 27648 next 18
2021-01-21 08:29:21.119706: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e0f900 of size 256 next 19
2021-01-21 08:29:21.119712: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e0fa00 of size 256 next 20
2021-01-21 08:29:21.119719: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e0fb00 of size 256 next 21
2021-01-21 08:29:21.119726: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e0fc00 of size 256 next 22
2021-01-21 08:29:21.119733: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e0fd00 of size 55296 next 23
2021-01-21 08:29:21.119740: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e1d500 of size 256 next 24
2021-01-21 08:29:21.119747: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e1d600 of size 256 next 25
2021-01-21 08:29:21.119753: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e1d700 of size 256 next 26
2021-01-21 08:29:21.119760: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e1d800 of size 256 next 27
2021-01-21 08:29:21.119767: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e1d900 of size 110592 next 28
2021-01-21 08:29:21.119782: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e38900 of size 256 next 29
2021-01-21 08:29:21.119789: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e38a00 of size 256 next 30
2021-01-21 08:29:21.119796: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e38b00 of size 256 next 31
2021-01-21 08:29:21.119803: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e38c00 of size 256 next 32
2021-01-21 08:29:21.119810: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e38d00 of size 110592 next 33
2021-01-21 08:29:21.119816: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e53d00 of size 256 next 34
2021-01-21 08:29:21.119823: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e53e00 of size 256 next 35
2021-01-21 08:29:21.119830: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e53f00 of size 256 next 36
2021-01-21 08:29:21.119837: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e54000 of size 256 next 37
2021-01-21 08:29:21.119844: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e54100 of size 221184 next 38
2021-01-21 08:29:21.119851: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e8a100 of size 256 next 39
2021-01-21 08:29:21.119857: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e8a200 of size 256 next 40
2021-01-21 08:29:21.119864: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e8a300 of size 256 next 41
2021-01-21 08:29:21.119871: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e8a400 of size 256 next 42
2021-01-21 08:29:21.119878: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e8a500 of size 482048 next 18446744073709551615
2021-01-21 08:29:21.119885: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 2097152
2021-01-21 08:29:21.119892: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5400000 of size 256 next 44
2021-01-21 08:29:21.119899: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5400100 of size 256 next 45
2021-01-21 08:29:21.119906: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5400200 of size 256 next 46
2021-01-21 08:29:21.119913: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5400300 of size 256 next 47
2021-01-21 08:29:21.119920: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5400400 of size 442368 next 48
2021-01-21 08:29:21.119926: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d546c400 of size 256 next 49
2021-01-21 08:29:21.119933: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d546c500 of size 256 next 50
2021-01-21 08:29:21.119940: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d546c600 of size 256 next 51
2021-01-21 08:29:21.119947: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d546c700 of size 256 next 52
2021-01-21 08:29:21.119954: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d546c800 of size 1652736 next 18446744073709551615
2021-01-21 08:29:21.119960: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 4194304
2021-01-21 08:29:21.119968: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5a00000 of size 512 next 54
2021-01-21 08:29:21.119975: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5a00200 of size 512 next 55
2021-01-21 08:29:21.119982: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5a00400 of size 512 next 56
2021-01-21 08:29:21.119988: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5a00600 of size 512 next 57
2021-01-21 08:29:21.119995: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5a00800 of size 1769472 next 58
2021-01-21 08:29:21.120010: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5bb0800 of size 512 next 59
2021-01-21 08:29:21.120017: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5bb0a00 of size 512 next 60
2021-01-21 08:29:21.120024: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5bb0c00 of size 512 next 61
2021-01-21 08:29:21.120031: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5bb0e00 of size 512 next 62
2021-01-21 08:29:21.120039: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5bb1000 of size 2420736 next 18446744073709551615
2021-01-21 08:29:21.120045: I tensorflow/core/common_runtime/bfc_allocator.cc:1031]      Summary of in-use Chunks by size: 
2021-01-21 08:29:21.120060: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 354 Chunks of size 256 totalling 88.5KiB
2021-01-21 08:29:21.120068: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 33 Chunks of size 512 totalling 16.5KiB
2021-01-21 08:29:21.120077: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 1280 totalling 1.2KiB
2021-01-21 08:29:21.120085: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 3584 totalling 10.5KiB
2021-01-21 08:29:21.120092: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4352 totalling 4.2KiB
2021-01-21 08:29:21.120100: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 5376 totalling 15.8KiB
2021-01-21 08:29:21.120108: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 13 Chunks of size 27648 totalling 351.0KiB
2021-01-21 08:29:21.120116: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 9 Chunks of size 55296 totalling 486.0KiB
2021-01-21 08:29:21.120124: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 12 Chunks of size 110592 totalling 1.27MiB
2021-01-21 08:29:21.120132: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 9 Chunks of size 221184 totalling 1.90MiB
2021-01-21 08:29:21.120139: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 11 Chunks of size 442368 totalling 4.64MiB
2021-01-21 08:29:21.120148: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 482048 totalling 470.8KiB
2021-01-21 08:29:21.120155: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 8 Chunks of size 884736 totalling 6.75MiB
2021-01-21 08:29:21.120163: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 1652736 totalling 1.58MiB
2021-01-21 08:29:21.120171: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 4 Chunks of size 1769472 totalling 6.75MiB
2021-01-21 08:29:21.120179: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 2105344 totalling 2.01MiB
2021-01-21 08:29:21.120186: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 2420736 totalling 2.31MiB
2021-01-21 08:29:21.120194: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 2548224 totalling 2.43MiB
2021-01-21 08:29:21.120202: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 5 Chunks of size 3538944 totalling 16.88MiB
2021-01-21 08:29:21.120210: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 7077888 totalling 6.75MiB
2021-01-21 08:29:21.120218: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 14 Chunks of size 14155776 totalling 189.00MiB
2021-01-21 08:29:21.120226: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 17694720 totalling 16.88MiB
2021-01-21 08:29:21.120234: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 19398656 totalling 18.50MiB
2021-01-21 08:29:21.120242: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 21233664 totalling 20.25MiB
2021-01-21 08:29:21.120250: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 28311552 totalling 54.00MiB
2021-01-21 08:29:21.120258: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 42467328 totalling 40.50MiB
2021-01-21 08:29:21.120274: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 10 Chunks of size 56623104 totalling 540.00MiB
2021-01-21 08:29:21.120283: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 70516736 totalling 67.25MiB
2021-01-21 08:29:21.120292: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 112721920 totalling 107.50MiB
2021-01-21 08:29:21.120299: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 134217728 totalling 128.00MiB
2021-01-21 08:29:21.120307: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 8 Chunks of size 226492416 totalling 1.69GiB
2021-01-21 08:29:21.120315: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 295174144 totalling 281.50MiB
2021-01-21 08:29:21.120323: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 310378496 totalling 296.00MiB
2021-01-21 08:29:21.120331: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 325582848 totalling 621.00MiB
2021-01-21 08:29:21.120339: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 452984832 totalling 432.00MiB
2021-01-21 08:29:21.120346: I tensorflow/core/common_runtime/bfc_allocator.cc:1038] Sum Total of in-use chunks: 4.49GiB
2021-01-21 08:29:21.120354: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] total_region_allocated_bytes_: 5243269376 memory_limit_: 5450214944 available bytes: 206945568 curr_region_allocation_bytes_: 8589934592
2021-01-21 08:29:21.120367: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Stats: 
Limit:                      5450214944
InUse:                      4818246656
MaxInUse:                   4874896640
NumAllocs:                        1394
MaxAllocSize:                639827968
Reserved:                            0
PeakReserved:                        0
LargestFreeBlock:                    0

2021-01-21 08:29:21.120409: W tensorflow/core/common_runtime/bfc_allocator.cc:439] *************___***************************************__********************x*********x************
2021-01-21 08:29:21.120440: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES fa2021-01-21 08:28:22.403984: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
WARNING:root:The label loss 0.0 is not positive.
2021-01-21 08:28:24.201600: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2021-01-21 08:28:24.254764: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:0e:00.0 name: Tesla V100-DGXS-32GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 31.75GiB deviceMemoryBandwidth: 836.37GiB/s
2021-01-21 08:28:24.255868: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 1 with properties: 
pciBusID: 0000:0f:00.0 name: Tesla V100-DGXS-32GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 31.75GiB deviceMemoryBandwidth: 836.37GiB/s
2021-01-21 08:28:24.255930: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-01-21 08:28:24.258280: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2021-01-21 08:28:24.260237: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2021-01-21 08:28:24.260646: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2021-01-21 08:28:24.262653: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2021-01-21 08:28:24.263794: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2021-01-21 08:28:24.268435: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2021-01-21 08:28:24.274688: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0, 1
2021-01-21 08:28:24.279448: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-01-21 08:28:24.296112: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2198815000 Hz
2021-01-21 08:28:24.298881: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x560874367c60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-01-21 08:28:24.298980: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2021-01-21 08:28:24.600993: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x560876198450 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-01-21 08:28:24.601059: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla V100-DGXS-32GB, Compute Capability 7.0
2021-01-21 08:28:24.601075: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (1): Tesla V100-DGXS-32GB, Compute Capability 7.0
2021-01-21 08:28:24.609998: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:0e:00.0 name: Tesla V100-DGXS-32GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 31.75GiB deviceMemoryBandwidth: 836.37GiB/s
2021-01-21 08:28:24.612014: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 1 with properties: 
pciBusID: 0000:0f:00.0 name: Tesla V100-DGXS-32GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 31.75GiB deviceMemoryBandwidth: 836.37GiB/s
2021-01-21 08:28:24.612095: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-01-21 08:28:24.612147: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2021-01-21 08:28:24.612226: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2021-01-21 08:28:24.612260: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2021-01-21 08:28:24.612291: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2021-01-21 08:28:24.612321: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2021-01-21 08:28:24.612352: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2021-01-21 08:28:24.620496: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0, 1
2021-01-21 08:28:24.620580: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-01-21 08:28:25.747883: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-01-21 08:28:25.747940: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 1 
2021-01-21 08:28:25.747950: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N Y 
2021-01-21 08:28:25.747956: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 1:   Y N 
2021-01-21 08:28:25.753233: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2021-01-21 08:28:25.753306: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30129 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0e:00.0, compute capability: 7.0)
2021-01-21 08:28:25.755071: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2021-01-21 08:28:25.755125: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 5197 MB memory) -> physical GPU (device: 1, name: Tesla V100-DGXS-32GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2021-01-21 08:28:35.413981: I tensorflow/core/profiler/lib/profiler_session.cc:164] Profiler session started.
2021-01-21 08:28:35.414073: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1391] Profiler found 2 GPUs
2021-01-21 08:28:35.416598: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcupti.so.10.1
2021-01-21 08:28:35.517085: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1441] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI_ERROR_INSUFFICIENT_PRIVILEGES
2021-01-21 08:29:06.332891: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2021-01-21 08:29:08.020853: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2021-01-21 08:29:10.983029: I tensorflow/stream_executor/cuda/cuda_driver.cc:775] failed to allocate 1.01G (1089187328 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-01-21 08:29:10.986457: I tensorflow/stream_executor/cuda/cuda_driver.cc:775] failed to allocate 934.86M (980268544 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-01-21 08:29:11.000381: W tensorflow/core/common_runtime/bfc_allocator.cc:246] Allocator (GPU_1_bfc) ran out of memory trying to allocate 232.00MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-01-21 08:29:11.000434: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once.
2021-01-21 08:29:21.115353: W tensorflow/core/common_runtime/bfc_allocator.cc:431] Allocator (GPU_1_bfc) ran out of memory trying to allocate 216.00MiB (rounded to 226492416)requested by op replica_1/RegistrationModel/functional_1/u_net/up_sample_resnet_block/residual3d_block_4/conv3d_block_9/conv3d_26/conv3d_27/Conv3D
Current allocation summary follows.
2021-01-21 08:29:21.115703: I tensorflow/core/common_runtime/bfc_allocator.cc:970] BFCAllocator dump for GPU_1_bfc
2021-01-21 08:29:21.115775: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (256):   Total Chunks: 354, Chunks in use: 354. 88.5KiB allocated for chunks. 88.5KiB in use in bin. 34.2KiB client-requested in use in bin.
2021-01-21 08:29:21.115799: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (512):   Total Chunks: 33, Chunks in use: 33. 16.5KiB allocated for chunks. 16.5KiB in use in bin. 16.5KiB client-requested in use in bin.
2021-01-21 08:29:21.115816: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (1024):  Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2021-01-21 08:29:21.115833: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (2048):  Total Chunks: 3, Chunks in use: 3. 10.5KiB allocated for chunks. 10.5KiB in use in bin. 10.1KiB client-requested in use in bin.
2021-01-21 08:29:21.115850: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (4096):  Total Chunks: 4, Chunks in use: 4. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 19.3KiB client-requested in use in bin.
2021-01-21 08:29:21.115866: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-01-21 08:29:21.115884: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (16384):         Total Chunks: 13, Chunks in use: 13. 351.0KiB allocated for chunks. 351.0KiB in use in bin. 351.0KiB client-requested in use in bin.
2021-01-21 08:29:21.115911: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (32768):         Total Chunks: 9, Chunks in use: 9. 486.0KiB allocated for chunks. 486.0KiB in use in bin. 486.0KiB client-requested in use in bin.
2021-01-21 08:29:21.115920: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (65536):         Total Chunks: 12, Chunks in use: 12. 1.27MiB allocated for chunks. 1.27MiB in use in bin. 1.27MiB client-requested in use in bin.
2021-01-21 08:29:21.115929: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (131072):        Total Chunks: 9, Chunks in use: 9. 1.90MiB allocated for chunks. 1.90MiB in use in bin. 1.90MiB client-requested in use in bin.
2021-01-21 08:29:21.115938: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (262144):        Total Chunks: 12, Chunks in use: 12. 5.10MiB allocated for chunks. 5.10MiB in use in bin. 5.06MiB client-requested in use in bin.
2021-01-21 08:29:21.115946: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (524288):        Total Chunks: 8, Chunks in use: 8. 6.75MiB allocated for chunks. 6.75MiB in use in bin. 6.75MiB client-requested in use in bin.
2021-01-21 08:29:21.115955: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (1048576):       Total Chunks: 5, Chunks in use: 5. 8.33MiB allocated for chunks. 8.33MiB in use in bin. 7.59MiB client-requested in use in bin.
2021-01-21 08:29:21.115965: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (2097152):       Total Chunks: 9, Chunks in use: 8. 26.96MiB allocated for chunks. 23.62MiB in use in bin. 21.94MiB client-requested in use in bin.
2021-01-21 08:29:21.115974: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 6.75MiB allocated for chunks. 6.75MiB in use in bin. 6.75MiB client-requested in use in bin.
2021-01-21 08:29:21.115983: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (8388608):       Total Chunks: 14, Chunks in use: 14. 189.00MiB allocated for chunks. 189.00MiB in use in bin. 189.00MiB client-requested in use in bin.
2021-01-21 08:29:21.116025: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (16777216):      Total Chunks: 5, Chunks in use: 5. 109.62MiB allocated for chunks. 109.62MiB in use in bin. 101.25MiB client-requested in use in bin.
2021-01-21 08:29:21.116037: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (33554432):      Total Chunks: 12, Chunks in use: 11. 634.50MiB allocated for chunks. 580.50MiB in use in bin. 580.50MiB client-requested in use in bin.
2021-01-21 08:29:21.116047: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (67108864):      Total Chunks: 2, Chunks in use: 2. 174.75MiB allocated for chunks. 174.75MiB in use in bin. 92.01MiB client-requested in use in bin.
2021-01-21 08:29:21.116055: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (134217728):     Total Chunks: 11, Chunks in use: 9. 2.15GiB allocated for chunks. 1.81GiB in use in bin. 1.81GiB client-requested in use in bin.
2021-01-21 08:29:21.116064: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (268435456):     Total Chunks: 5, Chunks in use: 5. 1.59GiB allocated for chunks. 1.59GiB in use in bin. 1.45GiB client-requested in use in bin.
2021-01-21 08:29:21.116073: I tensorflow/core/common_runtime/bfc_allocator.cc:993] Bin for 216.00MiB was 128.00MiB, Chunk State: 
2021-01-21 08:29:21.116088: I tensorflow/core/common_runtime/bfc_allocator.cc:999]   Size: 154.62MiB | Requested Size: 222.80MiB | in_use: 0 | bin_num: 19, prev:   Size: 432.00MiB | Requested Size: 432.00MiB | in_use: 1 | bin_num: -1
2021-01-21 08:29:21.116098: I tensorflow/core/common_runtime/bfc_allocator.cc:999]   Size: 193.37MiB | Requested Size: 0B | in_use: 0 | bin_num: 19, prev:   Size: 216.00MiB | Requested Size: 216.00MiB | in_use: 1 | bin_num: -1
2021-01-21 08:29:21.116105: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 882241792
2021-01-21 08:29:21.116117: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e12000000 of size 226492416 next 519
2021-01-21 08:29:21.116124: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e1f800000 of size 226492416 next 516
2021-01-21 08:29:21.116131: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e2d000000 of size 226492416 next 520
2021-01-21 08:29:21.116138: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f1e3a800000 of size 202764544 next 18446744073709551615
2021-01-21 08:29:21.116145: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 2147483648
2021-01-21 08:29:21.116152: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e5c000000 of size 56623104 next 432
2021-01-21 08:29:21.116159: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e5f600000 of size 56623104 next 433
2021-01-21 08:29:21.116166: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e62c00000 of size 56623104 next 440
2021-01-21 08:29:21.116173: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e66200000 of size 14155776 next 441
2021-01-21 08:29:21.116180: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e66f80000 of size 14155776 next 446
2021-01-21 08:29:21.116187: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e67d00000 of size 14155776 next 443
2021-01-21 08:29:21.116193: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e68a80000 of size 14155776 next 451
2021-01-21 08:29:21.116200: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e69800000 of size 14155776 next 448
2021-01-21 08:29:21.116207: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e6a580000 of size 14155776 next 456
2021-01-21 08:29:21.116214: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e6b300000 of size 3538944 next 461
2021-01-21 08:29:21.116221: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e6b660000 of size 3538944 next 458
2021-01-21 08:29:21.116238: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e6b9c0000 of size 3538944 next 466
2021-01-21 08:29:21.116246: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e6bd20000 of size 3538944 next 463
2021-01-21 08:29:21.116254: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e6c080000 of size 17694720 next 468
2021-01-21 08:29:21.116261: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e6d160000 of size 14155776 next 478
2021-01-21 08:29:21.116268: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e6dee0000 of size 14155776 next 471
2021-01-21 08:29:21.116274: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e6ec60000 of size 14155776 next 483
2021-01-21 08:29:21.116281: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e6f9e0000 of size 14155776 next 480
2021-01-21 08:29:21.116288: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e70760000 of size 14155776 next 488
2021-01-21 08:29:21.116295: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e714e0000 of size 56623104 next 485
2021-01-21 08:29:21.116302: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e74ae0000 of size 56623104 next 493
2021-01-21 08:29:21.116309: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e780e0000 of size 56623104 next 497
2021-01-21 08:29:21.116315: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e7b6e0000 of size 56623104 next 494
2021-01-21 08:29:21.116322: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e7ece0000 of size 56623104 next 502
2021-01-21 08:29:21.116329: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e822e0000 of size 325582848 next 499
2021-01-21 08:29:21.116336: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e95960000 of size 56623104 next 503
2021-01-21 08:29:21.116343: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f1e98f60000 of size 56623104 next 508
2021-01-21 08:29:21.116350: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1e9c560000 of size 226492416 next 505
2021-01-21 08:29:21.116357: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1ea9d60000 of size 226492416 next 513
2021-01-21 08:29:21.116364: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f1eb7560000 of size 452984832 next 510
2021-01-21 08:29:21.116371: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f1ed2560000 of size 162136064 next 18446744073709551615
2021-01-21 08:29:21.116378: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 1073741824
2021-01-21 08:29:21.116385: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2200000000 of size 325582848 next 412
2021-01-21 08:29:21.116392: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2213680000 of size 226492416 next 423
2021-01-21 08:29:21.116399: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2220e80000 of size 226492416 next 420
2021-01-21 08:29:21.116406: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f222e680000 of size 295174144 next 18446744073709551615
2021-01-21 08:29:21.116413: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 536870912
2021-01-21 08:29:21.116420: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2240000000 of size 226492416 next 408
2021-01-21 08:29:21.116427: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f224d800000 of size 310378496 next 18446744073709551615
2021-01-21 08:29:21.116434: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 268435456
2021-01-21 08:29:21.116441: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2290000000 of size 14155776 next 405
2021-01-21 08:29:21.116448: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2290d80000 of size 7077888 next 437
2021-01-21 08:29:21.116463: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2291440000 of size 3538944 next 453
2021-01-21 08:29:21.116470: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a0000 of size 512 next 457
2021-01-21 08:29:21.116477: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a0200 of size 512 next 459
2021-01-21 08:29:21.116484: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a0400 of size 512 next 460
2021-01-21 08:29:21.116491: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a0600 of size 512 next 462
2021-01-21 08:29:21.116498: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a0800 of size 512 next 464
2021-01-21 08:29:21.116504: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a0a00 of size 512 next 465
2021-01-21 08:29:21.116511: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a0c00 of size 512 next 467
2021-01-21 08:29:21.116518: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a0e00 of size 512 next 469
2021-01-21 08:29:21.116525: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1000 of size 512 next 470
2021-01-21 08:29:21.116532: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1200 of size 256 next 472
2021-01-21 08:29:21.116538: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1300 of size 256 next 474
2021-01-21 08:29:21.116545: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1400 of size 256 next 475
2021-01-21 08:29:21.116552: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1500 of size 256 next 473
2021-01-21 08:29:21.116559: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1600 of size 256 next 476
2021-01-21 08:29:21.116566: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1700 of size 256 next 477
2021-01-21 08:29:21.116572: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1800 of size 256 next 479
2021-01-21 08:29:21.116579: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1900 of size 256 next 481
2021-01-21 08:29:21.116586: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1a00 of size 256 next 482
2021-01-21 08:29:21.116593: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1b00 of size 256 next 484
2021-01-21 08:29:21.116600: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1c00 of size 256 next 486
2021-01-21 08:29:21.116606: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1d00 of size 256 next 487
2021-01-21 08:29:21.116613: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1e00 of size 256 next 489
2021-01-21 08:29:21.116620: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a1f00 of size 256 next 491
2021-01-21 08:29:21.116627: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2000 of size 256 next 492
2021-01-21 08:29:21.116634: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2100 of size 256 next 490
2021-01-21 08:29:21.116640: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2200 of size 256 next 495
2021-01-21 08:29:21.116647: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2300 of size 256 next 496
2021-01-21 08:29:21.116654: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2400 of size 256 next 498
2021-01-21 08:29:21.116661: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2500 of size 256 next 500
2021-01-21 08:29:21.116668: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2600 of size 256 next 501
2021-01-21 08:29:21.116674: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2700 of size 256 next 504
2021-01-21 08:29:21.116688: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2800 of size 256 next 506
2021-01-21 08:29:21.116696: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2900 of size 256 next 507
2021-01-21 08:29:21.116703: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2a00 of size 256 next 509
2021-01-21 08:29:21.116709: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2b00 of size 256 next 511
2021-01-21 08:29:21.116716: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2c00 of size 256 next 512
2021-01-21 08:29:21.116723: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2d00 of size 256 next 515
2021-01-21 08:29:21.116730: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2e00 of size 256 next 517
2021-01-21 08:29:21.116736: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a2f00 of size 256 next 518
2021-01-21 08:29:21.116744: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22917a3000 of size 27648 next 521
2021-01-21 08:29:21.116751: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at 7f22917a9c00 of size 3499008 next 403
2021-01-21 08:29:21.116758: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2291b00000 of size 28311552 next 406
2021-01-21 08:29:21.116765: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2293600000 of size 28311552 next 409
2021-01-21 08:29:21.116772: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2295100000 of size 14155776 next 414
2021-01-21 08:29:21.116779: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2295e80000 of size 56623104 next 428
2021-01-21 08:29:21.116786: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f2299480000 of size 112721920 next 18446744073709551615
2021-01-21 08:29:21.116793: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 134217728
2021-01-21 08:29:21.116801: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22b0000000 of size 42467328 next 371
2021-01-21 08:29:21.116808: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22b2880000 of size 21233664 next 402
2021-01-21 08:29:21.116815: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22b3cc0000 of size 70516736 next 18446744073709551615
2021-01-21 08:29:21.116822: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 134217728
2021-01-21 08:29:21.116830: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22b8000000 of size 134217728 next 18446744073709551615
2021-01-21 08:29:21.116836: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 33554432
2021-01-21 08:29:21.116844: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22ca000000 of size 14155776 next 307
2021-01-21 08:29:21.116851: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f22cad80000 of size 19398656 next 18446744073709551615
2021-01-21 08:29:21.116858: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 8388608
2021-01-21 08:29:21.116865: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a800000 of size 512 next 64
2021-01-21 08:29:21.116872: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a800200 of size 512 next 65
2021-01-21 08:29:21.116879: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a800400 of size 512 next 66
2021-01-21 08:29:21.116886: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a800600 of size 512 next 67
2021-01-21 08:29:21.116893: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a800800 of size 884736 next 68
2021-01-21 08:29:21.116900: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a8d8800 of size 256 next 69
2021-01-21 08:29:21.116914: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a8d8900 of size 256 next 70
2021-01-21 08:29:21.116921: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a8d8a00 of size 256 next 71
2021-01-21 08:29:21.116928: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a8d8b00 of size 256 next 72
2021-01-21 08:29:21.116935: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a8d8c00 of size 884736 next 73
2021-01-21 08:29:21.116942: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a9b0c00 of size 256 next 74
2021-01-21 08:29:21.116949: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a9b0d00 of size 256 next 75
2021-01-21 08:29:21.116955: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a9b0e00 of size 256 next 76
2021-01-21 08:29:21.116962: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a9b0f00 of size 256 next 77
2021-01-21 08:29:21.116970: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230a9b1000 of size 442368 next 78
2021-01-21 08:29:21.116976: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aa1d000 of size 256 next 79
2021-01-21 08:29:21.116983: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aa1d100 of size 256 next 80
2021-01-21 08:29:21.116990: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aa1d200 of size 256 next 81
2021-01-21 08:29:21.116997: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aa1d300 of size 256 next 82
2021-01-21 08:29:21.117004: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aa1d400 of size 442368 next 83
2021-01-21 08:29:21.117011: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aa89400 of size 256 next 84
2021-01-21 08:29:21.117017: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aa89500 of size 256 next 85
2021-01-21 08:29:21.117024: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aa89600 of size 256 next 86
2021-01-21 08:29:21.117031: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aa89700 of size 256 next 87
2021-01-21 08:29:21.117038: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aa89800 of size 221184 next 88
2021-01-21 08:29:21.117045: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aabf800 of size 256 next 89
2021-01-21 08:29:21.117052: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aabf900 of size 256 next 90
2021-01-21 08:29:21.117059: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aabfa00 of size 256 next 91
2021-01-21 08:29:21.117066: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aabfb00 of size 256 next 92
2021-01-21 08:29:21.117072: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aabfc00 of size 221184 next 93
2021-01-21 08:29:21.117079: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aaf5c00 of size 256 next 94
2021-01-21 08:29:21.117086: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aaf5d00 of size 256 next 95
2021-01-21 08:29:21.117093: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aaf5e00 of size 256 next 96
2021-01-21 08:29:21.117100: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aaf5f00 of size 256 next 97
2021-01-21 08:29:21.117107: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230aaf6000 of size 110592 next 98
2021-01-21 08:29:21.117114: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab11000 of size 256 next 99
2021-01-21 08:29:21.117121: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab11100 of size 256 next 100
2021-01-21 08:29:21.117128: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab11200 of size 256 next 101
2021-01-21 08:29:21.117134: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab11300 of size 256 next 102
2021-01-21 08:29:21.117148: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab11400 of size 110592 next 103
2021-01-21 08:29:21.117156: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab2c400 of size 256 next 104
2021-01-21 08:29:21.117163: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab2c500 of size 256 next 105
2021-01-21 08:29:21.117169: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab2c600 of size 256 next 106
2021-01-21 08:29:21.117176: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab2c700 of size 256 next 107
2021-01-21 08:29:21.117235: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab2c800 of size 55296 next 108
2021-01-21 08:29:21.117243: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab3a000 of size 256 next 109
2021-01-21 08:29:21.117250: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab3a100 of size 256 next 110
2021-01-21 08:29:21.117257: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab3a200 of size 256 next 111
2021-01-21 08:29:21.117264: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab3a300 of size 256 next 112
2021-01-21 08:29:21.117270: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab3a400 of size 55296 next 113
2021-01-21 08:29:21.117277: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab47c00 of size 256 next 114
2021-01-21 08:29:21.117284: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab47d00 of size 256 next 115
2021-01-21 08:29:21.117291: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab47e00 of size 256 next 116
2021-01-21 08:29:21.117298: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab47f00 of size 256 next 117
2021-01-21 08:29:21.117305: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab48000 of size 27648 next 118
2021-01-21 08:29:21.117312: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab4ec00 of size 256 next 119
2021-01-21 08:29:21.117318: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab4ed00 of size 256 next 120
2021-01-21 08:29:21.117325: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab4ee00 of size 256 next 121
2021-01-21 08:29:21.117332: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab4ef00 of size 256 next 122
2021-01-21 08:29:21.117339: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab4f000 of size 27648 next 123
2021-01-21 08:29:21.117346: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab55c00 of size 256 next 124
2021-01-21 08:29:21.117352: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab55d00 of size 256 next 125
2021-01-21 08:29:21.117359: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab55e00 of size 256 next 126
2021-01-21 08:29:21.117366: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab55f00 of size 256 next 127
2021-01-21 08:29:21.117373: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab56000 of size 5376 next 128
2021-01-21 08:29:21.117380: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab57500 of size 256 next 129
2021-01-21 08:29:21.117387: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab57600 of size 256 next 131
2021-01-21 08:29:21.117424: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab57700 of size 256 next 130
2021-01-21 08:29:21.117433: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab57800 of size 256 next 132
2021-01-21 08:29:21.117440: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab57900 of size 256 next 133
2021-01-21 08:29:21.117447: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab57a00 of size 256 next 134
2021-01-21 08:29:21.117462: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab57b00 of size 256 next 135
2021-01-21 08:29:21.117470: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab57c00 of size 256 next 136
2021-01-21 08:29:21.117477: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab57d00 of size 256 next 137
2021-01-21 08:29:21.117484: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab57e00 of size 256 next 138
2021-01-21 08:29:21.117491: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab57f00 of size 256 next 139
2021-01-21 08:29:21.117497: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab58000 of size 256 next 140
2021-01-21 08:29:21.117504: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab58100 of size 256 next 141
2021-01-21 08:29:21.117511: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab58200 of size 256 next 142
2021-01-21 08:29:21.117518: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab58300 of size 256 next 143
2021-01-21 08:29:21.117525: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab58400 of size 256 next 144
2021-01-21 08:29:21.117532: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab58500 of size 256 next 145
2021-01-21 08:29:21.117538: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab58600 of size 256 next 146
2021-01-21 08:29:21.117545: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab58700 of size 256 next 147
2021-01-21 08:29:21.117552: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab58800 of size 256 next 148
2021-01-21 08:29:21.117559: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab58900 of size 256 next 149
2021-01-21 08:29:21.117566: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab58a00 of size 3584 next 150
2021-01-21 08:29:21.117573: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab59800 of size 256 next 151
2021-01-21 08:29:21.117580: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab59900 of size 256 next 152
2021-01-21 08:29:21.117587: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab59a00 of size 27648 next 153
2021-01-21 08:29:21.117594: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab60600 of size 256 next 154
2021-01-21 08:29:21.117601: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab60700 of size 256 next 155
2021-01-21 08:29:21.117608: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab60800 of size 27648 next 156
2021-01-21 08:29:21.117615: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab67400 of size 256 next 157
2021-01-21 08:29:21.117622: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab67500 of size 256 next 158
2021-01-21 08:29:21.117629: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab67600 of size 55296 next 159
2021-01-21 08:29:21.117635: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab74e00 of size 256 next 160
2021-01-21 08:29:21.117642: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab74f00 of size 256 next 161
2021-01-21 08:29:21.117649: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab75000 of size 110592 next 162
2021-01-21 08:29:21.117656: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab90000 of size 256 next 163
2021-01-21 08:29:21.117663: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab90100 of size 256 next 164
2021-01-21 08:29:21.117670: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ab90200 of size 110592 next 165
2021-01-21 08:29:21.117676: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230abab200 of size 256 next 166
2021-01-21 08:29:21.117690: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230abab300 of size 256 next 167
2021-01-21 08:29:21.117698: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230abab400 of size 221184 next 168
2021-01-21 08:29:21.117705: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230abe1400 of size 256 next 169
2021-01-21 08:29:21.117712: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230abe1500 of size 256 next 170
2021-01-21 08:29:21.117719: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230abe1600 of size 442368 next 171
2021-01-21 08:29:21.117726: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ac4d600 of size 256 next 172
2021-01-21 08:29:21.117733: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ac4d700 of size 256 next 173
2021-01-21 08:29:21.117739: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ac4d800 of size 442368 next 174
2021-01-21 08:29:21.117746: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230acb9800 of size 256 next 175
2021-01-21 08:29:21.117753: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230acb9900 of size 256 next 176
2021-01-21 08:29:21.117760: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230acb9a00 of size 884736 next 177
2021-01-21 08:29:21.117767: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ad91a00 of size 512 next 178
2021-01-21 08:29:21.117774: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ad91c00 of size 512 next 179
2021-01-21 08:29:21.117781: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ad91e00 of size 2548224 next 18446744073709551615
2021-01-21 08:29:21.117789: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 16777216
2021-01-21 08:29:21.117796: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b000000 of size 512 next 181
2021-01-21 08:29:21.117803: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b000200 of size 512 next 182
2021-01-21 08:29:21.117810: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b000400 of size 1769472 next 183
2021-01-21 08:29:21.117817: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1b0400 of size 512 next 184
2021-01-21 08:29:21.117824: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1b0600 of size 512 next 185
2021-01-21 08:29:21.117831: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1b0800 of size 55296 next 186
2021-01-21 08:29:21.117837: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1be000 of size 256 next 187
2021-01-21 08:29:21.117844: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1be100 of size 256 next 188
2021-01-21 08:29:21.117851: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1be200 of size 27648 next 189
2021-01-21 08:29:21.117858: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1c4e00 of size 256 next 190
2021-01-21 08:29:21.117865: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1c4f00 of size 256 next 191
2021-01-21 08:29:21.117872: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1c5000 of size 27648 next 192
2021-01-21 08:29:21.117878: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1cbc00 of size 256 next 193
2021-01-21 08:29:21.117885: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1cbd00 of size 256 next 194
2021-01-21 08:29:21.117892: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1cbe00 of size 55296 next 195
2021-01-21 08:29:21.117899: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1d9600 of size 256 next 196
2021-01-21 08:29:21.117906: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1d9700 of size 256 next 197
2021-01-21 08:29:21.117920: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b1d9800 of size 221184 next 198
2021-01-21 08:29:21.117927: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b20f800 of size 256 next 199
2021-01-21 08:29:21.117934: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b20f900 of size 256 next 200
2021-01-21 08:29:21.117941: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b20fa00 of size 110592 next 201
2021-01-21 08:29:21.117948: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b22aa00 of size 256 next 202
2021-01-21 08:29:21.117955: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b22ab00 of size 256 next 203
2021-01-21 08:29:21.117962: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b22ac00 of size 110592 next 204
2021-01-21 08:29:21.117968: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b245c00 of size 256 next 205
2021-01-21 08:29:21.117975: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b245d00 of size 256 next 206
2021-01-21 08:29:21.117982: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b245e00 of size 221184 next 207
2021-01-21 08:29:21.117989: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b27be00 of size 256 next 208
2021-01-21 08:29:21.117996: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b27bf00 of size 256 next 209
2021-01-21 08:29:21.118003: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b27c000 of size 884736 next 210
2021-01-21 08:29:21.118010: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b354000 of size 256 next 211
2021-01-21 08:29:21.118017: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b354100 of size 256 next 212
2021-01-21 08:29:21.118023: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b354200 of size 442368 next 213
2021-01-21 08:29:21.118030: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b3c0200 of size 256 next 214
2021-01-21 08:29:21.118037: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b3c0300 of size 256 next 215
2021-01-21 08:29:21.118044: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b3c0400 of size 442368 next 216
2021-01-21 08:29:21.118051: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b42c400 of size 256 next 217
2021-01-21 08:29:21.118057: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b42c500 of size 256 next 218
2021-01-21 08:29:21.118064: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b42c600 of size 884736 next 219
2021-01-21 08:29:21.118071: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b504600 of size 256 next 220
2021-01-21 08:29:21.118078: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b504700 of size 256 next 221
2021-01-21 08:29:21.118085: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b504800 of size 5376 next 222
2021-01-21 08:29:21.118091: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b505d00 of size 256 next 223
2021-01-21 08:29:21.118098: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b505e00 of size 3584 next 224
2021-01-21 08:29:21.118105: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b506c00 of size 256 next 225
2021-01-21 08:29:21.118112: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b506d00 of size 256 next 226
2021-01-21 08:29:21.118119: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b506e00 of size 27648 next 227
2021-01-21 08:29:21.118125: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b50da00 of size 256 next 228
2021-01-21 08:29:21.118132: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b50db00 of size 256 next 229
2021-01-21 08:29:21.118146: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b50dc00 of size 27648 next 230
2021-01-21 08:29:21.118154: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b514800 of size 256 next 231
2021-01-21 08:29:21.118160: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b514900 of size 256 next 232
2021-01-21 08:29:21.118167: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b514a00 of size 55296 next 233
2021-01-21 08:29:21.118174: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b522200 of size 256 next 234
2021-01-21 08:29:21.118181: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b522300 of size 256 next 235
2021-01-21 08:29:21.118188: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b522400 of size 110592 next 236
2021-01-21 08:29:21.118195: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b53d400 of size 256 next 237
2021-01-21 08:29:21.118202: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b53d500 of size 256 next 238
2021-01-21 08:29:21.118208: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b53d600 of size 110592 next 239
2021-01-21 08:29:21.118215: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b558600 of size 256 next 240
2021-01-21 08:29:21.118222: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b558700 of size 256 next 241
2021-01-21 08:29:21.118229: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b558800 of size 221184 next 242
2021-01-21 08:29:21.118236: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b58e800 of size 256 next 243
2021-01-21 08:29:21.118243: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b58e900 of size 256 next 244
2021-01-21 08:29:21.118249: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b58ea00 of size 442368 next 245
2021-01-21 08:29:21.118256: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b5faa00 of size 256 next 246
2021-01-21 08:29:21.118263: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b5fab00 of size 256 next 247
2021-01-21 08:29:21.118270: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b5fac00 of size 442368 next 248
2021-01-21 08:29:21.118276: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b666c00 of size 256 next 249
2021-01-21 08:29:21.118283: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b666d00 of size 256 next 250
2021-01-21 08:29:21.118290: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b666e00 of size 884736 next 251
2021-01-21 08:29:21.118297: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b73ee00 of size 512 next 252
2021-01-21 08:29:21.118304: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b73f000 of size 512 next 253
2021-01-21 08:29:21.118311: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b73f200 of size 1769472 next 254
2021-01-21 08:29:21.118318: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b8ef200 of size 512 next 255
2021-01-21 08:29:21.118324: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b8ef400 of size 512 next 256
2021-01-21 08:29:21.118331: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230b8ef600 of size 1769472 next 257
2021-01-21 08:29:21.118338: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ba9f600 of size 512 next 258
2021-01-21 08:29:21.118345: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ba9f800 of size 512 next 259
2021-01-21 08:29:21.118352: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230ba9fa00 of size 55296 next 260
2021-01-21 08:29:21.118359: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230baad200 of size 256 next 261
2021-01-21 08:29:21.118372: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230baad300 of size 256 next 262
2021-01-21 08:29:21.118380: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230baad400 of size 27648 next 263
2021-01-21 08:29:21.118387: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bab4000 of size 256 next 264
2021-01-21 08:29:21.118394: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bab4100 of size 256 next 265
2021-01-21 08:29:21.118401: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bab4200 of size 27648 next 266
2021-01-21 08:29:21.118408: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230babae00 of size 256 next 267
2021-01-21 08:29:21.118414: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230babaf00 of size 256 next 268
2021-01-21 08:29:21.118421: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230babb000 of size 55296 next 269
2021-01-21 08:29:21.118428: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bac8800 of size 256 next 270
2021-01-21 08:29:21.118435: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bac8900 of size 256 next 271
2021-01-21 08:29:21.118442: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bac8a00 of size 221184 next 272
2021-01-21 08:29:21.118449: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bafea00 of size 256 next 273
2021-01-21 08:29:21.118456: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bafeb00 of size 256 next 274
2021-01-21 08:29:21.118463: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bafec00 of size 110592 next 275
2021-01-21 08:29:21.118470: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bb19c00 of size 256 next 276
2021-01-21 08:29:21.118476: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bb19d00 of size 256 next 277
2021-01-21 08:29:21.118483: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bb19e00 of size 110592 next 278
2021-01-21 08:29:21.118490: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bb34e00 of size 256 next 279
2021-01-21 08:29:21.118497: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bb34f00 of size 256 next 280
2021-01-21 08:29:21.118504: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bb35000 of size 221184 next 281
2021-01-21 08:29:21.118510: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bb6b000 of size 256 next 282
2021-01-21 08:29:21.118517: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bb6b100 of size 256 next 283
2021-01-21 08:29:21.118524: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bb6b200 of size 884736 next 284
2021-01-21 08:29:21.118531: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bc43200 of size 256 next 285
2021-01-21 08:29:21.118538: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bc43300 of size 256 next 286
2021-01-21 08:29:21.118545: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bc43400 of size 442368 next 287
2021-01-21 08:29:21.118552: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bcaf400 of size 256 next 288
2021-01-21 08:29:21.118558: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bcaf500 of size 256 next 289
2021-01-21 08:29:21.118565: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bcaf600 of size 442368 next 290
2021-01-21 08:29:21.118572: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bd1b600 of size 256 next 291
2021-01-21 08:29:21.118579: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bd1b700 of size 256 next 292
2021-01-21 08:29:21.118586: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bd1b800 of size 884736 next 293
2021-01-21 08:29:21.118600: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf3800 of size 256 next 294
2021-01-21 08:29:21.118608: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf3900 of size 256 next 295
2021-01-21 08:29:21.118615: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf3a00 of size 5376 next 296
2021-01-21 08:29:21.118622: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf4f00 of size 256 next 297
2021-01-21 08:29:21.118629: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf5000 of size 256 next 298
2021-01-21 08:29:21.118635: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf5100 of size 256 next 299
2021-01-21 08:29:21.118642: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf5200 of size 256 next 300
2021-01-21 08:29:21.118649: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf5300 of size 256 next 301
2021-01-21 08:29:21.118656: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf5400 of size 256 next 302
2021-01-21 08:29:21.118663: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf5500 of size 256 next 303
2021-01-21 08:29:21.118670: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf5600 of size 256 next 304
2021-01-21 08:29:21.118677: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf5700 of size 4352 next 305
2021-01-21 08:29:21.118684: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf6800 of size 256 next 309
2021-01-21 08:29:21.118690: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf6900 of size 256 next 310
2021-01-21 08:29:21.118697: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf6a00 of size 256 next 311
2021-01-21 08:29:21.118704: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf6b00 of size 256 next 312
2021-01-21 08:29:21.118711: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf6c00 of size 256 next 313
2021-01-21 08:29:21.118718: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf6d00 of size 256 next 314
2021-01-21 08:29:21.118725: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf6e00 of size 256 next 315
2021-01-21 08:29:21.118732: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf6f00 of size 256 next 316
2021-01-21 08:29:21.118738: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7000 of size 256 next 317
2021-01-21 08:29:21.118745: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7100 of size 256 next 318
2021-01-21 08:29:21.118752: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7200 of size 256 next 319
2021-01-21 08:29:21.118759: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7300 of size 256 next 320
2021-01-21 08:29:21.118766: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7400 of size 256 next 321
2021-01-21 08:29:21.118773: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7500 of size 256 next 322
2021-01-21 08:29:21.118780: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7600 of size 256 next 323
2021-01-21 08:29:21.118786: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7700 of size 256 next 324
2021-01-21 08:29:21.118793: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7800 of size 256 next 325
2021-01-21 08:29:21.118800: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7900 of size 256 next 326
2021-01-21 08:29:21.118807: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7a00 of size 256 next 327
2021-01-21 08:29:21.118814: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7b00 of size 256 next 328
2021-01-21 08:29:21.118828: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7c00 of size 256 next 329
2021-01-21 08:29:21.118836: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7d00 of size 256 next 330
2021-01-21 08:29:21.118842: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7e00 of size 256 next 331
2021-01-21 08:29:21.118849: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf7f00 of size 256 next 332
2021-01-21 08:29:21.118856: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8000 of size 256 next 333
2021-01-21 08:29:21.118863: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8100 of size 256 next 334
2021-01-21 08:29:21.118870: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8200 of size 256 next 335
2021-01-21 08:29:21.118876: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8300 of size 256 next 336
2021-01-21 08:29:21.118884: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8400 of size 256 next 337
2021-01-21 08:29:21.118890: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8500 of size 256 next 338
2021-01-21 08:29:21.118897: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8600 of size 256 next 339
2021-01-21 08:29:21.118904: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8700 of size 256 next 340
2021-01-21 08:29:21.118911: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8800 of size 256 next 341
2021-01-21 08:29:21.118918: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8900 of size 256 next 342
2021-01-21 08:29:21.118933: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8a00 of size 256 next 343
2021-01-21 08:29:21.118943: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8b00 of size 256 next 344
2021-01-21 08:29:21.118950: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8c00 of size 256 next 345
2021-01-21 08:29:21.118957: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8d00 of size 256 next 346
2021-01-21 08:29:21.118963: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8e00 of size 256 next 347
2021-01-21 08:29:21.118970: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf8f00 of size 256 next 348
2021-01-21 08:29:21.118977: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9000 of size 256 next 349
2021-01-21 08:29:21.118984: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9100 of size 256 next 350
2021-01-21 08:29:21.118991: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9200 of size 256 next 351
2021-01-21 08:29:21.118998: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9300 of size 256 next 352
2021-01-21 08:29:21.119004: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9400 of size 256 next 353
2021-01-21 08:29:21.119011: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9500 of size 256 next 354
2021-01-21 08:29:21.119018: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9600 of size 256 next 355
2021-01-21 08:29:21.119025: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9700 of size 256 next 356
2021-01-21 08:29:21.119032: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9800 of size 256 next 357
2021-01-21 08:29:21.119039: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9900 of size 256 next 358
2021-01-21 08:29:21.119046: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9a00 of size 256 next 359
2021-01-21 08:29:21.119052: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9b00 of size 256 next 360
2021-01-21 08:29:21.119059: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9c00 of size 256 next 361
2021-01-21 08:29:21.119088: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9d00 of size 256 next 362
2021-01-21 08:29:21.119096: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9e00 of size 256 next 363
2021-01-21 08:29:21.119103: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdf9f00 of size 256 next 364
2021-01-21 08:29:21.119110: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfa000 of size 256 next 365
2021-01-21 08:29:21.119117: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfa100 of size 256 next 366
2021-01-21 08:29:21.119124: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfa200 of size 256 next 367
2021-01-21 08:29:21.119130: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfa300 of size 256 next 368
2021-01-21 08:29:21.119137: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfa400 of size 256 next 369
2021-01-21 08:29:21.119144: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfa500 of size 256 next 372
2021-01-21 08:29:21.119151: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfa600 of size 256 next 373
2021-01-21 08:29:21.119158: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfa700 of size 256 next 374
2021-01-21 08:29:21.119164: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfa800 of size 256 next 375
2021-01-21 08:29:21.119171: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfa900 of size 256 next 376
2021-01-21 08:29:21.119178: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfaa00 of size 256 next 377
2021-01-21 08:29:21.119185: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfab00 of size 256 next 378
2021-01-21 08:29:21.119192: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfac00 of size 256 next 379
2021-01-21 08:29:21.119198: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfad00 of size 256 next 380
2021-01-21 08:29:21.119205: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfae00 of size 256 next 381
2021-01-21 08:29:21.119212: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfaf00 of size 256 next 382
2021-01-21 08:29:21.119219: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfb000 of size 256 next 383
2021-01-21 08:29:21.119226: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfb100 of size 256 next 384
2021-01-21 08:29:21.119233: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfb200 of size 256 next 385
2021-01-21 08:29:21.119240: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfb300 of size 256 next 386
2021-01-21 08:29:21.119246: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfb400 of size 256 next 387
2021-01-21 08:29:21.119253: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfb500 of size 256 next 388
2021-01-21 08:29:21.119260: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfb600 of size 256 next 389
2021-01-21 08:29:21.119267: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfb700 of size 256 next 390
2021-01-21 08:29:21.119274: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfb800 of size 256 next 391
2021-01-21 08:29:21.119281: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfb900 of size 256 next 392
2021-01-21 08:29:21.119287: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfba00 of size 256 next 393
2021-01-21 08:29:21.119294: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfbb00 of size 256 next 394
2021-01-21 08:29:21.119301: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfbc00 of size 256 next 395
2021-01-21 08:29:21.119319: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfbd00 of size 256 next 396
2021-01-21 08:29:21.119327: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfbe00 of size 256 next 397
2021-01-21 08:29:21.119334: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfbf00 of size 256 next 398
2021-01-21 08:29:21.119341: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfc000 of size 256 next 399
2021-01-21 08:29:21.119347: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfc100 of size 256 next 400
2021-01-21 08:29:21.119354: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfc200 of size 256 next 401
2021-01-21 08:29:21.119361: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfc300 of size 256 next 415
2021-01-21 08:29:21.119368: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfc400 of size 256 next 416
2021-01-21 08:29:21.119375: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfc500 of size 256 next 417
2021-01-21 08:29:21.119381: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfc600 of size 256 next 419
2021-01-21 08:29:21.119388: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfc700 of size 256 next 418
2021-01-21 08:29:21.119395: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfc800 of size 256 next 421
2021-01-21 08:29:21.119402: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfc900 of size 256 next 422
2021-01-21 08:29:21.119408: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfca00 of size 256 next 424
2021-01-21 08:29:21.119415: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfcb00 of size 256 next 426
2021-01-21 08:29:21.119422: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfcc00 of size 256 next 427
2021-01-21 08:29:21.119429: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfcd00 of size 256 next 429
2021-01-21 08:29:21.119436: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfce00 of size 256 next 430
2021-01-21 08:29:21.119442: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfcf00 of size 256 next 431
2021-01-21 08:29:21.119450: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfd000 of size 256 next 410
2021-01-21 08:29:21.119456: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfd100 of size 256 next 413
2021-01-21 08:29:21.119463: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfd200 of size 256 next 434
2021-01-21 08:29:21.119470: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfd300 of size 256 next 435
2021-01-21 08:29:21.119477: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfd400 of size 256 next 436
2021-01-21 08:29:21.119484: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfd500 of size 256 next 438
2021-01-21 08:29:21.119490: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfd600 of size 256 next 439
2021-01-21 08:29:21.119497: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfd700 of size 256 next 442
2021-01-21 08:29:21.119504: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfd800 of size 256 next 444
2021-01-21 08:29:21.119511: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfd900 of size 256 next 445
2021-01-21 08:29:21.119517: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfda00 of size 256 next 447
2021-01-21 08:29:21.119524: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfdb00 of size 256 next 449
2021-01-21 08:29:21.119531: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfdc00 of size 256 next 450
2021-01-21 08:29:21.119545: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfdd00 of size 256 next 452
2021-01-21 08:29:21.119553: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfde00 of size 256 next 454
2021-01-21 08:29:21.119560: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfdf00 of size 256 next 455
2021-01-21 08:29:21.119567: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f230bdfe000 of size 2105344 next 18446744073709551615
2021-01-21 08:29:21.119574: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 1048576
2021-01-21 08:29:21.119582: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e00000 of size 256 next 1
2021-01-21 08:29:21.119589: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e00100 of size 1280 next 2
2021-01-21 08:29:21.119596: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e00600 of size 256 next 3
2021-01-21 08:29:21.119603: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e00700 of size 256 next 4
2021-01-21 08:29:21.119610: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e00800 of size 256 next 5
2021-01-21 08:29:21.119617: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e00900 of size 256 next 6
2021-01-21 08:29:21.119623: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e00a00 of size 256 next 7
2021-01-21 08:29:21.119630: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e00b00 of size 3584 next 8
2021-01-21 08:29:21.119637: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e01900 of size 256 next 9
2021-01-21 08:29:21.119644: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e01a00 of size 256 next 10
2021-01-21 08:29:21.119651: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e01b00 of size 256 next 11
2021-01-21 08:29:21.119657: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e01c00 of size 256 next 12
2021-01-21 08:29:21.119664: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e01d00 of size 27648 next 13
2021-01-21 08:29:21.119671: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e08900 of size 256 next 14
2021-01-21 08:29:21.119678: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e08a00 of size 256 next 15
2021-01-21 08:29:21.119685: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e08b00 of size 256 next 16
2021-01-21 08:29:21.119692: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e08c00 of size 256 next 17
2021-01-21 08:29:21.119699: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e08d00 of size 27648 next 18
2021-01-21 08:29:21.119706: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e0f900 of size 256 next 19
2021-01-21 08:29:21.119712: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e0fa00 of size 256 next 20
2021-01-21 08:29:21.119719: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e0fb00 of size 256 next 21
2021-01-21 08:29:21.119726: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e0fc00 of size 256 next 22
2021-01-21 08:29:21.119733: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e0fd00 of size 55296 next 23
2021-01-21 08:29:21.119740: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e1d500 of size 256 next 24
2021-01-21 08:29:21.119747: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e1d600 of size 256 next 25
2021-01-21 08:29:21.119753: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e1d700 of size 256 next 26
2021-01-21 08:29:21.119760: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e1d800 of size 256 next 27
2021-01-21 08:29:21.119767: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e1d900 of size 110592 next 28
2021-01-21 08:29:21.119782: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e38900 of size 256 next 29
2021-01-21 08:29:21.119789: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e38a00 of size 256 next 30
2021-01-21 08:29:21.119796: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e38b00 of size 256 next 31
2021-01-21 08:29:21.119803: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e38c00 of size 256 next 32
2021-01-21 08:29:21.119810: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e38d00 of size 110592 next 33
2021-01-21 08:29:21.119816: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e53d00 of size 256 next 34
2021-01-21 08:29:21.119823: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e53e00 of size 256 next 35
2021-01-21 08:29:21.119830: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e53f00 of size 256 next 36
2021-01-21 08:29:21.119837: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e54000 of size 256 next 37
2021-01-21 08:29:21.119844: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e54100 of size 221184 next 38
2021-01-21 08:29:21.119851: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e8a100 of size 256 next 39
2021-01-21 08:29:21.119857: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e8a200 of size 256 next 40
2021-01-21 08:29:21.119864: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e8a300 of size 256 next 41
2021-01-21 08:29:21.119871: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e8a400 of size 256 next 42
2021-01-21 08:29:21.119878: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d4e8a500 of size 482048 next 18446744073709551615
2021-01-21 08:29:21.119885: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 2097152
2021-01-21 08:29:21.119892: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5400000 of size 256 next 44
2021-01-21 08:29:21.119899: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5400100 of size 256 next 45
2021-01-21 08:29:21.119906: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5400200 of size 256 next 46
2021-01-21 08:29:21.119913: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5400300 of size 256 next 47
2021-01-21 08:29:21.119920: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5400400 of size 442368 next 48
2021-01-21 08:29:21.119926: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d546c400 of size 256 next 49
2021-01-21 08:29:21.119933: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d546c500 of size 256 next 50
2021-01-21 08:29:21.119940: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d546c600 of size 256 next 51
2021-01-21 08:29:21.119947: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d546c700 of size 256 next 52
2021-01-21 08:29:21.119954: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d546c800 of size 1652736 next 18446744073709551615
2021-01-21 08:29:21.119960: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 4194304
2021-01-21 08:29:21.119968: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5a00000 of size 512 next 54
2021-01-21 08:29:21.119975: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5a00200 of size 512 next 55
2021-01-21 08:29:21.119982: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5a00400 of size 512 next 56
2021-01-21 08:29:21.119988: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5a00600 of size 512 next 57
2021-01-21 08:29:21.119995: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5a00800 of size 1769472 next 58
2021-01-21 08:29:21.120010: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5bb0800 of size 512 next 59
2021-01-21 08:29:21.120017: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5bb0a00 of size 512 next 60
2021-01-21 08:29:21.120024: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5bb0c00 of size 512 next 61
2021-01-21 08:29:21.120031: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5bb0e00 of size 512 next 62
2021-01-21 08:29:21.120039: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 7f23d5bb1000 of size 2420736 next 18446744073709551615
2021-01-21 08:29:21.120045: I tensorflow/core/common_runtime/bfc_allocator.cc:1031]      Summary of in-use Chunks by size: 
2021-01-21 08:29:21.120060: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 354 Chunks of size 256 totalling 88.5KiB
2021-01-21 08:29:21.120068: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 33 Chunks of size 512 totalling 16.5KiB
2021-01-21 08:29:21.120077: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 1280 totalling 1.2KiB
2021-01-21 08:29:21.120085: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 3584 totalling 10.5KiB
2021-01-21 08:29:21.120092: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4352 totalling 4.2KiB
2021-01-21 08:29:21.120100: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 5376 totalling 15.8KiB
2021-01-21 08:29:21.120108: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 13 Chunks of size 27648 totalling 351.0KiB
2021-01-21 08:29:21.120116: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 9 Chunks of size 55296 totalling 486.0KiB
2021-01-21 08:29:21.120124: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 12 Chunks of size 110592 totalling 1.27MiB
2021-01-21 08:29:21.120132: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 9 Chunks of size 221184 totalling 1.90MiB
2021-01-21 08:29:21.120139: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 11 Chunks of size 442368 totalling 4.64MiB
2021-01-21 08:29:21.120148: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 482048 totalling 470.8KiB
2021-01-21 08:29:21.120155: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 8 Chunks of size 884736 totalling 6.75MiB
2021-01-21 08:29:21.120163: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 1652736 totalling 1.58MiB
2021-01-21 08:29:21.120171: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 4 Chunks of size 1769472 totalling 6.75MiB
2021-01-21 08:29:21.120179: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 2105344 totalling 2.01MiB
2021-01-21 08:29:21.120186: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 2420736 totalling 2.31MiB
2021-01-21 08:29:21.120194: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 2548224 totalling 2.43MiB
2021-01-21 08:29:21.120202: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 5 Chunks of size 3538944 totalling 16.88MiB
2021-01-21 08:29:21.120210: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 7077888 totalling 6.75MiB
2021-01-21 08:29:21.120218: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 14 Chunks of size 14155776 totalling 189.00MiB
2021-01-21 08:29:21.120226: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 17694720 totalling 16.88MiB
2021-01-21 08:29:21.120234: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 19398656 totalling 18.50MiB
2021-01-21 08:29:21.120242: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 21233664 totalling 20.25MiB
2021-01-21 08:29:21.120250: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 28311552 totalling 54.00MiB
2021-01-21 08:29:21.120258: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 42467328 totalling 40.50MiB
2021-01-21 08:29:21.120274: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 10 Chunks of size 56623104 totalling 540.00MiB
2021-01-21 08:29:21.120283: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 70516736 totalling 67.25MiB
2021-01-21 08:29:21.120292: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 112721920 totalling 107.50MiB
2021-01-21 08:29:21.120299: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 134217728 totalling 128.00MiB
2021-01-21 08:29:21.120307: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 8 Chunks of size 226492416 totalling 1.69GiB
2021-01-21 08:29:21.120315: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 295174144 totalling 281.50MiB
2021-01-21 08:29:21.120323: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 310378496 totalling 296.00MiB
2021-01-21 08:29:21.120331: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 325582848 totalling 621.00MiB
2021-01-21 08:29:21.120339: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 452984832 totalling 432.00MiB
2021-01-21 08:29:21.120346: I tensorflow/core/common_runtime/bfc_allocator.cc:1038] Sum Total of in-use chunks: 4.49GiB
2021-01-21 08:29:21.120354: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] total_region_allocated_bytes_: 5243269376 memory_limit_: 5450214944 available bytes: 206945568 curr_region_allocation_bytes_: 8589934592
2021-01-21 08:29:21.120367: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Stats: 
Limit:                      5450214944
InUse:                      4818246656
MaxInUse:                   4874896640
NumAllocs:                        1394
MaxAllocSize:                639827968
Reserved:                            0
PeakReserved:                        0
LargestFreeBlock:                    0

2021-01-21 08:29:21.120409: W tensorflow/core/common_runtime/bfc_allocator.cc:439] *************___***************************************__********************x*********x************
2021-01-21 08:29:21.120440: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at conv_ops_3d.cc:327 : Resource exhausted: OOM when allocating tensor with shape[2,16,192,192,48] and type float on /job:localhost/replica:0/task:0/device:GPU:1 by allocator GPU_1_bfc

[mathpluscode]

@ciphercharly this is now OOM? I suggest you trying a smaller batch size.

[ciphercharly]

I did tried with 2 GPUs and batch_size: 2 will re-try with batch size 1, thanks for the tip

[mathpluscode]

I did tried with 2 GPUs and batch_size: 2 will re-try with batch size 1, thanks for the tip

Oh! Maybe you want to reduce the image size in your config, this is more efficient!

We plan to rollout mixed precision support within months.

[ciphercharly]

noted, I'd prefer not to miss resolution tho (have already reduced to 192,192,48 which is smaller than some of the original nifti files)

[mathpluscode]

noted, I'd prefer not to miss resolution tho (have already reduced to 192,192,48 which is smaller than some of the original nifti files)

Sure :) wanted to just verify that smaller images are not going to crash your run.

[ciphercharly]

right, good point

[mathpluscode]

Hi @ciphercharly, with the latest main we just added the back-compatibility for the config, hope this could be helpful ^^

https://github.com/DeepRegNet/DeepReg/blob/main/deepreg/config/parser.py#L101

[ciphercharly]

thanks for the heads-up!

[ciphercharly]

re-pulled and retried with batch_size 1 and 2 GPUs but still getting resource exhausted OOM error, with the size 192,192,48, will have to stick to 1 GPU for now if I don't want to down-size (will try that too for the sake of checking at some point but now I need to get some runs done and they are slow)

[mathpluscode]

re-pulled and retried with batch_size 1 and 2 GPUs but still getting resource exhausted OOM error, with the size 192,192,48, will have to stick to 1 GPU for now if I don't want to down-size (will try that too for the sake of checking at some point but now I need to get some runs done and they are slow)

Okay. That's strange? So you can use one GPU with which batch size?But you can't use two GPUs?

[mathpluscode]

re-pulled and retried with batch_size 1 and 2 GPUs but still getting resource exhausted OOM error, with the size 192,192,48, will have to stick to 1 GPU for now if I don't want to down-size (will try that too for the sake of checking at some point but now I need to get some runs done and they are slow)

@ciphercharly May I ask how many GPUs you have on your machine? I may have an idea why we get OOM now.

[ciphercharly]

4 Teslas, but I can't use all of them together, to leave some room for other people's tasks...

[mathpluscode]

4 Teslas, but I can't use all of them together, to leave some room for other people's tasks...

Sure. I think there's a bug. The OOM might be because it is trying to use all GPUs:) Will fix it

[ciphercharly]

ah interesting, great that you tracked it down if that's the case! :)

[mathpluscode]

ah interesting, great that you tracked it down if that's the case! :)

Hi again, after checking the code again, with

os.environ["CUDA_VISIBLE_DEVICES"] = gpu

it should not use other GPUs and also the logs above are also showing only 2 GPUs are used. Would you mind verifying with nvidia-smi that the GPUs you intended to use do have enough memory?

Otherwise, feel free to send me an email at yunguan.fu.18@ucl.ac.uk and we can have a live chat regarding this.

@ciphercharly

[ciphercharly]

ok I checked with nvidia-smi to pick the most free ones but it may just be that there's not enough free memory available right now, thanks a lot for the check!

[ciphercharly]

🤦 I had picked the wrong ones ofc, the indexes are 0,1,2,3 and I was assuming 1,2,3,4 without thinking.... it's running on 2 GPUs with batch_size 2 now and no errors

[mathpluscode]

I had picked the wrong ones ofc, the indexes are 0,1,2,3 and I was assuming 1,2,3,4 without thinking.... it's running on 2 GPUs with batch_size 2 now and no errors

Glad to know!