search

RasaHQ / rasa

💬 Open source machine learning framework to automate text- and voice-based conversations: NLU, dialogue management, connect to Slack, Facebook, and more - Create chatbots and voice assistants
https://rasa.com/docs/rasa/
Apache License 2.0
18.92k stars 4.63k forks source link

Investigate unusual memory usage in rasa test #8026

Closed koernerfelicia closed 3 years ago

koernerfelicia commented 3 years ago

Rasa version: 2.3.0

Python version: 3.7.2

Operating system (windows, osx, ...): Linux

Issue: rasa test fails due to OOM, even though rasa train completes successfully. This suggests that something about rasa test is inefficient. The OOM error originates from tensorflow code, and research will work to bring this down (https://github.com/RasaHQ/rasa/issues/8025). However, it's still strange that test pushes this over the edge, when train doesn't. The tensor referenced in the error needs to be created in both train and test, but we can increase train well above the additional numbers and still train successfully so something must be off about test.

318 dialogues train, 41 test: train and test successful 400 dialogues train, 51 test: train successful, test OOM 480 dialogues, 60 test: train successful, test OOM

Error (including full traceback): NOTE I suspect this error message isn't relevant, because it's a side-effect of rasa tests memory usage.

2021-02-04 03:45:01.282803: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at cwise_ops_common.h:134 : Resource exhausted: OOM when allocating tensor with shape[3374,49,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
  File "/opt/conda/bin/rasa", line 5, in <module>
    main()
  File "/home/fkr/rasa/rasa/__main__.py", line 116, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/home/fkr/rasa/rasa/cli/test.py", line 108, in run_core_test
    additional_arguments=vars(args),
  File "/home/fkr/rasa/rasa/test.py", line 135, in test_core
    _agent = Agent.load(unpacked_model)
  File "/home/fkr/rasa/rasa/core/agent.py", line 447, in load
    if core_model
  File "/home/fkr/rasa/rasa/core/policies/ensemble.py", line 374, in load
    policy = policy_cls.load(policy_path, **context)
  File "/home/fkr/rasa/rasa/core/policies/ted_policy.py", line 827, in load
    finetune_mode=should_finetune,
  File "/home/fkr/rasa/rasa/utils/tensorflow/models.py", line 402, in load
    False if finetune_mode else True
  File "/home/fkr/rasa/rasa/utils/tensorflow/models.py", line 242, in fit
    self.train_summary_writer,
  File "/home/fkr/rasa/rasa/utils/tensorflow/models.py", line 438, in _batch_loop
    call_model_function(batch_in)
  File "/home/fkr/rasa/rasa/utils/tensorflow/models.py", line 300, in train_on_batch
    prediction_gradients = tape.gradient(prediction_loss, self.trainable_variables)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/backprop.py", line 1073, in gradient
    unconnected_gradients=unconnected_gradients)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/imperative_grad.py", line 77, in imperative_grad
    compat.as_str(unconnected_gradients.value))
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/backprop.py", line 162, in _gradient_function
    return grad_fn(mock_op, *out_grads)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/math_grad.py", line 1456, in _PowGrad
    return grad * y * math_ops.pow(x, y - 1), None
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py", line 1124, in binary_op_wrapper
    return func(x, y, name=name)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py", line 1456, in _mul_dispatch
    return multiply(x, y, name=name)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py", line 201, in wrapper
    return target(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py", line 508, in multiply
    return gen_math_ops.mul(x, y, name)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/gen_math_ops.py", line 6166, in mul
    _ops.raise_from_not_ok_status(e, name)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3374,49,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Mul]

Command or request that led to error:

`rasa test core`

Definition of Done (added by Enable squad)

koernerfelicia commented 3 years ago

@Ghostvv anything you would add?

Ghostvv commented 3 years ago

@koernerfelicia could you explain that tensorflow OOM is unrelated in this case

koernerfelicia commented 3 years ago

@Ghostvv I can try! Reason why it's unrelated is because this isn't a problem when we train on a number of dialogues that is greater than the number of total dialogues (train + test) that first cause a problem for test. If it were a tensorflow problem, we'd expect:

  1. 318 dialogues train, 41 test: train and test successful
  2. 400 dialogues train, 51 test: train successful, test OOM
  3. 480 dialogues, 60 test: train OOM (unsuccessful )

however, we see:

  1. 318 dialogues train, 41 test: train and test successful
  2. 400 dialogues train, 51 test: train successful, test OOM
  3. 480 dialogues, 60 test: train successful, test OOM

From logging, we know that the tensor mentioned in the error message for setup 2. is smaller than tensors successfully created during train in setup 3.

twerkmeister commented 3 years ago

Just looking over this briefly two things come to mind:

  1. A tensor of shape [3374,49,512] has 3374 49 512 ~ 85 Million weights - definitely not a small tensor
  2. nevertheless it's probably not this tensor itself but it's just the last drop that pushes it into OOM
koernerfelicia commented 3 years ago

Yes, good point. We have an issue related to bringing down these tensors sizes as well (https://github.com/RasaHQ/rasa/issues/8025). We do know that tensors of this size are also being created during train successfully (and also for larger tensors). I can share those logs if you'd like. Let me know! :)

twerkmeister commented 3 years ago

Hey @koernerfelicia 👋 What dataset are you using where this happens? Would you be able to share it? Has this issue come up on other datasets as well before?

twerkmeister commented 3 years ago

Also couple more questions haha:

  1. How are you calling the test? just rasa test from the command line? And is the model already trained at this point?
  2. Is there by any chance a very long example in the test set? I am thinking the memory usage of tensorflow should not be affected as much by the number of stories. In the end your batch size determines how many stories you are handling at once. However, if you have a really big story in your test cases I could imagine it increasing some sequence lengths (and thus the size of some of your tensors) a lot; leading to OOM.
koernerfelicia commented 3 years ago

Hey! This has happened on MultiWOZ and Advising, specifically for e2e training and testing. I haven't tried other datasets, so I don't know if this is also a problem.

I don't think that there's a very long example in the test set, though I can get actual numbers for you. It seems unlikely for Advising, because those stories are pretty simple.

Here's my run.sh (converted to txt so that I can upload it in this comment) run.txt

koernerfelicia commented 3 years ago

Okay, looks like the max story length for the subset of Advising is 15 turns (bot and user), whereas for the subset of MultiWOZ it's 30 turns

twerkmeister commented 3 years ago

Thanks a lot Felicia for all the info, I have access to these two datasets. Were you just using the top N stories from those dataset splits as in the training data repo to create your crash statistics?

What's the config you are using?

twerkmeister commented 3 years ago

Given this happens in e2e training, potentially some of the message lengths could be the culprit. What do you think? At least in advising test some bot responses are very long:

- bot: 'Cryptography plays a fundamental role in building secure computing and communication systems. With its fascinating history through centuries and intriguing connections to deep mathematical ("how quickly can we factor an integer?") and philosophical ("what is randomness?") questions, Cryptography is an important and beautiful subject. With increasing concerns over privacy, security, and authenticity of data and communications in our wired (and wireless) society, cryptographic applications are bound to pervade our lives. Cryptography is, and will continue to be, a vast and exciting area of research in Computer Science and Mathematics. This course is an introduction to the art and science of cryptography. At the end of the course, students should be well-prepared to apply the core scientific principles of cryptography to build secure software and communication systems as well as to pursue more advanced courses and state of the art research in cryptography. This course will study fundamental concepts, algorithms, encryption schemes, and protocols in cryptography. Main topics include: symmetric (private key) encryption, public key encryption, hash functions, digital signatures, and key distribution. The course emphasizes a rigorous mathematical study of the various cryptographic schemes and their security in terms of algorithmic complexity. A nontrivial part of the course will be devoted to algorithmic and mathematical background from number theory and algebra needed to gain a solid understanding of cryptography. Popular cryptographic schemes such as AES and RSA will be highlighted and their security will be rigorously investigated. Detailed syllabus is available from the course web site (link below). This is a 4-credit course approved as an upper-level CS technical elective for undergraduate students in CS-ENGR and CS-LSA. This course is also approved as a cognate course for Math Majors. Advanced undergraduate and beginning graduate students in Computer Science and Engineering and Mathematics are invited to take this course. Graduate students in EECS can also take it as a 400-level elective course. Grading will be based on homework assignments, a mid-term, and a final project/term paper.'
koernerfelicia commented 3 years ago

I was using the number_of_dialogues flag from the dataset_to_rasa.py scripts.

The config is very simple:

language: en
pipeline:
  - name: WhitespaceTokenizer
  - name: CountVectorsFeaturizer
policies:
- name: TEDPolicy
  epochs: 200
koernerfelicia commented 3 years ago

Given this happens in e2e training, potentially some of the message lengths could be the culprit. What do you think?

I'm sure that doesn't help, but I'm not sure it explains why test fails when train doesn't. For MultiWOZ, the longest bot utterance is in the training data, not the test data. We're already convinced that these datasets and the tensors are large. What's weird is that this is a problem for test much sooner (in terms of dataset size progression) than it is for train. Do you know what I mean?

koernerfelicia commented 3 years ago

I should note that my numbers up there for number of stories are not quite correct. I'm sorry about this, it's an oversight on my part. In order to replicate this for MultiWOZ you'll need to set the flag to:

400
500
600

The script creates a train/val/test split, but some dialogues are ignored because they can't be neatly converted, which is why the numbers come out a bit wonky-looking. The number of dialogues I listed above isn't quite correct. These I think are right, though I don't think it should matter.

400 - 312, 41 test
500 - 393, 51 test
600 - 470 train, 63 test
twerkmeister commented 3 years ago

To me it seems that the number of stories is secondary here, as you are using batched training anyway - please correct me if I am wrong. So if you use a batch size of 32 you always get 32 examples into your memory no matter if your entire dataset has 100 or 10000 examples. However, what matters is the size of your examples - 32 big examples might take more memory than 32 small examples.

What I found is that the dataset_to_rasa script for the advising data set sorts the examples according to their length (which I think is the conversation length) https://github.com/RasaHQ/training-data/blob/395f75e58e092eedd31cb09fc622baa5de4dbce4/public/Advising/scripts/advising_loader.py#L100

It would be easily imaginable that there are many more short dialogues in the larger training set and than in the smaller test set. So if you take the shortest n stories from train, you might get a max len thats much shorter than when you take n stories from test.

Just thinking out loud here. Will investigate a bit more

twerkmeister commented 3 years ago

so those wonky numbers mean you are trying to take the top 400 but only get 312 train examples and 41 test examples?

twerkmeister commented 3 years ago

I think this could make sense with your observations - taking the shortest 500 examples from train -> max len a, taking the shortest 500 examples from test (which has fewer examples than train so you get into larger stories quicker) -> max len b. and b >> a -> OOM

twerkmeister commented 3 years ago

hmm for multiwoz there doesn't seem to be such sorting

koernerfelicia commented 3 years ago

so those wonky numbers mean you are trying to take the top 400 but only get 312 train examples and 41 test examples

I just assumed it was creating an 80/10/10 split, because that's what the numbers roughly look like. But now that I look closer it seems to be cutting off at "num_dialogues" and then splitting these by pulling the dialogue IDs from testListFile and valListFile (these come from MultiWOZ itself) here. I haven't figured out how it works out that there is a nice number of dialogues in test/val

koernerfelicia commented 3 years ago

I guess some dialogues are lost here

koernerfelicia commented 3 years ago

I think this could make sense with your observations

You'd expect test to succeed if we ran test on train data then, yes?

twerkmeister commented 3 years ago

You'd expect test to succeed if we ran test on train data then, yes?

That would indeed be a great test for my hypothesis haha

koernerfelicia commented 3 years ago 
rasa test core --debug --stories MultiWOZ/MW.500/data/stories.yml --no-plot --model experiments/MultiWOZ.MW.500.config_1/models/20210204-034433.tar.gz 
2021-03-03 10:45:05.529383: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
2021-03-03 10:45:10 INFO     rasa.model  - Loading model experiments/MultiWOZ.MW.500.config_1/models/20210204-034433.tar.gz...
2021-03-03 10:45:11 DEBUG    rasa.model  - Extracted model to '/tmp/tmpq2yrrgx4'.
/home/fkr/rasa/rasa/utils/train_utils.py:93: FutureWarning: `loss_type=softmax` is deprecated. Please update your configuration file to use`loss_type=cross_entropy` instead. (will be removed in 3.0.0)
  warn_until_version=NEXT_MAJOR_VERSION_FOR_DEPRECATIONS,
2021-03-03 10:45:16 DEBUG    rasa.utils.tensorflow.models  - Loading the model from /tmp/tmpq2yrrgx4/core/policy_0_TEDPolicy/ted_policy.tf_model with finetune_mode=False...
2021-03-03 10:45:16.846010: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2021-03-03 10:45:19.110765: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-03 10:45:19.111635: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:00:04.0 name: Tesla K80 computeCapability: 3.7
coreClock: 0.8235GHz coreCount: 13 deviceMemorySize: 11.17GiB deviceMemoryBandwidth: 223.96GiB/s
2021-03-03 10:45:19.111693: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
2021-03-03 10:45:19.166114: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
2021-03-03 10:45:19.192797: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2021-03-03 10:45:19.204650: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2021-03-03 10:45:19.254108: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2021-03-03 10:45:19.270168: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
2021-03-03 10:45:19.276527: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2021-03-03 10:45:19.276703: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-03 10:45:19.277529: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-03 10:45:19.278262: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2021-03-03 10:45:19.300611: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2300000000 Hz
2021-03-03 10:45:19.301738: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x562d78d405c0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-03-03 10:45:19.301768: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2021-03-03 10:45:19.556666: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-03 10:45:19.557584: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x562d7b7a22d0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-03-03 10:45:19.557622: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla K80, Compute Capability 3.7
2021-03-03 10:45:19.557918: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-03 10:45:19.558699: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:00:04.0 name: Tesla K80 computeCapability: 3.7
coreClock: 0.8235GHz coreCount: 13 deviceMemorySize: 11.17GiB deviceMemoryBandwidth: 223.96GiB/s
2021-03-03 10:45:19.558801: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
2021-03-03 10:45:19.558849: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
2021-03-03 10:45:19.558885: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2021-03-03 10:45:19.558924: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2021-03-03 10:45:19.558956: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2021-03-03 10:45:19.558989: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
2021-03-03 10:45:19.559033: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2021-03-03 10:45:19.559124: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-03 10:45:19.559942: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-03 10:45:19.560655: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2021-03-03 10:45:19.562542: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
2021-03-03 10:45:24.291850: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-03-03 10:45:24.291940: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 
2021-03-03 10:45:24.291961: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N 
2021-03-03 10:45:24.301735: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-03 10:45:24.302588: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-03 10:45:24.303391: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10637 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
2021-03-03 10:45:25.311424: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
TensorShape([3374, 49, 128])
TensorShape([3374, 49, 512])
TensorShape([3374, 49, 512])
TensorShape([3374, 49, 128])
TensorShape([3374, 49, 128])
TensorShape([1, 1, 128])
TensorShape([1, 1, 512])
TensorShape([1, 1, 512])
TensorShape([1, 1, 128])
TensorShape([1, 1, 128])
2021-03-03 10:45:37.729887: W tensorflow/core/common_runtime/bfc_allocator.cc:431] Allocator (GPU_0_bfc) ran out of memory trying to allocate 322.90MiB (rounded to 338587648)requested by op Mul
Current allocation summary follows.
2021-03-03 10:45:37.729972: I tensorflow/core/common_runtime/bfc_allocator.cc:970] BFCAllocator dump for GPU_0_bfc
2021-03-03 10:45:37.729990: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (256):   Total Chunks: 173, Chunks in use: 173. 43.2KiB allocated for chunks. 43.2KiB in use in bin. 4.2KiB client-requested in use in bin.
2021-03-03 10:45:37.730001: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (512):   Total Chunks: 85, Chunks in use: 84. 43.8KiB allocated for chunks. 43.2KiB in use in bin. 41.6KiB client-requested in use in bin.
2021-03-03 10:45:37.730011: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (1024):  Total Chunks: 13, Chunks in use: 12. 18.2KiB allocated for chunks. 17.2KiB in use in bin. 14.4KiB client-requested in use in bin.
2021-03-03 10:45:37.730020: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (2048):  Total Chunks: 24, Chunks in use: 24. 52.8KiB allocated for chunks. 52.8KiB in use in bin. 49.3KiB client-requested in use in bin.
2021-03-03 10:45:37.730045: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (4096):  Total Chunks: 8, Chunks in use: 8. 36.0KiB allocated for chunks. 36.0KiB in use in bin. 31.2KiB client-requested in use in bin.
2021-03-03 10:45:37.730065: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (8192):  Total Chunks: 6, Chunks in use: 6. 75.5KiB allocated for chunks. 75.5KiB in use in bin. 69.7KiB client-requested in use in bin.
2021-03-03 10:45:37.730079: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (16384):     Total Chunks: 6, Chunks in use: 6. 143.0KiB allocated for chunks. 143.0KiB in use in bin. 139.1KiB client-requested in use in bin.
2021-03-03 10:45:37.730091: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (32768):     Total Chunks: 5, Chunks in use: 5. 242.0KiB allocated for chunks. 242.0KiB in use in bin. 196.4KiB client-requested in use in bin.
2021-03-03 10:45:37.730101: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (65536):     Total Chunks: 30, Chunks in use: 30. 2.04MiB allocated for chunks. 2.04MiB in use in bin. 1.87MiB client-requested in use in bin.
2021-03-03 10:45:37.730112: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (131072):    Total Chunks: 4, Chunks in use: 4. 769.5KiB allocated for chunks. 769.5KiB in use in bin. 719.6KiB client-requested in use in bin.
2021-03-03 10:45:37.730122: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (262144):    Total Chunks: 16, Chunks in use: 16. 4.18MiB allocated for chunks. 4.18MiB in use in bin. 4.01MiB client-requested in use in bin.
2021-03-03 10:45:37.730131: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (524288):    Total Chunks: 21, Chunks in use: 21. 13.49MiB allocated for chunks. 13.49MiB in use in bin. 13.48MiB client-requested in use in bin.
2021-03-03 10:45:37.730157: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (1048576):   Total Chunks: 16, Chunks in use: 16. 20.45MiB allocated for chunks. 20.45MiB in use in bin. 18.61MiB client-requested in use in bin.
2021-03-03 10:45:37.730167: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (2097152):   Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-03-03 10:45:37.730182: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (4194304):   Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-03-03 10:45:37.730193: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (8388608):   Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-03-03 10:45:37.730205: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (16777216):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-03-03 10:45:37.730219: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (33554432):  Total Chunks: 2, Chunks in use: 1. 80.80MiB allocated for chunks. 39.10MiB in use in bin. 30.90MiB client-requested in use in bin.
2021-03-03 10:45:37.730232: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (67108864):  Total Chunks: 65, Chunks in use: 64. 5.30GiB allocated for chunks. 5.22GiB in use in bin. 5.16GiB client-requested in use in bin.
2021-03-03 10:45:37.730246: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (134217728):     Total Chunks: 1, Chunks in use: 0. 162.73MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-03-03 10:45:37.730259: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (268435456):     Total Chunks: 15, Chunks in use: 15. 4.81GiB allocated for chunks. 4.81GiB in use in bin. 4.73GiB client-requested in use in bin.
2021-03-03 10:45:37.730272: I tensorflow/core/common_runtime/bfc_allocator.cc:993] Bin for 322.90MiB was 256.00MiB, Chunk State: 
2021-03-03 10:45:37.730283: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 11154668800
2021-03-03 10:45:37.730300: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de0000 of size 1280 next 1
2021-03-03 10:45:37.730314: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de0500 of size 256 next 2
2021-03-03 10:45:37.730323: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de0600 of size 256 next 3
2021-03-03 10:45:37.730333: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de0700 of size 256 next 4
2021-03-03 10:45:37.730344: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de0800 of size 256 next 5
2021-03-03 10:45:37.730353: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de0900 of size 256 next 6
2021-03-03 10:45:37.730363: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de0a00 of size 256 next 7
2021-03-03 10:45:37.730373: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de0b00 of size 256 next 8
2021-03-03 10:45:37.730383: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de0c00 of size 256 next 9
2021-03-03 10:45:37.730393: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de0d00 of size 256 next 10
2021-03-03 10:45:37.730403: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de0e00 of size 256 next 11
2021-03-03 10:45:37.730412: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de0f00 of size 256 next 12
2021-03-03 10:45:37.730423: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de1000 of size 256 next 13
2021-03-03 10:45:37.730433: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de1100 of size 256 next 14
2021-03-03 10:45:37.730443: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de1200 of size 256 next 15
2021-03-03 10:45:37.730452: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de1300 of size 256 next 16
2021-03-03 10:45:37.730462: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de1400 of size 256 next 17
2021-03-03 10:45:37.730472: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de1500 of size 256 next 18
2021-03-03 10:45:37.730482: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de1600 of size 256 next 19
2021-03-03 10:45:37.730493: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de1700 of size 256 next 20
2021-03-03 10:45:37.730503: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de1800 of size 256 next 21
2021-03-03 10:45:37.730513: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de1900 of size 256 next 22
2021-03-03 10:45:37.730541: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de1a00 of size 256 next 23
2021-03-03 10:45:37.730550: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de1b00 of size 256 next 24
2021-03-03 10:45:37.730556: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de1c00 of size 256 next 25
2021-03-03 10:45:37.730563: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de1d00 of size 256 next 26
2021-03-03 10:45:37.730569: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de1e00 of size 256 next 27
2021-03-03 10:45:37.730576: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de1f00 of size 256 next 28
2021-03-03 10:45:37.730582: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de2000 of size 256 next 29
2021-03-03 10:45:37.730602: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de2100 of size 256 next 30
2021-03-03 10:45:37.730609: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de2200 of size 256 next 31
2021-03-03 10:45:37.730615: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de2300 of size 256 next 32
2021-03-03 10:45:37.730622: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de2400 of size 256 next 33
2021-03-03 10:45:37.730629: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de2500 of size 256 next 34
2021-03-03 10:45:37.730636: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de2600 of size 256 next 35
2021-03-03 10:45:37.730645: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de2700 of size 256 next 36
2021-03-03 10:45:37.730653: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de2800 of size 256 next 37
2021-03-03 10:45:37.730661: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de2900 of size 256 next 40
2021-03-03 10:45:37.730668: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de2a00 of size 256 next 43
2021-03-03 10:45:37.730678: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de2b00 of size 256 next 44
2021-03-03 10:45:37.730689: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de2c00 of size 256 next 38
2021-03-03 10:45:37.730697: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de2d00 of size 256 next 39
2021-03-03 10:45:37.730706: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de2e00 of size 1280 next 41
2021-03-03 10:45:37.730716: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de3300 of size 1024 next 42
2021-03-03 10:45:37.730758: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de3700 of size 1024 next 45
2021-03-03 10:45:37.730765: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de3b00 of size 256 next 49
2021-03-03 10:45:37.730779: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de3c00 of size 256 next 52
2021-03-03 10:45:37.730786: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de3d00 of size 3584 next 46
2021-03-03 10:45:37.730801: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de4b00 of size 256 next 47
2021-03-03 10:45:37.730809: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de4c00 of size 256 next 48
2021-03-03 10:45:37.730817: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de4d00 of size 4352 next 50
2021-03-03 10:45:37.730828: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de5e00 of size 4096 next 51
2021-03-03 10:45:37.730843: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de6e00 of size 4096 next 54
2021-03-03 10:45:37.730854: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de7e00 of size 2304 next 53
2021-03-03 10:45:37.730865: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de8700 of size 256 next 55
2021-03-03 10:45:37.730875: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de8800 of size 2304 next 56
2021-03-03 10:45:37.730885: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de9100 of size 256 next 57
2021-03-03 10:45:37.730896: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de9200 of size 256 next 58
2021-03-03 10:45:37.730906: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de9300 of size 256 next 59
2021-03-03 10:45:37.730914: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de9400 of size 256 next 60
2021-03-03 10:45:37.730924: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de9500 of size 2304 next 61
2021-03-03 10:45:37.730937: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de9e00 of size 256 next 62
2021-03-03 10:45:37.730948: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03de9f00 of size 2304 next 63
2021-03-03 10:45:37.730958: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03dea800 of size 2304 next 64
2021-03-03 10:45:37.730968: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03deb100 of size 256 next 67
2021-03-03 10:45:37.730978: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03deb200 of size 256 next 76
2021-03-03 10:45:37.730988: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03deb300 of size 256 next 81
2021-03-03 10:45:37.730998: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03deb400 of size 256 next 86
2021-03-03 10:45:37.731009: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03deb500 of size 256 next 87
2021-03-03 10:45:37.731019: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03deb600 of size 256 next 88
2021-03-03 10:45:37.731029: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03deb700 of size 768 next 65
2021-03-03 10:45:37.731039: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03deba00 of size 2304 next 66
2021-03-03 10:45:37.731049: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03dec300 of size 2304 next 68
2021-03-03 10:45:37.731059: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03decc00 of size 2304 next 69
2021-03-03 10:45:37.731069: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ded500 of size 13568 next 70
2021-03-03 10:45:37.731080: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03df0a00 of size 13568 next 77
2021-03-03 10:45:37.731091: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03df3f00 of size 512 next 103
2021-03-03 10:45:37.731105: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03df4100 of size 256 next 108
2021-03-03 10:45:37.731112: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03df4200 of size 84224 next 79
2021-03-03 10:45:37.731125: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03e08b00 of size 677120 next 74
2021-03-03 10:45:37.731135: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eae000 of size 256 next 101
2021-03-03 10:45:37.731148: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eae100 of size 256 next 114
2021-03-03 10:45:37.731159: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eae200 of size 256 next 115
2021-03-03 10:45:37.731169: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eae300 of size 512 next 116
2021-03-03 10:45:37.731180: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eae500 of size 256 next 121
2021-03-03 10:45:37.731190: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eae600 of size 256 next 122
2021-03-03 10:45:37.731199: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eae700 of size 256 next 123
2021-03-03 10:45:37.731221: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eae800 of size 256 next 125
2021-03-03 10:45:37.731231: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eae900 of size 256 next 126
2021-03-03 10:45:37.731241: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eaea00 of size 256 next 127
2021-03-03 10:45:37.731251: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eaeb00 of size 256 next 128
2021-03-03 10:45:37.731261: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eaec00 of size 512 next 137
2021-03-03 10:45:37.731271: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eaee00 of size 512 next 138
2021-03-03 10:45:37.731280: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eaf000 of size 256 next 163
2021-03-03 10:45:37.731290: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eaf100 of size 512 next 166
2021-03-03 10:45:37.731300: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eaf300 of size 512 next 176
2021-03-03 10:45:37.731310: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eaf500 of size 512 next 177
2021-03-03 10:45:37.731320: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eaf700 of size 256 next 189
2021-03-03 10:45:37.731330: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eaf800 of size 2048 next 188
2021-03-03 10:45:37.731340: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eb0000 of size 256 next 195
2021-03-03 10:45:37.731350: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eb0100 of size 256 next 196
2021-03-03 10:45:37.731361: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eb0200 of size 512 next 204
2021-03-03 10:45:37.731371: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eb0400 of size 512 next 216
2021-03-03 10:45:37.731377: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eb0600 of size 512 next 217
2021-03-03 10:45:37.731383: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eb0800 of size 256 next 228
2021-03-03 10:45:37.731389: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eb0900 of size 27136 next 235
2021-03-03 10:45:37.731400: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eb7300 of size 256 next 243
2021-03-03 10:45:37.731410: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eb7400 of size 256 next 244
2021-03-03 10:45:37.731420: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03eb7500 of size 13568 next 251
2021-03-03 10:45:37.731430: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebaa00 of size 512 next 285
2021-03-03 10:45:37.731440: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebac00 of size 512 next 286
2021-03-03 10:45:37.731450: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebae00 of size 512 next 287
2021-03-03 10:45:37.731460: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebb000 of size 512 next 283
2021-03-03 10:45:37.731470: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebb200 of size 512 next 288
2021-03-03 10:45:37.731480: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebb400 of size 512 next 289
2021-03-03 10:45:37.731488: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebb600 of size 256 next 290
2021-03-03 10:45:37.731497: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebb700 of size 256 next 291
2021-03-03 10:45:37.731507: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebb800 of size 256 next 292
2021-03-03 10:45:37.731517: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebb900 of size 512 next 293
2021-03-03 10:45:37.731534: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebbb00 of size 512 next 294
2021-03-03 10:45:37.731545: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebbd00 of size 512 next 295
2021-03-03 10:45:37.731553: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebbf00 of size 512 next 296
2021-03-03 10:45:37.731562: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebc100 of size 512 next 297
2021-03-03 10:45:37.731572: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebc300 of size 512 next 301
2021-03-03 10:45:37.731583: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebc500 of size 512 next 305
2021-03-03 10:45:37.731593: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebc700 of size 512 next 309
2021-03-03 10:45:37.731601: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebc900 of size 256 next 310
2021-03-03 10:45:37.731620: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebca00 of size 256 next 311
2021-03-03 10:45:37.731642: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebcb00 of size 256 next 312
2021-03-03 10:45:37.731652: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebcc00 of size 256 next 313
2021-03-03 10:45:37.731668: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebcd00 of size 256 next 314
2021-03-03 10:45:37.731679: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebce00 of size 256 next 315
2021-03-03 10:45:37.731686: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebcf00 of size 512 next 316
2021-03-03 10:45:37.731696: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebd100 of size 512 next 319
2021-03-03 10:45:37.731706: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebd300 of size 512 next 320
2021-03-03 10:45:37.731716: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebd500 of size 512 next 322
2021-03-03 10:45:37.731726: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebd700 of size 512 next 323
2021-03-03 10:45:37.731735: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebd900 of size 512 next 324
2021-03-03 10:45:37.731745: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebdb00 of size 512 next 325
2021-03-03 10:45:37.731755: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebdd00 of size 512 next 326
2021-03-03 10:45:37.731765: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebdf00 of size 512 next 327
2021-03-03 10:45:37.731774: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebe100 of size 768 next 117
2021-03-03 10:45:37.731784: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ebe400 of size 95488 next 71
2021-03-03 10:45:37.731794: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ed5900 of size 647936 next 72
2021-03-03 10:45:37.731805: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03f73c00 of size 65536 next 119
2021-03-03 10:45:37.731815: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03f83c00 of size 99840 next 78
2021-03-03 10:45:37.731825: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03f9c200 of size 65536 next 298
2021-03-03 10:45:37.731837: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03fac200 of size 65536 next 299
2021-03-03 10:45:37.731848: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03fbc200 of size 65536 next 300
2021-03-03 10:45:37.731858: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03fcc200 of size 65536 next 302
2021-03-03 10:45:37.731869: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03fdc200 of size 65536 next 303
2021-03-03 10:45:37.731882: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03fec200 of size 65536 next 304
2021-03-03 10:45:37.731892: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b03ffc200 of size 65536 next 306
2021-03-03 10:45:37.731902: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b0400c200 of size 65536 next 307
2021-03-03 10:45:37.731912: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b0401c200 of size 65536 next 308
2021-03-03 10:45:37.731922: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b0402c200 of size 71680 next 73
2021-03-03 10:45:37.731933: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b0403da00 of size 192768 next 75
2021-03-03 10:45:37.731944: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b0406cb00 of size 175872 next 94
2021-03-03 10:45:37.731956: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b04097a00 of size 25088 next 134
2021-03-03 10:45:37.731966: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b0409dc00 of size 65536 next 148
2021-03-03 10:45:37.731977: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b040adc00 of size 85248 next 98
2021-03-03 10:45:37.731987: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b040c2900 of size 226560 next 82
2021-03-03 10:45:37.731997: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b040f9e00 of size 1156352 next 83
2021-03-03 10:45:37.732005: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b04214300 of size 1156352 next 84
2021-03-03 10:45:37.732015: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b0432e800 of size 192768 next 85
2021-03-03 10:45:37.732026: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b0435d900 of size 771072 next 80
2021-03-03 10:45:37.732037: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b04419d00 of size 1347584 next 90
2021-03-03 10:45:37.732047: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b04562d00 of size 1059328 next 89
2021-03-03 10:45:37.732057: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b04665700 of size 82919424 next 91
2021-03-03 10:45:37.732068: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b09579700 of size 82919424 next 92
2021-03-03 10:45:37.732075: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b0e48d700 of size 82919424 next 93
2021-03-03 10:45:37.732082: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b133a1700 of size 103649280 next 95
2021-03-03 10:45:37.732089: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b1967a700 of size 82919424 next 96
2021-03-03 10:45:37.732099: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b1e58e700 of size 82919424 next 97
2021-03-03 10:45:37.732110: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b234a2700 of size 1054464 next 100
2021-03-03 10:45:37.732120: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b235a3e00 of size 1059328 next 102
2021-03-03 10:45:37.732130: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b236a6800 of size 1059328 next 99
2021-03-03 10:45:37.732142: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b237a9200 of size 1727488 next 104
2021-03-03 10:45:37.732153: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b2394ee00 of size 1727488 next 105
2021-03-03 10:45:37.732164: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b23af4a00 of size 1727488 next 106
2021-03-03 10:45:37.732174: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b23c9a600 of size 661504 next 139
2021-03-03 10:45:37.732183: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b23d3be00 of size 1065984 next 107
2021-03-03 10:45:37.732193: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b23e40200 of size 1727488 next 109
2021-03-03 10:45:37.732203: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b23fe5e00 of size 1727488 next 110
2021-03-03 10:45:37.732213: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b2418ba00 of size 84646912 next 111
2021-03-03 10:45:37.732224: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b29245600 of size 84646912 next 112
2021-03-03 10:45:37.732232: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b2e2ff200 of size 84646912 next 113
2021-03-03 10:45:37.732242: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b333b8e00 of size 84646912 next 118
2021-03-03 10:45:37.732252: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b38472a00 of size 84646912 next 120
2021-03-03 10:45:37.732270: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b3d52c600 of size 84646912 next 124
2021-03-03 10:45:37.732286: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b425e6200 of size 84646912 next 132
2021-03-03 10:45:37.732296: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4769fe00 of size 84646912 next 133
2021-03-03 10:45:37.732307: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4c759a00 of size 65536 next 150
2021-03-03 10:45:37.732317: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4c769a00 of size 65536 next 151
2021-03-03 10:45:37.732327: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4c779a00 of size 65536 next 152
2021-03-03 10:45:37.732335: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4c789a00 of size 65536 next 154
2021-03-03 10:45:37.732345: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4c799a00 of size 65536 next 155
2021-03-03 10:45:37.732355: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4c7a9a00 of size 65536 next 156
2021-03-03 10:45:37.732365: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4c7b9a00 of size 65536 next 170
2021-03-03 10:45:37.732375: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4c7c9a00 of size 65536 next 167
2021-03-03 10:45:37.732385: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4c7d9a00 of size 65536 next 168
2021-03-03 10:45:37.732395: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4c7e9a00 of size 256 next 237
2021-03-03 10:45:37.732405: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4c7e9b00 of size 256 next 238
2021-03-03 10:45:37.732416: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4c7e9c00 of size 25600 next 239
2021-03-03 10:45:37.732426: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4c7f0000 of size 45568 next 141
2021-03-03 10:45:37.732434: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4c7fb200 of size 661504 next 142
2021-03-03 10:45:37.732444: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4c89ca00 of size 661504 next 178
2021-03-03 10:45:37.732454: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4c93e200 of size 661504 next 180
2021-03-03 10:45:37.732466: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4c9dfa00 of size 262144 next 191
2021-03-03 10:45:37.732476: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4ca1fa00 of size 399360 next 181
2021-03-03 10:45:37.732487: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4ca81200 of size 661504 next 182
2021-03-03 10:45:37.732495: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cb22a00 of size 262144 next 190
2021-03-03 10:45:37.732504: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cb62a00 of size 262144 next 210
2021-03-03 10:45:37.732515: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cba2a00 of size 262144 next 208
2021-03-03 10:45:37.732529: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cbe2a00 of size 262144 next 209
2021-03-03 10:45:37.732542: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cc22a00 of size 661504 next 218
2021-03-03 10:45:37.732552: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4ccc4200 of size 661504 next 220
2021-03-03 10:45:37.732562: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd65a00 of size 25600 next 241
2021-03-03 10:45:37.732573: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6be00 of size 256 next 254
2021-03-03 10:45:37.732585: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6bf00 of size 256 next 259
2021-03-03 10:45:37.732596: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6c000 of size 256 next 260
2021-03-03 10:45:37.732609: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6c100 of size 256 next 261
2021-03-03 10:45:37.732616: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6c200 of size 256 next 262
2021-03-03 10:45:37.732625: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6c300 of size 256 next 263
2021-03-03 10:45:37.732637: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6c400 of size 256 next 264
2021-03-03 10:45:37.732649: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6c500 of size 256 next 265
2021-03-03 10:45:37.732659: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6c600 of size 256 next 266
2021-03-03 10:45:37.732668: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6c700 of size 256 next 267
2021-03-03 10:45:37.732679: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6c800 of size 256 next 268
2021-03-03 10:45:37.732686: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6c900 of size 256 next 269
2021-03-03 10:45:37.732696: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6ca00 of size 256 next 430
2021-03-03 10:45:37.732707: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6cb00 of size 256 next 270
2021-03-03 10:45:37.732717: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6cc00 of size 512 next 271
2021-03-03 10:45:37.732727: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6ce00 of size 512 next 272
2021-03-03 10:45:37.732737: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6d000 of size 512 next 276
2021-03-03 10:45:37.732747: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6d200 of size 512 next 277
2021-03-03 10:45:37.732756: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6d400 of size 256 next 278
2021-03-03 10:45:37.732766: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6d500 of size 256 next 279
2021-03-03 10:45:37.732774: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6d600 of size 512 next 280
2021-03-03 10:45:37.732783: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6d800 of size 512 next 282
2021-03-03 10:45:37.732794: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6da00 of size 256 next 284
2021-03-03 10:45:37.732804: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6db00 of size 256 next 281
2021-03-03 10:45:37.732814: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6dc00 of size 512 next 257
2021-03-03 10:45:37.732824: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6de00 of size 5376 next 252
2021-03-03 10:45:37.732835: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd6f300 of size 40704 next 253
2021-03-03 10:45:37.732843: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd79200 of size 512 next 328
2021-03-03 10:45:37.732853: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd79400 of size 256 next 329
2021-03-03 10:45:37.732862: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd79500 of size 256 next 330
2021-03-03 10:45:37.732872: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd79600 of size 512 next 331
2021-03-03 10:45:37.732881: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd79800 of size 512 next 332
2021-03-03 10:45:37.732890: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd79a00 of size 512 next 333
2021-03-03 10:45:37.732898: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd79c00 of size 512 next 334
2021-03-03 10:45:37.732908: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd79e00 of size 512 next 335
2021-03-03 10:45:37.732919: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd7a000 of size 2048 next 336
2021-03-03 10:45:37.732929: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd7a800 of size 2048 next 340
2021-03-03 10:45:37.732936: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd7b000 of size 2048 next 341
2021-03-03 10:45:37.732946: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd7b800 of size 2048 next 342
2021-03-03 10:45:37.732958: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd7c000 of size 256 next 343
2021-03-03 10:45:37.732969: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd7c100 of size 3328 next 275
2021-03-03 10:45:37.732977: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd7ce00 of size 107520 next 274
2021-03-03 10:45:37.732987: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cd97200 of size 61440 next 273
2021-03-03 10:45:37.732995: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cda6200 of size 86528 next 256
2021-03-03 10:45:37.733016: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cdbb400 of size 310784 next 221
2021-03-03 10:45:37.733027: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4ce07200 of size 661504 next 222
2021-03-03 10:45:37.733037: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4cea8a00 of size 1727488 next 236
2021-03-03 10:45:37.733045: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d04e600 of size 674816 next 240
2021-03-03 10:45:37.733056: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d0f3200 of size 674816 next 242
2021-03-03 10:45:37.733066: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d197e00 of size 674816 next 245
2021-03-03 10:45:37.733073: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d23ca00 of size 674816 next 246
2021-03-03 10:45:37.733083: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d2e1600 of size 674816 next 247
2021-03-03 10:45:37.733093: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d386200 of size 674816 next 248
2021-03-03 10:45:37.733104: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d42ae00 of size 674816 next 249
2021-03-03 10:45:37.733115: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d4cfa00 of size 674816 next 250
2021-03-03 10:45:37.733125: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d574600 of size 677120 next 255
2021-03-03 10:45:37.733135: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d619b00 of size 677120 next 258
2021-03-03 10:45:37.733145: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6bf000 of size 2048 next 344
2021-03-03 10:45:37.733155: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6bf800 of size 2048 next 345
2021-03-03 10:45:37.733166: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c0000 of size 2048 next 346
2021-03-03 10:45:37.733176: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c0800 of size 2048 next 347
2021-03-03 10:45:37.733184: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c1000 of size 2048 next 348
2021-03-03 10:45:37.733194: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c1800 of size 2048 next 349
2021-03-03 10:45:37.733204: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c2000 of size 512 next 350
2021-03-03 10:45:37.733215: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c2200 of size 512 next 357
2021-03-03 10:45:37.733225: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c2400 of size 512 next 359
2021-03-03 10:45:37.733233: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c2600 of size 512 next 360
2021-03-03 10:45:37.733242: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c2800 of size 512 next 351
2021-03-03 10:45:37.733250: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c2a00 of size 2048 next 352
2021-03-03 10:45:37.733260: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c3200 of size 2048 next 353
2021-03-03 10:45:37.733270: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c3a00 of size 512 next 361
2021-03-03 10:45:37.733281: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c3c00 of size 512 next 362
2021-03-03 10:45:37.733291: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c3e00 of size 512 next 363
2021-03-03 10:45:37.733302: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c4000 of size 512 next 364
2021-03-03 10:45:37.733312: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c4200 of size 512 next 365
2021-03-03 10:45:37.733323: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c4400 of size 256 next 366
2021-03-03 10:45:37.733331: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c4500 of size 256 next 367
2021-03-03 10:45:37.733341: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c4600 of size 256 next 368
2021-03-03 10:45:37.733351: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c4700 of size 512 next 369
2021-03-03 10:45:37.733361: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c4900 of size 512 next 370
2021-03-03 10:45:37.733369: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c4b00 of size 512 next 371
2021-03-03 10:45:37.733379: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c4d00 of size 512 next 372
2021-03-03 10:45:37.733386: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c4f00 of size 512 next 373
2021-03-03 10:45:37.733396: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c5100 of size 512 next 374
2021-03-03 10:45:37.733406: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c5300 of size 512 next 375
2021-03-03 10:45:37.733416: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c5500 of size 512 next 376
2021-03-03 10:45:37.733427: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c5700 of size 512 next 377
2021-03-03 10:45:37.733437: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c5900 of size 512 next 378
2021-03-03 10:45:37.733444: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c5b00 of size 512 next 379
2021-03-03 10:45:37.733454: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c5d00 of size 512 next 380
2021-03-03 10:45:37.733463: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c5f00 of size 256 next 381
2021-03-03 10:45:37.733473: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c6000 of size 256 next 382
2021-03-03 10:45:37.733483: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c6100 of size 256 next 383
2021-03-03 10:45:37.733493: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c6200 of size 256 next 385
2021-03-03 10:45:37.733500: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c6300 of size 1792 next 386
2021-03-03 10:45:37.733509: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c6a00 of size 256 next 387
2021-03-03 10:45:37.733524: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c6b00 of size 256 next 388
2021-03-03 10:45:37.733535: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c6c00 of size 256 next 389
2021-03-03 10:45:37.733545: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6c6d00 of size 17408 next 384
2021-03-03 10:45:37.733556: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6cb100 of size 16128 next 317
2021-03-03 10:45:37.733566: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6cf000 of size 65536 next 318
2021-03-03 10:45:37.733574: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6df000 of size 65536 next 321
2021-03-03 10:45:37.733584: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d6ef000 of size 262144 next 339
2021-03-03 10:45:37.733594: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d72f000 of size 262144 next 337
2021-03-03 10:45:37.733602: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d76f000 of size 262144 next 338
2021-03-03 10:45:37.733612: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7af000 of size 262144 next 356
2021-03-03 10:45:37.733622: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7ef000 of size 1792 next 390
2021-03-03 10:45:37.733632: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7ef700 of size 256 next 391
2021-03-03 10:45:37.733640: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7ef800 of size 256 next 392
2021-03-03 10:45:37.733650: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7ef900 of size 256 next 393
2021-03-03 10:45:37.733660: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7efa00 of size 1792 next 394
2021-03-03 10:45:37.733670: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f0100 of size 256 next 395
2021-03-03 10:45:37.733680: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f0200 of size 256 next 396
2021-03-03 10:45:37.733693: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f0300 of size 256 next 397
2021-03-03 10:45:37.733700: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f0400 of size 256 next 398
2021-03-03 10:45:37.733710: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f0500 of size 256 next 399
2021-03-03 10:45:37.733720: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f0600 of size 1792 next 400
2021-03-03 10:45:37.733728: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f0d00 of size 256 next 401
2021-03-03 10:45:37.733739: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f0e00 of size 256 next 402
2021-03-03 10:45:37.733749: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f0f00 of size 256 next 403
2021-03-03 10:45:37.733759: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f1000 of size 256 next 404
2021-03-03 10:45:37.733769: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f1100 of size 1792 next 405
2021-03-03 10:45:37.733781: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f1800 of size 256 next 406
2021-03-03 10:45:37.733792: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f1900 of size 256 next 407
2021-03-03 10:45:37.733802: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f1a00 of size 256 next 408
2021-03-03 10:45:37.733812: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f1b00 of size 256 next 409
2021-03-03 10:45:37.733822: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f1c00 of size 1792 next 410
2021-03-03 10:45:37.733833: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f2300 of size 256 next 411
2021-03-03 10:45:37.733843: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f2400 of size 256 next 412
2021-03-03 10:45:37.733853: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f2500 of size 256 next 413
2021-03-03 10:45:37.733861: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f2600 of size 256 next 414
2021-03-03 10:45:37.733871: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f2700 of size 256 next 415
2021-03-03 10:45:37.733881: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f2800 of size 256 next 416
2021-03-03 10:45:37.733891: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f2900 of size 256 next 417
2021-03-03 10:45:37.733899: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f2a00 of size 256 next 418
2021-03-03 10:45:37.733909: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f2b00 of size 512 next 419
2021-03-03 10:45:37.733918: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f2d00 of size 256 next 420
2021-03-03 10:45:37.733929: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f2e00 of size 256 next 421
2021-03-03 10:45:37.733939: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f2f00 of size 256 next 422
2021-03-03 10:45:37.733949: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f3000 of size 512 next 423
2021-03-03 10:45:37.733959: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f3200 of size 256 next 424
2021-03-03 10:45:37.733969: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f3300 of size 256 next 425
2021-03-03 10:45:37.733979: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f3400 of size 256 next 426
2021-03-03 10:45:37.733990: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f3500 of size 10240 next 427
2021-03-03 10:45:37.734000: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f5d00 of size 4096 next 428
2021-03-03 10:45:37.734008: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f6d00 of size 4096 next 429
2021-03-03 10:45:37.734018: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7f7d00 of size 25600 next 431
2021-03-03 10:45:37.734029: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7fe100 of size 256 next 432
2021-03-03 10:45:37.734039: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7fe200 of size 256 next 433
2021-03-03 10:45:37.734049: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7fe300 of size 256 next 434
2021-03-03 10:45:37.734059: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7fe400 of size 1024 next 435
2021-03-03 10:45:37.734069: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7fe800 of size 256 next 436
2021-03-03 10:45:37.734080: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7fe900 of size 256 next 437
2021-03-03 10:45:37.734090: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7fea00 of size 256 next 438
2021-03-03 10:45:37.734100: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7feb00 of size 256 next 440
2021-03-03 10:45:37.734111: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7fec00 of size 256 next 441
2021-03-03 10:45:37.734118: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7fed00 of size 256 next 442
2021-03-03 10:45:37.734128: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7fee00 of size 256 next 444
2021-03-03 10:45:37.734138: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7fef00 of size 256 next 445
2021-03-03 10:45:37.734149: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7ff000 of size 256 next 446
2021-03-03 10:45:37.734159: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d7ff100 of size 61440 next 447
2021-03-03 10:45:37.734170: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d80e100 of size 256 next 448
2021-03-03 10:45:37.734180: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d80e200 of size 256 next 449
2021-03-03 10:45:37.734191: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d80e300 of size 256 next 450
2021-03-03 10:45:37.734201: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d80e400 of size 65536 next 451
2021-03-03 10:45:37.734211: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d81e400 of size 256 next 452
2021-03-03 10:45:37.734221: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d81e500 of size 256 next 453
2021-03-03 10:45:37.734231: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d81e600 of size 256 next 454
2021-03-03 10:45:37.734241: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d81e700 of size 256 next 455
2021-03-03 10:45:37.734249: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d81e800 of size 256 next 456
2021-03-03 10:45:37.734259: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d81e900 of size 256 next 457
2021-03-03 10:45:37.734269: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d81ea00 of size 256 next 465
2021-03-03 10:45:37.734279: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d81eb00 of size 6656 next 469
2021-03-03 10:45:37.734290: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d820500 of size 256 next 468
2021-03-03 10:45:37.734300: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d820600 of size 768 next 459
2021-03-03 10:45:37.734311: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d820900 of size 10240 next 462
2021-03-03 10:45:37.734321: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d823100 of size 256 next 467
2021-03-03 10:45:37.734331: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d823200 of size 1280 next 472
2021-03-03 10:45:37.734341: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d823700 of size 512 next 477
2021-03-03 10:45:37.734351: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d823900 of size 768 next 482
2021-03-03 10:45:37.734362: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d823c00 of size 512 next 475
2021-03-03 10:45:37.734372: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d823e00 of size 256 next 470
2021-03-03 10:45:37.734382: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d823f00 of size 256 next 471
2021-03-03 10:45:37.734392: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d824000 of size 512 next 460
2021-03-03 10:45:37.734402: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d824200 of size 768 next 474
2021-03-03 10:45:37.734410: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d824500 of size 512 next 463
2021-03-03 10:45:37.734420: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d824700 of size 2048 next 478
2021-03-03 10:45:37.734430: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d824f00 of size 512 next 483
2021-03-03 10:45:37.734440: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d825100 of size 512 next 485
2021-03-03 10:45:37.734450: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d825300 of size 512 next 490
2021-03-03 10:45:37.734461: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at b4d825500 of size 1024 next 473
2021-03-03 10:45:37.734472: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d825900 of size 38656 next 354
2021-03-03 10:45:37.734489: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d82f000 of size 262144 next 355
2021-03-03 10:45:37.734496: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d86f000 of size 262144 next 358
2021-03-03 10:45:37.734507: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d8af000 of size 1059328 next 439
2021-03-03 10:45:37.734517: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4d9b1a00 of size 1059328 next 443
2021-03-03 10:45:37.734536: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4dab4400 of size 4096 next 476
2021-03-03 10:45:37.734547: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4dab5400 of size 262144 next 458
2021-03-03 10:45:37.734557: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4daf5400 of size 512 next 464
2021-03-03 10:45:37.734567: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4daf5600 of size 262144 next 484
2021-03-03 10:45:37.734578: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4db35600 of size 262144 next 481
2021-03-03 10:45:37.734589: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at b4db75600 of size 512 next 487
2021-03-03 10:45:37.734599: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b4db75800 of size 512 next 491
2021-03-03 10:45:37.734609: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at b4db75a00 of size 43721728 next 131
2021-03-03 10:45:37.734620: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b50527e00 of size 41000704 next 129
2021-03-03 10:45:37.734631: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b52c41d00 of size 84646912 next 130
2021-03-03 10:45:37.734642: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b57cfb900 of size 84646912 next 135
2021-03-03 10:45:37.734652: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b5cdb5500 of size 84646912 next 140
2021-03-03 10:45:37.734660: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b61e6f100 of size 84646912 next 143
2021-03-03 10:45:37.734670: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b66f28d00 of size 84646912 next 144
2021-03-03 10:45:37.734681: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b6bfe2900 of size 84646912 next 145
2021-03-03 10:45:37.734691: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b7109c500 of size 84646912 next 146
2021-03-03 10:45:37.734701: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b76156100 of size 84646912 next 147
2021-03-03 10:45:37.734712: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b7b20fd00 of size 84646912 next 149
2021-03-03 10:45:37.734741: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b802c9900 of size 84646912 next 153
2021-03-03 10:45:37.734753: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b85383500 of size 84646912 next 157
2021-03-03 10:45:37.734764: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b8a43d100 of size 84646912 next 158
2021-03-03 10:45:37.734774: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b8f4f6d00 of size 84646912 next 159
2021-03-03 10:45:37.734782: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b945b0900 of size 84646912 next 160
2021-03-03 10:45:37.734793: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at b9966a500 of size 129615616 next 161
2021-03-03 10:45:37.734803: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at ba1206c00 of size 129615616 next 162
2021-03-03 10:45:37.734814: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at ba8da3300 of size 129615616 next 164
2021-03-03 10:45:37.734824: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at bb093fa00 of size 129615616 next 165
2021-03-03 10:45:37.734834: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at bb84dc100 of size 84646912 next 169
2021-03-03 10:45:37.734845: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at bbd595d00 of size 84646912 next 171
2021-03-03 10:45:37.734855: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at bc264f900 of size 84646912 next 172
2021-03-03 10:45:37.734865: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at bc7709500 of size 84646912 next 173
2021-03-03 10:45:37.734875: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at bcc7c3100 of size 84646912 next 174
2021-03-03 10:45:37.734886: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at bd187cd00 of size 84646912 next 175
2021-03-03 10:45:37.734896: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at bd6936900 of size 84646912 next 179
2021-03-03 10:45:37.734907: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at bdb9f0500 of size 84646912 next 183
2021-03-03 10:45:37.734917: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at be0aaa100 of size 84646912 next 184
2021-03-03 10:45:37.734927: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at be5b63d00 of size 84646912 next 185
2021-03-03 10:45:37.734938: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at beac1d900 of size 84646912 next 186
2021-03-03 10:45:37.734945: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at befcd7500 of size 84646912 next 187
2021-03-03 10:45:37.734953: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at bf4d91100 of size 338587648 next 192
2021-03-03 10:45:37.734959: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at c09078100 of size 338587648 next 193
2021-03-03 10:45:37.734966: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at c1d35f100 of size 338587648 next 194
2021-03-03 10:45:37.734973: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at c31646100 of size 338587648 next 197
2021-03-03 10:45:37.734980: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at c4592d100 of size 338587648 next 198
2021-03-03 10:45:37.734986: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at c59c14100 of size 338587648 next 199
2021-03-03 10:45:37.734993: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at c6defb100 of size 338587648 next 200
2021-03-03 10:45:37.734999: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at c821e2100 of size 338587648 next 201
2021-03-03 10:45:37.735007: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at c964c9100 of size 338587648 next 202
2021-03-03 10:45:37.735013: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at caa7b0100 of size 338587648 next 203
2021-03-03 10:45:37.735020: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at cbea97100 of size 84646912 next 211
2021-03-03 10:45:37.735026: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at cc3b50d00 of size 84646912 next 212
2021-03-03 10:45:37.735033: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at cc8c0a900 of size 84646912 next 213
2021-03-03 10:45:37.735040: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at ccdcc4500 of size 84646912 next 214
2021-03-03 10:45:37.735047: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at cd2d7e100 of size 84646912 next 205
2021-03-03 10:45:37.735053: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at cd7e37d00 of size 338587648 next 206
2021-03-03 10:45:37.735059: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at cec11ed00 of size 338587648 next 207
2021-03-03 10:45:37.735065: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at d00405d00 of size 84646912 next 215
2021-03-03 10:45:37.735075: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at d054bf900 of size 84646912 next 219
2021-03-03 10:45:37.735086: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at d0a579500 of size 84646912 next 223
2021-03-03 10:45:37.735096: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at d0f633100 of size 84646912 next 224
2021-03-03 10:45:37.735107: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at d146ecd00 of size 84646912 next 225
2021-03-03 10:45:37.735117: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at d197a6900 of size 84646912 next 226
2021-03-03 10:45:37.735128: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at d1e860500 of size 84646912 next 227
2021-03-03 10:45:37.735139: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at d2391a100 of size 84646912 next 136
2021-03-03 10:45:37.735150: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at d289d3d00 of size 84646912 next 229
2021-03-03 10:45:37.735161: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at d2da8d900 of size 84646912 next 230
2021-03-03 10:45:37.735170: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at d32b47500 of size 84646912 next 231
2021-03-03 10:45:37.735181: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at d37c01100 of size 84646912 next 232
2021-03-03 10:45:37.735192: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at d3ccbad00 of size 84646912 next 233
2021-03-03 10:45:37.735202: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at d41d74900 of size 84646912 next 234
2021-03-03 10:45:37.735213: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at d46e2e500 of size 84646912 next 461
2021-03-03 10:45:37.735223: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at d4bee8100 of size 84646912 next 480
2021-03-03 10:45:37.735233: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at d50fa1d00 of size 423234560 next 479
2021-03-03 10:45:37.735244: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at d6a342900 of size 338587648 next 486
2021-03-03 10:45:37.735255: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at d7e629900 of size 338587648 next 489
2021-03-03 10:45:37.735271: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free  at d92910900 of size 170636288 next 18446744073709551615
2021-03-03 10:45:37.735282: I tensorflow/core/common_runtime/bfc_allocator.cc:1031]      Summary of in-use Chunks by size: 
2021-03-03 10:45:37.735302: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 173 Chunks of size 256 totalling 43.2KiB
2021-03-03 10:45:37.735315: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 79 Chunks of size 512 totalling 39.5KiB
2021-03-03 10:45:37.735326: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 5 Chunks of size 768 totalling 3.8KiB
2021-03-03 10:45:37.735338: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 1024 totalling 3.0KiB
2021-03-03 10:45:37.735349: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 1280 totalling 3.8KiB
2021-03-03 10:45:37.735360: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 6 Chunks of size 1792 totalling 10.5KiB
2021-03-03 10:45:37.735368: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 14 Chunks of size 2048 totalling 28.0KiB
2021-03-03 10:45:37.735379: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 8 Chunks of size 2304 totalling 18.0KiB
2021-03-03 10:45:37.735389: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 3328 totalling 3.2KiB
2021-03-03 10:45:37.735398: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 3584 totalling 3.5KiB
2021-03-03 10:45:37.735409: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 5 Chunks of size 4096 totalling 20.0KiB
2021-03-03 10:45:37.735420: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4352 totalling 4.2KiB
2021-03-03 10:45:37.735431: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 5376 totalling 5.2KiB
2021-03-03 10:45:37.735442: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 6656 totalling 6.5KiB
2021-03-03 10:45:37.735453: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 10240 totalling 20.0KiB
2021-03-03 10:45:37.735464: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 13568 totalling 39.8KiB
2021-03-03 10:45:37.735476: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 16128 totalling 15.8KiB
2021-03-03 10:45:37.735487: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 17408 totalling 17.0KiB
2021-03-03 10:45:37.735498: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 25088 totalling 24.5KiB
2021-03-03 10:45:37.735509: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 25600 totalling 75.0KiB
2021-03-03 10:45:37.735529: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 27136 totalling 26.5KiB
2021-03-03 10:45:37.735541: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 38656 totalling 37.8KiB
2021-03-03 10:45:37.735550: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 40704 totalling 39.8KiB
2021-03-03 10:45:37.735557: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 45568 totalling 44.5KiB
2021-03-03 10:45:37.735565: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 61440 totalling 120.0KiB
2021-03-03 10:45:37.735575: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 23 Chunks of size 65536 totalling 1.44MiB
2021-03-03 10:45:37.735582: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 71680 totalling 70.0KiB
2021-03-03 10:45:37.735590: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 84224 totalling 82.2KiB
2021-03-03 10:45:37.735602: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 85248 totalling 83.2KiB
2021-03-03 10:45:37.735613: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 86528 totalling 84.5KiB
2021-03-03 10:45:37.735621: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 95488 totalling 93.2KiB
2021-03-03 10:45:37.735629: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 99840 totalling 97.5KiB
2021-03-03 10:45:37.735636: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 107520 totalling 105.0KiB
2021-03-03 10:45:37.735644: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 175872 totalling 171.8KiB
2021-03-03 10:45:37.735653: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 192768 totalling 376.5KiB
2021-03-03 10:45:37.735660: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 226560 totalling 221.2KiB
2021-03-03 10:45:37.735668: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 14 Chunks of size 262144 totalling 3.50MiB
2021-03-03 10:45:37.735676: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 310784 totalling 303.5KiB
2021-03-03 10:45:37.735687: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 399360 totalling 390.0KiB
2021-03-03 10:45:37.735699: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 647936 totalling 632.8KiB
2021-03-03 10:45:37.735710: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 8 Chunks of size 661504 totalling 5.05MiB
2021-03-03 10:45:37.735721: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 8 Chunks of size 674816 totalling 5.15MiB
2021-03-03 10:45:37.735729: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 677120 totalling 1.94MiB
2021-03-03 10:45:37.735738: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 771072 totalling 753.0KiB
2021-03-03 10:45:37.735745: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 1054464 totalling 1.00MiB
2021-03-03 10:45:37.735752: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 5 Chunks of size 1059328 totalling 5.05MiB
2021-03-03 10:45:37.735766: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 1065984 totalling 1.02MiB
2021-03-03 10:45:37.735777: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 1156352 totalling 2.21MiB
2021-03-03 10:45:37.735788: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 1347584 totalling 1.29MiB
2021-03-03 10:45:37.735799: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 6 Chunks of size 1727488 totalling 9.88MiB
2021-03-03 10:45:37.735812: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 41000704 totalling 39.10MiB
2021-03-03 10:45:37.735823: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 5 Chunks of size 82919424 totalling 395.39MiB
2021-03-03 10:45:37.735838: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 54 Chunks of size 84646912 totalling 4.26GiB
2021-03-03 10:45:37.735849: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 103649280 totalling 98.85MiB
2021-03-03 10:45:37.735861: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 4 Chunks of size 129615616 totalling 494.44MiB
2021-03-03 10:45:37.735872: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 14 Chunks of size 338587648 totalling 4.41GiB
2021-03-03 10:45:37.735884: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 423234560 totalling 403.63MiB
2021-03-03 10:45:37.735896: I tensorflow/core/common_runtime/bfc_allocator.cc:1038] Sum Total of in-use chunks: 10.11GiB
2021-03-03 10:45:37.735906: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] total_region_allocated_bytes_: 11154668800 memory_limit_: 11154668928 available bytes: 128 curr_region_allocation_bytes_: 22309338112
2021-03-03 10:45:37.735925: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Stats: 
Limit:                     11154668928
InUse:                     10855662336
MaxInUse:                  10855663360
NumAllocs:                        1238
MaxAllocSize:                423234560
Reserved:                            0
PeakReserved:                        0
LargestFreeBlock:                    0

2021-03-03 10:45:37.735974: W tensorflow/core/common_runtime/bfc_allocator.cc:439] ***************************************************************************************************_
2021-03-03 10:45:37.736015: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at cwise_ops_common.h:134 : Resource exhausted: OOM when allocating tensor with shape[3374,49,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
  File "/opt/conda/bin/rasa", line 5, in <module>
    main()
  File "/home/fkr/rasa/rasa/__main__.py", line 116, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/home/fkr/rasa/rasa/cli/test.py", line 108, in run_core_test
    additional_arguments=vars(args),
  File "/home/fkr/rasa/rasa/test.py", line 135, in test_core
    _agent = Agent.load(unpacked_model)
  File "/home/fkr/rasa/rasa/core/agent.py", line 447, in load
    if core_model
  File "/home/fkr/rasa/rasa/core/policies/ensemble.py", line 374, in load
    policy = policy_cls.load(policy_path, **context)
  File "/home/fkr/rasa/rasa/core/policies/ted_policy.py", line 847, in load
    finetune_mode=should_finetune,
  File "/home/fkr/rasa/rasa/utils/tensorflow/models.py", line 404, in load
    False if finetune_mode else True
  File "/home/fkr/rasa/rasa/utils/tensorflow/models.py", line 244, in fit
    self.train_summary_writer,
  File "/home/fkr/rasa/rasa/utils/tensorflow/models.py", line 440, in _batch_loop
    call_model_function(batch_in)
  File "/home/fkr/rasa/rasa/utils/tensorflow/models.py", line 302, in train_on_batch
    prediction_gradients = tape.gradient(prediction_loss, self.trainable_variables)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/backprop.py", line 1073, in gradient
    unconnected_gradients=unconnected_gradients)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/imperative_grad.py", line 77, in imperative_grad
    compat.as_str(unconnected_gradients.value))
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/eager/backprop.py", line 162, in _gradient_function
    return grad_fn(mock_op, *out_grads)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/math_grad.py", line 1456, in _PowGrad
    return grad * y * math_ops.pow(x, y - 1), None
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py", line 1124, in binary_op_wrapper
    return func(x, y, name=name)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py", line 1456, in _mul_dispatch
    return multiply(x, y, name=name)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py", line 201, in wrapper
    return target(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py", line 508, in multiply
    return gen_math_ops.mul(x, y, name)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/gen_math_ops.py", line 6166, in mul
    _ops.raise_from_not_ok_status(e, name)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3374,49,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Mul]
twerkmeister commented 3 years ago

Well that hypothesis is dead then :D

koernerfelicia commented 3 years ago

😅 Do you have access to the rasa-research google cloud group? If not, it might make sense to ask Alan for access, then you could ssh into e2e-exp-fkr and try some things out

koernerfelicia commented 3 years ago

To me it seems that the number of stories is secondary here

Also, I think this isn't quite true. The tensor in question grows with the number of stories, as each unique bot utterance is added to action_names_or_texts, which influences the size of the tensor. Although I guess we have agreed that tensor is more a symptom than a cause... not sure

twerkmeister commented 3 years ago

To me it seems that the number of stories is secondary here

Also, I think this isn't quite true. The tensor in question grows with the number of stories, as each unique bot utterance is added to action_names_or_texts, which influences the size of the tensor. Although I guess we have agreed that tensor is more a symptom than a cause... not sure

That's good to know! Will think and investigate more

koernerfelicia commented 3 years ago

@twerkmeister I did some profiling for 100 dialogues, expecting that test would use more memory than train. As you can see, that's not the case.

test_MW100 train_MW100

I guess there may be something wrong with load/fit, because it seems that OOM is before the stories are featurized. test_MW500 The OOM crash occurs even if only one story is tested. test_MW500_1_story

I'd say this means we can close the issue and have someone from research dig into whether fit is doing something weird? What do you think?

twerkmeister commented 3 years ago

Hey @koernerfelicia 👋

This is fairly in line with my own findings. I was trying to replicate the issue and noticed that the memory consumption was not higher during test than during train for the Multiwoz subsets I tested with. Also I managed to run test on the 51 stories after training on the 390 stories, but I was using cpu only, and possibly there was no OOM due to swapping or sth. Just used 1 epoch during training to keep time manageable on the cpu.

The thing that was most striking looking at memory was that during training my memory consumption spiked significantly during the first dataset loading here in this line: https://github.com/RasaHQ/rasa/blob/2ef804000087197f070e1e6a6bd0b6f7a3925551/rasa/utils/tensorflow/models.py#L466

Had 16GB+ memory consumption for the 390 stories here. Whereas when I was using 40 stories it was just about 1.4GB. Might be worth investigating too.

Also, my reading of the error log is similar in the sense that the OOM already happens during the loading time. Would be interesting to check whether it also happens in other instances where you load the model, like for interactive training or so? In any case there could be something that happens before loading or during loading that isn't properly cleaned up and then clutters the memory

twerkmeister commented 3 years ago

@koernerfelicia did you use https://pypi.org/project/memory-profiler/ to do the profiling and plots? Does that tool actually capture gpu memory as well?

twerkmeister commented 3 years ago

@koernerfelicia So I looked into the eager execution, and it seems it is indeed active when loading the model:

As far as I can tell, this part just takes the default argument of the load function for should_finetune which is False. https://github.com/RasaHQ/rasa/blob/8b9238f9e0138c1e8594566d043e5d2e4f2e004d/rasa/core/policies/ted_policy.py#L857-L869

which then turns into eager mode here https://github.com/RasaHQ/rasa/blob/16aca813f5aaac5b3c1a65fa235c5b6ba91d995a/rasa/utils/tensorflow/models.py#L394-L406

which in turn seems to have some influence on the construction of the graph. Note that we are not seeing this debug "Building tensorflow {phase} graph..." output in your logs. So it seems eager mode was indeed activated. https://github.com/RasaHQ/rasa/blob/16aca813f5aaac5b3c1a65fa235c5b6ba91d995a/rasa/utils/tensorflow/models.py#L449-L470

I also noticed that the second code snippet, the call for model.fit, has changed completely in the main branch compared to 2.3.0 which you seem to be running on. Possibly that will already make a difference aswell!

koernerfelicia commented 3 years ago

Currently running train/test on 100 dialogues with nvidia-smi gpu query every second. I'll let you know what I find.

has changed completely in the main branch compared to 2.3.0 which you seem to be running on

That's my bad for not updating this issue, but I have somewhat periodically been pulling from main with no discernible change in this behaviour, so I don't think this has fixed itself in 2.4 :( . I pulled latest before starting the above experiment, so we will see soon whether this is a problem still.

koernerfelicia commented 3 years ago

@twerkmeister you were absolutely right about this:

I also noticed that the second code snippet, the call for model.fit, has changed completely in the main branch compared to 2.3.0 which you seem to be running on. Possibly that will already make a difference aswell!

test now runs successfully, so I think we can close this particular issue