search

sogou / SogouMRCToolkit

This toolkit was designed for the fast and efficient development of modern machine comprehension models, including both published models and original prototypes.
Apache License 2.0
746 stars 162 forks source link

Shuffle buffer filled #34

Open berton820 opened 5 years ago

berton820 commented 5 years ago

command: python run_bert_squadv2.py GPU: P40 CPU memory: 126G

log: 2019-07-28 01:28:56.654470: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N 2019-07-28 01:28:56.866822: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21139 MB memory) -> physical GPU (device: 0, name: Tesla P40, pci bus id: 0000:05:00.0, compute capability: 6.1) 2019-07-28 01:29:01.610923: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0 2019-07-28 01:29:01.611087: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-07-28 01:29:01.611125: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 2019-07-28 01:29:01.611143: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N 2019-07-28 01:29:01.792284: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21139 MB memory) -> physical GPU (device: 0, name: Tesla P40, pci bus id: 0000:05:00.0, compute capability: 6.1) 2019-07-28 01:29:04.056203: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0 2019-07-28 01:29:04.056312: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-07-28 01:29:04.056346: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 2019-07-28 01:29:04.056365: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N 2019-07-28 01:29:04.057344: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21139 MB memory) -> physical GPU (device: 0, name: Tesla P40, pci bus id: 0000:05:00.0, compute capability: 6.1) 2019-07-28 01:29:49.138936: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 7971 of 130497 2019-07-28 01:29:59.139110: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 15820 of 130497 2019-07-28 01:30:09.139348: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 23963 of 130497 2019-07-28 01:30:19.138907: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 31963 of 130497 2019-07-28 01:30:29.139424: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 39830 of 130497 2019-07-28 01:30:39.139672: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 48245 of 130497 2019-07-28 01:30:49.138921: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 56364 of 130497 2019-07-28 01:30:59.139422: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 64376 of 130497 2019-07-28 01:31:09.139234: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 72092 of 130497 2019-07-28 01:31:19.139490: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 79574 of 130497 2019-07-28 01:31:29.139453: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 87148 of 130497 2019-07-28 01:31:39.139663: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 94795 of 130497 2019-07-28 01:31:49.139455: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 102515 of 130497 2019-07-28 01:31:59.140305: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 110001 of 130497 2019-07-28 01:32:09.138877: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 117752 of 130497 2019-07-28 01:32:19.139864: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:98] Filling up shuffle buffer (this may take a while): 125678 of 130497 2019-07-28 01:32:25.347410: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:136] Shuffle buffer filled. ~

zww847204326 commented 4 years ago

I have the same trouble, do you solve it?

berton820 commented 4 years ago

I have the same trouble, do you solve it?

no