search

brightics / studio

Component based analytics studio on the web browser
Other
181 stars 85 forks source link

[BUG] Brightics Studio 의 DL 튜토리얼 에러 입니다. #815

Closed ogosengi closed 3 years ago

ogosengi commented 3 years ago

Brightics Studio DL 에서 다음의 튜토리얼을 따라하면 에러가 생기는데 해결방법을 알고 싶습니다.

튜토리얼 주소: https://www.brightics.ai/kr/docs/ai/dl3.7/tutorials/101_Modeler_Basic?type=insight

Brightics Studio 상 로그내용 첨부합니다.

c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\framework\dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\framework\dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\framework\dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\framework\dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\framework\dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\framework\dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) [2021-01-11 15:12:12,223] INFO BrighticsDL-1610345532223: Training job is starting... INFO:tensorflow:Using config: {'_model_dir': 'c:\brightics-studio\brightics-studio\brightics-deeplearning\workspace\models\BrighticsDL-1610345522468', '_tf_random_seed': None, '_save_summary_steps': 50, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': device_count { key: "GPU" value: 1 } gpu_options { allow_growth: true } allow_soft_placement: true , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x000001FB5FD21128>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} [2021-01-11 15:12:13,652] INFO BrighticsDL-1610345532223: Running Process(30460) for Training Job with Local Engine INFO:tensorflow:Calling model_fn. WARNING:tensorflow:From c:\brightics-studio\brightics-studio\brightics-server\functions\python\brightics\deeplearning\model_function\image\classification\network\slim\nets\resnet_v1.py:244: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead INFO:tensorflow:Create BrighticsTrainingHook. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Graph was finalized. 2021-01-11 15:12:21.137578: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 2021-01-11 15:12:21.287175: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1405] Found device 0 with properties: name: GeForce GTX 750 Ti major: 5 minor: 0 memoryClockRate(GHz): 1.1105 pciBusID: 0000:01:00.0 totalMemory: 2.00GiB freeMemory: 1.64GiB 2021-01-11 15:12:21.287256: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1484] Adding visible gpu devices: 0 2021-01-11 15:12:22.007285: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-01-11 15:12:22.007345: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:971] 0 2021-01-11 15:12:22.007367: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:984] 0: N 2021-01-11 15:12:22.007505: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1401 MB memory) -> physical GPU (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0, compute capability: 5.0) INFO:tensorflow:Running local_init_op. INFO:tensorflow:Done running local_init_op. INFO:tensorflow:Saving checkpoints for 0 into c:\brightics-studio\brightics-studio\brightics-deeplearning\workspace\models\BrighticsDL-1610345522468\model.ckpt. 2021-01-11 15:12:35.580388: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 450.19MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2021-01-11 15:12:37.014939: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 660.09MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2021-01-11 15:12:37.015043: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 299.63MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2021-01-11 15:12:47.727518: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 144.0KiB. Current allocation summary follows. 2021-01-11 15:12:47.727640: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (256): Total Chunks: 233, Chunks in use: 233. 58.3KiB allocated for chunks. 58.3KiB in use in bin. 24.8KiB client-requested in use in bin. 2021-01-11 15:12:47.727683: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (512): Total Chunks: 106, Chunks in use: 106. 53.0KiB allocated for chunks. 53.0KiB in use in bin. 53.0KiB client-requested in use in bin. 2021-01-11 15:12:47.727721: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (1024): Total Chunks: 213, Chunks in use: 213. 213.8KiB allocated for chunks. 213.8KiB in use in bin. 213.0KiB client-requested in use in bin. 2021-01-11 15:12:47.727746: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (2048): Total Chunks: 143, Chunks in use: 143. 286.0KiB allocated for chunks. 286.0KiB in use in bin. 286.0KiB client-requested in use in bin. 2021-01-11 15:12:47.727781: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (4096): Total Chunks: 58, Chunks in use: 58. 232.0KiB allocated for chunks. 232.0KiB in use in bin. 232.0KiB client-requested in use in bin. 2021-01-11 15:12:47.727805: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (8192): Total Chunks: 34, Chunks in use: 34. 272.0KiB allocated for chunks. 272.0KiB in use in bin. 272.0KiB client-requested in use in bin. 2021-01-11 15:12:47.727916: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (16384): Total Chunks: 5, Chunks in use: 4. 95.3KiB allocated for chunks. 64.0KiB in use in bin. 64.0KiB client-requested in use in bin. 2021-01-11 15:12:47.727942: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (32768): Total Chunks: 2, Chunks in use: 1. 73.5KiB allocated for chunks. 36.8KiB in use in bin. 36.8KiB client-requested in use in bin. 2021-01-11 15:12:47.727980: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (65536): Total Chunks: 13, Chunks in use: 12. 832.0KiB allocated for chunks. 768.0KiB in use in bin. 768.0KiB client-requested in use in bin. 2021-01-11 15:12:47.728018: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (131072): Total Chunks: 8, Chunks in use: 8. 1.09MiB allocated for chunks. 1.09MiB in use in bin. 1.09MiB client-requested in use in bin. 2021-01-11 15:12:47.728062: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (262144): Total Chunks: 14, Chunks in use: 14. 3.50MiB allocated for chunks. 3.50MiB in use in bin. 3.50MiB client-requested in use in bin. 2021-01-11 15:12:47.728115: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (524288): Total Chunks: 12, Chunks in use: 12. 6.50MiB allocated for chunks. 6.50MiB in use in bin. 6.50MiB client-requested in use in bin. 2021-01-11 15:12:47.728148: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (1048576): Total Chunks: 22, Chunks in use: 22. 23.40MiB allocated for chunks. 23.40MiB in use in bin. 22.00MiB client-requested in use in bin. 2021-01-11 15:12:47.728204: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (2097152): Total Chunks: 16, Chunks in use: 16. 35.00MiB allocated for chunks. 35.00MiB in use in bin. 35.00MiB client-requested in use in bin. 2021-01-11 15:12:47.728248: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (4194304): Total Chunks: 10, Chunks in use: 10. 40.00MiB allocated for chunks. 40.00MiB in use in bin. 40.00MiB client-requested in use in bin. 2021-01-11 15:12:47.728282: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (8388608): Total Chunks: 7, Chunks in use: 7. 63.00MiB allocated for chunks. 63.00MiB in use in bin. 61.00MiB client-requested in use in bin. 2021-01-11 15:12:47.728326: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (16777216): Total Chunks: 1, Chunks in use: 1. 16.00MiB allocated for chunks. 16.00MiB in use in bin. 9.00MiB client-requested in use in bin. 2021-01-11 15:12:47.728359: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (33554432): Total Chunks: 3, Chunks in use: 3. 147.00MiB allocated for chunks. 147.00MiB in use in bin. 147.00MiB client-requested in use in bin. 2021-01-11 15:12:47.728403: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (67108864): Total Chunks: 2, Chunks in use: 2. 148.65MiB allocated for chunks. 148.65MiB in use in bin. 87.75MiB client-requested in use in bin. 2021-01-11 15:12:47.728448: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (134217728): Total Chunks: 3, Chunks in use: 3. 648.00MiB allocated for chunks. 648.00MiB in use in bin. 588.00MiB client-requested in use in bin. 2021-01-11 15:12:47.728503: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (268435456): Total Chunks: 1, Chunks in use: 1. 266.96MiB allocated for chunks. 266.96MiB in use in bin. 196.00MiB client-requested in use in bin. 2021-01-11 15:12:47.728543: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:646] Bin for 144.0KiB was 128.0KiB, Chunk State: 2021-01-11 15:12:47.728586: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000701880000 of size 1280 2021-01-11 15:12:47.728630: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000701880500 of size 256 2021-01-11 15:12:47.728670: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000701880600 of size 256 2021-01-11 15:12:47.728709: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000701880700 of size 256 2021-01-11 15:12:47.728731: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000701880800 of size 256 2021-01-11 15:12:47.728753: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000701880900 of size 256 2021-01-11 15:12:47.728784: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000701880A00 of size 256 2021-01-11 15:12:47.728814: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000701880B00 of size 256 중략 - 같은 내용이 반복 됩니다. 2021-01-11 15:12:47.748341: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000742C30000 of size 205520896 2021-01-11 15:12:47.748360: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000074F030000 of size 51380224 2021-01-11 15:12:47.748380: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000752130000 of size 51380224 2021-01-11 15:12:47.748403: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000755230000 of size 88191744 2021-01-11 15:12:47.748422: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:671] Summary of in-use Chunks by size: 2021-01-11 15:12:47.748445: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 233 Chunks of size 256 totalling 58.3KiB 2021-01-11 15:12:47.748466: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 106 Chunks of size 512 totalling 53.0KiB 2021-01-11 15:12:47.748486: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 211 Chunks of size 1024 totalling 211.0KiB 2021-01-11 15:12:47.748507: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 1280 totalling 1.3KiB 2021-01-11 15:12:47.748527: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 1536 totalling 1.5KiB 2021-01-11 15:12:47.748547: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 143 Chunks of size 2048 totalling 286.0KiB 2021-01-11 15:12:47.748567: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 58 Chunks of size 4096 totalling 232.0KiB 2021-01-11 15:12:47.748587: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 34 Chunks of size 8192 totalling 272.0KiB 2021-01-11 15:12:47.748608: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 4 Chunks of size 16384 totalling 64.0KiB 2021-01-11 15:12:47.748628: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 37632 totalling 36.8KiB 2021-01-11 15:12:47.748648: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 12 Chunks of size 65536 totalling 768.0KiB 2021-01-11 15:12:47.748668: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 2 Chunks of size 131072 totalling 256.0KiB 2021-01-11 15:12:47.748688: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 6 Chunks of size 147456 totalling 864.0KiB 2021-01-11 15:12:47.748709: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 14 Chunks of size 262144 totalling 3.50MiB 2021-01-11 15:12:47.748729: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 4 Chunks of size 524288 totalling 2.00MiB 2021-01-11 15:12:47.748749: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 8 Chunks of size 589824 totalling 4.50MiB 2021-01-11 15:12:47.748771: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 20 Chunks of size 1048576 totalling 20.00MiB 2021-01-11 15:12:47.748793: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 1571072 totalling 1.50MiB 2021-01-11 15:12:47.748813: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 1990912 totalling 1.90MiB 2021-01-11 15:12:47.748833: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 4 Chunks of size 2097152 totalling 8.00MiB 2021-01-11 15:12:47.748854: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 12 Chunks of size 2359296 totalling 27.00MiB 2021-01-11 15:12:47.748874: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 10 Chunks of size 4194304 totalling 40.00MiB 2021-01-11 15:12:47.748894: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 2 Chunks of size 8388608 totalling 16.00MiB 2021-01-11 15:12:47.748914: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 4 Chunks of size 9437184 totalling 36.00MiB 2021-01-11 15:12:47.748934: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 11534336 totalling 11.00MiB 2021-01-11 15:12:47.748955: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 16777216 totalling 16.00MiB 2021-01-11 15:12:47.748975: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 3 Chunks of size 51380224 totalling 147.00MiB 2021-01-11 15:12:47.748995: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 67677696 totalling 64.54MiB 2021-01-11 15:12:47.749015: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 88191744 totalling 84.11MiB 2021-01-11 15:12:47.749035: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 2 Chunks of size 205520896 totalling 392.00MiB 2021-01-11 15:12:47.749056: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 268434432 totalling 256.00MiB 2021-01-11 15:12:47.749076: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 279932160 totalling 266.96MiB 2021-01-11 15:12:47.749096: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:678] Sum Total of in-use chunks: 1.37GiB 2021-01-11 15:12:47.749119: I T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:680] Stats: Limit: 1469231924 InUse: 1469096704 MaxInUse: 1469096704 NumAllocs: 929 MaxAllocSize: 279932160

2021-01-11 15:12:47.749173: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:279] **xxxx****xxxxx*****xx 2021-01-11 15:12:47.749204: W T:\src\github\tensorflow\tensorflow\core\framework\op_kernel.cc:1275] OP_REQUIRES failed at conv_ops.cc:682 : Resource exhausted: OOM when allocating tensor with shape[64,64,3,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

[Job Failed] Exception occurs from TrainingJob with Local Engine : OOM when allocating tensor with shape[64,64,3,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/Conv2D = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/Relu, resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/weights/read/_2861)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[Node: gradients/AddN_43/_3131 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_3821_gradients/AddN_43", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op 'resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/Conv2D', defined at: File "", line 1, in File "c:\brightics-studio\brightics-studio\brightics-server\functions\python\brightics\deeplearning\runner\local_job_runner.py", line 242, in internal_run classifier.train(input_fn=self.train_input_fn, steps=iteration, hooks=train_hooks) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\estimator\estimator.py", line 376, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1145, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1170, in _train_model_default features, labels, model_fn_lib.ModeKeys.TRAIN, self.config) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1133, in _call_model_fn model_fn_results = self._model_fn(features=features, kwargs) File "c:\brightics-studio\brightics-studio\brightics-deeplearning\workspace\models\BrighticsDL-1610345522468\resources\model_function.py", line 50, in brightics_model_fn returned = model_function(features, labels, mode, params) File "c:\brightics-studio\brightics-studio\brightics-deeplearning\workspace\models\BrighticsDL-1610345522468\resources\model_function.py", line 11, in model_function return ImageClassificationModelFunction(json.loads(json_param)).model_function(features, labels, mode, params) File "c:\brightics-studio\brightics-studio\brightics-server\functions\python\brightics\deeplearning\model_function\image\classification\function.py", line 58, in model_function logits, end_points = model.run(inputs=inputs, is_training=is_training) File "c:\brightics-studio\brightics-studio\brightics-server\functions\python\brightics\deeplearning\model_function\image\classification\network\base.py", line 71, in run net, end_points = self.slim_model(inputs=inputs_reshape, is_training=is_training, model_params) File "c:\brightics-studio\brightics-studio\brightics-server\functions\python\brightics\deeplearning\model_function\image\classification\network\resnet.py", line 22, in slim_model return self.resnet_model_function(inputs=inputs, is_training=is_training, kwargs) File "c:\brightics-studio\brightics-studio\brightics-server\functions\python\brightics\deeplearning\model_function\image\classification\network\slim\nets\resnet_v1.py", line 309, in resnet_v1_50 reuse=reuse, scope=scope) File "c:\brightics-studio\brightics-studio\brightics-server\functions\python\brightics\deeplearning\model_function\image\classification\network\slim\nets\resnet_v1.py", line 237, in resnet_v1 store_non_strided_activations) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 183, in func_with_args return func(args, current_args) File "c:\brightics-studio\brightics-studio\brightics-server\functions\python\brightics\deeplearning\model_function\image\classification\network\slim\nets\resnet_utils.py", line 199, in stack_blocks_dense net = block.unit_fn(net, rate=1, unit) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 183, in func_with_args return func(args, current_args) File "c:\brightics-studio\brightics-studio\brightics-server\functions\python\brightics\deeplearning\model_function\image\classification\network\slim\nets\resnet_v1.py", line 126, in bottleneck rate=rate, scope='conv2') File "c:\brightics-studio\brightics-studio\brightics-server\functions\python\brightics\deeplearning\model_function\image\classification\network\slim\nets\resnet_utils.py", line 113, in conv2d_same padding='SAME', scope=scope) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 183, in func_with_args return func(*args, current_args) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1154, in convolution2d conv_dims=2) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 183, in func_with_args return func(*args, *current_args) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1057, in convolution outputs = layer.apply(inputs) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 805, in apply return self.call(inputs, args, kwargs) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\layers\base.py", line 362, in call outputs = super(Layer, self).call(inputs, *args, kwargs) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 736, in call outputs = self.call(inputs, *args, *kwargs) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\keras\layers\convolutional.py", line 186, in call outputs = self._convolution_op(inputs, self.kernel) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 868, in call return self.conv_op(inp, filter) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 520, in call return self.call(inp, filter) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 204, in call name=self.name) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 1042, in conv2d data_format=data_format, dilations=dilations, name=name) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\util\deprecation.py", line 454, in new_func return func(args, kwargs) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\framework\ops.py", line 3155, in create_op op_def=op_def) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\framework\ops.py", line 1717, in init self._traceback = tf_stack.extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[64,64,3,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/Conv2D = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/Relu, resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/weights/read/_2861)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[Node: gradients/AddN_43/_3131 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_3821_gradients/AddN_43", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Traceback (most recent call last): File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\client\session.py", line 1278, in _do_call return fn(*args) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\client\session.py", line 1263, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[64,64,3,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/Conv2D = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/Relu, resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/weights/read/_2861)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[Node: gradients/AddN_43/_3131 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_3821_gradients/AddN_43", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "c:\brightics-studio\brightics-studio\brightics-server\functions\python\brightics\deeplearning\runner\local_job_runner.py", line 242, in internal_run classifier.train(input_fn=self.train_input_fn, steps=iteration, hooks=train_hooks) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\estimator\estimator.py", line 376, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1145, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1173, in _train_model_default saving_listeners) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1451, in _train_with_estimatorspec , loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss]) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\training\monitored_session.py", line 583, in run run_metadata=run_metadata) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1059, in run run_metadata=run_metadata) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1150, in run raise six.reraise(original_exc_info) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\six.py", line 696, in reraise raise value File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1135, in run return self._sess.run(args, *kwargs) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1207, in run run_metadata=run_metadata) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\training\monitored_session.py", line 987, in run return self._sess.run(args, **kwargs) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\client\session.py", line 877, in run run_metadata_ptr) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\client\session.py", line 1100, in _run feed_dict_tensor, options, run_metadata) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\client\session.py", line 1272, in _do_run run_metadata) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\client\session.py", line 1291, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[64,64,3,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/Conv2D = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/Relu, resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/weights/read/_2861)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[Node: gradients/AddN_43/_3131 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_3821_gradients/AddN_43", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op 'resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/Conv2D', defined at: File "", line 1, in File "c:\brightics-studio\brightics-studio\brightics-server\functions\python\brightics\deeplearning\runner\local_job_runner.py", line 242, in internal_run classifier.train(input_fn=self.train_input_fn, steps=iteration, hooks=train_hooks) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\estimator\estimator.py", line 376, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1145, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1170, in _train_model_default features, labels, model_fn_lib.ModeKeys.TRAIN, self.config) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1133, in _call_model_fn model_fn_results = self._model_fn(features=features, kwargs) File "c:\brightics-studio\brightics-studio\brightics-deeplearning\workspace\models\BrighticsDL-1610345522468\resources\model_function.py", line 50, in brightics_model_fn returned = model_function(features, labels, mode, params) File "c:\brightics-studio\brightics-studio\brightics-deeplearning\workspace\models\BrighticsDL-1610345522468\resources\model_function.py", line 11, in model_function return ImageClassificationModelFunction(json.loads(json_param)).model_function(features, labels, mode, params) File "c:\brightics-studio\brightics-studio\brightics-server\functions\python\brightics\deeplearning\model_function\image\classification\function.py", line 58, in model_function logits, end_points = model.run(inputs=inputs, is_training=is_training) File "c:\brightics-studio\brightics-studio\brightics-server\functions\python\brightics\deeplearning\model_function\image\classification\network\base.py", line 71, in run net, end_points = self.slim_model(inputs=inputs_reshape, is_training=is_training, model_params) File "c:\brightics-studio\brightics-studio\brightics-server\functions\python\brightics\deeplearning\model_function\image\classification\network\resnet.py", line 22, in slim_model return self.resnet_model_function(inputs=inputs, is_training=is_training, kwargs) File "c:\brightics-studio\brightics-studio\brightics-server\functions\python\brightics\deeplearning\model_function\image\classification\network\slim\nets\resnet_v1.py", line 309, in resnet_v1_50 reuse=reuse, scope=scope) File "c:\brightics-studio\brightics-studio\brightics-server\functions\python\brightics\deeplearning\model_function\image\classification\network\slim\nets\resnet_v1.py", line 237, in resnet_v1 store_non_strided_activations) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 183, in func_with_args return func(args, current_args) File "c:\brightics-studio\brightics-studio\brightics-server\functions\python\brightics\deeplearning\model_function\image\classification\network\slim\nets\resnet_utils.py", line 199, in stack_blocks_dense net = block.unit_fn(net, rate=1, unit) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 183, in func_with_args return func(args, current_args) File "c:\brightics-studio\brightics-studio\brightics-server\functions\python\brightics\deeplearning\model_function\image\classification\network\slim\nets\resnet_v1.py", line 126, in bottleneck rate=rate, scope='conv2') File "c:\brightics-studio\brightics-studio\brightics-server\functions\python\brightics\deeplearning\model_function\image\classification\network\slim\nets\resnet_utils.py", line 113, in conv2d_same padding='SAME', scope=scope) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 183, in func_with_args return func(*args, current_args) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1154, in convolution2d conv_dims=2) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 183, in func_with_args return func(*args, *current_args) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1057, in convolution outputs = layer.apply(inputs) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 805, in apply return self.call(inputs, args, kwargs) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\layers\base.py", line 362, in call outputs = super(Layer, self).call(inputs, *args, kwargs) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 736, in call outputs = self.call(inputs, *args, *kwargs) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\keras\layers\convolutional.py", line 186, in call outputs = self._convolution_op(inputs, self.kernel) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 868, in call return self.conv_op(inp, filter) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 520, in call return self.call(inp, filter) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 204, in call name=self.name) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 1042, in conv2d data_format=data_format, dilations=dilations, name=name) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\util\deprecation.py", line 454, in new_func return func(args, kwargs) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\framework\ops.py", line 3155, in create_op op_def=op_def) File "c:\brightics-studio\brightics-studio\lib\python_dl\lib\site-packages\tensorflow\python\framework\ops.py", line 1717, in init self._traceback = tf_stack.extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[64,64,3,3] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/Conv2D = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/Relu, resnet_v1_50/block1/unit_1/bottleneck_v1/conv2/weights/read/_2861)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[Node: gradients/AddN_43/_3131 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_3821_gradients/AddN_43", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

github-actions[bot] commented 3 years ago

Awesome! Thanks for taking the time to open an issue. We will have a look and answer as soon as we can.' first issue

shovsj commented 3 years ago

gpu memory가 넉넉하지 않은 듯 합니다. 생성한 두 input function의 batch size를 조금 줄여보시면 될 것 같습니다.

ogosengi commented 3 years ago

gpu memory가 넉넉하지 않은 듯 합니다. 생성한 두 input function의 batch size를 조금 줄여보시면 될 것 같습니다.

감사합니다. learning 이 되고 있는 듯 합니다. train 파일과 Test 파일 모두 batch size 가 64였는데요. 32로 변경 -> 실패 16으로 변경 -> 실패 8로 변경 -> leaning 진행중 입니다.

shovsj commented 3 years ago

감사합니다. 다른 분들도 사례를 공유할 수 있도록 gpu memory정보도 있으면 좋을 것 같습니다. GeForce GTX 750 Ti를 검색하니 2Gi인걸로 나오는데 맞나요?

krazyeom commented 3 years ago

감사합니다. 다른 분들도 사례를 공유할 수 있도록 gpu memory정보도 있으면 좋을 것 같습니다. GeForce GTX 750 Ti를 검색하니 2Gi인걸로 나오는데 맞나요?

TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1401 MB memory) -> physical GPU (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0, compute capability: 5.0)

로그상에 1401MB로 찍히는거 보니, 2Gi 모델이 맞나보네요.

ogosengi commented 3 years ago

네 맞습니다.