search

harbecke / HexHex

AlphaGo Zero adaptation for Hex
GNU General Public License v3.0
20 stars 4 forks source link

data creation slows down dramatically #13

Closed harbecke closed 5 years ago

harbecke commented 5 years ago

I added logger.info(f"{len(board_states)}") after https://github.com/harbecke/hex/blob/c82fb86594e90d4d5acb4bf8b6a185bf3dc839e2/hex/creation/create_data.py#L34.

Results for batch_size=32 and samples_per_file = 40000 are:

2019-07-18 19:46:47,530 - INFO - === creating data from self play === 2019-07-18 19:46:49,915 - INFO - 3603 2019-07-18 19:46:52,931 - INFO - 3549 2019-07-18 19:46:58,871 - INFO - 3558 2019-07-18 19:47:06,903 - INFO - 3519 2019-07-18 19:47:16,569 - INFO - 3482 2019-07-18 19:47:28,136 - INFO - 3580 2019-07-18 19:48:16,708 - INFO - 3548 2019-07-18 19:49:23,159 - INFO - 3571 2019-07-18 19:50:48,866 - INFO - 3562 2019-07-18 19:52:26,660 - INFO - 3465 2019-07-18 19:54:13,377 - INFO - 3559 2019-07-18 19:56:16,024 - INFO - 3487

This is probably due to memory allocation for concatenation of tensors.

harbecke commented 5 years ago

Solved with db9e9357b03bc32e3698a7f461156d48e0fa833b:

new results:

2019-07-18 20:27:32,783 - INFO - === creating data from self play === 2019-07-18 20:27:35,066 - INFO - 3526 2019-07-18 20:27:37,384 - INFO - 3465 2019-07-18 20:27:39,698 - INFO - 3462 2019-07-18 20:27:42,002 - INFO - 3554 2019-07-18 20:27:44,350 - INFO - 3495 2019-07-18 20:27:46,674 - INFO - 3516 2019-07-18 20:27:49,009 - INFO - 3584 2019-07-18 20:27:51,298 - INFO - 3328 2019-07-18 20:27:53,611 - INFO - 3510 2019-07-18 20:27:55,925 - INFO - 3523 2019-07-18 20:27:58,244 - INFO - 3494 2019-07-18 20:28:00,563 - INFO - 3477