search

tkipf / gcn

Implementation of Graph Convolutional Networks in TensorFlow
MIT License
7.08k stars 2k forks source link

Out Of Memory Error when training with higher support or more kernels #17

Open schulter opened 6 years ago

schulter commented 6 years ago

Hi,

I tried to train your GCN network using the Chebychev polynomials. However, on my network and features (~10.000 nodes, ~90.000 edges, 24 feature dimensions), my graphics card seems to quickly run out of memory when using either higher suport polynomials (>3) or more filters (hidden1 > 20).

I am using a NVIDIA GeForce TITAN X card with 12 GB memory.

Do you think the reason for that lies in the implementation and can be tweaked or is it a natural limitation?

The exact error I get is:

totalMemory: 11.90GiB freeMemory: 11.76GiB
2017-11-15 14:14:23.596353: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:81:00.0, compute capability: 6.1)
2017-11-15 14:14:36.928227: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 649.86MiB.  Current allocation summary follows.
2017-11-15 14:14:36.928341: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (256):   Total Chunks: 39, Chunks in use: 37. 9.8KiB allocated for chunks. 9.2KiB in use in bin. 3.2KiB client-requested in use in bin.
2017-11-15 14:14:36.928366: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (512):   Total Chunks: 1, Chunks in use: 0. 512B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-11-15 14:14:36.928380: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (1024):  Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2017-11-15 14:14:36.928392: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (2048):  Total Chunks: 16, Chunks in use: 16. 40.0KiB allocated for chunks. 40.0KiB in use in bin. 37.5KiB client-requested in use in bin.
2017-11-15 14:14:36.928401: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (4096):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-11-15 14:14:36.928410: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-11-15 14:14:36.928419: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (16384):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-11-15 14:14:36.928429: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (32768):     Total Chunks: 3, Chunks in use: 2. 118.0KiB allocated for chunks. 78.5KiB in use in bin. 78.1KiB client-requested in use in bin.
2017-11-15 14:14:36.928440: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (65536):     Total Chunks: 6, Chunks in use: 5. 469.5KiB allocated for chunks. 391.2KiB in use in bin. 390.3KiB client-requested in use in bin.
2017-11-15 14:14:36.928459: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (131072):    Total Chunks: 2, Chunks in use: 1. 379.2KiB allocated for chunks. 156.2KiB in use in bin. 156.1KiB client-requested in use in bin.
2017-11-15 14:14:36.928470: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (262144):    Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-11-15 14:14:36.928480: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (524288):    Total Chunks: 13, Chunks in use: 11. 11.29MiB allocated for chunks. 9.47MiB in use in bin. 9.34MiB client-requested in use in bin.
2017-11-15 14:14:36.928490: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (1048576):   Total Chunks: 6, Chunks in use: 5. 8.69MiB allocated for chunks. 7.36MiB in use in bin. 6.66MiB client-requested in use in bin.
2017-11-15 14:14:36.928501: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (2097152):   Total Chunks: 1, Chunks in use: 1. 2.64MiB allocated for chunks. 2.64MiB in use in bin. 2.64MiB client-requested in use in bin.
2017-11-15 14:14:36.928512: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (4194304):   Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-11-15 14:14:36.928524: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (8388608):   Total Chunks: 4, Chunks in use: 4. 43.52MiB allocated for chunks. 43.52MiB in use in bin. 43.52MiB client-requested in use in bin.
2017-11-15 14:14:36.928534: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (16777216):  Total Chunks: 2, Chunks in use: 2. 42.51MiB allocated for chunks. 42.51MiB in use in bin. 42.51MiB client-requested in use in bin.
2017-11-15 14:14:36.928547: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (33554432):  Total Chunks: 4, Chunks in use: 4. 207.96MiB allocated for chunks. 207.96MiB in use in bin. 207.96MiB client-requested in use in bin.
2017-11-15 14:14:36.928559: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (67108864):  Total Chunks: 1, Chunks in use: 1. 103.98MiB allocated for chunks. 103.98MiB in use in bin. 103.98MiB client-requested in use in bin.
2017-11-15 14:14:36.928572: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (134217728):     Total Chunks: 1, Chunks in use: 1. 181.08MiB allocated for chunks. 181.08MiB in use in bin. 181.08MiB client-requested in use in bin.
2017-11-15 14:14:36.928584: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (268435456):     Total Chunks: 6, Chunks in use: 5. 10.58GiB allocated for chunks. 10.24GiB in use in bin. 6.19GiB client-requested in use in bin.
2017-11-15 14:14:36.928595: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin for 649.86MiB was 256.00MiB, Chunk State: 
2017-11-15 14:14:36.928613: I tensorflow/core/common_runtime/bfc_allocator.cc:649]   Size: 344.70MiB | Requested Size: 0B | in_use: 0, prev:   Size: 16.52MiB | Requested Size: 16.52MiB | in_use: 1, next:   Size: 362.17MiB | Requested Size: 362.17MiB | in_use: 1
2017-11-15 14:14:36.928630: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d600000 of size 1280
2017-11-15 14:14:36.928643: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d600500 of size 256
2017-11-15 14:14:36.928649: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d600600 of size 256
2017-11-15 14:14:36.928654: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d600700 of size 256
2017-11-15 14:14:36.928660: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d600800 of size 256
2017-11-15 14:14:36.928665: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d600900 of size 256
2017-11-15 14:14:36.928671: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d600a00 of size 256
2017-11-15 14:14:36.928677: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d600b00 of size 256
2017-11-15 14:14:36.928682: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d600c00 of size 256
2017-11-15 14:14:36.928687: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d600d00 of size 256
2017-11-15 14:14:36.928692: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d600e00 of size 2560
2017-11-15 14:14:36.928698: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d601800 of size 2560
2017-11-15 14:14:36.928704: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d602200 of size 2560
2017-11-15 14:14:36.928722: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d602c00 of size 256
2017-11-15 14:14:36.928727: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d602d00 of size 2560
2017-11-15 14:14:36.928733: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d603700 of size 2560
2017-11-15 14:14:36.928739: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d604100 of size 2560
2017-11-15 14:14:36.928745: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d604b00 of size 2560
2017-11-15 14:14:36.928751: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d605500 of size 256
2017-11-15 14:14:36.928757: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d605600 of size 256
2017-11-15 14:14:36.928763: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d605700 of size 256
2017-11-15 14:14:36.928787: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d605800 of size 256
2017-11-15 14:14:36.928792: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d605900 of size 256
2017-11-15 14:14:36.928798: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d605a00 of size 256
2017-11-15 14:14:36.928803: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d605b00 of size 256
2017-11-15 14:14:36.928809: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d605d00 of size 256
2017-11-15 14:14:36.928815: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d605e00 of size 256
2017-11-15 14:14:36.928821: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d605f00 of size 256
2017-11-15 14:14:36.928827: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606000 of size 256
2017-11-15 14:14:36.928832: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606100 of size 256
2017-11-15 14:14:36.928839: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606200 of size 256
2017-11-15 14:14:36.928845: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606300 of size 256
2017-11-15 14:14:36.928850: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606400 of size 256
2017-11-15 14:14:36.928855: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606500 of size 256
2017-11-15 14:14:36.928860: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606600 of size 256
2017-11-15 14:14:36.928865: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606700 of size 256
2017-11-15 14:14:36.928871: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606800 of size 256
2017-11-15 14:14:36.928876: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606900 of size 256
2017-11-15 14:14:36.928881: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606a00 of size 256
2017-11-15 14:14:36.928887: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606b00 of size 2560
2017-11-15 14:14:36.928893: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d607500 of size 2560
2017-11-15 14:14:36.928898: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d607f00 of size 2560
2017-11-15 14:14:36.928903: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d608900 of size 2560
2017-11-15 14:14:36.928908: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d609300 of size 2560
2017-11-15 14:14:36.928914: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d609d00 of size 2560
2017-11-15 14:14:36.928919: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d60a700 of size 256
2017-11-15 14:14:36.928925: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d60a800 of size 256
2017-11-15 14:14:36.928931: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d60a900 of size 2560
2017-11-15 14:14:36.928938: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d60b300 of size 2560
2017-11-15 14:14:36.928944: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d60bd00 of size 2560
2017-11-15 14:14:36.928950: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d60c700 of size 40192
2017-11-15 14:14:36.928957: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d616400 of size 160000
2017-11-15 14:14:36.928963: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d63d500 of size 692992
2017-11-15 14:14:36.928968: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d6e6800 of size 2771200
2017-11-15 14:14:36.928975: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d98b100 of size 27257344
2017-11-15 14:14:36.928981: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f389b00 of size 109028608
2017-11-15 14:14:36.928988: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10215b84000 of size 189882112
2017-11-15 14:14:36.928994: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10221099f00 of size 759528448
2017-11-15 14:14:36.929000: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024e5d0900 of size 80128
2017-11-15 14:14:36.929006: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024e5ee000 of size 256
2017-11-15 14:14:36.929012: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024e5ee100 of size 256
2017-11-15 14:14:36.929018: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024e5ee400 of size 256
2017-11-15 14:14:36.929033: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024e5ee500 of size 80128
2017-11-15 14:14:36.929039: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024e601e00 of size 80128
2017-11-15 14:14:36.929052: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024e615700 of size 80128
2017-11-15 14:14:36.929059: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024e63c900 of size 590848
2017-11-15 14:14:36.929066: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024e6ccd00 of size 54514432
2017-11-15 14:14:36.929072: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10251aca000 of size 54514432
2017-11-15 14:14:36.929078: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10254ec7300 of size 80128
2017-11-15 14:14:36.929086: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10254edac00 of size 40192
2017-11-15 14:14:36.929091: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10254ee4a00 of size 256
2017-11-15 14:14:36.929097: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10254f1c700 of size 54514432
2017-11-15 14:14:36.929103: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10258319a00 of size 54514432
2017-11-15 14:14:36.929109: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1025b716d00 of size 999168
2017-11-15 14:14:36.929126: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1025b80ac00 of size 17319936
2017-11-15 14:14:36.929146: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10272142b00 of size 379764224
2017-11-15 14:14:36.929157: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10288b6e900 of size 1385728
2017-11-15 14:14:36.929162: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10288cc0e00 of size 1385728
2017-11-15 14:14:36.929167: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10288e13300 of size 1385728
2017-11-15 14:14:36.929173: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102890b7d00 of size 1825536
2017-11-15 14:14:36.929178: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10289275800 of size 999168
2017-11-15 14:14:36.929183: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10289369700 of size 1739520
2017-11-15 14:14:36.929188: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10289512200 of size 912896
2017-11-15 14:14:36.929193: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102895f1000 of size 912896
2017-11-15 14:14:36.929197: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102896cfe00 of size 912896
2017-11-15 14:14:36.929202: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102897aec00 of size 912896
2017-11-15 14:14:36.929207: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1028988da00 of size 379764224
2017-11-15 14:14:36.929212: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102a02b9800 of size 379764224
2017-11-15 14:14:36.929217: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102b6ce5600 of size 11408640
2017-11-15 14:14:36.929222: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102b77c6b00 of size 11408640
2017-11-15 14:14:36.929226: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102b82a8000 of size 11408640
2017-11-15 14:14:36.929231: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102b8d89500 of size 11408640
2017-11-15 14:14:36.929235: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102b995e900 of size 999168
2017-11-15 14:14:36.929240: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102b9a52800 of size 999168
2017-11-15 14:14:36.929244: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102b9b46700 of size 999168
2017-11-15 14:14:36.929250: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102b9c3a600 of size 9100342016
2017-11-15 14:14:36.929255: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x1020d605c00 of size 256
2017-11-15 14:14:36.929260: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x1024e4f1b00 of size 912896
2017-11-15 14:14:36.929265: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x1024e5e4200 of size 40448
2017-11-15 14:14:36.929270: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x1024e5ee200 of size 512
2017-11-15 14:14:36.929274: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x1024e629000 of size 80128
2017-11-15 14:14:36.929278: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x10254ee4900 of size 256
2017-11-15 14:14:36.929283: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x10254ee4b00 of size 228352
2017-11-15 14:14:36.929288: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x1025c88f400 of size 361445120
2017-11-15 14:14:36.929292: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x10288f65800 of size 1385728
2017-11-15 14:14:36.929297: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x102b986aa00 of size 999168
2017-11-15 14:14:36.929301: I tensorflow/core/common_runtime/bfc_allocator.cc:676]      Summary of in-use Chunks by size: 
2017-11-15 14:14:36.929310: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 37 Chunks of size 256 totalling 9.2KiB
2017-11-15 14:14:36.929316: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 1280 totalling 1.2KiB
2017-11-15 14:14:36.929322: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 16 Chunks of size 2560 totalling 40.0KiB
2017-11-15 14:14:36.929328: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 2 Chunks of size 40192 totalling 78.5KiB
2017-11-15 14:14:36.929334: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 5 Chunks of size 80128 totalling 391.2KiB
2017-11-15 14:14:36.929339: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 160000 totalling 156.2KiB
2017-11-15 14:14:36.929345: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 590848 totalling 577.0KiB
2017-11-15 14:14:36.929350: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 692992 totalling 676.8KiB
2017-11-15 14:14:36.929356: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 4 Chunks of size 912896 totalling 3.48MiB
2017-11-15 14:14:36.929361: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 5 Chunks of size 999168 totalling 4.76MiB
2017-11-15 14:14:36.929366: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 3 Chunks of size 1385728 totalling 3.96MiB
2017-11-15 14:14:36.929371: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 1739520 totalling 1.66MiB
2017-11-15 14:14:36.929377: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 1825536 totalling 1.74MiB
2017-11-15 14:14:36.929382: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 2771200 totalling 2.64MiB
2017-11-15 14:14:36.929387: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 4 Chunks of size 11408640 totalling 43.52MiB
2017-11-15 14:14:36.929393: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 17319936 totalling 16.52MiB
2017-11-15 14:14:36.929399: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 27257344 totalling 25.99MiB
2017-11-15 14:14:36.929404: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 4 Chunks of size 54514432 totalling 207.96MiB
2017-11-15 14:14:36.929410: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 109028608 totalling 103.98MiB
2017-11-15 14:14:36.929415: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 189882112 totalling 181.08MiB
2017-11-15 14:14:36.929421: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 3 Chunks of size 379764224 totalling 1.06GiB
2017-11-15 14:14:36.929426: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 759528448 totalling 724.34MiB
2017-11-15 14:14:36.929432: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 9100342016 totalling 8.47GiB
2017-11-15 14:14:36.929437: I tensorflow/core/common_runtime/bfc_allocator.cc:683] Sum Total of in-use chunks: 10.83GiB
2017-11-15 14:14:36.929451: I tensorflow/core/common_runtime/bfc_allocator.cc:685] Stats: 
Limit:                 11992553882
InUse:                 11627460864
MaxInUse:              11991371008
NumAllocs:                     131
MaxAllocSize:           9100342016

2017-11-15 14:14:36.929470: W tensorflow/core/common_runtime/bfc_allocator.cc:277] ************__**************************************************xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2017-11-15 14:14:36.929507: W tensorflow/core/framework/op_kernel.cc:1192] Resource exhausted: OOM when allocating tensor with shape[6814273,25]
2017-11-15 14:14:46.930598: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 362.17MiB.  Current allocation summary follows.
2017-11-15 14:14:46.930676: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (256):   Total Chunks: 39, Chunks in use: 37. 9.8KiB allocated for chunks. 9.2KiB in use in bin. 3.2KiB client-requested in use in bin.
2017-11-15 14:14:46.930686: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (512):   Total Chunks: 1, Chunks in use: 0. 512B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-11-15 14:14:46.930694: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (1024):  Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2017-11-15 14:14:46.930702: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (2048):  Total Chunks: 16, Chunks in use: 16. 40.0KiB allocated for chunks. 40.0KiB in use in bin. 37.5KiB client-requested in use in bin.
2017-11-15 14:14:46.930709: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (4096):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-11-15 14:14:46.930715: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-11-15 14:14:46.930721: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (16384):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-11-15 14:14:46.930729: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (32768):     Total Chunks: 3, Chunks in use: 2. 118.0KiB allocated for chunks. 78.5KiB in use in bin. 78.1KiB client-requested in use in bin.
2017-11-15 14:14:46.930737: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (65536):     Total Chunks: 10, Chunks in use: 10. 782.5KiB allocated for chunks. 782.5KiB in use in bin. 780.5KiB client-requested in use in bin.
2017-11-15 14:14:46.930744: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (131072):    Total Chunks: 2, Chunks in use: 1. 301.0KiB allocated for chunks. 156.2KiB in use in bin. 156.1KiB client-requested in use in bin.
2017-11-15 14:14:46.930751: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (262144):    Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-11-15 14:14:46.930758: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (524288):    Total Chunks: 11, Chunks in use: 10. 9.16MiB allocated for chunks. 8.52MiB in use in bin. 8.39MiB client-requested in use in bin.
2017-11-15 14:14:46.930765: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (1048576):   Total Chunks: 7, Chunks in use: 6. 10.59MiB allocated for chunks. 9.27MiB in use in bin. 7.61MiB client-requested in use in bin.
2017-11-15 14:14:46.930772: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (2097152):   Total Chunks: 1, Chunks in use: 1. 2.64MiB allocated for chunks. 2.64MiB in use in bin. 2.64MiB client-requested in use in bin.
2017-11-15 14:14:46.930779: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (4194304):   Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-11-15 14:14:46.930786: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (8388608):   Total Chunks: 4, Chunks in use: 4. 43.52MiB allocated for chunks. 43.52MiB in use in bin. 43.52MiB client-requested in use in bin.
2017-11-15 14:14:46.930794: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (16777216):  Total Chunks: 2, Chunks in use: 2. 42.51MiB allocated for chunks. 42.51MiB in use in bin. 42.51MiB client-requested in use in bin.
2017-11-15 14:14:46.930802: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (33554432):  Total Chunks: 4, Chunks in use: 3. 207.96MiB allocated for chunks. 155.97MiB in use in bin. 155.97MiB client-requested in use in bin.
2017-11-15 14:14:46.930809: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (67108864):  Total Chunks: 1, Chunks in use: 1. 103.98MiB allocated for chunks. 103.98MiB in use in bin. 103.98MiB client-requested in use in bin.
2017-11-15 14:14:46.930817: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (134217728):     Total Chunks: 1, Chunks in use: 1. 181.08MiB allocated for chunks. 181.08MiB in use in bin. 181.08MiB client-requested in use in bin.
2017-11-15 14:14:46.930825: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (268435456):     Total Chunks: 6, Chunks in use: 5. 10.58GiB allocated for chunks. 10.24GiB in use in bin. 6.19GiB client-requested in use in bin.
2017-11-15 14:14:46.930832: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin for 362.17MiB was 256.00MiB, Chunk State: 
2017-11-15 14:14:46.930848: I tensorflow/core/common_runtime/bfc_allocator.cc:649]   Size: 344.70MiB | Requested Size: 0B | in_use: 0, prev:   Size: 16.52MiB | Requested Size: 16.52MiB | in_use: 1, next:   Size: 362.17MiB | Requested Size: 362.17MiB | in_use: 1
2017-11-15 14:14:46.930860: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d600000 of size 1280
2017-11-15 14:14:46.930865: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d600500 of size 256
2017-11-15 14:14:46.930870: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d600600 of size 256
2017-11-15 14:14:46.930874: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d600700 of size 256
2017-11-15 14:14:46.930879: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d600800 of size 256
2017-11-15 14:14:46.930883: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d600900 of size 256
2017-11-15 14:14:46.930888: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d600a00 of size 256
2017-11-15 14:14:46.930893: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d600b00 of size 256
2017-11-15 14:14:46.930897: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d600c00 of size 256
2017-11-15 14:14:46.930902: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d600d00 of size 256
2017-11-15 14:14:46.930906: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d600e00 of size 2560
2017-11-15 14:14:46.930911: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d601800 of size 2560
2017-11-15 14:14:46.930916: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d602200 of size 2560
2017-11-15 14:14:46.930920: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d602c00 of size 256
2017-11-15 14:14:46.930925: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d602d00 of size 2560
2017-11-15 14:14:46.930930: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d603700 of size 2560
2017-11-15 14:14:46.930935: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d604100 of size 2560
2017-11-15 14:14:46.930939: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d604b00 of size 2560
2017-11-15 14:14:46.930944: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d605500 of size 256
2017-11-15 14:14:46.930948: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d605600 of size 256
2017-11-15 14:14:46.930953: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d605700 of size 256
2017-11-15 14:14:46.930958: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d605800 of size 256
2017-11-15 14:14:46.930962: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d605900 of size 256
2017-11-15 14:14:46.930967: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d605a00 of size 256
2017-11-15 14:14:46.930972: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d605b00 of size 256
2017-11-15 14:14:46.930976: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d605d00 of size 256
2017-11-15 14:14:46.930981: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d605e00 of size 256
2017-11-15 14:14:46.930986: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d605f00 of size 256
2017-11-15 14:14:46.930990: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606000 of size 256
2017-11-15 14:14:46.930995: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606100 of size 256
2017-11-15 14:14:46.930999: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606200 of size 256
2017-11-15 14:14:46.931004: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606300 of size 256
2017-11-15 14:14:46.931009: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606400 of size 256
2017-11-15 14:14:46.931013: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606500 of size 256
2017-11-15 14:14:46.931018: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606600 of size 256
2017-11-15 14:14:46.931023: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606700 of size 256
2017-11-15 14:14:46.931028: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606800 of size 256
2017-11-15 14:14:46.931033: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606900 of size 256
2017-11-15 14:14:46.931038: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606a00 of size 256
2017-11-15 14:14:46.931042: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d606b00 of size 2560
2017-11-15 14:14:46.931047: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d607500 of size 2560
2017-11-15 14:14:46.931051: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d607f00 of size 2560
2017-11-15 14:14:46.931056: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d608900 of size 2560
2017-11-15 14:14:46.931061: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d609300 of size 2560
2017-11-15 14:14:46.931066: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d609d00 of size 2560
2017-11-15 14:14:46.931070: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d60a700 of size 256
2017-11-15 14:14:46.931075: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d60a800 of size 256
2017-11-15 14:14:46.931079: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d60a900 of size 2560
2017-11-15 14:14:46.931084: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d60b300 of size 2560
2017-11-15 14:14:46.931089: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d60bd00 of size 2560
2017-11-15 14:14:46.931094: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d60c700 of size 40192
2017-11-15 14:14:46.931099: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d616400 of size 160000
2017-11-15 14:14:46.931104: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d63d500 of size 692992
2017-11-15 14:14:46.931140: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d6e6800 of size 2771200
2017-11-15 14:14:46.931147: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d98b100 of size 27257344
2017-11-15 14:14:46.931154: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f389b00 of size 109028608
2017-11-15 14:14:46.931169: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10215b84000 of size 189882112
2017-11-15 14:14:46.931175: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10221099f00 of size 759528448
2017-11-15 14:14:46.931191: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024e4f1b00 of size 80128
2017-11-15 14:14:46.931196: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024e505400 of size 80128
2017-11-15 14:14:46.931202: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024e518d00 of size 80128
2017-11-15 14:14:46.931208: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024e5d0900 of size 80128
2017-11-15 14:14:46.931214: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024e5ee000 of size 256
2017-11-15 14:14:46.931220: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024e5ee100 of size 256
2017-11-15 14:14:46.931225: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024e5ee400 of size 256
2017-11-15 14:14:46.931230: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024e5ee500 of size 80128
2017-11-15 14:14:46.931237: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024e601e00 of size 80128
2017-11-15 14:14:46.931243: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024e615700 of size 80128
2017-11-15 14:14:46.931248: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024e629000 of size 80128
2017-11-15 14:14:46.931254: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024e63c900 of size 590848
2017-11-15 14:14:46.931270: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1024e6ccd00 of size 54514432
2017-11-15 14:14:46.931275: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10254ec7300 of size 80128
2017-11-15 14:14:46.931281: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10254edac00 of size 40192
2017-11-15 14:14:46.931286: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10254ee4a00 of size 256
2017-11-15 14:14:46.931292: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10254ee4b00 of size 80128
2017-11-15 14:14:46.931298: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10254f1c700 of size 54514432
2017-11-15 14:14:46.931305: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10258319a00 of size 54514432
2017-11-15 14:14:46.931310: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1025b716d00 of size 999168
2017-11-15 14:14:46.931316: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1025b80ac00 of size 17319936
2017-11-15 14:14:46.931323: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10272142b00 of size 379764224
2017-11-15 14:14:46.931330: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10288b6e900 of size 1385728
2017-11-15 14:14:46.931343: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10288cc0e00 of size 1385728
2017-11-15 14:14:46.931349: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10288f65800 of size 1385728
2017-11-15 14:14:46.931355: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102890b7d00 of size 1825536
2017-11-15 14:14:46.931361: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10289275800 of size 999168
2017-11-15 14:14:46.931367: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10289369700 of size 1739520
2017-11-15 14:14:46.931374: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10289512200 of size 912896
2017-11-15 14:14:46.931381: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102895f1000 of size 912896
2017-11-15 14:14:46.931388: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102896cfe00 of size 912896
2017-11-15 14:14:46.931397: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102897aec00 of size 912896
2017-11-15 14:14:46.931402: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1028988da00 of size 379764224
2017-11-15 14:14:46.931409: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102a02b9800 of size 379764224
2017-11-15 14:14:46.931418: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102b6ce5600 of size 11408640
2017-11-15 14:14:46.931424: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102b77c6b00 of size 11408640
2017-11-15 14:14:46.931432: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102b82a8000 of size 11408640
2017-11-15 14:14:46.931439: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102b8d89500 of size 11408640
2017-11-15 14:14:46.931451: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102b986aa00 of size 1998336
2017-11-15 14:14:46.931458: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102b9a52800 of size 999168
2017-11-15 14:14:46.931464: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102b9b46700 of size 999168
2017-11-15 14:14:46.931470: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102b9c3a600 of size 9100342016
2017-11-15 14:14:46.931480: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x1020d605c00 of size 256
2017-11-15 14:14:46.931489: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x1024e52c600 of size 672512
2017-11-15 14:14:46.931495: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x1024e5e4200 of size 40448
2017-11-15 14:14:46.931501: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x1024e5ee200 of size 512
2017-11-15 14:14:46.931506: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x10251aca000 of size 54514432
2017-11-15 14:14:46.931515: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x10254ee4900 of size 256
2017-11-15 14:14:46.931520: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x10254ef8400 of size 148224
2017-11-15 14:14:46.931526: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x1025c88f400 of size 361445120
2017-11-15 14:14:46.931532: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x10288e13300 of size 1385728
2017-11-15 14:14:46.931538: I tensorflow/core/common_runtime/bfc_allocator.cc:676]      Summary of in-use Chunks by size: 
2017-11-15 14:14:46.931548: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 37 Chunks of size 256 totalling 9.2KiB
2017-11-15 14:14:46.931559: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 1280 totalling 1.2KiB
2017-11-15 14:14:46.931567: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 16 Chunks of size 2560 totalling 40.0KiB
2017-11-15 14:14:46.931577: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 2 Chunks of size 40192 totalling 78.5KiB
2017-11-15 14:14:46.931585: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 10 Chunks of size 80128 totalling 782.5KiB
2017-11-15 14:14:46.931591: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 160000 totalling 156.2KiB
2017-11-15 14:14:46.931598: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 590848 totalling 577.0KiB
2017-11-15 14:14:46.931605: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 692992 totalling 676.8KiB
2017-11-15 14:14:46.931612: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 4 Chunks of size 912896 totalling 3.48MiB
2017-11-15 14:14:46.931619: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 4 Chunks of size 999168 totalling 3.81MiB
2017-11-15 14:14:46.931628: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 3 Chunks of size 1385728 totalling 3.96MiB
2017-11-15 14:14:46.931638: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 1739520 totalling 1.66MiB
2017-11-15 14:14:46.931644: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 1825536 totalling 1.74MiB
2017-11-15 14:14:46.931650: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 1998336 totalling 1.91MiB
2017-11-15 14:14:46.931657: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 2771200 totalling 2.64MiB
2017-11-15 14:14:46.931664: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 4 Chunks of size 11408640 totalling 43.52MiB
2017-11-15 14:14:46.931671: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 17319936 totalling 16.52MiB
2017-11-15 14:14:46.931679: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 27257344 totalling 25.99MiB
2017-11-15 14:14:46.931689: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 3 Chunks of size 54514432 totalling 155.97MiB
2017-11-15 14:14:46.931696: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 109028608 totalling 103.98MiB
2017-11-15 14:14:46.931703: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 189882112 totalling 181.08MiB
2017-11-15 14:14:46.931710: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 3 Chunks of size 379764224 totalling 1.06GiB
2017-11-15 14:14:46.931719: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 759528448 totalling 724.34MiB
2017-11-15 14:14:46.931728: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 9100342016 totalling 8.47GiB
2017-11-15 14:14:46.931736: I tensorflow/core/common_runtime/bfc_allocator.cc:683] Sum Total of in-use chunks: 10.78GiB
2017-11-15 14:14:46.931747: I tensorflow/core/common_runtime/bfc_allocator.cc:685] Stats: 
Limit:                 11992553882
InUse:                 11574346240
MaxInUse:              11991371008
NumAllocs:                     146
MaxAllocSize:           9100342016

2017-11-15 14:14:46.931778: W tensorflow/core/common_runtime/bfc_allocator.cc:277] ************__**************************************************xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2017-11-15 14:14:46.931803: W tensorflow/core/framework/op_kernel.cc:1192] Resource exhausted: OOM when allocating tensor with shape[47470525,2]
2017-11-15 14:14:46.932584: W tensorflow/core/framework/op_kernel.cc:1192] Resource exhausted: OOM when allocating tensor with shape[6814273,25]
     [[Node: gradients/graphconvolution_1/SparseTensorDenseMatMul_5/SparseTensorDenseMatMul_grad/Gather_1 = Gather[Tindices=DT_INT64, Tparams=DT_FLOAT, validate_indices=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graphconvolution_1/SparseTensorDenseMatMul_4/SparseTensorDenseMatMul, gradients/graphconvolution_1/SparseTensorDenseMatMul_5/SparseTensorDenseMatMul_grad/strided_slice_1/_43)]]
2017-11-15 14:14:46.932653: W tensorflow/core/framework/op_kernel.cc:1192] Resource exhausted: OOM when allocating tensor with shape[6814273,25]
     [[Node: gradients/graphconvolution_1/SparseTensorDenseMatMul_5/SparseTensorDenseMatMul_grad/Gather_1 = Gather[Tindices=DT_INT64, Tparams=DT_FLOAT, validate_indices=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graphconvolution_1/SparseTensorDenseMatMul_4/SparseTensorDenseMatMul, gradients/graphconvolution_1/SparseTensorDenseMatMul_5/SparseTensorDenseMatMul_grad/strided_slice_1/_43)]]
2017-11-15 14:14:46.932694: W tensorflow/core/framework/op_kernel.cc:1192] Resource exhausted: OOM when allocating tensor with shape[6814273,25]
     [[Node: gradients/graphconvolution_1/SparseTensorDenseMatMul_5/SparseTensorDenseMatMul_grad/Gather_1 = Gather[Tindices=DT_INT64, Tparams=DT_FLOAT, validate_indices=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graphconvolution_1/SparseTensorDenseMatMul_4/SparseTensorDenseMatMul, gradients/graphconvolution_1/SparseTensorDenseMatMul_5/SparseTensorDenseMatMul_grad/strided_slice_1/_43)]]
2017-11-15 14:14:46.932814: W tensorflow/core/framework/op_kernel.cc:1192] Resource exhausted: OOM when allocating tensor with shape[6814273,25]
     [[Node: gradients/graphconvolution_1/SparseTensorDenseMatMul_5/SparseTensorDenseMatMul_grad/Gather_1 = Gather[Tindices=DT_INT64, Tparams=DT_FLOAT, validate_indices=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graphconvolution_1/SparseTensorDenseMatMul_4/SparseTensorDenseMatMul, gradients/graphconvolution_1/SparseTensorDenseMatMul_5/SparseTensorDenseMatMul_grad/strided_slice_1/_43)]]
2017-11-15 14:14:46.932867: W tensorflow/core/framework/op_kernel.cc:1192] Resource exhausted: OOM when allocating tensor with shape[6814273,25]
     [[Node: gradients/graphconvolution_1/SparseTensorDenseMatMul_5/SparseTensorDenseMatMul_grad/Gather_1 = Gather[Tindices=DT_INT64, Tparams=DT_FLOAT, validate_indices=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graphconvolution_1/SparseTensorDenseMatMul_4/SparseTensorDenseMatMul, gradients/graphconvolution_1/SparseTensorDenseMatMul_5/SparseTensorDenseMatMul_grad/strided_slice_1/_43)]]
2017-11-15 14:14:46.932955: W tensorflow/core/framework/op_kernel.cc:1192] Resource exhausted: OOM when allocating tensor with shape[6814273,25]
     [[Node: gradients/graphconvolution_1/SparseTensorDenseMatMul_5/SparseTensorDenseMatMul_grad/Gather_1 = Gather[Tindices=DT_INT64, Tparams=DT_FLOAT, validate_indices=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graphconvolution_1/SparseTensorDenseMatMul_4/SparseTensorDenseMatMul, gradients/graphconvolution_1/SparseTensorDenseMatMul_5/SparseTensorDenseMatMul_grad/strided_slice_1/_43)]]
Traceback (most recent call last):
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1323, in _do_call
    return fn(*args)
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1302, in _run_fn
    status, run_metadata)
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[6814273,25]
     [[Node: gradients/graphconvolution_1/SparseTensorDenseMatMul_5/SparseTensorDenseMatMul_grad/Gather_1 = Gather[Tindices=DT_INT64, Tparams=DT_FLOAT, validate_indices=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graphconvolution_1/SparseTensorDenseMatMul_4/SparseTensorDenseMatMul, gradients/graphconvolution_1/SparseTensorDenseMatMul_5/SparseTensorDenseMatMul_grad/strided_slice_1/_43)]]
     [[Node: Mean_3/_115 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1165_Mean_3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train_gcn.py", line 134, in <module>
    outs = sess.run([model.opt_op, model.loss, model.accuracy], feed_dict=feed_dict)
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 889, in run
    run_metadata_ptr)
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1120, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run
    options, run_metadata)
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[6814273,25]
     [[Node: gradients/graphconvolution_1/SparseTensorDenseMatMul_5/SparseTensorDenseMatMul_grad/Gather_1 = Gather[Tindices=DT_INT64, Tparams=DT_FLOAT, validate_indices=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graphconvolution_1/SparseTensorDenseMatMul_4/SparseTensorDenseMatMul, gradients/graphconvolution_1/SparseTensorDenseMatMul_5/SparseTensorDenseMatMul_grad/strided_slice_1/_43)]]
     [[Node: Mean_3/_115 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1165_Mean_3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Caused by op 'gradients/graphconvolution_1/SparseTensorDenseMatMul_5/SparseTensorDenseMatMul_grad/Gather_1', defined at:
  File "train_gcn.py", line 112, in <module>
    model = GCN(placeholders, input_dim=features[2][1], logging=True)
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/gcn-1.0-py3.5.egg/gcn/models.py", line 144, in __init__
    self.build()
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/gcn-1.0-py3.5.egg/gcn/models.py", line 58, in build
    self.opt_op = self.optimizer.minimize(self.loss)
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/optimizer.py", line 343, in minimize
    grad_loss=grad_loss)
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/training/optimizer.py", line 414, in compute_gradients
    colocate_gradients_with_ops=colocate_gradients_with_ops)
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/gradients_impl.py", line 581, in gradients
    grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/gradients_impl.py", line 353, in _MaybeCompile
    return grad_fn()  # Exit early
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/gradients_impl.py", line 581, in <lambda>
    grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/sparse_grad.py", line 170, in _SparseTensorDenseMatMulGrad
    cols if not adj_a else rows)
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py", line 2486, in gather
    params, indices, validate_indices=validate_indices, name=name)
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1834, in gather
    validate_indices=validate_indices, name=name)
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
    op_def=op_def)
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1470, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

...which was originally created as op 'graphconvolution_1/SparseTensorDenseMatMul_5/SparseTensorDenseMatMul', defined at:
  File "train_gcn.py", line 112, in <module>
    model = GCN(placeholders, input_dim=features[2][1], logging=True)
[elided 0 identical lines from previous traceback]
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/gcn-1.0-py3.5.egg/gcn/models.py", line 144, in __init__
    self.build()
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/gcn-1.0-py3.5.egg/gcn/models.py", line 46, in build
    hidden = layer(self.activations[-1])
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/gcn-1.0-py3.5.egg/gcn/layers.py", line 75, in __call__
    outputs = self._call(inputs)
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/gcn-1.0-py3.5.egg/gcn/layers.py", line 180, in _call
    support = dot(self.support[i], pre_sup, sparse=True)
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/gcn-1.0-py3.5.egg/gcn/layers.py", line 33, in dot
    res = tf.sparse_tensor_dense_matmul(x, y)
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/sparse_ops.py", line 1719, in sparse_tensor_dense_matmul
    adjoint_b=adjoint_b)
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/gen_sparse_ops.py", line 1818, in _sparse_tensor_dense_mat_mul
    name=name)
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
    op_def=op_def)
  File "/home/sasse/anaconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1470, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[6814273,25]
     [[Node: gradients/graphconvolution_1/SparseTensorDenseMatMul_5/SparseTensorDenseMatMul_grad/Gather_1 = Gather[Tindices=DT_INT64, Tparams=DT_FLOAT, validate_indices=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graphconvolution_1/SparseTensorDenseMatMul_4/SparseTensorDenseMatMul, gradients/graphconvolution_1/SparseTensorDenseMatMul_5/SparseTensorDenseMatMul_grad/strided_slice_1/_43)]]
     [[Node: Mean_3/_115 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1165_Mean_3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Thanks,

Roman

tkipf commented 6 years ago

Hi Roman, thanks for reporting this.

It is certainly possible to optimize the memory efficiency of the model (e.g. computing higher-order terms from the Chebyshev polynomial on the fly and discarding them after they are used in a certain operation), yet these things are quite cumbersome to implement. Another option is to avoid storing the whole graph in memory by doing some form of subsampling.

Currently, there is no published paper about such a subsampling scheme with respect to polynomial filters (such as in Defferard's work), but the GCN model can be very simply optimized by doing either random subsampling (https://arxiv.org/abs/1706.02216) or using some form of importance sampling (https://openreview.net/forum?id=rytstxWAW, https://openreview.net/forum?id=rylejExC-).

Edit: Maybe the PyTorch version of the model is a bit more memory-efficient? https://github.com/tkipf/pygcn

schulter commented 6 years ago

Hi, thanks for letting me know. I'll try one or the other form of that. Since mini-batches in node classification are not really an option, I'll have to find other ways here.

Thanks for the links and the great package,

Roman

pinkfloyd06 commented 6 years ago

Hi @tkipf,

Talking about the Chebyshev polynomial order K. What is the intuition behind the way we set K-order ?

In the paper entitled "Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering" it is shown that filters based on Chebyshev are localized . "Strictly localized filters.

Enhancing [4], the proposed spectral filters are provable to be strictly localized in a ball of radius K, i.e. K hops from the central vertex

In this paper, the graph convnet is test on MNIST dataset. Which is a particular form of graph (Grid). The number of nodes is fixed, number of edges for each node is apriori knwon 8 and Chebyshev polynomial order is set to K=25.

For an arbitrary graph structure : variable size of number nodes, variable number of edges per node. How can we choose the value K ?

According to the definition given : Enhancing [4], the proposed spectral filters are provable to be strictly localized in a ball of radius K, i.e. K hops from the central vertex

The K may be intrinsically related to the neighborhood of the central vertex of a given graph .

If i understand well : the less K is, the more filters are localized . AM l wrong ?

Let me explain the idea : If l have a graph of 20 nodes and the max number of edges per nodes is 3. Then setting K=25 results a filters which are not localized. It's like a convolution filter on a grid. Mnist example: rather to set filter of 33 on image of 2828 , when filters are not localized the have the dimension of an image 28*28.

Thank you for your clarification

tkipf commented 6 years ago

If i understand well : the less K is, the more filters are localized . AM l wrong ?

Yes, exactly. Filters that are polynomials in the adjacency matrix or the graph Laplacian (like Chebychev filters) are K-localized where K is the order of the polynomial. That means that information is gathered from up to K hops away. Similarly you can increase the receptive field size of a model by stacking more layers. With filter order of K=1 you can stack L layers and have a receptive field size of L. Most recent works design filters of order K=1 and achieve larger receptive field size by simply stacking layers (if you stack deep, then make sure to use gating or residual connections).

m-saksena commented 5 years ago

Hi, I was wondering how you assign the tensors to the GPU in the first place? I'm not sure how this works with a placeholder dict / feed dict. I have access to a CPU and a GPU.

tkipf commented 5 years ago

See here: https://www.tensorflow.org/guide/using_gpu

On Thu 14. Feb 2019 at 16:47 Malika Saksena notifications@github.com wrote:

Hi, I was wondering how you assign the tensors to the GPU in the first place? I'm not sure how this works with a placeholder dict / feed dict. I have access to a CPU and a GPU.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/tkipf/gcn/issues/17#issuecomment-463677041, or mute the thread https://github.com/notifications/unsubscribe-auth/AHAcYHXuSA7ILIPgaEi5G9xxpTgGYdnCks5vNYUEgaJpZM4Qe7yG .

ICDI0906 commented 5 years ago

how does the parameter K reflect in the code, it means the ’support‘?