search

Skuldur / Classical-Piano-Composer

MIT License
602 stars 318 forks source link

loss becomes nan #18

Closed GeorgeShimakawa closed 5 years ago

GeorgeShimakawa commented 5 years ago

When I switched cpu based Keras to PaidML based Keras, loss values became "nan" while running Istm.py. It was not the case before. Would you guess what the reason is ?

GeorgeShimakawa commented 5 years ago

Pardon for my ambiguous question. I changed backend to an AMD GPU "PlaidML" by adding a line (os.environ["KERAS_BACKEND"] = "plaidml.keras.backend"). Platform is iMac Late2015, OSX-Mojave, Python 3.6.5 After parsing about 60 midi files, first loss was indicated as 'nan' as follows: . . . Parsing midi_songs/sonat-5.mid Parsing midi_songs/352.mid Parsing midi_songs/384.mid Parsing midi_songs/5112.mid Parsing midi_songs/sonat-4.mid INFO:plaidml:Opening device "metal_amd_radeon_r9_m390.0" Epoch 1/200 INFO:plaidml:Analyzing Ops: 2104 of 29742 operations complete INFO:plaidml:Analyzing Ops: 5776 of 29742 operations complete INFO:plaidml:Analyzing Ops: 9603 of 29742 operations complete INFO:plaidml:Analyzing Ops: 14197 of 29742 operations complete INFO:plaidml:Analyzing Ops: 19025 of 29742 operations complete INFO:plaidml:Analyzing Ops: 24197 of 29742 operations complete INFO:plaidml:Analyzing Ops: 27148 of 29742 operations complete 720/43019 [..............................] - ETA: 1:39:10 - loss: nan

Even when I continued further the the loss continues to be nan.

GeorgeShimakawa commented 5 years ago

OpenCL version is 1.2 and the gpu is AMD Radeon R9 M390. CPU is Intel Core i5. PlaidML is 0.3.5. Memory is 16GB.