search

cindyxinyiwang / deep-latent-sequence-model

Pytorch implementation of "A Probabilistic Formulation of Unsupervised Text Style Transfer" by He. et. al. at ICLR 2020
163 stars 26 forks source link

RuntimeError: CUDA error: device-side assert triggered #2

Closed lzzhaha closed 4 years ago

lzzhaha commented 4 years ago

Hi, when I tried to train the yelp model, the following error occurred:

/pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [128,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [129,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [130,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [131,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [132,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [133,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [134,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [135,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [136,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [137,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [138,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [139,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [140,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [141,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [142,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [143,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [144,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [145,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [146,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [147,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [148,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [149,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [150,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [151,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [152,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [153,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [154,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [155,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [156,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [157,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [158,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [159,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [96,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [97,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [98,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [99,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [100,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [101,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [102,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [103,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [104,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [105,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [106,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [107,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [108,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [109,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [110,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [111,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [112,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [113,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [114,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [115,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [116,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [117,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [118,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [119,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [120,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [121,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [122,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [123,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [124,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [125,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [126,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [127,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [0,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [1,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [2,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [3,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [4,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [5,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [6,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [7,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [8,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [9,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [10,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [11,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [12,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [13,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [14,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [15,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [16,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [17,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [18,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [19,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [20,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [21,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [22,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [23,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [24,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [25,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [26,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [27,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [28,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [29,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [30,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /pytorch/aten/src/THC/THCTensorRandom.cuh:193: void sampleMultinomialOnce(long , long, int, T , T , int, int) [with T = float, AccT = float]: block: [14,0,0], thread: [31,0,0] Assertion THCNumerics<T>::ge(val, zero) failed. /home/zlinai/anaconda3/envs/py3.7-env/lib/python3.7/site-packages/torch/nn/modules/rnn.py:54: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.3 and num_layers=1 "num_layers={}".format(dropout, num_layers)) /home/zlinai/anaconda3/envs/py3.7-env/lib/python3.7/site-packages/torch/serialization.py:454: SourceChangeWarning: source code of class 'torch.nn.modules.sparse.Embedding' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /home/zlinai/anaconda3/envs/py3.7-env/lib/python3.7/site-packages/torch/serialization.py:454: SourceChangeWarning: source code of class 'torch.nn.modules.dropout.Dropout' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /home/zlinai/anaconda3/envs/py3.7-env/lib/python3.7/site-packages/torch/serialization.py:454: SourceChangeWarning: source code of class 'torch.nn.modules.rnn.LSTM' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /home/zlinai/anaconda3/envs/py3.7-env/lib/python3.7/site-packages/torch/serialization.py:454: SourceChangeWarning: source code of class 'torch.nn.modules.linear.Linear' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /home/zlinai/anaconda3/envs/py3.7-env/lib/python3.7/site-packages/torch/serialization.py:454: SourceChangeWarning: source code of class 'torch.nn.modules.loss.CrossEntropyLoss' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /home/zlinai/anaconda3/envs/py3.7-env/lib/python3.7/site-packages/torch/serialization.py:454: SourceChangeWarning: source code of class 'torch.nn.modules.container.ModuleList' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /home/zlinai/anaconda3/envs/py3.7-env/lib/python3.7/site-packages/torch/serialization.py:454: SourceChangeWarning: source code of class 'torch.nn.modules.conv.Conv1d' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /home/zlinai/anaconda3/envs/py3.7-env/lib/python3.7/site-packages/torch/nn/_reduction.py:46: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead. warnings.warn(warning.format(ret)) /home/zlinai/anaconda3/envs/py3.7-env/lib/python3.7/site-packages/torch/nn/functional.py:1374: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead. warnings.warn("nn.functional.tanh is deprecated. Use torch.tanh instead.") Traceback (most recent call last): File "src/main.py", line 773, in main() File "src/main.py", line 770, in main train() File "src/main.py", line 600, in train trans_logits, noise_logits, KL_loss, lm_len, trans_len = model.forward(x_train, x_mask, x_len, x_pos_emb_idxs, y_train, y_mask, y_len, y_pos_emb_idxs, y_sampled, y_sampled_mask, y_sampled_len) File "/home/data/zlinai/COMP_5212_Project/Baselines/deep-latent-sequence-model/src/model.py", line 252, in forward x_trans, x_trans_mask, x_trans_len, index = self.get_translations(x_train, x_mask, x_len, y_sampled, y_sampled_mask, y_sampled_len, temperature) File "/home/data/zlinai/COMP_5212_Project/Baselines/deep-latent-sequence-model/src/model.py", line 363, in get_translations temperature=temperature, sampling=True) File "/home/data/zlinai/COMP_5212_Project/Baselines/deep-latent-sequence-model/src/model.py", line 550, in translate hyps = self.sampling_translate(x_train, x_mask, x_len, y, y_mask, y_len, max_len=max_len, temperature=temperature) File "/home/data/zlinai/COMP_5212_Project/Baselines/deep-latent-sequence-model/src/model.py", line 631, in sampling_translate if mask[i].item(): RuntimeError: CUDA error: device-side assert triggered

But when I train it on CPU, the code executed without errors. Could you tell me how to fix this issue? My cuda version is 9.0, python version is 3.7 and pytorch version is 1.1.0.

Thx!

lzzhaha commented 4 years ago

Fixed this issue by upgrading pytorch to 1.4.0.