search

vikigenius / conditional_text_generation

Adversarial Latent Space model for dialog generation
Creative Commons Zero v1.0 Universal
3 stars 1 forks source link

RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED #3

Closed Demfier closed 5 years ago

Demfier commented 5 years ago

I am getting this error when I run allennlp train experiments/vae.jsonnet -s models/dialog_vae --include-package src I tried using different GPU IDs but it didn't work.

2019-07-21 17:45:17,012 - INFO - pytorch_pretrained_bert.modeling - Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex .
2019-07-21 17:45:17,665 - INFO - allennlp.common.params - random_seed = 0
2019-07-21 17:45:17,665 - INFO - allennlp.common.params - numpy_seed = 0
2019-07-21 17:45:17,665 - INFO - allennlp.common.params - pytorch_seed = 0
2019-07-21 17:45:17,720 - INFO - allennlp.common.checks - Pytorch version: 1.1.0
2019-07-21 17:45:17,722 - INFO - allennlp.common.params - evaluate_on_test = False
2019-07-21 17:45:17,722 - INFO - allennlp.common.params - trainer.type = callback
2019-07-21 17:45:17,722 - INFO - allennlp.common.params - validation_dataset_reader = None
2019-07-21 17:45:17,722 - INFO - allennlp.common.from_params - instantiating class <class 'allennlp.data.dataset_readers.dataset_reader.DatasetReader'> from params {'type': 'autoencoder'} and extras set()
2019-07-21 17:45:17,722 - INFO - allennlp.common.params - dataset_reader.type = autoencoder
2019-07-21 17:45:17,722 - INFO - allennlp.common.from_params - instantiating class <class 'src.data.dataset.AutoencoderDatasetReader'> from params {} and extras set()
2019-07-21 17:45:17,722 - INFO - allennlp.common.params - dataset_reader.source_token_indexers = <allennlp.common.params.Params object at 0x7fa15c5736d8>
2019-07-21 17:45:17,722 - INFO - allennlp.common.params - dataset_reader.target_token_indexers = <allennlp.common.params.Params object at 0x7fa15c5736d8>
2019-07-21 17:45:17,722 - INFO - allennlp.common.params - dataset_reader.source_add_start_token = True
2019-07-21 17:45:17,722 - INFO - allennlp.common.params - dataset_reader.delimiter =    
2019-07-21 17:45:17,722 - INFO - allennlp.common.params - dataset_reader.lazy = False
2019-07-21 17:45:17,806 - INFO - allennlp.common.params - train_data_path = data/interim/dialog/train_sentences.tsv
2019-07-21 17:45:17,806 - INFO - allennlp.training.util - Reading training data from data/interim/dialog/train_sentences.tsv
0it [00:00, ?it/s]2019-07-21 17:45:17,806 - INFO - src.data.dataset - Reading instances from lines in file at: data/interim/dialog/train_sentences.tsv
152104it [00:26, 5848.90it/s]
2019-07-21 17:45:43,812 - INFO - allennlp.common.params - validation_data_path = data/interim/dialog/valid_sentences.tsv
2019-07-21 17:45:43,812 - INFO - allennlp.training.util - Reading validation data from data/interim/dialog/valid_sentences.tsv
0it [00:00, ?it/s]2019-07-21 17:45:43,812 - INFO - src.data.dataset - Reading instances from lines in file at: data/interim/dialog/valid_sentences.tsv
8932it [00:01, 6413.16it/s]
2019-07-21 17:45:45,205 - INFO - allennlp.common.params - test_data_path = None
2019-07-21 17:45:45,381 - INFO - allennlp.training.trainer_pieces - From dataset instances, train, validation will be considered for vocabulary creation.
2019-07-21 17:45:45,381 - INFO - allennlp.common.params - vocabulary.type = None
2019-07-21 17:45:45,381 - INFO - allennlp.common.params - vocabulary.extend = False
2019-07-21 17:45:45,381 - INFO - allennlp.common.params - vocabulary.directory_path = None
2019-07-21 17:45:45,381 - INFO - allennlp.common.params - vocabulary.min_count = None
2019-07-21 17:45:45,381 - INFO - allennlp.common.params - vocabulary.max_vocab_size = None
2019-07-21 17:45:45,381 - INFO - allennlp.common.params - vocabulary.non_padded_namespaces = ('*tags', '*labels')
2019-07-21 17:45:45,382 - INFO - allennlp.common.params - vocabulary.min_pretrained_embeddings = None
2019-07-21 17:45:45,382 - INFO - allennlp.common.params - vocabulary.only_include_pretrained_words = False
2019-07-21 17:45:45,382 - INFO - allennlp.common.params - vocabulary.tokens_to_add = None
2019-07-21 17:45:45,382 - INFO - allennlp.data.vocabulary - Fitting token dictionary from dataset.
161036it [00:01, 85823.61it/s]
2019-07-21 17:45:47,290 - INFO - allennlp.common.from_params - instantiating class <class 'allennlp.models.model.Model'> from params {'decoder': {'latent_dim': 128, 'rnn': {'hidden_size': 512, 'input_size': 428, 'num_layers': 1, 'type': 'lstm'}, 'target_embedder': {'token_embedders': {'tokens': {'embedding_dim': 300, 'type': 'embedding', 'vocab_namespace': 'tokens'}}}, 'type': 'variational_decoder'}, 'initializer': [['.*', {'a': -0.1, 'b': 0.1, 'type': 'uniform'}]], 'kl_weight': {'max_weight': 0.5, 'min_weight': 0, 'num_iter_to_max': 3500, 'slope': 0.5, 'type': 'sigmoid_annealed', 'warmup': 500}, 'temperature': 1e-05, 'type': 'vae', 'variational_encoder': {'encoder': {'bidirectional': True, 'hidden_size': 512, 'input_size': 300, 'num_layers': 1, 'type': 'lstm'}, 'latent_dim': 128, 'text_field_embedder': {'token_embedders': {'tokens': {'embedding_dim': 300, 'type': 'embedding', 'vocab_namespace': 'tokens'}}}, 'type': 'gaussian'}} and extras {'vocab'}
2019-07-21 17:45:47,290 - INFO - allennlp.common.params - model.type = vae
2019-07-21 17:45:47,290 - INFO - allennlp.common.from_params - instantiating class <class 'src.models.vae.VAE'> from params {'decoder': {'latent_dim': 128, 'rnn': {'hidden_size': 512, 'input_size': 428, 'num_layers': 1, 'type': 'lstm'}, 'target_embedder': {'token_embedders': {'tokens': {'embedding_dim': 300, 'type': 'embedding', 'vocab_namespace': 'tokens'}}}, 'type': 'variational_decoder'}, 'initializer': [['.*', {'a': -0.1, 'b': 0.1, 'type': 'uniform'}]], 'kl_weight': {'max_weight': 0.5, 'min_weight': 0, 'num_iter_to_max': 3500, 'slope': 0.5, 'type': 'sigmoid_annealed', 'warmup': 500}, 'temperature': 1e-05, 'variational_encoder': {'encoder': {'bidirectional': True, 'hidden_size': 512, 'input_size': 300, 'num_layers': 1, 'type': 'lstm'}, 'latent_dim': 128, 'text_field_embedder': {'token_embedders': {'tokens': {'embedding_dim': 300, 'type': 'embedding', 'vocab_namespace': 'tokens'}}}, 'type': 'gaussian'}} and extras {'vocab'}
2019-07-21 17:45:47,290 - INFO - allennlp.common.from_params - instantiating class <class 'src.modules.encoders.variational_encoder.VariationalEncoder'> from params {'encoder': {'bidirectional': True, 'hidden_size': 512, 'input_size': 300, 'num_layers': 1, 'type': 'lstm'}, 'latent_dim': 128, 'text_field_embedder': {'token_embedders': {'tokens': {'embedding_dim': 300, 'type': 'embedding', 'vocab_namespace': 'tokens'}}}, 'type': 'gaussian'} and extras {'vocab'}
2019-07-21 17:45:47,290 - INFO - allennlp.common.params - model.variational_encoder.type = gaussian
2019-07-21 17:45:47,290 - INFO - allennlp.common.from_params - instantiating class <class 'src.modules.encoders.gaussian_encoder.GaussianEncoder'> from params {'encoder': {'bidirectional': True, 'hidden_size': 512, 'input_size': 300, 'num_layers': 1, 'type': 'lstm'}, 'latent_dim': 128, 'text_field_embedder': {'token_embedders': {'tokens': {'embedding_dim': 300, 'type': 'embedding', 'vocab_namespace': 'tokens'}}}} and extras {'vocab'}
2019-07-21 17:45:47,290 - INFO - allennlp.common.from_params - instantiating class <class 'allennlp.modules.text_field_embedders.text_field_embedder.TextFieldEmbedder'> from params {'token_embedders': {'tokens': {'embedding_dim': 300, 'type': 'embedding', 'vocab_namespace': 'tokens'}}} and extras {'vocab'}
2019-07-21 17:45:47,290 - INFO - allennlp.common.params - model.variational_encoder.text_field_embedder.type = basic
2019-07-21 17:45:47,291 - INFO - allennlp.common.params - model.variational_encoder.text_field_embedder.embedder_to_indexer_map = None
2019-07-21 17:45:47,291 - INFO - allennlp.common.params - model.variational_encoder.text_field_embedder.allow_unmatched_keys = False
2019-07-21 17:45:47,291 - INFO - allennlp.common.from_params - instantiating class <class 'allennlp.modules.token_embedders.token_embedder.TokenEmbedder'> from params {'embedding_dim': 300, 'type': 'embedding', 'vocab_namespace': 'tokens'} and extras {'vocab'}
2019-07-21 17:45:47,291 - INFO - allennlp.common.params - model.variational_encoder.text_field_embedder.token_embedders.tokens.type = embedding
2019-07-21 17:45:47,291 - INFO - allennlp.common.params - model.variational_encoder.text_field_embedder.token_embedders.tokens.num_embeddings = None
2019-07-21 17:45:47,291 - INFO - allennlp.common.params - model.variational_encoder.text_field_embedder.token_embedders.tokens.vocab_namespace = tokens
2019-07-21 17:45:47,291 - INFO - allennlp.common.params - model.variational_encoder.text_field_embedder.token_embedders.tokens.embedding_dim = 300
2019-07-21 17:45:47,291 - INFO - allennlp.common.params - model.variational_encoder.text_field_embedder.token_embedders.tokens.pretrained_file = None
2019-07-21 17:45:47,291 - INFO - allennlp.common.params - model.variational_encoder.text_field_embedder.token_embedders.tokens.projection_dim = None
2019-07-21 17:45:47,291 - INFO - allennlp.common.params - model.variational_encoder.text_field_embedder.token_embedders.tokens.trainable = True
2019-07-21 17:45:47,291 - INFO - allennlp.common.params - model.variational_encoder.text_field_embedder.token_embedders.tokens.padding_index = None
2019-07-21 17:45:47,291 - INFO - allennlp.common.params - model.variational_encoder.text_field_embedder.token_embedders.tokens.max_norm = None
2019-07-21 17:45:47,291 - INFO - allennlp.common.params - model.variational_encoder.text_field_embedder.token_embedders.tokens.norm_type = 2.0
2019-07-21 17:45:47,291 - INFO - allennlp.common.params - model.variational_encoder.text_field_embedder.token_embedders.tokens.scale_grad_by_freq = False
2019-07-21 17:45:47,291 - INFO - allennlp.common.params - model.variational_encoder.text_field_embedder.token_embedders.tokens.sparse = False
2019-07-21 17:45:47,327 - INFO - allennlp.common.from_params - instantiating class <class 'allennlp.modules.seq2vec_encoders.seq2vec_encoder.Seq2VecEncoder'> from params {'bidirectional': True, 'hidden_size': 512, 'input_size': 300, 'num_layers': 1, 'type': 'lstm'} and extras {'vocab'}
2019-07-21 17:45:47,328 - INFO - allennlp.common.params - model.variational_encoder.encoder.type = lstm
2019-07-21 17:45:47,328 - INFO - allennlp.common.params - model.variational_encoder.encoder.batch_first = True
2019-07-21 17:45:47,328 - INFO - allennlp.common.params - Converting Params object to dict; logging of default values will not occur when dictionary parameters are used subsequently.
2019-07-21 17:45:47,328 - INFO - allennlp.common.params - CURRENTLY DEFINED PARAMETERS: 
2019-07-21 17:45:47,328 - INFO - allennlp.common.params - model.variational_encoder.encoder.bidirectional = True
2019-07-21 17:45:47,328 - INFO - allennlp.common.params - model.variational_encoder.encoder.hidden_size = 512
2019-07-21 17:45:47,328 - INFO - allennlp.common.params - model.variational_encoder.encoder.input_size = 300
2019-07-21 17:45:47,328 - INFO - allennlp.common.params - model.variational_encoder.encoder.num_layers = 1
2019-07-21 17:45:47,328 - INFO - allennlp.common.params - model.variational_encoder.encoder.batch_first = True
2019-07-21 17:45:47,347 - INFO - allennlp.common.params - model.variational_encoder.latent_dim = 128
2019-07-21 17:45:47,349 - INFO - allennlp.common.from_params - instantiating class <class 'src.modules.decoders.decoder.Decoder'> from params {'latent_dim': 128, 'rnn': {'hidden_size': 512, 'input_size': 428, 'num_layers': 1, 'type': 'lstm'}, 'target_embedder': {'token_embedders': {'tokens': {'embedding_dim': 300, 'type': 'embedding', 'vocab_namespace': 'tokens'}}}, 'type': 'variational_decoder'} and extras {'vocab'}
2019-07-21 17:45:47,349 - INFO - allennlp.common.params - model.decoder.type = variational_decoder
2019-07-21 17:45:47,349 - INFO - allennlp.common.from_params - instantiating class <class 'src.modules.decoders.variational_decoder.VariationalDecoder'> from params {'latent_dim': 128, 'rnn': {'hidden_size': 512, 'input_size': 428, 'num_layers': 1, 'type': 'lstm'}, 'target_embedder': {'token_embedders': {'tokens': {'embedding_dim': 300, 'type': 'embedding', 'vocab_namespace': 'tokens'}}}} and extras {'vocab'}
2019-07-21 17:45:47,349 - INFO - allennlp.common.from_params - instantiating class <class 'allennlp.modules.text_field_embedders.text_field_embedder.TextFieldEmbedder'> from params {'token_embedders': {'tokens': {'embedding_dim': 300, 'type': 'embedding', 'vocab_namespace': 'tokens'}}} and extras {'vocab'}
2019-07-21 17:45:47,349 - INFO - allennlp.common.params - model.decoder.target_embedder.type = basic
2019-07-21 17:45:47,349 - INFO - allennlp.common.params - model.decoder.target_embedder.embedder_to_indexer_map = None
2019-07-21 17:45:47,349 - INFO - allennlp.common.params - model.decoder.target_embedder.allow_unmatched_keys = False
2019-07-21 17:45:47,350 - INFO - allennlp.common.from_params - instantiating class <class 'allennlp.modules.token_embedders.token_embedder.TokenEmbedder'> from params {'embedding_dim': 300, 'type': 'embedding', 'vocab_namespace': 'tokens'} and extras {'vocab'}
2019-07-21 17:45:47,350 - INFO - allennlp.common.params - model.decoder.target_embedder.token_embedders.tokens.type = embedding
2019-07-21 17:45:47,350 - INFO - allennlp.common.params - model.decoder.target_embedder.token_embedders.tokens.num_embeddings = None
2019-07-21 17:45:47,350 - INFO - allennlp.common.params - model.decoder.target_embedder.token_embedders.tokens.vocab_namespace = tokens
2019-07-21 17:45:47,350 - INFO - allennlp.common.params - model.decoder.target_embedder.token_embedders.tokens.embedding_dim = 300
2019-07-21 17:45:47,350 - INFO - allennlp.common.params - model.decoder.target_embedder.token_embedders.tokens.pretrained_file = None
2019-07-21 17:45:47,350 - INFO - allennlp.common.params - model.decoder.target_embedder.token_embedders.tokens.projection_dim = None
2019-07-21 17:45:47,350 - INFO - allennlp.common.params - model.decoder.target_embedder.token_embedders.tokens.trainable = True
2019-07-21 17:45:47,350 - INFO - allennlp.common.params - model.decoder.target_embedder.token_embedders.tokens.padding_index = None
2019-07-21 17:45:47,350 - INFO - allennlp.common.params - model.decoder.target_embedder.token_embedders.tokens.max_norm = None
2019-07-21 17:45:47,350 - INFO - allennlp.common.params - model.decoder.target_embedder.token_embedders.tokens.norm_type = 2.0
2019-07-21 17:45:47,350 - INFO - allennlp.common.params - model.decoder.target_embedder.token_embedders.tokens.scale_grad_by_freq = False
2019-07-21 17:45:47,350 - INFO - allennlp.common.params - model.decoder.target_embedder.token_embedders.tokens.sparse = False
2019-07-21 17:45:47,386 - INFO - allennlp.common.from_params - instantiating class <class 'allennlp.modules.seq2seq_encoders.seq2seq_encoder.Seq2SeqEncoder'> from params {'hidden_size': 512, 'input_size': 428, 'num_layers': 1, 'type': 'lstm'} and extras {'vocab'}
2019-07-21 17:45:47,386 - INFO - allennlp.common.params - model.decoder.rnn.type = lstm
2019-07-21 17:45:47,386 - INFO - allennlp.common.params - model.decoder.rnn.batch_first = True
2019-07-21 17:45:47,386 - INFO - allennlp.common.params - model.decoder.rnn.stateful = False
2019-07-21 17:45:47,386 - INFO - allennlp.common.params - Converting Params object to dict; logging of default values will not occur when dictionary parameters are used subsequently.
2019-07-21 17:45:47,386 - INFO - allennlp.common.params - CURRENTLY DEFINED PARAMETERS: 
2019-07-21 17:45:47,386 - INFO - allennlp.common.params - model.decoder.rnn.hidden_size = 512
2019-07-21 17:45:47,386 - INFO - allennlp.common.params - model.decoder.rnn.input_size = 428
2019-07-21 17:45:47,386 - INFO - allennlp.common.params - model.decoder.rnn.num_layers = 1
2019-07-21 17:45:47,386 - INFO - allennlp.common.params - model.decoder.rnn.batch_first = True
2019-07-21 17:45:47,398 - INFO - allennlp.common.params - model.decoder.latent_dim = 128
2019-07-21 17:45:47,398 - INFO - allennlp.common.params - model.decoder.dropout_p = 0.5
2019-07-21 17:45:47,460 - INFO - allennlp.common.from_params - instantiating class <class 'src.modules.annealer.LossWeight'> from params {'max_weight': 0.5, 'min_weight': 0, 'num_iter_to_max': 3500, 'slope': 0.5, 'type': 'sigmoid_annealed', 'warmup': 500} and extras {'vocab'}
2019-07-21 17:45:47,460 - INFO - allennlp.common.params - model.kl_weight.type = sigmoid_annealed
2019-07-21 17:45:47,460 - INFO - allennlp.common.from_params - instantiating class <class 'src.modules.annealer.SigmoidAnnealedWeight'> from params {'max_weight': 0.5, 'min_weight': 0, 'num_iter_to_max': 3500, 'slope': 0.5, 'warmup': 500} and extras {'vocab'}
2019-07-21 17:45:47,460 - INFO - allennlp.common.params - model.kl_weight.min_weight = 0
2019-07-21 17:45:47,460 - INFO - allennlp.common.params - model.kl_weight.max_weight = 0.5
2019-07-21 17:45:47,460 - INFO - allennlp.common.params - model.kl_weight.warmup = 500
2019-07-21 17:45:47,460 - INFO - allennlp.common.params - model.kl_weight.num_iter_to_max = 3500
2019-07-21 17:45:47,460 - INFO - allennlp.common.params - model.kl_weight.slope = 0.5
2019-07-21 17:45:47,460 - INFO - allennlp.common.params - model.temperature = 1e-05
2019-07-21 17:45:47,460 - INFO - allennlp.common.params - model.initializer = [['.*', {'a': -0.1, 'b': 0.1, 'type': 'uniform'}]]
2019-07-21 17:45:47,460 - INFO - allennlp.common.from_params - instantiating class <class 'allennlp.nn.initializers.Initializer'> from params {'a': -0.1, 'b': 0.1, 'type': 'uniform'} and extras set()
2019-07-21 17:45:47,460 - INFO - allennlp.common.params - model.initializer.0.1.type = uniform
2019-07-21 17:45:47,460 - INFO - allennlp.common.params - Converting Params object to dict; logging of default values will not occur when dictionary parameters are used subsequently.
2019-07-21 17:45:47,460 - INFO - allennlp.common.params - CURRENTLY DEFINED PARAMETERS: 
2019-07-21 17:45:47,460 - INFO - allennlp.common.params - model.initializer.0.1.a = -0.1
2019-07-21 17:45:47,460 - INFO - allennlp.common.params - model.initializer.0.1.b = 0.1
2019-07-21 17:45:47,461 - INFO - allennlp.nn.initializers - Initializing parameters
2019-07-21 17:45:47,461 - INFO - allennlp.nn.initializers - Initializing _encoder._text_field_embedder.token_embedder_tokens.weight using .* initializer
2019-07-21 17:45:47,490 - INFO - allennlp.nn.initializers - Initializing _encoder._encoder._module.weight_ih_l0 using .* initializer
2019-07-21 17:45:47,493 - INFO - allennlp.nn.initializers - Initializing _encoder._encoder._module.weight_hh_l0 using .* initializer
2019-07-21 17:45:47,497 - INFO - allennlp.nn.initializers - Initializing _encoder._encoder._module.bias_ih_l0 using .* initializer
2019-07-21 17:45:47,498 - INFO - allennlp.nn.initializers - Initializing _encoder._encoder._module.bias_hh_l0 using .* initializer
2019-07-21 17:45:47,498 - INFO - allennlp.nn.initializers - Initializing _encoder._encoder._module.weight_ih_l0_reverse using .* initializer
2019-07-21 17:45:47,500 - INFO - allennlp.nn.initializers - Initializing _encoder._encoder._module.weight_hh_l0_reverse using .* initializer
2019-07-21 17:45:47,505 - INFO - allennlp.nn.initializers - Initializing _encoder._encoder._module.bias_ih_l0_reverse using .* initializer
2019-07-21 17:45:47,505 - INFO - allennlp.nn.initializers - Initializing _encoder._encoder._module.bias_hh_l0_reverse using .* initializer
2019-07-21 17:45:47,505 - INFO - allennlp.nn.initializers - Initializing _encoder._latent_to_mean.weight using .* initializer
2019-07-21 17:45:47,506 - INFO - allennlp.nn.initializers - Initializing _encoder._latent_to_mean.bias using .* initializer
2019-07-21 17:45:47,506 - INFO - allennlp.nn.initializers - Initializing _encoder._latent_to_logvar.weight using .* initializer
2019-07-21 17:45:47,507 - INFO - allennlp.nn.initializers - Initializing _encoder._latent_to_logvar.bias using .* initializer
2019-07-21 17:45:47,507 - INFO - allennlp.nn.initializers - Initializing _decoder._target_embedder.token_embedder_tokens.weight using .* initializer
2019-07-21 17:45:47,536 - INFO - allennlp.nn.initializers - Initializing _decoder.rnn._module.weight_ih_l0 using .* initializer
2019-07-21 17:45:47,540 - INFO - allennlp.nn.initializers - Initializing _decoder.rnn._module.weight_hh_l0 using .* initializer
2019-07-21 17:45:47,545 - INFO - allennlp.nn.initializers - Initializing _decoder.rnn._module.bias_ih_l0 using .* initializer
2019-07-21 17:45:47,545 - INFO - allennlp.nn.initializers - Initializing _decoder.rnn._module.bias_hh_l0 using .* initializer
2019-07-21 17:45:47,545 - INFO - allennlp.nn.initializers - Initializing _decoder._latent_to_dec_hidden.weight using .* initializer
2019-07-21 17:45:47,545 - INFO - allennlp.nn.initializers - Initializing _decoder._latent_to_dec_hidden.bias using .* initializer
2019-07-21 17:45:47,545 - INFO - allennlp.nn.initializers - Initializing _decoder._dec_linear.weight using .* initializer
2019-07-21 17:45:47,595 - INFO - allennlp.nn.initializers - Initializing _decoder._dec_linear.bias using .* initializer
2019-07-21 17:45:47,595 - INFO - allennlp.nn.initializers - Done initializing parameters; the following parameters are using their default initialization from their code
2019-07-21 17:45:47,626 - INFO - allennlp.common.from_params - instantiating class <class 'allennlp.data.iterators.data_iterator.DataIterator'> from params {'batch_size': 64, 'sorting_keys': [['source_tokens', 'num_tokens']], 'type': 'bucket'} and extras set()
2019-07-21 17:45:47,626 - INFO - allennlp.common.params - iterator.type = bucket
2019-07-21 17:45:47,626 - INFO - allennlp.common.from_params - instantiating class <class 'allennlp.data.iterators.bucket_iterator.BucketIterator'> from params {'batch_size': 64, 'sorting_keys': [['source_tokens', 'num_tokens']]} and extras set()
2019-07-21 17:45:47,626 - INFO - allennlp.common.params - iterator.sorting_keys = [['source_tokens', 'num_tokens']]
2019-07-21 17:45:47,626 - INFO - allennlp.common.params - iterator.padding_noise = 0.1
2019-07-21 17:45:47,626 - INFO - allennlp.common.params - iterator.biggest_batch_first = False
2019-07-21 17:45:47,626 - INFO - allennlp.common.params - iterator.batch_size = 64
2019-07-21 17:45:47,626 - INFO - allennlp.common.params - iterator.instances_per_epoch = None
2019-07-21 17:45:47,626 - INFO - allennlp.common.params - iterator.max_instances_in_memory = None
2019-07-21 17:45:47,626 - INFO - allennlp.common.params - iterator.cache_instances = False
2019-07-21 17:45:47,626 - INFO - allennlp.common.params - iterator.track_epoch = False
2019-07-21 17:45:47,626 - INFO - allennlp.common.params - iterator.maximum_samples_per_batch = None
2019-07-21 17:45:47,626 - INFO - allennlp.common.params - validation_iterator = None
2019-07-21 17:45:47,626 - INFO - allennlp.common.params - trainer.no_grad = ()
2019-07-21 17:45:47,626 - INFO - allennlp.training.trainer_pieces - Following parameters are Frozen  (without gradient):
2019-07-21 17:45:47,626 - INFO - allennlp.training.trainer_pieces - Following parameters are Tunable (with gradient):
2019-07-21 17:45:47,626 - INFO - allennlp.training.trainer_pieces - _encoder._text_field_embedder.token_embedder_tokens.weight
2019-07-21 17:45:47,626 - INFO - allennlp.training.trainer_pieces - _encoder._encoder._module.weight_ih_l0
2019-07-21 17:45:47,626 - INFO - allennlp.training.trainer_pieces - _encoder._encoder._module.weight_hh_l0
2019-07-21 17:45:47,626 - INFO - allennlp.training.trainer_pieces - _encoder._encoder._module.bias_ih_l0
2019-07-21 17:45:47,626 - INFO - allennlp.training.trainer_pieces - _encoder._encoder._module.bias_hh_l0
2019-07-21 17:45:47,626 - INFO - allennlp.training.trainer_pieces - _encoder._encoder._module.weight_ih_l0_reverse
2019-07-21 17:45:47,626 - INFO - allennlp.training.trainer_pieces - _encoder._encoder._module.weight_hh_l0_reverse
2019-07-21 17:45:47,627 - INFO - allennlp.training.trainer_pieces - _encoder._encoder._module.bias_ih_l0_reverse
2019-07-21 17:45:47,627 - INFO - allennlp.training.trainer_pieces - _encoder._encoder._module.bias_hh_l0_reverse
2019-07-21 17:45:47,627 - INFO - allennlp.training.trainer_pieces - _encoder._latent_to_mean.weight
2019-07-21 17:45:47,627 - INFO - allennlp.training.trainer_pieces - _encoder._latent_to_mean.bias
2019-07-21 17:45:47,627 - INFO - allennlp.training.trainer_pieces - _encoder._latent_to_logvar.weight
2019-07-21 17:45:47,627 - INFO - allennlp.training.trainer_pieces - _encoder._latent_to_logvar.bias
2019-07-21 17:45:47,627 - INFO - allennlp.training.trainer_pieces - _decoder._target_embedder.token_embedder_tokens.weight
2019-07-21 17:45:47,627 - INFO - allennlp.training.trainer_pieces - _decoder.rnn._module.weight_ih_l0
2019-07-21 17:45:47,627 - INFO - allennlp.training.trainer_pieces - _decoder.rnn._module.weight_hh_l0
2019-07-21 17:45:47,627 - INFO - allennlp.training.trainer_pieces - _decoder.rnn._module.bias_ih_l0
2019-07-21 17:45:47,627 - INFO - allennlp.training.trainer_pieces - _decoder.rnn._module.bias_hh_l0
2019-07-21 17:45:47,627 - INFO - allennlp.training.trainer_pieces - _decoder._latent_to_dec_hidden.weight
2019-07-21 17:45:47,627 - INFO - allennlp.training.trainer_pieces - _decoder._latent_to_dec_hidden.bias
2019-07-21 17:45:47,627 - INFO - allennlp.training.trainer_pieces - _decoder._dec_linear.weight
2019-07-21 17:45:47,627 - INFO - allennlp.training.trainer_pieces - _decoder._dec_linear.bias
2019-07-21 17:45:47,627 - INFO - allennlp.common.params - trainer.shuffle = True
2019-07-21 17:45:47,627 - INFO - allennlp.common.params - trainer.num_epochs = 30
2019-07-21 17:45:47,627 - INFO - allennlp.common.params - trainer.cuda_device = 1
Traceback (most recent call last):
  File "/home/gsahu/dev/bin/allennlp", line 11, in <module>
    load_entry_point('allennlp', 'console_scripts', 'allennlp')()
  File "/home/gsahu/allennlp/allennlp/run.py", line 18, in run
    main(prog="allennlp")
  File "/home/gsahu/allennlp/allennlp/commands/__init__.py", line 102, in main
    args.func(args)
  File "/home/gsahu/allennlp/allennlp/commands/train.py", line 117, in train_model_from_args
    args.cache_prefix)
  File "/home/gsahu/allennlp/allennlp/commands/train.py", line 161, in train_model_from_file
    cache_directory, cache_prefix)
  File "/home/gsahu/allennlp/allennlp/commands/train.py", line 239, in train_model
    trainer = TrainerBase.from_params(params, serialization_dir, recover)
  File "/home/gsahu/allennlp/allennlp/training/trainer_base.py", line 85, in from_params
    return klass.from_params(params, serialization_dir, recover)
  File "/home/gsahu/allennlp/allennlp/training/callback_trainer.py", line 285, in from_params
    model = model.cuda(model_device)
  File "/home/gsahu/dev/lib/python3.6/site-packages/torch/nn/modules/module.py", line 265, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/home/gsahu/dev/lib/python3.6/site-packages/torch/nn/modules/module.py", line 193, in _apply
    module._apply(fn)
  File "/home/gsahu/dev/lib/python3.6/site-packages/torch/nn/modules/module.py", line 193, in _apply
    module._apply(fn)
  File "/home/gsahu/dev/lib/python3.6/site-packages/torch/nn/modules/module.py", line 193, in _apply
    module._apply(fn)
  File "/home/gsahu/dev/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 127, in _apply
    self.flatten_parameters()
  File "/home/gsahu/dev/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 123, in flatten_parameters
    self.batch_first, bool(self.bidirectional))
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
Demfier commented 5 years ago

Allennlp version that I have setup --> 0.8.5-unreleased

vikigenius commented 5 years ago

I am not sure if this is a version issue. Just install it by cloning the repo onto your local. Activate your environment and use pip install -e .

It is working for me. Either change the device in experiments/vae.jsonnet by changing the line local CUDA = 1

or leave it as 0 and use EXPORT CUDA_VISIBLE_DEVICES to control it. It is working for me.

vikigenius commented 5 years ago

Test your pytorch installation again if you still have a problem. This reads more like a pytorch or GPU issue and not an allennlp issue

vikigenius commented 5 years ago

Closing this as this is a PYTORCH, CUDNN version issue