Closed dwinkler1 closed 5 years ago
"Or would it be better to put them in a Json or csv file with one entry per file even though I do not have labels?" Yeah we currently require a line-by-line data format. You don't need to have a labels column (if you don't have one it will be automatically interpretted that the label is -1). You will however need a header for the text column (the default text column name is sentence).
You should just be able to plug in that csv and watch it train.
You may have to edit the --num_shards
and --split
arguments to reflect your dataset size and if you want to have the training set split into train/val/test.
Thank you very much for your quick response! The model is working now with a test dataset. However, if I want to run it with lazy loading I get the following error:
python main.py --data all.csv --text_key obs --lazy
configuring data
Traceback (most recent call last):
File "main.py", line 145, in <module>
train_data, val_data, test_data = data_config.apply(args)
File "/home/imsm/Documents/daniel_tmp/sentimentNvidia/sentiment-discovery-master/configure_data.py", line 16, in apply
return make_loaders(opt)
File "/home/imsm/Documents/daniel_tmp/sentimentNvidia/sentiment-discovery-master/configure_data.py", line 61, in make_loaders
train = data_utils.make_dataset(**data_set_args)
File "/home/imsm/Documents/daniel_tmp/sentimentNvidia/sentiment-discovery-master/data_utils/__init__.py", line 82, in make_dataset
num_shards=num_shards)
File "/home/imsm/Documents/daniel_tmp/sentimentNvidia/sentiment-discovery-master/data_utils/__init__.py", line 62, in post_process_ds
ds = unsupervised_dataset(ds, seq_length, persist_state=persist_state, num_shards=shards)
File "/home/imsm/Documents/daniel_tmp/sentimentNvidia/sentiment-discovery-master/data_utils/datasets.py", line 343, in __init__
self.str_ends = list(accumulate(ds.lens))
TypeError: 'int' object is not iterable
Any ideas of what I am doing wrong?
Few questions to help me better understand the situation:
1) How long is your dataset?
2) Can you try with --num_shards 12
--split 10,1,1
3) can you print out ds.lens right before that line gets called and share the printout with me?
Thank you for your help!!
yeah so the problem was that the dataset was being split into 102 parts by default so some of them had no data
Thank your for the awesome model. I would like to train it on a number of longer text documents and was wondering in what format I should pass the texts to the script. Can I just put them all in a single text file and pass that to main.py? Or would it be better to put them in a Json or csv file with one entry per file even though I do not have labels? Sorry, I am kind of confused since the model is unsupervised but the datasets still have labels.