Closed adryyandc closed 6 years ago
Would you mind trying this for the make lazy function?
def make_lazy(path, strs, data_type='data'):
"""make lazy version of file"""
lazypath = get_lazy_path(path)
if not os.path.exists(lazypath):
os.makedirs(lazypath)
datapath = os.path.join(lazypath, data_type)
lenpath = os.path.join(lazypath, data_type+'.len.pkl')
if not torch.distributed._initialized or torch.distributed.get_rank() == 0:
with open(datapath, 'w') as f:
str_ends = []
str_cnt = 0
for s in strs:
f.write(s)
str_cnt += len(s)
str_ends.append(str_cnt)
pkl.dump(str_ends, open(lenpath, 'wb'))
else:
while not os.path.exists(lenpath):
time.sleep(1)
That function fixed the error, but i get another error:
root@ub-16-sentiment:~/work/sentiment-discovery# python3 main.py --data /home/adrianc/work/Sentiment/dataset/amazon/aggressive_dedup.json --lazy --loose_json --text_key reviewText --label_key overall --num_shards 1002 --optim Adam --split 1000,1,1
configuring data
Creating mlstm
* number of parameters: 86294784
Traceback (most recent call last):
File "main.py", line 303, in <module>
val_loss = train(total_iters)
File "main.py", line 233, in train
for i, batch in enumerate(train_data):
File "/root/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 451, in __iter__
return _DataLoaderIter(self)
File "/root/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 239, in __init__
w.start()
File "/root/anaconda3/lib/python3.6/multiprocessing/process.py", line 105, in start
self._popen = self._Popen(self)
File "/root/anaconda3/lib/python3.6/multiprocessing/context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/root/anaconda3/lib/python3.6/multiprocessing/context.py", line 277, in _Popen
return Popen(process_obj)
File "/root/anaconda3/lib/python3.6/multiprocessing/popen_fork.py", line 26, in __init__
self._launch(process_obj)
File "/root/anaconda3/lib/python3.6/multiprocessing/popen_fork.py", line 73, in _launch
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
Adrian
Let me find a machine with 64GB to repro. What is LXD?
lxd --version 2.18 with ubuntu 16.04 and installed tools via conda.
Do you think this is related to GPU memory? I can do some verification!
Never mind, it got fixed when i installed the latest GPU driver:
wget https://us.download.nvidia.com/XFree86/Linux-x86_64/390.67/NVIDIA-Linux-x86_64-390.67.run
Now i get this error:
root@ub-16-sentiment:~/work/sentiment-discovery# python3 main.py --data /home/adrianc/work/Sentiment/dataset/amazon/aggressive_dedup.json --lazy --loose_json --text_key reviewText --label_key overall --num_shards 1002 --optim Adam --split 1000,1,1
configuring data
Creating mlstm
* number of parameters: 86294784
main.py:266: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
cur_loss = total_loss[0] / args.log_interval
| epoch 1 | 0/1180027 batches | lr 5.00E-04 | ms/batch 5.407E+01 | loss 5.55E-02 | ppl 1.06 | loss scale 1.00
Traceback (most recent call last):
File "main.py", line 303, in <module>
val_loss = train(total_iters)
File "main.py", line 233, in train
for i, batch in enumerate(train_data):
File "/root/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 286, in __next__
return self._process_next_batch(batch)
File "/root/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 307, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
TypeError: function takes exactly 5 arguments (1 given)
Hmmm. Someone else seems to be getting the same issue with the latest pytorch version. Let me look into this a little bit more.
Ok. I installed it with conda to be truth: root@ub-16-sentiment:~/work/sentiment-discovery# conda list | grep pytorch cuda90 1.0 h6433d27_0 pytorch pytorch 0.4.0 py36_cuda9.0.176_cudnn7.1.2_1 [cuda90] pytorch
So I am not able to get the same error while on master in my container.
There's several things to try here:
python main.py --lazy
and let me know what you get.pip
instead of conda
num_workers: 0
and/or pin_memory: False
in line 34
data_loader_args = {'num_workers': 1, 'shuffle': opt.shuffle, 'batch_size': batch_size,
'pin_memory': True, 'transpose': opt.transpose, 'distributed': opt.world_size > 1,
'rank': opt.rank, 'world_size': opt.world_size, 'drop_last': opt.world_size > 1}```
I will tomorrow when i get to work.
Let me also know if this PR works for you. https://github.com/NVIDIA/sentiment-discovery/pull/32
This works for me on the public pt0.4 container.
With the lazy patch i get the following error:
root@ub-16-sentiment:~/work/sentiment-discovery# python3 main.py --data /home/adrianc/work/Sentiment/dataset/amazon/aggressive_dedup.json --lazy --loose_json --text_key reviewText --label_key overall --num_shards 1002 --optim Adam --split 1000,1,1
configuring data
Traceback (most recent call last):
File "main.py", line 135, in <module>
train_data, val_data, test_data = data_config.apply(args)
File "/root/work/sentiment-discovery/configure_data.py", line 16, in apply
return make_loaders(opt)
File "/root/work/sentiment-discovery/configure_data.py", line 63, in make_loaders
train = data_utils.make_dataset(**data_set_args)
File "/root/work/sentiment-discovery/data_utils/__init__.py", line 133, in make_dataset
binarize_sent=binarize_sent, delim=delim, drop_unlabeled=drop_unlabeled, loose=loose)
File "/root/work/sentiment-discovery/data_utils/__init__.py", line 112, in handle_lazy
binarize_sent=binarize_sent, delim=delim, drop_unlabeled=drop_unlabeled)
File "/root/work/sentiment-discovery/data_utils/__init__.py", line 56, in get_lazy
return lazy_array_loader(processed_path, data_type=data_shard)
File "/root/work/sentiment-discovery/data_utils/lazy_loader.py", line 68, in __init__
self.read_lock = Lock()
NameError: name 'Lock' is not defined
P.S.: Fixed it with: import threading [...] and threading.Lock()
But i still get the error:
File "/root/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 307, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
TypeError: function takes exactly 5 arguments (1 given)
I will try compiling PyTorch from sources. What PyTorch commit hash do you use?
1) Running python main.py --lazy works just fine:
root@ub-16-sentiment:~/work/sentiment-discovery# python main.py --lazy
configuring data
Creating mlstm
* number of parameters: 86294784
main.py:266: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
cur_loss = total_loss[0] / args.log_interval
| epoch 1 | 0/ 1930 batches | lr 5.00E-04 | ms/batch 3.202E+01 | loss 5.54E-02 | ppl 1.06 | loss scale 1.00
| epoch 1 | 100/ 1930 batches | lr 4.74E-04 | ms/batch 2.947E+03 | loss 2.46E+00 | ppl 11.68 | loss scale 1.00
2) Running with running with num_workers: 0
i get error:
Traceback (most recent call last):
File "main.py", line 303, in <module>
val_loss = train(total_iters)
File "main.py", line 233, in train
for i, batch in enumerate(train_data):
File "/root/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 264, in __next__
batch = self.collate_fn([self.dataset[i] for i in indices])
File "/root/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 264, in <listcomp>
batch = self.collate_fn([self.dataset[i] for i in indices])
File "/root/work/sentiment-discovery/data_utils/datasets.py", line 375, in __getitem__
s = self.all_strs[other_str_idx]
File "/root/work/sentiment-discovery/data_utils/lazy_loader.py", line 79, in __getitem__
return self.file_read(start, end)
File "/root/work/sentiment-discovery/data_utils/lazy_loader.py", line 109, in file_read
rtn = rtn.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 2826-2827: unexpected end of data
3) Running with pin_memory: False
i get same error:
File "/root/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 307, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
TypeError: function takes exactly 5 arguments (1 given)
My apologies. I just added back the import statement now.
The first command indicates to me that this is specific to the amazon dataset or --loose_json
flag. Let me re download a a fresh copy of the amazon dataset, and test it myself.
Thanks for your patience.
No problem!
Maybe is my conda installation a problem. I tried with smaller dataset (amazon 5-core) and i get another error:
root@ub-16-sentiment:~/work/sentiment-discovery# python3 main.py --data /home/adrianc/work/Sentiment/dataset/amazon/kcore5.json --lazy --loose_json --text_key reviewText --label_key overall --num_shards 1002 --optim Adam --split 1000,1,1
configuring data
Creating mlstm
* number of parameters: 86294784
main.py:266: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
cur_loss = total_loss[0] / args.log_interval
| epoch 1 | 0/677428 batches | lr 5.00E-04 | ms/batch 4.036E+01 | loss 5.55E-02 | ppl 1.06 | loss scale 1.00
Traceback (most recent call last):
File "main.py", line 303, in <module>
val_loss = train(total_iters)
File "main.py", line 233, in train
for i, batch in enumerate(train_data):
File "/root/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 286, in __next__
return self._process_next_batch(batch)
File "/root/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 307, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
RuntimeError: Traceback (most recent call last):
File "/root/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 57, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/root/work/sentiment-discovery/data_utils/loaders.py", line 65, in default_collate
return [default_collate(samples) for samples in transposed]
File "/root/work/sentiment-discovery/data_utils/loaders.py", line 65, in <listcomp>
return [default_collate(samples) for samples in transposed]
File "/root/work/sentiment-discovery/data_utils/loaders.py", line 51, in default_collate
return torch.stack([torch.from_numpy(b) for b in batch], 0)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 257 and 261 in dimension 1 at /opt/conda/conda-bld/pytorch_1524586445097/work/aten/src/TH/generic/THTensorMath.c:3586
It is not related to --loose_json flag since i get same error with or without.
try #32 again. Fixed the UnicodeDecodeError I believe.
Thank you! Seems working now, at least for a few hundred of batches.
Awesome. Going to merge the change in then.
Hi,
I get memory error while trying to train on Amazon aggressively deduplicated data. I have 64 GB of memory installed on my system and a 1080ti installed.
I run command inside an LXD container.
The problem is memory usage get's beyond 64 GB.
Regards, Adrian