search

lucidrains / h-transformer-1d

Implementation of H-Transformer-1D, Hierarchical Attention for Sequence Learning
MIT License
154 stars 21 forks source link

Masking not working in training, thanks #18

Closed junyongyou closed 2 years ago

junyongyou commented 2 years ago

Hi, I have tried to train the model on GPU with masking enabled. The line 94 t = torch.flip(t, dims = (2,)) reports an error: RuntimeError: "flip_cuda" not implemented for 'Bool', even though I have tried to move mask to CPU.

Any ideas to solve the problem? Thanks a lot.

lucidrains commented 2 years ago

@junyongyou Hi Junyong! Thanks for catching this tidbit. I didn't know about this limitation. Should be good with https://github.com/lucidrains/h-transformer-1d/commit/8afd75cc6bc41754620bb6ab3737176cb69bdf93

lucidrains commented 2 years ago

@junyongyou is this architecture working for any of your tasks?

junyongyou commented 2 years ago

@junyongyou Hi Junyong! Thanks for catching this tidbit. I didn't know about this limitation. Should be good with 8afd75c

Thanks a lot for your efforts. I have updated the code, but have not tested it yet. I will test it tomorrow and will report back here. Thank you so much.

junyongyou commented 2 years ago

Dear @lucidrains I can confirm that the new change fixed the problem of masking, my train can go without problems. Thanks a lot for your efforts.

However, there is another issue when I train the model on multi-GPUs. \site-packages\rotary_embedding_torch\rotary_embedding_torch.py", line 45, in apply_rotary_emb t = (t * freqs.cos()) + (rotate_half(t) * freqs.sin()) RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!

I am not sure if this is a known issue. But at least I can train it on a single GPU. So we can close this issue. Thank you very much.

lucidrains commented 2 years ago

@junyongyou ohh you caught another bug, this time with another library - do you want to check if 0.1.2 https://github.com/lucidrains/rotary-embedding-torch fixed this issue?

junyongyou commented 2 years ago

@junyongyou ohh you caught another bug, this time with another library - do you want to check if 0.1.2 https://github.com/lucidrains/rotary-embedding-torch fixed this issue?

Thanks a lot for the info. I will check that. If not working, I will post an issue again. Let's close this issue and thanks again for your help.