Trained a transformer model ,but cannot infer (CUDA ERROR): invalid device function

VincentChen95 commented 2 years ago

Hello team, I trained a model using the lightseq-train command. Then i converted it to "pb" format. Finally, I run the infer with this code:

import lightseq.inference as lsi
import sys

def main():
    file_name = sys.argv[1]
    model = lsi.Transformer(file_name, 8)
    output = model.infer([[1, 2, 3], [4, 5, 6]])

main()

Received this error:

Parsing protobuf: lightseq/examples/inference/python/export/transformer.pb
Finish loading src_emb_wei from host to device
Finish loading trg_emb_wei from host to device
Finish loading enc_wei from host to device
Finish loading dec_wei from host to device
Finish loading all weight from host to device
***model config***
encoder layers: 4
decoder layers: 4
hidden size: 512
inner size: 1024
head number: 8
dim per head: 64
src vocab size: 176
trg vocab size: 168
is_post_ln: 0
no_scale_embedding: 0
use_gelu: 0
start_id: 2
end_id: 6
padding_id: 2
is_multilingual: 0

***generator config***
beam size: 4
extra decode length(max decode length - src input length): 50
length penalty: 0.6
diverse lambda: 0
sampling method: beam_search
topk: 1
topp: 0.75
Allocated 576MB GPU buffer for transformer
decoder buffer init start
Traceback (most recent call last):
  File "infer.py", line 12, in <module>
    main()
  File "infer.py", line 8, in main
    model = lsi.Transformer(file_name, 8)
RuntimeError: [CUDA][ERROR] /tmp/build-via-sdist-lcobild4/lightseq-2.1.4/lightseq/inference/model/decoder.cc.cu(205): invalid device function

could you please help me take a look? thank you!

VincentChen95 commented 2 years ago

my configuration: lightseq 2.1.4 ninja 1.10.2.1 torch 1.7.1 torchaudio 0.7.0a0+a853dff torchvision 0.8.2

VincentChen95 commented 2 years ago

+-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+

Taka152 commented 2 years ago

The pre-build pypi version only supports 61 70 and 75 arch. Since p100 is 60, you could change this line https://github.com/bytedance/lightseq/blob/4d370305656fe602f8628fc3bc8aa7b81464075b/CMakeLists.txt#L4 and build from source.

VincentChen95 commented 2 years ago

hello @Taka152 thank you for your reply how could i build from source? do you mean that git clone first, make the change, then do "pip install -e ."? thank you

Taka152 commented 2 years ago

Yes, you can follow this doc

bytedance / lightseq

Trained a transformer model ,but cannot infer (CUDA ERROR): invalid device function #194