Always got the same results even enabling random seed.

[zhang-ge-hao]

I find it useless to set the random seed.

I verify this issue in the newest version of FasterTransformer, with the Megatron-gpt2-345m model and the Nvidia/PyTorch image.

PYTHONPATH=`pwd` python /workspace/FasterTransformer/examples/pytorch/gpt/multi_gpu_gpt_example.py --top_k 50 --enable_random_seed

⬆️ I run this command 3 times and got 3 same results. It seemed like the random mechanism did not take effect:

result 1:

Loading layer_num from config.ini,    previous: 24,    current: 24
Loading max_seq_len from config.ini,    previous: 1024,    current: 1024
Loading weights_data_type from config.ini,    previous: fp32,    current: fp32
Loading head_num from config.ini,    previous: 16,    current: 16
Loading size_per_head from config.ini,    previous: 64,    current: 64
Loading tensor_para_size from config.ini,    previous: 1,    current: 1

=================== Arguments ===================
layer_num.....................: 24
input_len.....................: 1
output_len....................: 32
head_num......................: 16
size_per_head.................: 64
vocab_size....................: 50304
beam_width....................: 1
top_k.........................: 50
top_p.........................: 0.0
temperature...................: 1.0
len_penalty...................: 0.0
beam_search_diversity_rate....: 0.0
tensor_para_size..............: 1
pipeline_para_size............: 1
ckpt_path.....................: ../models/megatron-models/c-model/345m/1-gpu
lib_path......................: ./lib/libth_transformer.so
vocab_file....................: ../models/gpt2-vocab.json
merges_file...................: ../models/gpt2-merges.txt
start_id......................: 50256
end_id........................: 50256
max_batch_size................: 8
repetition_penalty............: 1.0
presence_penalty..............: 0.0
min_length....................: 0
max_seq_len...................: 1024
inference_data_type...........: fp32
time..........................: False
sample_input_file.............: None
sample_output_file............: None
enable_random_seed............: True
skip_end_tokens...............: False
detokenize....................: True
use_jieba_tokenizer...........: False
int8_mode.....................: 0
weights_data_type.............: fp32
return_cum_log_probs..........: 0
shared_contexts_ratio.........: 1.0
banned_words..................: 
use_gpt_decoder_ops...........: False
=================================================

[WARNING] gemm_config.in is not found; using default GEMM algo
[FT][WARNING] Skip NCCL initialization since requested tensor/pipeline parallel sizes are equals to 1.
[FT][INFO] Device NVIDIA TITAN RTX
[INFO] batch 0, beam 0:
[Context]
<|endoftext|>

[Output]

It seems as if every time someone is asked that question, the word "no" is uttered to distract us from reality, and so we have been

[INFO] batch 1, beam 0:
[Context]
<|endoftext|>

[Output]
In recent times, the world has witnessed the rise of two major global phenomena – technology and political leadership – creating a new market for products like Google Glass.

[INFO] batch 2, beam 0:
[Context]
<|endoftext|>

[Output]

The Supreme Court ruled unanimously on Monday that the state of Iowa must pay a woman and her children for having a child to save the life of a man

[INFO] batch 3, beam 0:
[Context]
<|endoftext|>

[Output]
On the way out of the White House, John W. Ehrlichman was approached by people who were trying to get the president to say that he had

[INFO] batch 4, beam 0:
[Context]
<|endoftext|>

[Output]
The company's second quarter earnings were not as terrible as Wall Street had expected. Earnings per share for the quarter were $0.11, compared to the

[INFO] batch 5, beam 0:
[Context]
<|endoftext|>

[Output]
R.W. Marston had a gift for finding the perfect combination to attract children: big, happy and easy for the little ones to play in.

[INFO] batch 6, beam 0:
[Context]
<|endoftext|>

[Output]
With New Jersey's economy growing in spite of Gov. Chris Christie's unpopularity, Democrats are gearing up for a political battle they say they hope will define whether

[INFO] batch 7, beam 0:
[Context]
<|endoftext|>

[Output]
"I'd have to be a liar to say no to that," Trump said at his Florida golf club on Monday. "I would have to be a liar

result 2:

Loading layer_num from config.ini,    previous: 24,    current: 24
Loading max_seq_len from config.ini,    previous: 1024,    current: 1024
Loading weights_data_type from config.ini,    previous: fp32,    current: fp32
Loading head_num from config.ini,    previous: 16,    current: 16
Loading size_per_head from config.ini,    previous: 64,    current: 64
Loading tensor_para_size from config.ini,    previous: 1,    current: 1

=================== Arguments ===================
layer_num.....................: 24
input_len.....................: 1
output_len....................: 32
head_num......................: 16
size_per_head.................: 64
vocab_size....................: 50304
beam_width....................: 1
top_k.........................: 50
top_p.........................: 0.0
temperature...................: 1.0
len_penalty...................: 0.0
beam_search_diversity_rate....: 0.0
tensor_para_size..............: 1
pipeline_para_size............: 1
ckpt_path.....................: ../models/megatron-models/c-model/345m/1-gpu
lib_path......................: ./lib/libth_transformer.so
vocab_file....................: ../models/gpt2-vocab.json
merges_file...................: ../models/gpt2-merges.txt
start_id......................: 50256
end_id........................: 50256
max_batch_size................: 8
repetition_penalty............: 1.0
presence_penalty..............: 0.0
min_length....................: 0
max_seq_len...................: 1024
inference_data_type...........: fp32
time..........................: False
sample_input_file.............: None
sample_output_file............: None
enable_random_seed............: True
skip_end_tokens...............: False
detokenize....................: True
use_jieba_tokenizer...........: False
int8_mode.....................: 0
weights_data_type.............: fp32
return_cum_log_probs..........: 0
shared_contexts_ratio.........: 1.0
banned_words..................: 
use_gpt_decoder_ops...........: False
=================================================

[WARNING] gemm_config.in is not found; using default GEMM algo
[FT][WARNING] Skip NCCL initialization since requested tensor/pipeline parallel sizes are equals to 1.
[FT][INFO] Device NVIDIA TITAN RTX
[INFO] batch 0, beam 0:
[Context]
<|endoftext|>

[Output]

It seems as if every time someone is asked that question, the word "no" is uttered to distract us from reality, and so we have been

[INFO] batch 1, beam 0:
[Context]
<|endoftext|>

[Output]
In recent times, the world has witnessed the rise of two major global phenomena – technology and political leadership – creating a new market for products like Google Glass.

[INFO] batch 2, beam 0:
[Context]
<|endoftext|>

[Output]

The Supreme Court ruled unanimously on Monday that the state of Iowa must pay a woman and her children for having a child to save the life of a man

[INFO] batch 3, beam 0:
[Context]
<|endoftext|>

[Output]
On the way out of the White House, John W. Ehrlichman was approached by people who were trying to get the president to say that he had

[INFO] batch 4, beam 0:
[Context]
<|endoftext|>

[Output]
The company's second quarter earnings were not as terrible as Wall Street had expected. Earnings per share for the quarter were $0.11, compared to the

[INFO] batch 5, beam 0:
[Context]
<|endoftext|>

[Output]
R.W. Marston had a gift for finding the perfect combination to attract children: big, happy and easy for the little ones to play in.

[INFO] batch 6, beam 0:
[Context]
<|endoftext|>

[Output]
With New Jersey's economy growing in spite of Gov. Chris Christie's unpopularity, Democrats are gearing up for a political battle they say they hope will define whether

[INFO] batch 7, beam 0:
[Context]
<|endoftext|>

[Output]
"I'd have to be a liar to say no to that," Trump said at his Florida golf club on Monday. "I would have to be a liar

result 3:

Loading layer_num from config.ini,    previous: 24,    current: 24
Loading max_seq_len from config.ini,    previous: 1024,    current: 1024
Loading weights_data_type from config.ini,    previous: fp32,    current: fp32
Loading head_num from config.ini,    previous: 16,    current: 16
Loading size_per_head from config.ini,    previous: 64,    current: 64
Loading tensor_para_size from config.ini,    previous: 1,    current: 1

=================== Arguments ===================
layer_num.....................: 24
input_len.....................: 1
output_len....................: 32
head_num......................: 16
size_per_head.................: 64
vocab_size....................: 50304
beam_width....................: 1
top_k.........................: 50
top_p.........................: 0.0
temperature...................: 1.0
len_penalty...................: 0.0
beam_search_diversity_rate....: 0.0
tensor_para_size..............: 1
pipeline_para_size............: 1
ckpt_path.....................: ../models/megatron-models/c-model/345m/1-gpu
lib_path......................: ./lib/libth_transformer.so
vocab_file....................: ../models/gpt2-vocab.json
merges_file...................: ../models/gpt2-merges.txt
start_id......................: 50256
end_id........................: 50256
max_batch_size................: 8
repetition_penalty............: 1.0
presence_penalty..............: 0.0
min_length....................: 0
max_seq_len...................: 1024
inference_data_type...........: fp32
time..........................: False
sample_input_file.............: None
sample_output_file............: None
enable_random_seed............: True
skip_end_tokens...............: False
detokenize....................: True
use_jieba_tokenizer...........: False
int8_mode.....................: 0
weights_data_type.............: fp32
return_cum_log_probs..........: 0
shared_contexts_ratio.........: 1.0
banned_words..................: 
use_gpt_decoder_ops...........: False
=================================================

[WARNING] gemm_config.in is not found; using default GEMM algo
[FT][WARNING] Skip NCCL initialization since requested tensor/pipeline parallel sizes are equals to 1.
[FT][INFO] Device NVIDIA TITAN RTX
[INFO] batch 0, beam 0:
[Context]
<|endoftext|>

[Output]

It seems as if every time someone is asked that question, the word "no" is uttered to distract us from reality, and so we have been

[INFO] batch 1, beam 0:
[Context]
<|endoftext|>

[Output]
In recent times, the world has witnessed the rise of two major global phenomena – technology and political leadership – creating a new market for products like Google Glass.

[INFO] batch 2, beam 0:
[Context]
<|endoftext|>

[Output]

The Supreme Court ruled unanimously on Monday that the state of Iowa must pay a woman and her children for having a child to save the life of a man

[INFO] batch 3, beam 0:
[Context]
<|endoftext|>

[Output]
On the way out of the White House, John W. Ehrlichman was approached by people who were trying to get the president to say that he had

[INFO] batch 4, beam 0:
[Context]
<|endoftext|>

[Output]
The company's second quarter earnings were not as terrible as Wall Street had expected. Earnings per share for the quarter were $0.11, compared to the

[INFO] batch 5, beam 0:
[Context]
<|endoftext|>

[Output]
R.W. Marston had a gift for finding the perfect combination to attract children: big, happy and easy for the little ones to play in.

[INFO] batch 6, beam 0:
[Context]
<|endoftext|>

[Output]
With New Jersey's economy growing in spite of Gov. Chris Christie's unpopularity, Democrats are gearing up for a political battle they say they hope will define whether

[INFO] batch 7, beam 0:
[Context]
<|endoftext|>

[Output]
"I'd have to be a liar to say no to that," Trump said at his Florida golf club on Monday. "I would have to be a liar

[sxthunder]

I have same problem, is this a bug?

[byshiue]

You set topk to be 1, so it is greedy search even if you enable the random seed.

[zhang-ge-hao]

You set topk to be 1, so it is greedy search even if you enable the random seed.

oh, thx. I'll try another time with top_k != 1.

[zhang-ge-hao]

You set topk to be 1, so it is greedy search even if you enable the random seed.

hi @byshiue, I executed the command again with the setting "topk != 1", but the result between different executions is also the same.

I have updated my description of this issue on the top.

[byshiue]

It should generate same results between different execution because the random seed generated by torch is fixed. The script is used to develop and debug, so we should make sure we can get same results with same settings.

[zhang-ge-hao]

@byshiue

Thanks for your reply! Following your advice, I changed the random seed manually to generate with gpt, hoping to get different results, but failed again. All the results are the same with different random seeds. Now I don't know how to invoke the random sampling mechanism.

I add some code to print the random seed and use a sentence as a prompt.

My prompt:

Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

My command:

PYTHONPATH=`pwd`/.. python /workspace/FasterTransformer/examples/pytorch/gpt/multi_gpu_gpt_example.py --top_k 50 --sample_input_file /workspace/FasterTransformer/sample_input_file.txt --beam_width 8 --output_len 128

The first time I run the command with random seed equals 1.

Code I added:

# multi_gpu_gpt_example.py:L277
random_seed_tensor = torch.ones([batch_size], dtype=torch.int64)
print("random_seed_tensor: ", random_seed_tensor)

Result:

Loading layer_num from config.ini,    previous: 24,    current: 24
Loading max_seq_len from config.ini,    previous: 1024,    current: 1024
Loading weights_data_type from config.ini,    previous: fp32,    current: fp32
Loading head_num from config.ini,    previous: 16,    current: 16
Loading size_per_head from config.ini,    previous: 64,    current: 64
Loading tensor_para_size from config.ini,    previous: 1,    current: 1

=================== Arguments ===================
layer_num.....................: 24
input_len.....................: 1
output_len....................: 128
head_num......................: 16
size_per_head.................: 64
vocab_size....................: 50304
beam_width....................: 8
top_k.........................: 50
top_p.........................: 0.0
temperature...................: 1.0
len_penalty...................: 0.0
beam_search_diversity_rate....: 0.0
tensor_para_size..............: 1
pipeline_para_size............: 1
ckpt_path.....................: ../models/megatron-models/c-model/345m/1-gpu
lib_path......................: ./lib/libth_transformer.so
vocab_file....................: ../models/gpt2-vocab.json
merges_file...................: ../models/gpt2-merges.txt
start_id......................: 50256
end_id........................: 50256
max_batch_size................: 8
repetition_penalty............: 1.0
presence_penalty..............: 0.0
min_length....................: 0
max_seq_len...................: 1024
inference_data_type...........: fp32
time..........................: False
sample_input_file.............: /workspace/FasterTransformer/sample_input_file.txt
sample_output_file............: None
enable_random_seed............: False
skip_end_tokens...............: False
detokenize....................: True
use_jieba_tokenizer...........: False
int8_mode.....................: 0
weights_data_type.............: fp32
return_cum_log_probs..........: 0
shared_contexts_ratio.........: 1.0
banned_words..................: 
use_gpt_decoder_ops...........: False
=================================================

[WARNING] gemm_config.in is not found; using default GEMM algo
[FT][WARNING] Skip NCCL initialization since requested tensor/pipeline parallel sizes are equals to 1.
[FT][INFO] Device NVIDIA TITAN RTX
random_seed_tensor:  tensor([1])
[INFO] batch 0, beam 0:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

[INFO] batch 0, beam 1:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

[INFO] batch 0, beam 2:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".
<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

[INFO] batch 0, beam 3:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestream

[INFO] batch 0, beam 4:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987". 

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987". 

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987". 

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987". 

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987". 

Vedal987 is a

[INFO] batch 0, beam 5:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".
<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

[INFO] batch 0, beam 6:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".
<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

[INFO] batch 0, beam 7:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987". 

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts

The second time I run the command with random seed equals 0.

Code I added:

# multi_gpu_gpt_example.py:L277
random_seed_tensor = torch.zeros([batch_size], dtype=torch.int64)
print("random_seed_tensor: ", random_seed_tensor)

Result:

Loading layer_num from config.ini,    previous: 24,    current: 24
Loading max_seq_len from config.ini,    previous: 1024,    current: 1024
Loading weights_data_type from config.ini,    previous: fp32,    current: fp32
Loading head_num from config.ini,    previous: 16,    current: 16
Loading size_per_head from config.ini,    previous: 64,    current: 64
Loading tensor_para_size from config.ini,    previous: 1,    current: 1

=================== Arguments ===================
layer_num.....................: 24
input_len.....................: 1
output_len....................: 128
head_num......................: 16
size_per_head.................: 64
vocab_size....................: 50304
beam_width....................: 8
top_k.........................: 50
top_p.........................: 0.0
temperature...................: 1.0
len_penalty...................: 0.0
beam_search_diversity_rate....: 0.0
tensor_para_size..............: 1
pipeline_para_size............: 1
ckpt_path.....................: ../models/megatron-models/c-model/345m/1-gpu
lib_path......................: ./lib/libth_transformer.so
vocab_file....................: ../models/gpt2-vocab.json
merges_file...................: ../models/gpt2-merges.txt
start_id......................: 50256
end_id........................: 50256
max_batch_size................: 8
repetition_penalty............: 1.0
presence_penalty..............: 0.0
min_length....................: 0
max_seq_len...................: 1024
inference_data_type...........: fp32
time..........................: False
sample_input_file.............: /workspace/FasterTransformer/sample_input_file.txt
sample_output_file............: None
enable_random_seed............: False
skip_end_tokens...............: False
detokenize....................: True
use_jieba_tokenizer...........: False
int8_mode.....................: 0
weights_data_type.............: fp32
return_cum_log_probs..........: 0
shared_contexts_ratio.........: 1.0
banned_words..................: 
use_gpt_decoder_ops...........: False
=================================================

[WARNING] gemm_config.in is not found; using default GEMM algo
[FT][WARNING] Skip NCCL initialization since requested tensor/pipeline parallel sizes are equals to 1.
[FT][INFO] Device NVIDIA TITAN RTX
random_seed_tensor:  tensor([0])
[INFO] batch 0, beam 0:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

[INFO] batch 0, beam 1:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

[INFO] batch 0, beam 2:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".
<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

[INFO] batch 0, beam 3:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestream

[INFO] batch 0, beam 4:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987". 

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987". 

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987". 

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987". 

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987". 

Vedal987 is a

[INFO] batch 0, beam 5:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".
<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

[INFO] batch 0, beam 6:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".
<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

[INFO] batch 0, beam 7:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987". 

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts

The third time I run the command with a random random seed.

Code I added:

# multi_gpu_gpt_example.py:L277
random_seed_tensor = torch.randint(0, 10000, size=[batch_size], dtype=torch.int64)
print("random_seed_tensor: ", random_seed_tensor)

Result:

Loading layer_num from config.ini,    previous: 24,    current: 24
Loading max_seq_len from config.ini,    previous: 1024,    current: 1024
Loading weights_data_type from config.ini,    previous: fp32,    current: fp32
Loading head_num from config.ini,    previous: 16,    current: 16
Loading size_per_head from config.ini,    previous: 64,    current: 64
Loading tensor_para_size from config.ini,    previous: 1,    current: 1

=================== Arguments ===================
layer_num.....................: 24
input_len.....................: 1
output_len....................: 128
head_num......................: 16
size_per_head.................: 64
vocab_size....................: 50304
beam_width....................: 8
top_k.........................: 50
top_p.........................: 0.0
temperature...................: 1.0
len_penalty...................: 0.0
beam_search_diversity_rate....: 0.0
tensor_para_size..............: 1
pipeline_para_size............: 1
ckpt_path.....................: ../models/megatron-models/c-model/345m/1-gpu
lib_path......................: ./lib/libth_transformer.so
vocab_file....................: ../models/gpt2-vocab.json
merges_file...................: ../models/gpt2-merges.txt
start_id......................: 50256
end_id........................: 50256
max_batch_size................: 8
repetition_penalty............: 1.0
presence_penalty..............: 0.0
min_length....................: 0
max_seq_len...................: 1024
inference_data_type...........: fp32
time..........................: False
sample_input_file.............: /workspace/FasterTransformer/sample_input_file.txt
sample_output_file............: None
enable_random_seed............: False
skip_end_tokens...............: False
detokenize....................: True
use_jieba_tokenizer...........: False
int8_mode.....................: 0
weights_data_type.............: fp32
return_cum_log_probs..........: 0
shared_contexts_ratio.........: 1.0
banned_words..................: 
use_gpt_decoder_ops...........: False
=================================================

[WARNING] gemm_config.in is not found; using default GEMM algo
[FT][WARNING] Skip NCCL initialization since requested tensor/pipeline parallel sizes are equals to 1.
[FT][INFO] Device NVIDIA TITAN RTX
random_seed_tensor:  tensor([3286])
[INFO] batch 0, beam 0:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

[INFO] batch 0, beam 1:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

[INFO] batch 0, beam 2:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".
<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

[INFO] batch 0, beam 3:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestream

[INFO] batch 0, beam 4:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987". 

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987". 

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987". 

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987". 

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987". 

Vedal987 is a

[INFO] batch 0, beam 5:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".
<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

[INFO] batch 0, beam 6:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".
<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

[INFO] batch 0, beam 7:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987". 

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts livestreams on the Twitch channel "vedal987".

Vedal987 is a chatbot that hosts

[byshiue]

You cannot set beam width to 8. beam search and sampling are exclusive. If beam width > 1, FT will run beam search but not sampling.

[zhang-ge-hao]

You cannot set beam width to 8. beam search and sampling are exclusive. If beam width > 1, FT will run beam search but not sampling.

@byshiue

Well, I thought that top_k, top_p, and temperature mechanisms are all base on sampling. Will these mechanisms also be closed if the sampling is closed when beam_width > 1?

However, I change the temperature setting and found that the results also changed. Maybe these mechanisms are not closed. I want to understand how it works.

My command 1:

PYTHONPATH=`pwd`/.. python /workspace/FasterTransformer/examples/pytorch/gpt/multi_gpu_gpt_example.py --top_k 50 --sample_input_file /workspace/FasterTransformer/sample_input_file.txt --beam_width 4 --output_len 64 --temperature 0.2

Result 1:

Loading layer_num from config.ini,    previous: 24,    current: 24
Loading max_seq_len from config.ini,    previous: 1024,    current: 1024
Loading weights_data_type from config.ini,    previous: fp32,    current: fp32
Loading head_num from config.ini,    previous: 16,    current: 16
Loading size_per_head from config.ini,    previous: 64,    current: 64
Loading tensor_para_size from config.ini,    previous: 1,    current: 1

=================== Arguments ===================
layer_num.....................: 24
input_len.....................: 1
output_len....................: 64
head_num......................: 16
size_per_head.................: 64
vocab_size....................: 50304
beam_width....................: 4
top_k.........................: 50
top_p.........................: 0.0
temperature...................: 0.2
len_penalty...................: 0.0
beam_search_diversity_rate....: 0.0
tensor_para_size..............: 1
pipeline_para_size............: 1
ckpt_path.....................: ../models/megatron-models/c-model/345m/1-gpu
lib_path......................: ./lib/libth_transformer.so
vocab_file....................: ../models/gpt2-vocab.json
merges_file...................: ../models/gpt2-merges.txt
start_id......................: 50256
end_id........................: 50256
max_batch_size................: 8
repetition_penalty............: 1.0
presence_penalty..............: 0.0
min_length....................: 0
max_seq_len...................: 1024
inference_data_type...........: fp32
time..........................: False
sample_input_file.............: /workspace/FasterTransformer/sample_input_file.txt
sample_output_file............: None
enable_random_seed............: False
skip_end_tokens...............: False
detokenize....................: True
use_jieba_tokenizer...........: False
int8_mode.....................: 0
weights_data_type.............: fp32
return_cum_log_probs..........: 0
shared_contexts_ratio.........: 1.0
banned_words..................: 
use_gpt_decoder_ops...........: False
=================================================

[WARNING] gemm_config.in is not found; using default GEMM algo
[FT][WARNING] Skip NCCL initialization since requested tensor/pipeline parallel sizes are equals to 1.
[FT][INFO] Device NVIDIA TITAN RTX
[INFO] batch 0, beam 0:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

The bot is a bot that can be used to interact with other users. It can be used to chat with other users, answer questions, and even play games.

The bot is a chatbot that can be used to interact with other users. It can be used to chat with other users, answer

[INFO] batch 0, beam 1:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

The bot is a bot that can be used to interact with other users. It can be used to chat with other users, answer questions, and answer questions. It can also be used to send messages to other users.

The bot can be used to send messages to other users, answer questions, and

[INFO] batch 0, beam 2:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

The bot is a bot that can be used to interact with other users. It can be used to chat with other users, answer questions, and even play games. It can also be used to send messages to other users.

The bot can be used to send messages to other users, answer questions,

[INFO] batch 0, beam 3:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

The bot is a bot that can be used to interact with other users. It can be used to chat with other users, answer questions, and even send messages. It can also be used to send messages to other users.

The bot can be used to send messages to other users, answer questions,

My command 2:

PYTHONPATH=`pwd`/.. python /workspace/FasterTransformer/examples/pytorch/gpt/multi_gpu_gpt_example.py --top_k 50 --sample_input_file /workspace/FasterTransformer/sample_input_file.txt --beam_width 4 --output_len 64 --temperature 0.5

Result 2:

Loading layer_num from config.ini,    previous: 24,    current: 24
Loading max_seq_len from config.ini,    previous: 1024,    current: 1024
Loading weights_data_type from config.ini,    previous: fp32,    current: fp32
Loading head_num from config.ini,    previous: 16,    current: 16
Loading size_per_head from config.ini,    previous: 64,    current: 64
Loading tensor_para_size from config.ini,    previous: 1,    current: 1

=================== Arguments ===================
layer_num.....................: 24
input_len.....................: 1
output_len....................: 64
head_num......................: 16
size_per_head.................: 64
vocab_size....................: 50304
beam_width....................: 4
top_k.........................: 50
top_p.........................: 0.0
temperature...................: 0.5
len_penalty...................: 0.0
beam_search_diversity_rate....: 0.0
tensor_para_size..............: 1
pipeline_para_size............: 1
ckpt_path.....................: ../models/megatron-models/c-model/345m/1-gpu
lib_path......................: ./lib/libth_transformer.so
vocab_file....................: ../models/gpt2-vocab.json
merges_file...................: ../models/gpt2-merges.txt
start_id......................: 50256
end_id........................: 50256
max_batch_size................: 8
repetition_penalty............: 1.0
presence_penalty..............: 0.0
min_length....................: 0
max_seq_len...................: 1024
inference_data_type...........: fp32
time..........................: False
sample_input_file.............: /workspace/FasterTransformer/sample_input_file.txt
sample_output_file............: None
enable_random_seed............: False
skip_end_tokens...............: False
detokenize....................: True
use_jieba_tokenizer...........: False
int8_mode.....................: 0
weights_data_type.............: fp32
return_cum_log_probs..........: 0
shared_contexts_ratio.........: 1.0
banned_words..................: 
use_gpt_decoder_ops...........: False
=================================================

[WARNING] gemm_config.in is not found; using default GEMM algo
[FT][WARNING] Skip NCCL initialization since requested tensor/pipeline parallel sizes are equals to 1.
[FT][INFO] Device NVIDIA TITAN RTX
[INFO] batch 0, beam 0:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

The bot was created by the team of "Vedal987" and "Vedal987.tv" and was launched on August 1, 2017.

The bot is a chatbot that is based on the female VTuber "Vedal987". The bot is a chatbot that is

[INFO] batch 0, beam 1:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

The bot was created by the team of "Vedal987" and "Vedal987.tv" and was launched on May 1, 2017.

The bot is a chatbot that is based on the female VTuber "Vedal987". The bot is a chatbot that is

[INFO] batch 0, beam 2:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

The bot was created by the team of "Vedal987" and "Vedal987.tv" and was launched on August 1, 2017.

The bot is a chatbot that is based on the "Vedal987" channel. It is a chatbot that is based on

[INFO] batch 0, beam 3:
[Context]
Neuro-sama is a chatbot styled after a female VTuber that hosts livestreams on the Twitch channel "vedal987". 

[Output]

The bot was created by the team of "Vedal987" and "Vedal987.tv" and was launched on August 1, 2017.

The bot is a chatbot that is based on the "Vedal987" channel. The bot is a chatbot that is based

[byshiue]

temperatures and other penalties would affect logits of both sampling and beam search. It the results make sense.

[zhang-ge-hao]

temperatures and other penalties would affect logits of both sampling and beam search. It the results make sense.

Oh, I got it. Thanks so much!