ggerganov / llama.cpp

LLM inference in C/C++
MIT License
65.3k stars 9.36k forks source link

Bug: Batched inference generates corrupted repeatative texts #8833

Closed hafezmg48 closed 2 weeks ago

hafezmg48 commented 1 month ago

What happened?

I am trying to run the llama-batched. It worked fine for small text sizes and small number of batches. But when having large batch numbers, after a certain amount of correct tokens, the sequences start to repeat themselves. In my example, I tried, -np 50 to have 50 batches and -n 251 to generate 251 tokens. I gave a single shared prompt "One of the important historic" to all of them. The tested model is llama3.1-8B and using CUDA gpu.

I tried to be careful with the context size and set --ctx-size 12800 to make sure there is enough shared kv cache for all batches, but it didnt help and many of the sequences are corrupted repetitions. For example:

sequence 47: One of the important historic buildings of the city, the building was built in 1896 and was used as a primary school. The building was restored in 2002 and used as a cultural center. One of the important historic buildings of the city, the building was built in 1896 and was used as a primary school. The building was restored in 2002 and used as a cultural center. The building was built in 1896 and was used as a primary school. The building was restored in 2002 and used as a cultural center. The building was built in 1896 and was used as a primary school. The building was restored in 2002 and used as a cultural center. The building was built in 1896 and was used as a primary school. The building was restored in 2002 and used as a cultural center.

or

sequence 30: One of the important historic buildings in the city of Qom is the shrine of Imamzadeh Abdollah. This shrine is located in the city of Qom and is a historic building of the Safavid era. The shrine of Imamzadeh Abdollah is located in the city of Qom and is one of the most important pilgrimage places for the people of Qom and the surrounding areas. The shrine of Imamzadeh Abdollah is located in the city of Qom and is one of the most important pilgrimage places for the people of Qom and the surrounding areas. The shrine of Imamzadeh Abdollah is located in the city of Qom and is one of the most important pilgrimage places for the people of Qom and the surrounding areas. The shrine of Imamzadeh Abdollah is located in the city of Qom and is one of the most important pilgrimage places for the people of Qom and the surrounding areas. The shrine of Imamzadeh Abdollah is located in the city of Qom and is one of the most important pilgrimage places for the people of Qom and the surrounding areas. The shrine of Imamzadeh Abdollah is located in the city of Qom and is one of

below I also provided the full log of the run.

Name and Version

llama-batched -m ./llama3.1-8B-F16.gguf -p "One of the important historic" -np 50 -ngl 99 --ctx-size 12800 -n 251

What operating system are you seeing the problem on?

Linux

Relevant log output

llama_model_loader: loaded meta data with 27 key-value pairs and 291 tensors from ./llama3.1-8B-F16.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Llama3.1 8b
llama_model_loader: - kv   3:                           general.basename str              = llama3.1
llama_model_loader: - kv   4:                         general.size_label str              = 8B
llama_model_loader: - kv   5:                            general.license str              = llama3
llama_model_loader: - kv   6:                               general.tags arr[str,6]       = ["facebook", "meta", "pytorch", "llam...
llama_model_loader: - kv   7:                          general.languages arr[str,1]       = ["en"]
llama_model_loader: - kv   8:                          llama.block_count u32              = 32
llama_model_loader: - kv   9:                       llama.context_length u32              = 8192
llama_model_loader: - kv  10:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv  11:                  llama.feed_forward_length u32              = 14336
llama_model_loader: - kv  12:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv  13:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  14:                       llama.rope.freq_base f32              = 500000.000000
llama_model_loader: - kv  15:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  16:                          general.file_type u32              = 1
llama_model_loader: - kv  17:                           llama.vocab_size u32              = 128256
llama_model_loader: - kv  18:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  19:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  20:                         tokenizer.ggml.pre str              = llama-bpe
llama_model_loader: - kv  21:                      tokenizer.ggml.tokens arr[str,128256]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  22:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  23:                      tokenizer.ggml.merges arr[str,280147]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  24:                tokenizer.ggml.bos_token_id u32              = 128000
llama_model_loader: - kv  25:                tokenizer.ggml.eos_token_id u32              = 128001
llama_model_loader: - kv  26:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type  f16:  226 tensors
llm_load_vocab: special tokens cache size = 256
llm_load_vocab: token to piece cache size = 0.8000 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 128256
llm_load_print_meta: n_merges         = 280147
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 8192
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 4
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 14336
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 500000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 8192
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 8B
llm_load_print_meta: model ftype      = F16
llm_load_print_meta: model params     = 8.03 B
llm_load_print_meta: model size       = 14.96 GiB (16.00 BPW)
llm_load_print_meta: general.name     = Llama3.1 8b
llm_load_print_meta: BOS token        = 128000 '<|begin_of_text|>'
llm_load_print_meta: EOS token        = 128001 '<|end_of_text|>'
llm_load_print_meta: LF token         = 128 'Ä'
llm_load_print_meta: EOT token        = 128009 '<|eot_id|>'
llm_load_print_meta: max token length = 256
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: Tesla V100-PCIE-32GB, compute capability 7.0, VMM: yes
llm_load_tensors: ggml ctx size =    0.27 MiB
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors:        CPU buffer size =  1002.00 MiB
llm_load_tensors:      CUDA0 buffer size = 14315.02 MiB
.........................................................................................
llama_new_context_with_model: n_ctx      = 12256
llama_new_context_with_model: n_batch    = 251
llama_new_context_with_model: n_ubatch   = 251
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 500000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:      CUDA0 KV buffer size =  1532.00 MiB
llama_new_context_with_model: KV self size  = 1532.00 MiB, K (f16):  766.00 MiB, V (f16):  766.00 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =    24.46 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   403.18 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    15.89 MiB
llama_new_context_with_model: graph nodes  = 1030
llama_new_context_with_model: graph splits = 2

main: n_predict = 251, n_ctx = 12256, n_batch = 251, n_parallel = 50, n_kv_req = 12256

<|begin_of_text|>One of the important historic

main: generating 50 sequences ...

main: stream 32 finished at n_cur = 65
main: stream 15 finished at n_cur = 82
main: stream 48 finished at n_cur = 86
main: stream 29 finished at n_cur = 96
main: stream 11 finished at n_cur = 104
main: stream 31 finished at n_cur = 111
main: stream 20 finished at n_cur = 120
main: stream 47 finished at n_cur = 176
main: stream 5 finished at n_cur = 200
main: stream 33 finished at n_cur = 204
main: stream 18 finished at n_cur = 206
main: stream 9 finished at n_cur = 216
main: stream 23 finished at n_cur = 235
main: stream 12 finished at n_cur = 248
main: stream 0 finished at n_cur = 251
main: stream 1 finished at n_cur = 251
main: stream 2 finished at n_cur = 251
main: stream 3 finished at n_cur = 251
main: stream 4 finished at n_cur = 251
main: stream 6 finished at n_cur = 251
main: stream 7 finished at n_cur = 251
main: stream 8 finished at n_cur = 251
main: stream 10 finished at n_cur = 251
main: stream 13 finished at n_cur = 251
main: stream 14 finished at n_cur = 251
main: stream 16 finished at n_cur = 251
main: stream 17 finished at n_cur = 251
main: stream 19 finished at n_cur = 251
main: stream 21 finished at n_cur = 251
main: stream 22 finished at n_cur = 251
main: stream 24 finished at n_cur = 251
main: stream 25 finished at n_cur = 251
main: stream 26 finished at n_cur = 251
main: stream 27 finished at n_cur = 251
main: stream 28 finished at n_cur = 251
main: stream 30 finished at n_cur = 251
main: stream 34 finished at n_cur = 251
main: stream 35 finished at n_cur = 251
main: stream 36 finished at n_cur = 251
main: stream 37 finished at n_cur = 251
main: stream 38 finished at n_cur = 251
main: stream 39 finished at n_cur = 251
main: stream 40 finished at n_cur = 251
main: stream 41 finished at n_cur = 251
main: stream 42 finished at n_cur = 251
main: stream 43 finished at n_cur = 251
main: stream 44 finished at n_cur = 251
main: stream 45 finished at n_cur = 251
main: stream 46 finished at n_cur = 251
main: stream 49 finished at n_cur = 251

sequence 0:

One of the important historic buildings in the city of Kerman is the Shah Nematollah Vali Mausoleum. This building is located in Mahan, 25 km. from Kerman. The building was built in 1954 and the dome was added to it in 1963. The building is a square shape with a height of 40 meters and a diameter of 25 meters. The building is built of brick and plaster and has a dome and a minaret. The building has a dome with a diameter of 16 meters and a height of 40 meters. The building has a dome and a minaret. The building has a dome with a diameter of 16 meters and a height of 40 meters. The building has a dome and a minaret. The building has a dome with a diameter of 16 meters and a height of 40 meters. The building has a dome and a minaret. The building has a dome with a diameter of 16 meters and a height of 40 meters. The building has a dome and a minaret. The building has a dome with a diameter of 16 meters and a height of 40 meters. The building has a dome and a minaret. The

sequence 1:

One of the important historic sites in the city of Isfahan is the historical house of the Qavam family. This house is located in the center of Isfahan city and is one of the important historical houses of Isfahan. The Qavam family was one of the most important families in the Qajar period. The Qavam family was one of the most prominent and influential families in the Qajar period. The Qavam family was one of the most prominent and influential families in the Qajar period. The Qavam family was one of the most prominent and influential families in the Qajar period. The Qavam family was one of the most prominent and influential families in the Qajar period. The Qavam family was one of the most prominent and influential families in the Qajar period. The Qavam family was one of the most prominent and influential families in the Qajar period. The Qavam family was one of the most prominent and influential families in the Qajar period. The Qavam family was one of the most prominent and influential families in the Qajar period. The Qavam family was one of the most prominent and influential families in the Qajar period. The Qav

sequence 2:

One of the important historic sites in the city of Isfahan is the tomb of Sheikh Bahai, which is located in the neighborhood of Sheikh Bahai. The tomb of Sheikh Bahai is located in the neighborhood of Sheikh Bahai in the city of Isfahan. This historic site is located in the neighborhood of Sheikh Bahai in the city of Isfahan. Sheikh Bahai is one of the most famous figures of the Safavid era. Sheikh Bahai was a student of Mir Damad and was the first person to teach philosophy and theology in Isfahan. Sheikh Bahai is one of the most important figures of the Safavid period. Sheikh Bahai was a student of Mir Damad and was the first person to teach philosophy and theology in Isfahan. Sheikh Bahai is one of the most important figures of the Safavid period. Sheikh Bahai was a student of Mir Damad and was the first person to teach philosophy and theology in Isfahan.
The tomb of Sheikh Bahai is located in the neighborhood of Sheikh Bahai in the city of Isfahan. This historic site is located in the neighborhood of Sheikh Bahai in the city of Isfahan. Sheikh Bahai is one of the most famous

sequence 3:

One of the important historic buildings in the city, the old Town Hall was built in 1905. The building was originally designed by the architect A. A. Gassner and built by the builder O. R. Kellner. The building was built in the style of the Vienna Secession. The building was designed as a multi-purpose building for the city administration, the city library, the city archive and the city museum. The building was also designed to serve as a meeting place for the local community. The building was built in the style of the Vienna Secession, which was a style of architecture that was popular in the early 20th century. The style was characterized by its use of curved lines, bright colors, and decorative details. The building was designed to be a symbol of the city's prosperity and progress. The building was also designed to be a symbol of the city's cultural and artistic heritage. The building was built in the style of the Vienna Secession, which was a style of architecture that was popular in the early 20th century. The style was characterized by its use of curved lines, bright colors, and decorative details. The building was designed to be a symbol of the city's prosperity and progress.

sequence 4:

One of the important historic buildings in the city of Hama, the Great Mosque, which is located in the center of the city, was built by the Ayyubid sultan Al-Malik Al-Aziz Uthman bin Ali bin Ayyub in 1170 AD.
The Great Mosque is located in the center of the city of Hama and is one of the most important landmarks of the city. It is one of the most important historic buildings in the city of Hama, and the Great Mosque is located in the center of the city, and was built by the Ayyubid sultan Al-Malik Al-Aziz Uthman bin Ali bin Ayyub in 1170 AD. It is considered one of the most important and largest mosques in Syria, and it is considered one of the most important historic buildings in the city of Hama.
The Great Mosque is located in the center of the city of Hama, and it was built in the Islamic Ayyubid era, and it is considered one of the most important and largest mosques in Syria, and it is considered one of the most important historic buildings in the city of Hama. The Great Mosque is located in the center

sequence 5:

One of the important historic buildings in the city of Kermanshah is the "Khosrow Abad" caravanserai, which is located on the Kermanshah-Ilam road. It is a two-story building, with a courtyard in the middle. The entrance to the caravanserai is from the west. The building has been built on the slope of the hill and has a special view of the city. The building has been built in the shape of a rectangle. The courtyard of the caravanserai has been covered by a dome, which is a special feature of the caravanserai. The building has been built in the Qajar period and is one of the most beautiful caravanserais in Kermanshah.
The caravanserai of Khosrow Abad is located on the Kermanshah-Ilam road, 15 km from Kermanshah city.

sequence 6:

One of the important historic sites in the city of Jeddah, the famous old Jeddah gate, which is located in the center of the city, and is one of the most important tourist attractions in the city, as it is considered one of the most famous historical buildings in the city of Jeddah, and it is one of the most important tourist attractions in the city, which is visited by many tourists and visitors to the city of Jeddah, and the old Jeddah gate is one of the most important tourist attractions in the city of Jeddah, as it is one of the most famous historical buildings in the city, and it is one of the most famous historical buildings in the city of Jeddah, as it is one of the most famous historical buildings in the city of Jeddah, as it is one of the most famous historical buildings in the city of Jeddah, as it is one of the most famous historical buildings in the city of Jeddah, as it is one of the most famous historical buildings in the city of Jeddah, as it is one of the most famous historical buildings in the city of Jeddah, as it is one of the most famous historical buildings in

sequence 7:

One of the important historic sites in the city of Tehran is the old Tabriz bazaar, which is located in the heart of Tehran and is one of the most important places in the city of Tehran. The Tabriz bazaar of Tehran is located in the north of Tehran and is one of the most important places in the city of Tehran. The Tabriz bazaar of Tehran is located in the north of Tehran and is one of the most important places in the city of Tehran. The Tabriz bazaar of Tehran is located in the north of Tehran and is one of the most important places in the city of Tehran. The Tabriz bazaar of Tehran is located in the north of Tehran and is one of the most important places in the city of Tehran. The Tabriz bazaar of Tehran is located in the north of Tehran and is one of the most important places in the city of Tehran. The Tabriz bazaar of Tehran is located in the north of Tehran and is one of the most important places in the city of Tehran. The Tabriz bazaar of Tehran is located in the north of Tehran and is one of the most important places in the city of Tehran. The Tabriz bazaar of Tehran is located in the north of

sequence 8:

One of the important historic buildings of the city, the building that was built as a school in 1892, was used as a school until 1964. The building, which was abandoned for a while after the school was closed, was restored in 1993 and opened as a museum. The building, which was restored in the 1990s, is now one of the most important historical buildings of the city.
The building, which was built as a school in 1892, was used as a school until 1964. The building, which was abandoned for a while after the school was closed, was restored in 1993 and opened as a museum. The building, which was restored in the 1990s, is now one of the most important historical buildings of the city.
The building, which was built as a school in 1892, was used as a school until 1964. The building, which was abandoned for a while after the school was closed, was restored in 1993 and opened as a museum. The building, which was restored in the 1990s, is now one of the most important historical buildings of the city.
The building, which was built as a school in

sequence 9:

One of the important historic buildings in the city of Mardin is the Mardin Castle, which was built in 1515. The castle is located on a high hill in Mardin and has a very beautiful view. The castle was built by the Mardin Emirate, who ruled the city of Mardin in the 15th century. The castle is located in the center of the city and is surrounded by a wall. The castle has a large courtyard and several towers. The castle is now a museum and is open to the public. The castle is a popular tourist attraction and is a great place to learn about the history of Mardin.
The Mardin Castle is one of the most important historical buildings in the city of Mardin. The castle was built in 1515 and is located on a high hill in the city. The castle is surrounded by a wall and has a large courtyard. The castle is now a museum and is open to the public. The castle is a popular tourist attraction and is a great place to learn about the history of Mardin.

sequence 10:

One of the important historic sites in the city of Tehran is the historical building of the former Parliament, which is located in the center of the city and is one of the most important tourist attractions in the capital. The building of the former Parliament of Iran is located in the center of Tehran, and is one of the most important tourist attractions in the capital. This building is located in the center of Tehran and is one of the most important tourist attractions in the capital. This building is located in the center of Tehran and is one of the most important tourist attractions in the capital. This building is located in the center of Tehran and is one of the most important tourist attractions in the capital. This building is located in the center of Tehran and is one of the most important tourist attractions in the capital. This building is located in the center of Tehran and is one of the most important tourist attractions in the capital. This building is located in the center of Tehran and is one of the most important tourist attractions in the capital. This building is located in the center of Tehran and is one of the most important tourist attractions in the capital. This building is located in the center of Tehran and is one of the most important tourist attractions in the capital. This building is

sequence 11:

One of the important historic sites of the city, the Church of St. George is located in the heart of the city, on the main square. The church was built in 1818-1820 on the site of the former church of the same name. The church was built in the style of the late classicism. The building is a three-aisled basilica with a bell tower. The temple is decorated with a rich interior decoration. The church is one of the main attractions of the city.

sequence 12:

One of the important historic sites in the area is the old town of Kavala, which is located on the hill of Panagia. The old town is built on the ruins of the ancient city of Neapolis, which was founded in the 7th century BC. In the old town of Kavala, there are many monuments of history and architecture, such as the Byzantine castle, the ancient theater, the ancient agora, the ancient aqueduct, the ancient walls, the ancient baths, the ancient temple of Apollo and the ancient stadium.
The old town of Kavala is a great place to visit if you are interested in history and architecture. The old town is also a great place to take a walk and enjoy the views of the city and the sea.
The old town of Kavala is a great place to visit if you are interested in history and architecture. The old town is also a great place to take a walk and enjoy the views of the city and the sea.
The old town of Kavala is a great place to visit if you are interested in history and architecture. The old town is also a great place to take a walk and enjoy the views of the city and the sea.

sequence 13:

One of the important historic buildings in the city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of Sanaa, the old city of

sequence 14:

One of the important historic buildings in the city of Tehran is the old mosque of Tehran, which is located in the historical district of Tehran and is one of the oldest mosques in Tehran. The old mosque of Tehran is located in the historical district of Tehran and is one of the oldest mosques in Tehran. The old mosque of Tehran is located in the historical district of Tehran and is one of the oldest mosques in Tehran.
The old mosque of Tehran is located in the historical district of Tehran and is one of the oldest mosques in Tehran. The old mosque of Tehran is located in the historical district of Tehran and is one of the oldest mosques in Tehran. The old mosque of Tehran is located in the historical district of Tehran and is one of the oldest mosques in Tehran.
The old mosque of Tehran is located in the historical district of Tehran and is one of the oldest mosques in Tehran. The old mosque of Tehran is located in the historical district of Tehran and is one of the oldest mosques in Tehran. The old mosque of Tehran is located in the historical district of Tehran and is one of the oldest mosques in Tehran.
The old mosque of Tehran is located in the historical district of Tehran and is one of the oldest mosques in Tehran. The old mosque of Tehran is

sequence 15:

One of the important historic buildings in the centre of the city, the former Town Hall was built in 1903-1905 in the Neo-Renaissance style. The architect was Józef Płoszko. The building is decorated with a richly ornamented portal and a tower. The town hall was used by the municipal authorities until 1945. It is currently a hotel.

sequence 16:

One of the important historic sites in the city of Kerman is the tomb of Sheikh Ahmad-e Jam. This tomb is located in the city of Jam and is one of the most important and most visited tourist attractions in Kerman province.
The tomb of Sheikh Ahmad-e Jam is located in the city of Jam, 60 km from the city of Kerman. The tomb is located in the center of the city and is considered as one of the most important historical sites in Kerman province. The tomb of Sheikh Ahmad-e Jam is one of the most important places of pilgrimage for the people of Kerman province and the people of the city of Jam.
The tomb of Sheikh Ahmad-e Jam is located in the center of the city of Jam, in the Kerman province. The tomb is located in the center of the city and is considered as one of the most important historical sites in Kerman province. The tomb of Sheikh Ahmad-e Jam is one of the most important places of pilgrimage for the people of Kerman province and the people of the city of Jam.
The tomb of Sheikh Ahmad-e Jam is located in the center of the city of Jam, in the Kerman province. The tomb is located in the center of the city and is considered as

sequence 17:

One of the important historic buildings in the city of Shiraz is the tomb of the famous poet, Hafez. The building is located in the center of the city and is one of the most important tourist attractions of the city. This building has been registered as one of the national monuments of Iran. The building was built in 1935 and is one of the most important buildings in the city of Shiraz. The building is located in the center of the city and is one of the most important tourist attractions of the city. The building was built in 1935 and is one of the most important buildings in the city of Shiraz. The building is located in the center of the city and is one of the most important tourist attractions of the city. The building was built in 1935 and is one of the most important buildings in the city of Shiraz. The building is located in the center of the city and is one of the most important tourist attractions of the city. The building was built in 1935 and is one of the most important buildings in the city of Shiraz. The building is located in the center of the city and is one of the most important tourist attractions of the city. The building was built in

sequence 18:

One of the important historic buildings in the city of Qazvin is the Mirza Ali Akbar Khan Mosque, which is located on the east side of the main street of the city of Qazvin, the Imam Khomeini Street, and is one of the most important mosques of the city. The mosque was built in 1295 AH by Mirza Ali Akbar Khan, the son of Mirza Ali Asghar Khan, the son of Fath Ali Shah Qajar. The mosque is a double-story structure, with a courtyard, a dome, and a minaret. The main building of the mosque is made of stone and brick, and the courtyard is made of brick and plaster. The mosque has a large courtyard with a pool in the center, and a large dome on the roof. The mosque has a simple and elegant design, with a large entrance gate and a large courtyard. The mosque is a popular place for prayer and worship, and is also a popular tourist attraction in Qazvin.

sequence 19:

One of the important historic sites in the city of Tehran is the mausoleum of the late Ayatollah Behbahani. This building is located in the Behbahani neighborhood in the northern part of Tehran. The mausoleum of Ayatollah Behbahani is one of the most important historical attractions of Tehran. The mausoleum of Ayatollah Behbahani is located in the Behbahani neighborhood in the northern part of Tehran, which is one of the most important historical attractions of Tehran. The mausoleum of Ayatollah Behbahani is located in the Behbahani neighborhood in the northern part of Tehran, which is one of the most important historical attractions of Tehran. The mausoleum of Ayatollah Behbahani is located in the Behbahani neighborhood in the northern part of Tehran, which is one of the most important historical attractions of Tehran. The mausoleum of Ayatollah Behbahani is located in the Behbahani neighborhood in the northern part of Tehran, which is one of the most important historical attractions of Tehran. The mausoleum of Ayatollah Behbahani is located in the Behbahani neighborhood in the northern part of Tehran,

sequence 20:

One of the important historic sites in the area is the Old Stone House. Built in 1765, it is the oldest structure in the county. The house was originally located on the south side of the road, but was moved to its present location in 1939. The house was used for a time as a tavern and a stagecoach stop. The house is open to the public on a limited basis. The house is on the National Register of Historic Places.
The Old Stone House is located on the south side of Route 1, just west of the intersection with Route 100.

sequence 21:

One of the important historic buildings in the city of Isfahan, the Safavid period, is the Shah Mosque, which is located in the square of Naqsh-e Jahan.
The Shah Mosque, also known as the Imam Mosque, is one of the most important architectural masterpieces of Safavid Iran and the most famous monument and historical building of Isfahan. It is located in the southern side of Naqsh-e Jahan Square, opposite to Ali Qapu Palace, and has a wonderful view of the square. The construction of the mosque was started in the year 1611 by the order of Shah Abbas I and was completed in the year 1629. The architect of the mosque was Mohammad Reza Isfahani. The Shah Mosque is the first building in the world to use seven-color mosaic tiles in its construction. The mosque has a rectangular plan with a length of 109 m and a width of 156 m. The main entrance of the mosque is located in the north and has two large porches. The main portal of the mosque is decorated with a large inscription in the Kufic script, which is one of the most famous inscriptions in the world. The inscription is written by Ali Re

sequence 22:

One of the important historic sites in the city of Kerman is the old house of the Mirza Hassan Khan. This house has been built in 1250 AH. and its architecture is similar to the Safavid architecture. This house is located in the middle of the city of Kerman and is one of the attractions of this city. This house has been registered in the national list of Iran's national monuments.
The old house of Mirza Hassan Khan is located in the middle of the city of Kerman and is one of the attractions of this city. This house has been registered in the national list of Iran's national monuments.
This house has been built in 1250 AH. and its architecture is similar to the Safavid architecture. This house is located in the middle of the city of Kerman and is one of the attractions of this city. This house has been registered in the national list of Iran's national monuments.
One of the important historic sites in the city of Kerman is the old house of the Mirza Hassan Khan. This house has been built in 1250 AH. and its architecture is similar to the Safavid architecture. This house is located in the middle of the city of Kerman and is one of the

sequence 23:

One of the important historic buildings in the city is the old city hall. It was built in 1892 and designed by architect J. M. S. de Bazel. The building is a fine example of the Amsterdam School architecture, a style that was developed in Amsterdam during the early 20th century. The building is located on the Damrak, a street that runs from the Central Station to the Dam Square.
The old city hall is a three-storey building with a large clock tower. The tower is 75 metres tall and has a spire that is decorated with statues. The building is made of brick and has a number of decorative elements, including a large archway and a number of statues.
The building was used as the city hall until 1987, when a new city hall was built. Since then, the old city hall has been used for a variety of purposes, including as a museum and as a concert hall.
The building is a popular tourist attraction and is open to the public. Visitors can take a tour of the building and learn about its history and architecture. The building is also a popular venue for concerts and other events.

sequence 24:

One of the important historic buildings in the city of Gorgan is the building of the old mosque of Gorgan. This building is located in the old part of Gorgan and its construction dates back to the Safavid period. The building of the old mosque of Gorgan is a two-story building with a rectangular shape and a dome over it.
The old mosque of Gorgan is one of the most important buildings of the Safavid period in the city of Gorgan. This building was built in the 11th century AH and is one of the oldest buildings in the city of Gorgan. The building of the old mosque of Gorgan is located in the old part of the city and is one of the most important buildings of the Safavid period in the city of Gorgan.
The building of the old mosque of Gorgan is a two-story building with a rectangular shape and a dome over it. The building of the old mosque of Gorgan is located in the old part of the city and is one of the most important buildings of the Safavid period in the city of Gorgan. The building of the old mosque of Gorgan is a two-story building with a rectangular shape and a dome over it. The building of the old mosque of

sequence 25:

One of the important historic buildings in the city of Izmir is the Clock Tower, which was built in 1901. The Clock Tower is located in the square in front of the Kemeralti Bazaar and is 25 meters high. The Clock Tower is one of the most important symbols of Izmir. The Clock Tower is a symbol of the city and is a popular tourist attraction.
The Clock Tower is a beautiful building with a clock on top. It is a popular meeting place for locals and tourists alike. The Clock Tower is a great place to take photos and enjoy the view of the city. There are also many shops and restaurants located around the Clock Tower, making it a great place to spend an afternoon.
The Clock Tower is a landmark in Izmir and is a popular tourist attraction. The Clock Tower is a great place to take photos and enjoy the view of the city. There are also many shops and restaurants located around the Clock Tower, making it a great place to spend an afternoon.
The Clock Tower is a beautiful building with a clock on top. It is a popular meeting place for locals and tourists alike. The Clock Tower is a great place to take photos and enjoy the view of the city. There are also many

sequence 26:

One of the important historic buildings in the city of Shiraz is the "Khan of Nai" or "Nai Khan" which is located in the center of the city and has a long history. This building has been registered as a national monument in the list of national monuments of Iran.
The Khan of Nai is one of the old buildings in the city of Shiraz and is located near the old bazaar of the city. This building is located in the center of the city and has a long history. The building was built in the 11th century AH and was used as a caravanserai. The building has been registered as a national monument in the list of national monuments of Iran.
The building has a rectangular shape and is built of brick and gypsum. The building is located on a rectangular platform and has a large yard. The building has a large courtyard and a large number of rooms. The building has a large number of rooms and is used as a caravanserai. The building has a large number of rooms and is used as a caravanserai. The building has a large number of rooms and is used as a caravanserai. The building has a large number of

sequence 27:

One of the important historic sites in the city of Istanbul is the Sultan Ahmed Mosque, which is also known as the Blue Mosque, which is located in the city of Istanbul in Turkey, and it is one of the most famous tourist attractions, and it is worth noting that the mosque is located near the Hagia Sophia Museum, and it is one of the most famous tourist areas in the city of Istanbul, and the mosque was built in the year 1616 AD, and it was built by Sultan Ahmed the first, and the mosque is characterized by its wonderful design, and it is one of the most important Islamic mosques in the city of Istanbul.
The Blue Mosque is characterized by its wonderful design and its beautiful decorations, and it is worth noting that the mosque was built in the year 1616 AD, and it is considered one of the most important tourist attractions in the city of Istanbul, and the mosque is characterized by its wonderful design and its beautiful decorations, and the mosque is characterized by its wonderful design and its beautiful decorations, and the mosque is characterized by its wonderful design and its beautiful decorations, and the mosque is characterized by its wonderful design and its beautiful decorations, and the mosque is characterized by its wonderful design and its beautiful decorations, and

sequence 28:

One of the important historic sites of the city of Konya is the Mevlana Museum, which is the tomb of the famous Sufi mystic and poet Mevlana Jalaluddin Rumi. The museum is located in the city center and is visited by thousands of people every year.
The Mevlana Museum is located in the center of the city of Konya, in a building that was originally built as a dervish lodge in the 13th century. The building was later converted into a museum and opened to the public in 1926. The museum contains a collection of manuscripts, books, and artifacts related to Mevlana and the Mevlevi order, as well as a mausoleum where Mevlana and his family are buried.
The Mevlana Museum is a popular destination for tourists and pilgrims alike, and is considered one of the most important cultural and historical sites in Turkey. Visitors can learn about the life and teachings of Mevlana, as well as the history of the Mevlevi order, through the exhibits and artifacts on display in the museum.
The museum also features a beautiful courtyard and garden, which is a peaceful and serene place to relax and reflect. In addition,

sequence 29:

One of the important historic buildings of the city, the former “Hotel de Ville” (Town Hall), was built in the 16th century. It was the seat of the city government until 1945. The building was severely damaged during the Second World War and was rebuilt in the 1950s. The interior of the building has been restored to its original appearance. Today, the building houses the “Museum of the History of the City of Strasbourg”.

sequence 30:

One of the important historic buildings in the city of Qom is the shrine of Imamzadeh Abdollah. This shrine is located in the city of Qom and is a historic building of the Safavid era. The shrine of Imamzadeh Abdollah is located in the city of Qom and is one of the most important pilgrimage places for the people of Qom and the surrounding areas. The shrine of Imamzadeh Abdollah is located in the city of Qom and is one of the most important pilgrimage places for the people of Qom and the surrounding areas.
The shrine of Imamzadeh Abdollah is located in the city of Qom and is one of the most important pilgrimage places for the people of Qom and the surrounding areas.
The shrine of Imamzadeh Abdollah is located in the city of Qom and is one of the most important pilgrimage places for the people of Qom and the surrounding areas.
The shrine of Imamzadeh Abdollah is located in the city of Qom and is one of the most important pilgrimage places for the people of Qom and the surrounding areas.
The shrine of Imamzadeh Abdollah is located in the city of Qom and is one of

sequence 31:

One of the important historic buildings in the city of Antalya, the Yivli Minaret was built by the Seljuk Sultan Alaaddin Keykubat in the 13th century. The minaret is 38 meters high and has a diameter of 4 meters at the base. The minaret was restored in 1990. The Yivli Minaret Mosque, which was built in 1230, is the oldest mosque in the city. The mosque, which was restored in 1990, is open to worship.

sequence 32:

One of the important historic sites in the area, the 18th century St. Paul's Anglican Church is located on the corner of Church and Main Streets. The church was built in 1787 by the Loyalists who came to the area after the American Revolution. The church is a National Historic Site.

sequence 33:

One of the important historic buildings in the town of Zamość is the Town Hall. It is located in the market square and is a Renaissance building. The construction of the Town Hall was started in 1587, and the building was completed in 1593. The Town Hall in Zamość is a building with a rich history. It has been used for many different purposes over the years, including as a courthouse, a prison, and a museum. Today, the Town Hall is open to the public and is a popular tourist destination.
The Town Hall in Zamość is a beautiful building that is well worth a visit. The interior of the building is just as impressive as the exterior, and there are many interesting things to see. The Town Hall is a great place to learn about the history of Zamość and the people who lived there. The Town Hall is a great place to visit if you are in Zamość. The building is beautiful and the history of the town is fascinating.

sequence 34:

One of the important historic buildings in the city of Kermanshah is the Sufi shrine of Baba Taher, which is located in the city of Kermanshah, in Baba Taher neighborhood. The shrine of Baba Taher is one of the most important historical monuments of Kermanshah and is located in the city of Kermanshah, in Baba Taher neighborhood. The shrine of Baba Taher is one of the most important historical monuments of Kermanshah and is located in the city of Kermanshah, in Baba Taher neighborhood. The shrine of Baba Taher is one of the most important historical monuments of Kermanshah and is located in the city of Kermanshah, in Baba Taher neighborhood. The shrine of Baba Taher is one of the most important historical monuments of Kermanshah and is located in the city of Kermanshah, in Baba Taher neighborhood. The shrine of Baba Taher is one of the most important historical monuments of Kermanshah and is located in the city of Kermanshah, in Baba Taher neighborhood. The shrine of Baba Taher is one of the

sequence 35:

One of the important historic buildings in the city of Tehran is the building of the former parliament. This building was constructed in the year 1316 AH. The building of the former parliament is located in the south of the city of Tehran and is one of the most important and beautiful buildings in the city. The building of the former parliament is one of the most important and beautiful buildings in the city of Tehran. The building of the former parliament is one of the most important and beautiful buildings in the city of Tehran. The building of the former parliament is one of the most important and beautiful buildings in the city of Tehran. The building of the former parliament is one of the most important and beautiful buildings in the city of Tehran. The building of the former parliament is one of the most important and beautiful buildings in the city of Tehran. The building of the former parliament is one of the most important and beautiful buildings in the city of Tehran. The building of the former parliament is one of the most important and beautiful buildings in the city of Tehran. The building of the former parliament is one of the most important and beautiful buildings in the city of Tehran. The building of the former parliament is one of the most important and beautiful buildings in the city of Tehran.

sequence 36:

One of the important historic buildings in the city of Kerman is the Jameh Mosque of Kerman, which is located in the middle of the city. This mosque is one of the oldest mosques in Iran and has been built in different periods. The mosque has been repaired and renovated many times. The most important renovation of this mosque was in the Safavid era. The mosque has a courtyard, a porch, a prayer hall, a dome and a minaret. The courtyard of the mosque is located in the middle of the mosque and is square. The porch of the mosque is located in the north and south of the courtyard. The prayer hall is located in the middle of the mosque and is square. The dome of the mosque is located in the middle of the prayer hall and is decorated with plasterwork. The minaret of the mosque is located in the middle of the courtyard and is decorated with plasterwork. The mosque is a beautiful and historic place that is worth visiting.
The Jameh Mosque of Kerman is one of the most important historical and religious monuments of Iran and is located in the city of Kerman. This mosque is one of the oldest mosques in Iran and has been built in different periods. The mosque has been repaired and renovated

sequence 37:

One of the important historic sites in the world is the ancient city of Petra in Jordan. The city is located in the Ma’an Governorate in the south of the country. The city is located in a valley between the mountains of Jordan and Saudi Arabia. Petra is one of the most important archaeological sites in the world. Petra is a UNESCO World Heritage Site and has been on the list since 1985.
The city of Petra is the capital of the Nabataean Kingdom and the Nabataean civilization. The city is located at the crossroads of many important trade routes. The city was founded in the 6th century BC and flourished in the Hellenistic and Roman periods. The city was abandoned in the 7th century AD. The city was rediscovered by the Swiss explorer Johann Ludwig Burckhardt in 1812.
The city of Petra is famous for its rock-cut architecture and tombs. The most famous building in the city is the Khazneh or the Treasury. The building is a huge rock-cut tomb with a facade of columns and statues. The building is 40 meters high and is one of the most popular tourist attractions in the city. Other famous buildings in the city include the Monastery, the Royal

sequence 38:

One of the important historic sites in the city of Isfahan is the Shahrestan Bridge which is located in the southern part of the city. The bridge is located in the Shahrestan neighborhood of the city and is one of the most important bridges in the city. The bridge is located on the Zayandeh Rood River and is one of the most important bridges in the city.
This bridge is the oldest bridge in Isfahan and is one of the most important bridges in the city. The bridge is located on the Zayandeh Rood River and is one of the most important bridges in the city. The bridge is located on the Zayandeh Rood River and is one of the most important bridges in the city. The bridge is located on the Zayandeh Rood River and is one of the most important bridges in the city. The bridge is located on the Zayandeh Rood River and is one of the most important bridges in the city. The bridge is located on the Zayandeh Rood River and is one of the most important bridges in the city. The bridge is located on the Zayandeh Rood River and is one of the most important bridges in the

sequence 39:

One of the important historic buildings in the city of Konya is the Karatay Medrese, which was built in 1251 by the Seljuk Sultan Alaaddin Keykubat. The building was built as a madrasa and was used as a madrasa for a long time. The building was converted into a museum in 1966 and opened to visitors. The building is now used as a museum and is one of the most important historical buildings in the city of Konya.
The Karatay Medrese is one of the most important historical buildings in the city of Konya and is a must-see for anyone visiting the city. The building is known for its beautiful architecture and its unique style, which is a mix of Seljuk and Islamic architecture. The building is also known for its beautiful tile work, which is one of the most important features of the building.
The Karatay Medrese is located in the heart of the city of Konya and is easily accessible by public transportation. The building is open to visitors from Monday to Saturday from 9:00 to 17:00 and on Sundays from 13:00 to 17:00. Admission to the building is free of charge.
The

sequence 40:

One of the important historic places of the city of Isfahan is the Jame Mosque of Isfahan. It is located in the center of the city, and is one of the most important historical buildings of the city.
The Jame Mosque of Isfahan, or the Friday Mosque of Isfahan, is one of the oldest mosques in the world. It is also one of the most important historical monuments of the city of Isfahan. The Jame Mosque of Isfahan is located in the center of the city, and is one of the most important historical buildings of the city.
The Jame Mosque of Isfahan is one of the oldest mosques in the world. It is also one of the most important historical monuments of the city of Isfahan. The Jame Mosque of Isfahan is located in the center of the city, and is one of the most important historical buildings of the city.
The Jame Mosque of Isfahan is one of the oldest mosques in the world. It is also one of the most important historical monuments of the city of Isfahan. The Jame Mosque of Isfahan is located in the center of the city, and is one of the most important historical buildings of the city.
The Jame Mosque

sequence 41:

One of the important historic buildings in the city of Kerman is the Shah Nematollah Vali Mausoleum. The mausoleum is located in the city of Mahan, 30 km to the north of Kerman. The construction of this building started in 1437 and it was completed in 1445. This building is a masterpiece of Iranian architecture and it is the tomb of Shah Nematollah Vali, the great mystic poet and philosopher of the 9th century AH. The mausoleum is a square building with a dome on top. The dome is supported by eight pillars. The dome is made of bricks and it is 21 meters in diameter. The dome is decorated with tiles and the entrance is decorated with plasterwork. The entrance is located on the eastern side of the building and it is made of brick and plaster. The walls of the building are made of brick and plaster. The building has a courtyard and a garden. The garden is planted with trees and flowers. The building is a UNESCO World Heritage Site. The mausoleum is a popular tourist destination and it is visited by thousands of people every year.
The Shah Nematollah Vali Mausoleum is a

sequence 42:

One of the important historic sites in the city of Isfahan is the Jameh Mosque of Isfahan. This mosque is located in the center of Isfahan city and is one of the most important mosques in Iran. The history of this mosque goes back to the time of the Sassanid era and the first mosque was built in the Sassanid era and later it was rebuilt in the Seljuk era. The name of this mosque is also known as the Friday Mosque of Isfahan and the name of the mosque is also known as the Jameh Mosque of Isfahan. The Jameh Mosque of Isfahan is one of the oldest mosques in Iran and is one of the most important historical monuments of Iran. The Jameh Mosque of Isfahan is located in the center of Isfahan city and is one of the most important mosques in Iran.
The Jameh Mosque of Isfahan is one of the most important mosques in Iran and is one of the most important historical monuments of Iran. The Jameh Mosque of Isfahan is located in the center of Isfahan city and is one of the most important mosques in Iran. The Jameh Mosque of Isfahan is one of the most important

sequence 43:

One of the important historic buildings in the city of Isfahan is the historical building of the former Zoroastrian fire temple of Isfahan. This building has been known as the fire temple of Isfahan since the Safavid era and has been used as a fire temple for many years. The name of this fire temple is the fire temple of Isfahan. This historical building is located in the city of Isfahan, in the middle of the city of Isfahan. The building of this fire temple is located in the middle of the city of Isfahan, in the middle of the city of Isfahan. The building of this fire temple is located in the middle of the city of Isfahan, in the middle of the city of Isfahan. The building of this fire temple is located in the middle of the city of Isfahan, in the middle of the city of Isfahan. The building of this fire temple is located in the middle of the city of Isfahan, in the middle of the city of Isfahan. The building of this fire temple is located in the middle of the city of Isfahan, in the middle of the city of Isfahan.

sequence 44:

One of the important historic sites in the region of the ancient city of Pergamon is the ancient city of Teos. The ancient city of Teos, which is located on the coast of the Gulf of Izmir, is one of the most important cities of the Ionian League. The city, which was founded in the 10th century BC, was an important center of trade and culture in the 6th century BC. The city, which was ruled by the Persians in the 5th century BC, was conquered by Alexander the Great in 334 BC. The city, which was ruled by the Romans in the 2nd century BC, was an important center of trade and culture in the 1st century BC. The city, which was ruled by the Byzantines in the 4th century AD, was an important center of trade and culture in the 5th century AD. The city, which was ruled by the Ottomans in the 15th century AD, was an important center of trade and culture in the 16th century AD. The city, which was ruled by the French in the 18th century AD, was an important center of trade and culture in the 19th

sequence 45:

One of the important historic sites in the city of Tehran is the tomb of Hafez, which is located in the historical and cultural complex of Hafez in the city of Shiraz. This complex has a large garden with a beautiful and beautiful view. The tomb of Hafez has been made in the form of a dome and is located in the middle of the garden. The tomb of Hafez is one of the most important historical and cultural monuments of Iran, which has been visited by many tourists from all over the world. The tomb of Hafez is located in the city of Shiraz, in the province of Fars. This tomb is one of the most important historical and cultural sites in Iran, which has been visited by many tourists from all over the world. The tomb of Hafez is located in the city of Shiraz, in the province of Fars. This tomb is one of the most important historical and cultural sites in Iran, which has been visited by many tourists from all over the world. The tomb of Hafez is located in the city of Shiraz, in the province of Fars. This tomb is one of the most important historical and cultural sites in Iran, which has been visited

sequence 46:

One of the important historic buildings in the town of Szigliget is the castle of the same name. The castle was built on a steep, 150-metre-high basalt rock, which is connected to the surrounding hills by a narrow strip of land. The castle was built by the Zala family in the 13th century. The castle was first mentioned in 1268, when the Zala family received it as a gift from King Béla IV. The fortress was destroyed by the Mongols in 1285, and was rebuilt by the Zala family in the 14th century. The castle was owned by the Zala family until 1397, when it was conquered by the Habsburgs. The castle was then used as a residence by the Habsburgs, and was later used as a prison. The castle was damaged during the Turkish invasion in the 16th century, but was rebuilt in the 17th century. The castle was used as a military fortress until the 19th century, and was later used as a hunting lodge by the Habsburgs. The castle was declared a national monument in 1956, and is now open to the public. The castle is

sequence 47:

One of the important historic buildings of the city, the building was built in 1896 and was used as a primary school. The building was restored in 2002 and used as a cultural center.
One of the important historic buildings of the city, the building was built in 1896 and was used as a primary school. The building was restored in 2002 and used as a cultural center.
The building was built in 1896 and was used as a primary school. The building was restored in 2002 and used as a cultural center.
The building was built in 1896 and was used as a primary school. The building was restored in 2002 and used as a cultural center.
The building was built in 1896 and was used as a primary school. The building was restored in 2002 and used as a cultural center.

sequence 48:

One of the important historic buildings in the city of Jeddah, the former residence of the late King Abdulaziz, was restored and converted into a museum.
The museum, which was opened in 1999, includes the King Abdulaziz’s personal belongings and the rooms he used during his stay in Jeddah.
The museum also includes a number of old cars that belonged to the late King Abdulaziz.

sequence 49:

One of the important historic buildings in the city of Tehran is the building of the former Ministry of Finance. This building is located in the center of Tehran and is one of the most important buildings of the Pahlavi era in the capital. The building of the former Ministry of Finance is located on the north side of the park of the former Ministry of Finance, on the north side of the park of the former Ministry of Finance, on the north side of the park of the former Ministry of Finance, on the north side of the park of the former Ministry of Finance, on the north side of the park of the former Ministry of Finance, on the north side of the park of the former Ministry of Finance, on the north side of the park of the former Ministry of Finance, on the north side of the park of the former Ministry of Finance, on the north side of the park of the former Ministry of Finance, on the north side of the park of the former Ministry of Finance, on the north side of the park of the former Ministry of Finance, on the north side of the park of the former Ministry of Finance, on the north side of the park of the former Ministry of Finance, on the north side of the park of the former

main: decoded 10885 tokens in 38.71 s, speed: 281.20 t/s

llama_print_timings:        load time =    4937.99 ms
llama_print_timings:      sample time =    1304.71 ms / 10935 runs   (    0.12 ms per token,  8381.19 tokens per second)
llama_print_timings: prompt eval time =   26691.69 ms / 10891 tokens (    2.45 ms per token,   408.03 tokens per second)
llama_print_timings:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings:       total time =   43644.58 ms / 10892 tokens
gbloisi-openaire commented 2 weeks ago

I noticed similar behaviour on Silicon mac with llama 3 instruct. It looks like it is solved by adding the assistant header (see https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3) ./llama-batched -m ~/aitest/Meta-Llama-3.1-8B-Instruct-Q6_K.gguf -p "One of the important historic<|eot_id|><|start_header_id|>assistant<|end_header_id|>" -np 50 -ngl 99 --ctx-size 12800 -n 251 so it might be more related to the base llama 3.1 model than to llama.cpp inference

ggerganov commented 2 weeks ago

This is expected behaviour for a base model with the sampling configuration used by the llama-batched example:

https://github.com/ggerganov/llama.cpp/blob/bdf314f38a2c90e18285f7d7067e8d736a14000a/examples/batched/batched.cpp#L177-L187

You don't see it with llama-cli because it uses higher temperature by default: 0.8.