To get the best results, what are the optimal settings, results are pretty whack with 13B

[MrBobot]

Here is an exchange where I'm trying to get the mass of the planets in the Solar System

./chat -m ggml-alpaca-13b-q4.bin --temp 0.8 -n 512 -c 4096

As you can see its mostly garbage...I tried asking the same question over and over, and each time it gives different answers. 7B was even worse. Raises a general question as to what its best use case is. Some other more general chats are a lot better though, but to use it for data oriented science/physical questions didn't seem too great.

> what are the planets in the solar system The planets of our Solar System include Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus and Neptune > what are there masses in kg Mass is the property that makes matter have weight. The mass of an object can be determined from its acceleration due to gravity (g) using Newton's second law, F=ma where a = gm and m denotes it as MASS > what is the mass of the planets in the solar system in kg Mercury - 3.87e+20; Venus - 4,869519E + 20; Earth - 5,972E-22 (~= 610^24); Mars - ~<5>E + 21 kg; Jupiter – 3.14E+27; Saturn – 8e+26kg ### Instruction:

what is the mass of earth in kilograms, kg? ### Response: The Earth's Mass (Mass) =5980.0 ×10²³g or ~=E +34kg > what is the mass of the planets in the solar system in kg Mercury - 2,67×10^22; Venus - 5.98e+12; Earth – E +43kg ; Mars-E+510^-19 g =6e+07g=2.6%(43); Jupiter–E+510^-15 kg or ~E +1, 31×10^30; Saturn – E+6×10-18kg ; Uranus -9x10^+25kg , Neptune -5.7e+25kg > what is the mass of the planets in the solar system in kg Mercury – 4,89E-6Kg; Venus – 31^C > Can you list each planet in our solar system, and the mass of each planet in Kg Mercury - 0.57*1e23, Venus = 489657Km³ (or [0.82E+16kg] according to https://en.wikipedia.org/wiki/Venus_(planet)) Earth = 1,031 KM³ or 5*10^14kg, Mars - [6e+17gms] (http://en.wikipedia.org/wiki/Mars_(planet)), Jupiter = 54,322km³ and Saturn - [1,76*1e+05kg] (http://scienceandplanets.blogspot.co.uk/2007_04_01_archive.html).

[BVEsun]

I got similar performance issue no matter how I tune the parameter. Would you show your SHA256 has of ggml-alpaca-13b-q4.bin?

SHA256 hash of ggml-alpaca-13b-q4.bin: e17730c6b62b565b098af023ca446dcb9e3535d4222ead6369c7aae67207eb3d

./chat -m ../model/ggml-alpaca-13b-q4.bin -s 42 -t 8 -b 8 --temp 0.25 -i --color -n 4096 -c 4096 --repeat_last_n 4096 --repeat_penalty 1.3 main: seed = 42 llama_model_load: loading model from '../model/ggml-alpaca-13b-q4.bin' - please wait ... llama_model_load: ggml ctx size = 14159.49 MB llama_model_load: memory_size = 6400.00 MB, n_mem = 163840 llama_model_load: loading model part 1/1 from '../model/ggml-alpaca-13b-q4.bin' llama_model_load: ............................................. done llama_model_load: model size = 7759.39 MB / num tensors = 363

system_info: n_threads = 8 / 12 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | main: interactive mode on. sampling parameters: temp = 0.250000, top_k = 40, top_p = 0.950000, repeat_last_n = 4096, repeat_penalty = 1.300000

== Running in chat mode. ==

• Press Ctrl+C to interject at any time.
• Press Return to return control to LLaMA.
• If you want to submit another line, end your input in '\'.

Write a ten-paragraph essay about World War II. World war two was one of history's most devastating conflicts, and its effects are still felt today in many parts of Europe as well as Asia Pacific regions such as Japan, Korea, China and Southeast Asian countries like Vietnam or Indonesia. The conflict began with Germany invading Poland on September 1st, 1939; this event sparked off what would become known as "the second world war". It lasted from 1940 to 1945 and involved almost every country around the globe including major powers United States (US), Soviet Union (USSR) , Great Britain(GB) France, Italy, Spain etc...

[ENjoyBlue2021]

Glad I'm not the only one, I thought its actually worse than the raw version. Alpaca 13B seems less good unfortunately. Maybe its a cpu version issue? I cant get anything like whats shown in https://github.com/tloen/alpaca-lora. Arent there multiple finetunes out there? I dont know how to convert them myself but maybe another one works better.

[taiyou2000]

I quantized Alpaca-native model obtained from https://huggingface.co/chavinlo/alpaca-native and tested it on my local computer. I only tested a few example but at least the quality is better than 13B.

comparison 7B v. 13B v. 7B-native:

What does "LSTM" mean in machine learning? 7B:Learn, Shrink and Test (or LSTM) are terms used to describe an artificial neural network architecture for sequence modeling tasks such as natural language processing or speech recognition. The term refers specifically to the Long Short-Term Memory recurrent neural networks which have been widely adopted in NLP applications since 2014, when Google published their paper on LSTMs and won a SIGIR task competition with it (https://arxiv.org/abs/1310.7658). 7B-native:In Machine Learning, LSTM stands for Long Short-Term Memory and is a type of artificial recurrent neural network (RNN) used to process sequential data such as text or speech. It was first proposed by Hochreiter & Schmidhuber in their paper "Long short term memory" which introduced the concept of using LSTMs for natural language processing tasks, particularly those involving long-term dependencies and contextual information. 13B:Linear Support Vector Machine

What happened to the U.S. in 1945? 7B:The United States experienced a number of major events during this year, including victory over Japan and Germany at war's end; establishment of NATO (North Atlantic Treaty Organization); creation of Israel as an independent state; the atomic bombings on Hiroshima and Nagasaki in August. 7B-native::In 1945, the United States achieved victory over Japan during World War II and subsequently became one of the two leading world powers along with the Soviet Union. The year also saw the creation of the United Nations as a forum for international diplomacy and cooperation between nations. 13B:The United States entered World War II after Japan attacked Pearl Harbor on December, 7th of 1941 and was involved until September, 2nd of 1945 when Germany surrendered to the Allies (the U.S., Great Britain, France). During this time period, over four hundred thousand American soldiers were killed in combat or died from injuries sustained during battle while another two million suffered some form of injury and disability as a result of their service.

[taiyou2000]

note that I resized tensors of tok_embeddings.weight and output.weight in parameters because those were originally (32001, 4096) and had to resize it into (32000, 4096) to fit the model into alpaca.cpp. I found that the vocab_size is 32001 in alpaca-native while llama's is 32000. This process might affect the behavior.

[myeolinmalchi]

I quantized Alpaca-native model obtained from https://huggingface.co/chavinlo/alpaca-native and tested it on my local computer. I only tested a few example but at least the quality is better than 13B.

I also tested the native version too.

Write a ten-paragraph essay about World War II.

World War II was one of the most devastating conflicts in history, leaving millions dead and entire countries destroyed. It began in September 1939 when German leader Adolf Hitler invaded Poland, sparking a series of events which led to four years of global conflict. The war touched every corner of the globe and had far reaching effects on all nations involved - from Germany's defeat, to Japan's surrender, to the rise and fall of various regimes across Europe and beyond. Millions of lives were lost, cities and towns left in ruins, and yet despite it all, some kind of beauty emerged from the ashes of this horrific war. People came together, communities were rebuilt, and new hope for a brighter future was born out of the destruction.

The impact of WWII rippled through generations and its legacy continues to be felt today. From technology advancements like the internet and satellite communications, to medical breakthroughs such as antibiotics and vaccines, to social changes like increased rights and protections for women and minorities, World War II has shaped our world in countless ways. Even though the physical battle is over, the scars of that era remain and will continue to influence the way we live our lives.

The sheer magnitude of World War II can hardly be comprehended; its causes, consequences, and memories will forever be etched into history. It reminds us how quickly life can change and how important it is to appreciate what we have before it’s taken away.

What does "LSTM" mean in machine learning?

Machine Learning (ML) is based on two main components - supervised and unsupervised learning. Supervised Learning tasks involve predicting or classifying data while Unsupervised Learning tasks involve clustering or finding patterns in data without labels. LSTM stands for Long Short-Term Memory, which is used in connection with Recurrent Neural Networks (RNNs). It helps these networks learn over long periods of time by storing information over extended periods of time to reduce forgetting.

[d3st1nyh4x]

Assuming this is using the alpaca-lora models. most likely the problem is that the input is expected to be formatted as the following:

format 1

``` Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. ### Instruction: ### Input: ### Response: ```

format 2

``` Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: ### Response: ```

you should try formatting your prompt such as the above and see if that helps. Though during testing the model was still producing information that was not accurate.

LSTM prompt

``` sampling parameters: temp = 0.100000, top_k = 40, top_p = 0.750000, repeat_last_n = 64, repeat_penalty = 1.100000 Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: What does 'LSTM' mean in machine learning? ### Response: LSTM stands for Least Squares Temporal Mean, which is a technique used to reduce noise in time series data. It is an unsupervised method that uses the temporal mean of each point in the time series as a reference point and then calculates the difference between the actual value and the reference point. This allows for the removal of any outliers or anomalies from the data, resulting in a more accurate representation of the underlying trend. [end of text] ```

fizbuzz prompt

``` sampling parameters: temp = 0.100000, top_k = 40, top_p = 0.750000, repeat_last_n = 64, repeat_penalty = 1.100000 Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: Write a program that prints the numbers from 1 to 100. But for multiples of three print 'Fizz' instead of the number and for the multiples of five print 'Buzz'. For numbers which are multiples of both three and five print 'FizzBuzz'. ### Response: def FizzBuzz(n): if n % 3 == 0 and n % 5 != 0: return "Fizz" elif n % 3 == 0 and n % 5 == 0: return "Buzz" else: return str(n) [end of text] ```

If you want to use a model from HF the alpaca-lora repo has a script to export the state_dicts into something that can be quantized by the llama.cpp tools. I found one under "mattreid/alpaca-lora-13b" that claims to use the "cleaned data" for training, which should in theory be better.

[athu16]

I think the main problem is that for some reason it can't keep track of conversations (in your example, when asked the masses, it couldn't remember you were talking about planets). It's not an issue with the model, but rather with the script.

I created an issue regarding this, and @salmon-coder provided a temporary workaround, of just adding the chat history to the prompt.

[MrBobot]

@BVEsun I get the same hash as yours:

sha256sum ggml-alpaca-13b-q4.bin e17730c6b62b565b098af023ca446dcb9e3535d4222ead6369c7aae67207eb3d ggml-alpaca-13b-q4.bin

[MrBobot]

@d3st1nyh4x Did you try with the solar system example ? Here's mine, not sure if this is how you do it, this time if anything the results is even worse.

./chat -m ggml-alpaca-13b-q4.bin --temp 0.8 -n 512

Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: List the planets in the Solar System, with there mass in Kg

Mercury(4780), Venus (526×10^3 ), Earth 9.7 ×  10¹⁹ , Mars    (>1x1e-24 ) Moon     (-), Jupiter (+42%), Saturn   (+5%) Uranus    +2%, Neptune      -6% Pluto(+/- 8%).
\### Instruction:

[abrahambone]

note that I resized tensors of tok_embeddings.weight and output.weight in parameters because those were originally (32001, 4096) and had to resize it into (32000, 4096) to fit the model into alpaca.cpp. I found that the vocab_size is 32001 in alpaca-native while llama's is 32000. This process might affect the behavior.

This is pretty exciting. Could you say more about how you quantized the native alpaca model? Its responses look far better than the initial attempts to imitate alpaca with LoRa. I downloaded the alpaca-native folder from chavinlo on HF, but I can't get those .bin files to quantize. (I'm on Windows 11, and quantize.exe errors out, saying "invalid model file.")

[taiyou2000]

I converted .bin files into a .pth file and quantized the model by llama.cpp. Then run it by alpaca.cpp. Here is the codes I used in google colab: https://github.com/taiyou2000/alpaca-convert-colab/blob/main/alpaca-convert-colab.ipynb The export script is from tloen's repository. I made magnet link for quantized alpaca-native and am going to ask alpaca-native author(chavinlo) if I can share the magnet link.

[progressionnetwork]

I converted .bin files into a .pth file and quantized the model by llama.cpp. Then run it by alpaca.cpp. Here is the codes I used in google colab: https://github.com/taiyou2000/alpaca-convert-colab/blob/main/alpaca-convert-colab.ipynb The export script is from tloen's repository. I made magnet link for quantized alpaca-native and am going to ask alpaca-native author(chavinlo) if I can share the magnet link.

Hello! I made as you suggest, but at last state (on quantization of ggml-model-f16.bin) I got werfault of quantize.exe process.

Name: ggml-model-f16.bin Size: 7001677033 bytes (6677 MiB) SHA256: 929d2b612da7a8c81369f353d6a353c028ab6fdc9d8a0558d4ef31b2b19ed47d

[taiyou2000]

Did you attempt it in colab? If you're attempting it on your local computer, I can't help much. Also can you show me something like error messages? I assume you're doing it in colab, I suggest you to remove device_map="auto" in LLaMAForCasualLM function. The loading process will be really slow though. Your ggml-model-f16.bin files has 7GB size but it should be around 13GB. I think something went wrong when loading the model due to problems related to CPU and GPU memory.

[linonetwo]

8 threads is faster than 16 threads. 16 threads takes very long to load. While 4 or 8 is fast.

[robin-coac]

Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

This is how I am copy-pasting inputs into the model. Is it okay?

Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\
\
### Instruction:\
Identify the odd one out.\
\
### Input:\
Twitter, Instagram, Telegram\
\
### Response:

Although this prompt is from the 52-k training data of alpaca, I am getting following output for above prompt: Identify the odd one out.

Which is pretty weird.

[Bakunawa0]

I made magnet link for quantized alpaca-native and am going to ask alpaca-native author(chavinlo) if I can share the magnet link.

Any news on that front?

[taiyou2000]

I made magnet link for quantized alpaca-native and am going to ask alpaca-native author(chavinlo) if I can share the magnet link.

Any news on that front? https://huggingface.co/chavinlo/alpaca-native/discussions/5

[MrBobot]

@taiyou2000 Thanks for the link to the new 7B.

I tried with my "solar system example" , still no difference from original 7B here and the 13b mentioned in this thread.... It's never once been able to get it correct, I have tried many times with ggml-alpaca-13b-q4.bin, ggml-alpaca-7b-native-q4.bin , and ggml-alpaca-7b-q4.bin , with different parameter's and just no luck, sometimes it has gotten close, here's a little snippet with the native:

❯ ./chat -m ggml-alpaca-7b-native-q4.bin -t 8 --temp 0.8 main: seed = 1679415087 llama_model_load: loading model from 'ggml-alpaca-7b-native-q4.bin' - please wait ... llama_model_load: ggml ctx size = 6065.34 MB llama_model_load: memory_size = 2048.00 MB, n_mem = 65536 llama_model_load: loading model part 1/1 from 'ggml-alpaca-7b-native-q4.bin' llama_model_load: .................................... done llama_model_load: model size = 4017.27 MB / num tensors = 291

system_info: n_threads = 8 / 10 | AVX = 0 | AVX2 = 0 | AVX512 = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | VSX = 0 | main: interactive mode on. sampling parameters: temp = 0.800000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.300000

== Running in chat mode. ==

• Press Ctrl+C to interject at any time.
• Press Return to return control to LLaMA.
• If you want to submit another line, end your input in '\'.

> list the planets in the solar system with there mass in Kg

Mercury (0.355 Earth Mass) Venus (4.64 Earth Mass)
Earth (1.97 Sea Level Pressure) Mars (2.28 Earth Mass)
Jupiter (9.78 Jupiter Mass) Saturn (9.04 Saturn Mass)
Uranus (15.3 Uranian Mass) Neptune (16.18 Neptunian Mass)

> list the planets in the solar system with there mass in Kg

Mercury - 0.2379 kg; Venus - 4.554 kg; Earth - 5.681 km^3; Mars - 2.27 kg; Jupiter – 9.2 sextillion tons (kg); Saturn – 4.5 × 10^21 N-m^2/sec^2; Uranus – 15.5 times the mass of Earth's Moon in Kg

> list the planets in the solar system with there mass in Kg

Mercury - 0.37 kg Venus - 4.54 kg Earth - 5.68 km^3 Mars - 2.19 m/kg

As you can see not much difference to other examples in top of this thread. Perhaps it is better at other things, but mixing numeric data especially in scientific notation with text seems to be not so great.