search

cmp-nct / ggllm.cpp

Falcon LLM ggml framework with CPU and GPU support
Other
244 stars 21 forks source link

Could we also get the Makefile updated to build a libfalcon.so #64

Open linuxmagic-mp opened 1 year ago

linuxmagic-mp commented 1 year ago

Still would be nice when everything starts getting merged into llama, for one place to maintain.. I just don't like touch the C files and embarassing my self, but should be able to make it a drop in replacement for libllama.so, correct? Just thinking in terms of projects like llama_cpp

cmp-nct commented 1 year ago

It's quite an irony that I've always been a big fan of Makefiles as opposed to configure/cmake and now it's all focused on cmake. I've pushed an update to add libfalcon.so, should work.

Regarding merging, I don't see it happening any time soon. The changes in ggllm.cpp (which probably will be renamed into falcon centric at a later time) are a lot deeper than just an evaluation routine. I'm quite sure llama.cpp will show basic support for falcon soon but it won't include the same features and improvements we have here. "ggllm" was originally chosen because that is what I wanted llama.cpp to become, it was meant as a "lobbying fork". By now we have deviated too far from that original intention.

Stopwords, deep changes into cuda memory management and calculation, 16 bit cublas, a more complex kvcache, different instruction mode, a new tokenizer, finetuning detection, system prompt support, etc. you can't just integrate that into llama.cpp without a ton of work. Also many of the changes likely conflict with the vision of the core developers of the respective llama modules.

Both projects have a different focus, llama.cpp is more backend ggml oriented while ggllm.cpp is aiming to put falcon into best possible realworld use as soon as possible.

maddes8cht commented 1 year ago

To get closer to a goal of a "ggllm.cpp" the design of a new and "final" ggml file format has been started on llama.cpp. Will "your" ggllm.cpp adopt this new file format, or rather not? I think there will be more and more functions in ggml that will refer to the new format?

To get closer to a goal of a "ggllm.cpp" the design of a new and "final" ggml file format has been started on llama.cpp. Will "your" ggllm.cpp adopt this new file format, or rather not? I think there will be more and more functions in ggml that will refer to the new format? The new file format could eventually become the common denominator from which a common base is built, which then splits more specifically into the functions of individual models.

cmp-nct commented 1 year ago

Regarding the new format, there won't be any benefits adopting it right away. Sticking with GGCC also won't remove any new features coming in from ggml. If the new format turns out to flexible enough I think it makes sense going into that direction but as a lower priority development. It's quite a bit of work and my focus for now is performance and new features.

The name choice of ggllm was a mistake, I'm just sticking to it for now due to the lack of a great new name.