65a
commented
1 month ago Ok, so the problem was mostly user error: I had ggml binary coming in from llama.cpp, but the memory layout of the included ggml headers here must be different enough to cause problems. With some hacks I was able to get the headers from llama.cpp/ggml to be used, and the result doesn't have any issues or segfaulting. Updating ggml submodule here is also fine, just needs some paths adjusted. I think the ggml here is still out of date and should be updated, but it might be nice to have the CMake allow for an external source for ggml (such as one that's provided by the llama.cpp cmakelists, etc)
Cannot link this with binary linking llama.cpp, because the newer version of ggml seems to cause issues with the way this library prefers to directly access
.dataon ggml tensors. It seems like newer versions of ggml would prefer using the helper functions (e.g.ggml_set...) rather than direct storage access. One fix is required in the alpha cumulative product function, which is straightforward, but model.cpp's tensor loader needs more work.