Closed 65a closed 3 weeks ago
Ok, so the problem was mostly user error: I had ggml binary coming in from llama.cpp, but the memory layout of the included ggml headers here must be different enough to cause problems. With some hacks I was able to get the headers from llama.cpp/ggml to be used, and the result doesn't have any issues or segfaulting. Updating ggml submodule here is also fine, just needs some paths adjusted. I think the ggml here is still out of date and should be updated, but it might be nice to have the CMake allow for an external source for ggml (such as one that's provided by the llama.cpp cmakelists, etc)
This looks fixed by 4a6e36edc586779918535e12b4fbe0583044ee6f so marking closed. I can send some PRs after #324 to make it possible to provide an alternative GGML header directory so that static linking with a binary that already links a different one (e.g. llama.cpp) can work consistently.
Cannot link this with binary linking llama.cpp, because the newer version of ggml seems to cause issues with the way this library prefers to directly access
.data
on ggml tensors. It seems like newer versions of ggml would prefer using the helper functions (e.g.ggml_set...
) rather than direct storage access. One fix is required in the alpha cumulative product function, which is straightforward, but model.cpp's tensor loader needs more work.