search

AndreasFischer1985 / code-snippets

miscellaneous code snippets
21 stars 3 forks source link

AssertionError on Llama model-loading #1

Closed matsuobasho closed 1 year ago

matsuobasho commented 1 year ago

Hi @AndreasFischer1985 , really nice examples here.

I'm going through your notebook on LangChain with agents, and getting an AssertionError on line 93:

llamallm = Llama(model_path="./weights.bin",n_ctx=2048)

Any thoughts on what may be happening? I tried it with l7b and wiz but still having this issue. I've installed llama_cpp, running on Windows.

Thank you.

AndreasFischer1985 commented 1 year ago

Dear @matsuobasho,

thank you for bringing this up! Unfortunately there was an update of the llama-cpp-library that requires the newer gguf-models instead of the ggml-models I referred to in my code. If you are interested in using the bin-files referred to in my code, you may want to try the ctransformers package - otherwise you may just choose one of the newer gguf-models from hugingface.co?search=gguf.

I updated the code accordingly: https://github.com/AndreasFischer1985/code-snippets/blob/master/py/LangChain_HuggingFace_examples.py#L70-L81

Best regards,

Andreas Fischer

matsuobasho commented 1 year ago

Thanks @AndreasFischer1985. So the contents of weights.bin is now HTML code, consequently still get the assertion error when I run llamallm = Llama(model_path="./weights.bin",n_ctx=2048):

<!DOCTYPE html>
<html class="">
    <head>
        <meta charset="utf-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=no" />
        <meta name="description" content="We’re on a journey to advance and democratize artificial intelligence through open source and open science." />
        <meta property="fb:app_id" content="1321688464574422" />
        <meta name="twitter:card" content="summary_large_image" />
        <meta name="twitter:site" content="@huggingface" />
        <meta property="og:title" content="wizardlm-13b-v1.2.Q4_0.gguf · TheBloke/WizardLM-13B-V1.2-GGUF at main" />
        <meta property="og:type" content="website" />
        <meta property="og:url" content="https://huggingface.co/TheBloke/WizardLM-13B-V1.2-GGUF/blob/main/wizardlm-13b-v1.2.Q4_0.gguf" />
        <meta property="og:image" content="https://cdn-thumbnails.huggingface.co/social-thumbnails/models/TheBloke/WizardLM-13B-V1.2-GGUF.png" />

        <link rel="stylesheet" href="/front/build/kube-8e07b48/style.css" />

        <link rel="preconnect" href="https://fonts.gstatic.com" />
        <link
            href="https://fonts.googleapis.com/css2?family=Source+Sans+Pro:ital,wght@0,200;0,300;0,400;0,600;0,700;0,900;1,200;1,300;1,400;1,600;1,700;1,900&display=swap"
            rel="stylesheet"
        />
        <link
            href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@400;600;700&display=swap"
            rel="stylesheet"
        />

        <link
            rel="preload"
            href="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.12.0/katex.min.css"
            as="style"
            onload="this.onload=null;this.rel='stylesheet'"
        />
        <noscript>
            <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.12.0/katex.min.css" />
        </noscript>

        <title>wizardlm-13b-v1.2.Q4_0.gguf · TheBloke/WizardLM-13B-V1.2-GGUF at main</title>

        <script defer data-domain="huggingface.co" src="/js/script.js"></script>
        <script type="text/javascript" src="https://de5282c3ca0c.edge.sdk.awswaf.com/de5282c3ca0c/526cf06acb0d/challenge.js" defer></script>
    </head>
    <body class="flex flex-col min-h-screen bg-white dark:bg-gray-950 text-black ViewerBlobPage">
        <div class="flex min-h-screen flex-col">
    <div class="SVELTE_HYDRATER contents" data-props="{&quot;classNames&quot;:&quot;&quot;,&quot;isWide&quot;:false,&quot;isZh&quot;:false}" data-target="MainHeader"><header class="border-b border-gray-100 "><div class="w-full px-4 container flex h-16 items-center"><div class="flex flex-1 items-center"><a class="mr-5 flex flex-none items-center lg:mr-6" href="/"><img alt="Hugging Face's logo" class="w-7 md:mr-2" src="/front/assets/huggingface_logo-noborder.svg">
                <span class="hidden whitespace-nowrap text-lg font-bold md:block">Hugging Face</span></a>
            <div class="relative flex-1 lg:max-w-sm mr-2 sm:mr-4 lg:mr-6"><input autocomplete="off" class="w-full dark:bg-gray-950 pl-8 form-input-alt h-9 pr-3 focus:shadow-xl" name="" placeholder="Search models, datasets, users..."  spellcheck="false" type="text" value="">
    <svg class="absolute left-2.5 text-gray-400 top-1/2 transform -translate-y-1/2" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M30 28.59L22.45 21A11 11 0 1 0 21 22.45L28.59 30zM5 14a9 9 0 1 1 9 9a9 9 0 0 1-9-9z" fill="currentColor"></path></svg>
    </div>
            <div class="flex flex-none items-center justify-center p-0.5 place-self-stretch lg:hidden"><button class="relative z-40 flex h-6 w-8 items-center justify-center" type="button"><svg width="1em" height="1em" viewBox="0 0 10 10" class="text-xl" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" focusable="false" role="img" preserveAspectRatio="xMidYMid meet" fill="currentColor"><path fill-rule="evenodd" clip-rule="evenodd" d="M1.65039 2.9999C1.65039 2.8066 1.80709 2.6499 2.00039 2.6499H8.00039C8.19369 2.6499 8.35039 2.8066 8.35039 2.9999C8.35039 3.1932 8.19369 3.3499 8.00039 3.3499H2.00039C1.80709 3.3499 1.65039 3.1932 1.65039 2.9999ZM1.65039 4.9999C1.65039 4.8066 1.80709 4.6499 2.00039 4.6499H8.00039C8.19369 4.6499 8.35039 4.8066 8.35039 4.9999C8.35039 5.1932 8.19369 5.3499 8.00039 5.3499H2.00039C1.80709 5.3499 1.65039 5.1932 1.65039 4.9999ZM2.00039 6.6499C1.80709 6.6499 1.65039 6.8066 1.65039 6.9999C1.65039 7.1932 1.80709 7.3499 2.00039 7.3499H8.00039C8.19369 7.3499 8.35039 7.1932 8.35039 6.9999C8.35039 6.8066 8.19369 6.6499 8.00039 6.6499H2.00039Z"></path></svg>
        </button>
AndreasFischer1985 commented 1 year ago

I*m sorry, there was a typo in the URL provided and the result should also be saved as .gguf, not as .bin - I updated the code accordingly. As downloading the weights takes quite some time you may prefer downloading the gguf-files manually or via a tool that provides information on the download-progress (e.g., wget).

AndreasFischer1985 commented 1 year ago

P.S. I just finished downloading and everything seems to work now. Saving the file as .bin also seems to work (at least on Linux)

matsuobasho commented 1 year ago

Thank you so much for the update, it works for me too now. For my knowledge, how did you know that this is the correct url? On the HF model card, there instructions are through the HF cli among others, but I don't see a link to the url you used: huggingface-cli download TheBloke/WizardLM-13B-V1.2-GGUF wizardlm-13b-v1.2.q4_K_M.gguf --local-dir . --local-dir-use-symlinks False

AndreasFischer1985 commented 1 year ago

For the URL I simply inspected the link to the gguf-files in my web-browser: From the model card, just click on "Files and versions" and choose one of the gguf-files (e.g., I usually choose the version ending with Q4_0.gguf) and click on it. Then click on "download" to download the gguf-file (or right-click the download-button to copy the link to the file for using it in your code).