mishushakov / llm-scraper

Turn any webpage into structured data using LLMs
MIT License
2.41k stars 147 forks source link

Attempted Fix: Sometimes `tool_calls` is undefined and crashes server #21

Open Ademsk1 opened 6 months ago

Ademsk1 commented 6 months ago

Hi

This might help with issue #20 . I'm hoping to test it for openai later on today, but I don't have access to a local language model right now, are there any models you recommend I use?

Thanks

mishushakov commented 6 months ago

Nice. Have you tested it?

Ademsk1 commented 6 months ago

@mishushakov tested for the openai section, doesn't crash the server anymore :D Screenshot 2024-05-02 at 11 18 39

I haven't tested the Llama one, but I think it should work the same. I started working on one last night, so will let you know of the results later on this evening.

mishushakov commented 6 months ago

Very good! I will run some tests too. Looking forward to get it merged 😄

Ademsk1 commented 6 months ago

@mishushakov Been trying to test using this model as my model, but my machine cannot handle it. What model are you using?

mishushakov commented 6 months ago

TinyLlama

mishushakov commented 6 months ago

I can test for you, no worries

Ademsk1 commented 6 months ago

Tested this with TinyLlama and no crashes :D Screenshot 2024-05-03 at 13 37 33

that being said, there weren't any crashes without the try/catch block for the local model, because we don't reference any tool_calls or the like. Still I think it's practical to have it there just in case.

Ademsk1 commented 6 months ago

While I have you @mishushakov , maybe you have more experience, with these models, I seem to get this error when trying to scrape data from a fairly large website. GGML_ASSERT: ... n_tokens_all <= cparams.n_batch Did you have this error at all? Any advice?

mishushakov commented 6 months ago

I believe you're hitting the input token limit. We need to think about a way of splitting websites into multiple chunks, maybe. Can you open an issue for this?