search

tjake / Jlama

Jlama is a modern LLM inference engine for Java
Apache License 2.0
665 stars 62 forks source link

Error while running locally using run-cli.sh #45

Open ved-asole opened 2 months ago

ved-asole commented 2 months ago 
[ERROR] No plugin found for prefix 'jlama-core' in the current project and in the plugin groups [org.apache.maven.plugins, org.codehaus.mojo] available from the repositories [local (/home/ved/.m2/repository), central (https://repo.maven.apache.org/maven2)] -> [Help 1]
org.apache.maven.plugin.prefix.NoPluginFoundForPrefixException: No plugin found for prefix 'jlama-core' in the current project and in the plugin groups [org.apache.maven.plugins, org.codehaus.mojo] available from the repositories [local (/home/ved/.m2/repository), central (https://repo.maven.apache.org/maven2)]
tjake commented 2 months ago

Hi @ved-asole can you give some more details like what you did to run this? did you just checkout the code and run-cli.sh?

ved-asole commented 2 months ago

As per the documentation I used the below commands :

./run-cli.sh download gpt2-medium
./run-cli.sh download -t XXXXXXXX meta-llama/Llama-2-7b-chat-hf
./run-cli.sh download intfloat/e5-small-v2
tjake commented 2 months ago

Thanks, I recently added support for multiple jvms which messed this up. I'll fix. In the meantime use the code from the latest release tag git checkout v0.2.1 and it should work

ved-asole commented 2 months ago

Thanks for the update. I will use the older v0.2.1 version for now

ved-asole commented 2 months ago

@tjake I'm not sure where I can get the older v0.2.1 version/ Could you please suggest how i can get it?

ved-asole commented 2 months ago

Getting below error while running v0.2.1 :

Command : sudo ./run-cli.sh download gpt2-medium

An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "make" (in directory "/home/ved/Downloads/temp/Jlama-0.2.1/jlama-native"): error=2, No such file or directory
[ERROR] around Ant part ...<exec resolveexecutable="true" failonerror="true" executable="make">... @ 4:71 in /home/ved/Downloads/temp/Jlama-0.2.1/jlama-native/target/antrun/build-main.xml
tjake commented 2 months ago

You need to build the native extensions requires make and gcc.

I realize this is all a PITA. I'm going to add release artifacts for all the platforms when I get a chance.

tjake commented 2 months ago

The easiest way to "just run a model" is with langchain4j integrations. Try this: https://github.com/langchain4j/langchain4j-examples/tree/main/jlama-examples

ved-asole commented 2 months ago

I want to run the models locally. Will it run with the langchain4j implementations?

tjake commented 2 months ago

Yes, it uses the maven artifacts which are pre-built

ved-asole commented 2 months ago

I got the below error while using it :

2024-09-02T15:41:34.997+05:30  INFO 94027 --- [pdf-assistant] [           main] c.g.t.j.safetensors.SafeTensorSupport    : Downloaded file: /home/ved/.jlama/models/TinyLlama-1.1B-Chat-v1.0-Jlama-Q4/tokenizer_config.json
2024-09-02T15:41:44.526+05:30  WARN 94027 --- [pdf-assistant] [           main] c.github.tjake.jlama.util.MachineSpec    : Unknown vector type: 128
2024-09-02T15:41:44.612+05:30  WARN 94027 --- [pdf-assistant] [           main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'chatController' defined in file [/mnt/hdd1/Coding/springWorks/pdf-assistant/target/classes/com/vedasole/pdf_assistant/ChatController.class]: Unsatisfied dependency expressed through constructor parameter 0: Error creating bean with name 'conversationalRetrievalChain' defined in com.vedasole.pdf_assistant.PdfAssistantConfig: Failed to instantiate [dev.langchain4j.chain.ConversationalRetrievalChain]: Factory method 'conversationalRetrievalChain' threw exception with message: java.lang.reflect.InvocationTargetException
2024-09-02T15:41:44.615+05:30  INFO 94027 --- [pdf-assistant] [           main] o.apache.catalina.core.StandardService   : Stopping service [Tomcat]
2024-09-02T15:41:44.638+05:30  INFO 94027 --- [pdf-assistant] [           main] .s.b.a.l.ConditionEvaluationReportLogger : 

Error starting ApplicationContext. To display the condition evaluation report re-run your application with 'debug' enabled.
2024-09-02T15:41:44.670+05:30 ERROR 94027 --- [pdf-assistant] [           main] o.s.boot.SpringApplication               : Application run failed

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'chatController' defined in file [/mnt/hdd1/Coding/springWorks/pdf-assistant/target/classes/com/vedasole/pdf_assistant/ChatController.class]: Unsatisfied dependency expressed through constructor parameter 0: Error creating bean with name 'conversationalRetrievalChain' defined in com.vedasole.pdf_assistant.PdfAssistantConfig: Failed to instantiate [dev.langchain4j.chain.ConversationalRetrievalChain]: Factory method 'conversationalRetrievalChain' threw exception with message: java.lang.reflect.InvocationTargetException
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:795) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:237) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1375) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1212) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:562) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:522) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:337) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:335) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:975) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:971) ~[spring-context-6.1.12.jar:6.1.12]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:625) ~[spring-context-6.1.12.jar:6.1.12]
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) ~[spring-boot-3.3.3.jar:3.3.3]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754) ~[spring-boot-3.3.3.jar:3.3.3]
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:456) ~[spring-boot-3.3.3.jar:3.3.3]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:335) ~[spring-boot-3.3.3.jar:3.3.3]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1363) ~[spring-boot-3.3.3.jar:3.3.3]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1352) ~[spring-boot-3.3.3.jar:3.3.3]
    at com.vedasole.pdf_assistant.PdfAssistantApplication.main(PdfAssistantApplication.java:44) ~[classes/:na]
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'conversationalRetrievalChain' defined in com.vedasole.pdf_assistant.PdfAssistantConfig: Failed to instantiate [dev.langchain4j.chain.ConversationalRetrievalChain]: Factory method 'conversationalRetrievalChain' threw exception with message: java.lang.reflect.InvocationTargetException
    at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:648) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:485) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1355) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1185) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:562) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:522) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:337) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:335) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1443) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1353) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:904) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:782) ~[spring-beans-6.1.12.jar:6.1.12]
    ... 19 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [dev.langchain4j.chain.ConversationalRetrievalChain]: Factory method 'conversationalRetrievalChain' threw exception with message: java.lang.reflect.InvocationTargetException
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:178) ~[spring-beans-6.1.12.jar:6.1.12]
    at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:644) ~[spring-beans-6.1.12.jar:6.1.12]
    ... 33 common frames omitted
Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at com.github.tjake.jlama.model.ModelSupport.loadModel(ModelSupport.java:161) ~[jlama-core-0.2.1.jar:na]
    at dev.langchain4j.model.jlama.JlamaModel$Loader.load(JlamaModel.java:92) ~[langchain4j-jlama-0.33.0.jar:na]
    at dev.langchain4j.model.jlama.JlamaChatModel.<init>(JlamaChatModel.java:51) ~[langchain4j-jlama-0.33.0.jar:na]
    at dev.langchain4j.model.jlama.JlamaChatModel$JlamaChatModelBuilder.build(JlamaChatModel.java:29) ~[langchain4j-jlama-0.33.0.jar:na]
    at com.vedasole.pdf_assistant.PdfAssistantConfig.conversationalRetrievalChain(PdfAssistantConfig.java:56) ~[classes/:na]
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:146) ~[spring-beans-6.1.12.jar:6.1.12]
    ... 34 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
    at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74) ~[na:na]
    at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[na:na]
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[na:na]
    at com.github.tjake.jlama.model.ModelSupport.loadModel(ModelSupport.java:153) ~[jlama-core-0.2.1.jar:na]
    ... 41 common frames omitted
Caused by: java.lang.UnsupportedOperationException: null
    at com.github.tjake.jlama.tensor.operations.PanamaTensorOperations.quantize(PanamaTensorOperations.java:1484) ~[jlama-core-0.2.1.jar:na]
    at com.github.tjake.jlama.tensor.operations.NativeTensorOperations.quantize(NativeTensorOperations.java:355) ~[jlama-native-0.2.1-linux-x86_64.jar:na]
    at com.github.tjake.jlama.model.AbstractModel.<init>(AbstractModel.java:117) ~[jlama-core-0.2.1.jar:na]
    at com.github.tjake.jlama.model.llama.LlamaModel.<init>(LlamaModel.java:55) ~[jlama-core-0.2.1.jar:na]
    at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[na:na]
    ... 44 common frames omitted
tjake commented 2 months ago

Which model are you loading and what kind of cpu are you loading it on?

tjake commented 2 months ago

Looking at the line it seems to be a non-arm AVX which I can support but currently needs a code change. I can fix that.