Open jwmatthews opened 4 months ago
To work around above, I needed to delete the locally cached file and retry request from IDE.
cd kai/data/vcr/coolstore/mistralai/mixtral-8x7b-instruct-v0-1
$ ls
-Users-jmatthews-git-jwmatthews-kai-example-coolstore-src-main-java-com-redhat-coolstore-service-OrderServiceMDB.java.yaml
-Users-jmatthews-git-jwmatthews-kai-example-coolstore-src-main-java-com-redhat-coolstore-service-ShoppingCartOrderProcessor.java.yaml
pom.xml.yaml
src-main-java-com-redhat-coolstore-model-Order.java.yaml
src-main-java-com-redhat-coolstore-model-OrderItem.java.yaml
src-main-java-com-redhat-coolstore-model-ShoppingCart.java.yaml
src-main-java-com-redhat-coolstore-persistence-Resources.java.yaml
src-main-java-com-redhat-coolstore-rest-CartEndpoint.java.yaml
src-main-java-com-redhat-coolstore-rest-OrderEndpoint.java.yaml
src-main-java-com-redhat-coolstore-rest-ProductEndpoint.java.yaml
src-main-java-com-redhat-coolstore-rest-RestApplication.java.yaml
src-main-java-com-redhat-coolstore-service-CatalogService.java.yaml
src-main-java-com-redhat-coolstore-service-InventoryNotificationMDB.java.yaml
src-main-java-com-redhat-coolstore-service-OrderService.java.yaml
src-main-java-com-redhat-coolstore-service-OrderServiceMDB.java.yaml
src-main-java-com-redhat-coolstore-service-ProductService.java.yaml
src-main-java-com-redhat-coolstore-service-PromoService.java.yaml
src-main-java-com-redhat-coolstore-service-ShippingService.java.yaml
src-main-java-com-redhat-coolstore-service-ShoppingCartOrderProcessor.java.yaml
src-main-java-com-redhat-coolstore-service-ShoppingCartService.java.yaml
src-main-java-com-redhat-coolstore-utils-DataBaseMigrationStartup.java.yaml
src-main-java-com-redhat-coolstore-utils-Producers.java.yaml
src-main-java-com-redhat-coolstore-utils-StartupListener.java.yaml
src-main-java-com-redhat-coolstore-utils-Transformers.java.yaml
src-main-resources-META-INF-persistence.xml.yaml
src-main-webapp-WEB-INF-beans.xml.yaml
src-main-webapp-WEB-INF-web.xml.yaml
rm -- "-Users-jmatthews-git-jwmatthews-kai-example-coolstore-src-main-java-com-redhat-coolstore-service-ShoppingCartOrderProcessor.java.yaml"
Notice that until we address #150 the cached responses are indexed as full file names from the IDE side
These errors:
WARNING - 2024-04-09 07:10:35,222 - [server.py:576 - get_incident_solutions_for_file() ] - Request to model failed for batch 1/1 for /Users/jmatthews/git/jwmatthews/kai/example/coolstore/src/main/java/com/redhat/coolstore/service/ShoppingCartOrderProcessor.java with exception, retrying in 10s
Can't overwrite existing cassette ('/Users/jmatthews/git/jwmatthews/kai/kai/data/vcr/coolstore/mistralai/mixtral-8x7b-instruct-v0-1/-Users-jmatthews-git-jwmatthews-kai-example-coolstore-src-main-java-com-redhat-coolstore-service-ShoppingCartOrderProcessor.java.yaml') in your current record mode ('once').
No match for the request (<Request (POST) https://bam-api.res.ibm.com/v2/text/chat_stream?version=2024-01-10>) was found.
Found 1 similar requests with 2 different matcher(s) :
1 - (<Request (POST) https://bam-api.res.ibm.com/v2/text/chat_stream?version=2024-01-10>).
Matchers succeeded : ['uri', 'method', 'scheme', 'host', 'port', 'path', 'query']
Matchers failed :
headers - assertion failure :
{'host': 'bam-api.res.ibm.com', 'accept': '*/*, text/event-stream', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'x-request-origin': 'python-sdk/2.2.0', 'user-agent': 'python-sdk/2.2.0', 'cache-control': 'no-store', 'content-length': '4826', 'content-type': 'application/json'} != {'accept': '*/*, text/event-stream', 'accept-encoding': 'gzip, deflate', 'cache-control': 'no-store', 'connection': 'keep-alive', 'content-length': '4952', 'content-type': 'application/json', 'host': 'bam-api.res.ibm.com', 'user-agent': 'python-sdk/2.2.0', 'x-request-origin': 'python-sdk/2.2.0'}
body - assertion failure :
mean the request was detected as a duplicate request with a different request body, which errors out by default. In #159 I've made it so that it no longer matches on request body, so it should always return a cached response. This does mean we need to be more deliberate about re-recording our responses as the prompts change
@jwmatthews can we close this one?
I may be seeing problems where if the LLM fails to give a good response while we are using DEMO_MODE=true, we get stuck in a spot where we can not recover. This is not confirmed and is only a guess based on my trying several times to generate a response for a specific file
service/ShoppingCartOrderProcessor.java
.I am guessing that the situation is:
Perhaps we need some logic that will only cache a response for a successful request from LLM?
I am running with:
DEMO_MODE=true
kai/config.toml:
I am using the IDE and attempting to general a 'fix all' for
service/ShoppingCartOrderProcessor.java