Ensure DEMO_MODE is able to cope with LLM server side errors

jwmatthews commented 2 months ago

I may be seeing problems where if the LLM fails to give a good response while we are using DEMO_MODE=true, we get stuck in a spot where we can not recover. This is not confirmed and is only a guess based on my trying several times to generate a response for a specific file service/ShoppingCartOrderProcessor.java.

I am guessing that the situation is:

Start with NO cached response
Server attempts to get response from LLM, it goes through several retries and fails each time
We exhaust retries and don't have a response
We cache the bad response or get into a bad state
All subsequent requests to this file fail

Perhaps we need some logic that will only cache a response for a successful request from LLM?

I am running with:

DEMO_MODE=true

kai/config.toml:

provider = "IBMOpenSource"
args = { model_id = "mistralai/mixtral-8x7b-instruct-v0-1" }

I am using the IDE and attempting to general a 'fix all' for service/ShoppingCartOrderProcessor.java


ERROR - 2024-04-08 15:11:48,525 - [server.py:582 - get_incident_solutions_for_file() ] - /Users/jmatthews/git/jwmatthews/kai/example/coolstore/src/main/java/com/redhat/coolstore/service/ShoppingCartOrderProcessor.java failed to migrate
INFO - 2024-04-08 16:08:29,205 - [server.py:470 - get_incident_solutions_for_file() ] - START - App: 'coolstore', File: '/Users/jmatthews/git/jwmatthews/kai/example/coolstore/src/main/java/com/redhat/coolstore/service/ShoppingCartOrderProcessor.java' with 10 incidents'
INFO - 2024-04-08 16:08:29,264 - [server.py:548 - get_incident_solutions_for_file() ] - Processing incident batch 1/1 for /Users/jmatthews/git/jwmatthews/kai/example/coolstore/src/main/java/com/redhat/coolstore/service/ShoppingCartOrderProcessor.java
WARNING - 2024-04-08 16:08:29,276 - [server.py:576 - get_incident_solutions_for_file() ] - Request to model failed for batch 1/1 for /Users/jmatthews/git/jwmatthews/kai/example/coolstore/src/main/java/com/redhat/coolstore/service/ShoppingCartOrderProcessor.java with exception, retrying in 10s
Can't overwrite existing cassette ('/Users/jmatthews/git/jwmatthews/kai/kai/data/vcr/coolstore/mistralai/mixtral-8x7b-instruct-v0-1/-Users-jmatthews-git-jwmatthews-kai-example-coolstore-src-main-java-com-redhat-coolstore-service-ShoppingCartOrderProcessor.java.yaml') in your current record mode ('once').
No match for the request (<Request (POST) https://bam-api.res.ibm.com/v2/text/chat_stream?version=2024-01-10>) was found.
Found 1 similar requests with 2 different matcher(s) :

1 - (<Request (POST) https://bam-api.res.ibm.com/v2/text/chat_stream?version=2024-01-10>).
Matchers succeeded : ['uri', 'method', 'scheme', 'host', 'port', 'path', 'query']
Matchers failed :
headers - assertion failure :
{'host': 'bam-api.res.ibm.com', 'accept': '*/*, text/event-stream', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'x-request-origin': 'python-sdk/2.2.0', 'user-agent': 'python-sdk/2.2.0', 'cache-control': 'no-store', 'content-length': '4826', 'content-type': 'application/json'} != {'accept': '*/*, text/event-stream', 'accept-encoding': 'gzip, deflate', 'cache-control': 'no-store', 'connection': 'keep-alive', 'content-length': '4952', 'content-type': 'application/json', 'host': 'bam-api.res.ibm.com', 'user-agent': 'python-sdk/2.2.0', 'x-request-origin': 'python-sdk/2.2.0'}
body - assertion failure :

WARNING - 2024-04-08 16:08:39,311 - [server.py:576 - get_incident_solutions_for_file() ] - Request to model failed for batch 1/1 for /Users/jmatthews/git/jwmatthews/kai/example/coolstore/src/main/java/com/redhat/coolstore/service/ShoppingCartOrderProcessor.java with exception, retrying in 10s
Can't overwrite existing cassette ('/Users/jmatthews/git/jwmatthews/kai/kai/data/vcr/coolstore/mistralai/mixtral-8x7b-instruct-v0-1/-Users-jmatthews-git-jwmatthews-kai-example-coolstore-src-main-java-com-redhat-coolstore-service-ShoppingCartOrderProcessor.java.yaml') in your current record mode ('once').
No match for the request (<Request (POST) https://bam-api.res.ibm.com/v2/text/chat_stream?version=2024-01-10>) was found.
Found 1 similar requests with 2 different matcher(s) :

1 - (<Request (POST) https://bam-api.res.ibm.com/v2/text/chat_stream?version=2024-01-10>).
Matchers succeeded : ['uri', 'method', 'scheme', 'host', 'port', 'path', 'query']
Matchers failed :
headers - assertion failure :
{'host': 'bam-api.res.ibm.com', 'accept': '*/*, text/event-stream', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'x-request-origin': 'python-sdk/2.2.0', 'user-agent': 'python-sdk/2.2.0', 'cache-control': 'no-store', 'content-length': '4826', 'content-type': 'application/json'} != {'accept': '*/*, text/event-stream', 'accept-encoding': 'gzip, deflate', 'cache-control': 'no-store', 'connection': 'keep-alive', 'content-length': '4952', 'content-type': 'application/json', 'host': 'bam-api.res.ibm.com', 'user-agent': 'python-sdk/2.2.0', 'x-request-origin': 'python-sdk/2.2.0'}
body - assertion failure :

WARNING - 2024-04-08 16:08:49,345 - [server.py:576 - get_incident_solutions_for_file() ] - Request to model failed for batch 1/1 for /Users/jmatthews/git/jwmatthews/kai/example/coolstore/src/main/java/com/redhat/coolstore/service/ShoppingCartOrderProcessor.java with exception, retrying in 10s
Can't overwrite existing cassette ('/Users/jmatthews/git/jwmatthews/kai/kai/data/vcr/coolstore/mistralai/mixtral-8x7b-instruct-v0-1/-Users-jmatthews-git-jwmatthews-kai-example-coolstore-src-main-java-com-redhat-coolstore-service-ShoppingCartOrderProcessor.java.yaml') in your current record mode ('once').
No match for the request (<Request (POST) https://bam-api.res.ibm.com/v2/text/chat_stream?version=2024-01-10>) was found.
Found 1 similar requests with 2 different matcher(s) :

1 - (<Request (POST) https://bam-api.res.ibm.com/v2/text/chat_stream?version=2024-01-10>).
Matchers succeeded : ['uri', 'method', 'scheme', 'host', 'port', 'path', 'query']
Matchers failed :
headers - assertion failure :
{'host': 'bam-api.res.ibm.com', 'accept': '*/*, text/event-stream', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'x-request-origin': 'python-sdk/2.2.0', 'user-agent': 'python-sdk/2.2.0', 'cache-control': 'no-store', 'content-length': '4826', 'content-type': 'application/json'} != {'accept': '*/*, text/event-stream', 'accept-encoding': 'gzip, deflate', 'cache-control': 'no-store', 'connection': 'keep-alive', 'content-length': '4952', 'content-type': 'application/json', 'host': 'bam-api.res.ibm.com', 'user-agent': 'python-sdk/2.2.0', 'x-request-origin': 'python-sdk/2.2.0'}
body - assertion failure :

WARNING - 2024-04-08 16:08:59,375 - [server.py:576 - get_incident_solutions_for_file() ] - Request to model failed for batch 1/1 for /Users/jmatthews/git/jwmatthews/kai/example/coolstore/src/main/java/com/redhat/coolstore/service/ShoppingCartOrderProcessor.java with exception, retrying in 10s
Can't overwrite existing cassette ('/Users/jmatthews/git/jwmatthews/kai/kai/data/vcr/coolstore/mistralai/mixtral-8x7b-instruct-v0-1/-Users-jmatthews-git-jwmatthews-kai-example-coolstore-src-main-java-com-redhat-coolstore-service-ShoppingCartOrderProcessor.java.yaml') in your current record mode ('once').
No match for the request (<Request (POST) https://bam-api.res.ibm.com/v2/text/chat_stream?version=2024-01-10>) was found.
Found 1 similar requests with 2 different matcher(s) :

1 - (<Request (POST) https://bam-api.res.ibm.com/v2/text/chat_stream?version=2024-01-10>).
Matchers succeeded : ['uri', 'method', 'scheme', 'host', 'port', 'path', 'query']
Matchers failed :
headers - assertion failure :
{'host': 'bam-api.res.ibm.com', 'accept': '*/*, text/event-stream', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'x-request-origin': 'python-sdk/2.2.0', 'user-agent': 'python-sdk/2.2.0', 'cache-control': 'no-store', 'content-length': '4826', 'content-type': 'application/json'} != {'accept': '*/*, text/event-stream', 'accept-encoding': 'gzip, deflate', 'cache-control': 'no-store', 'connection': 'keep-alive', 'content-length': '4952', 'content-type': 'application/json', 'host': 'bam-api.res.ibm.com', 'user-agent': 'python-sdk/2.2.0', 'x-request-origin': 'python-sdk/2.2.0'}
body - assertion failure :

WARNING - 2024-04-08 16:09:09,415 - [server.py:576 - get_incident_solutions_for_file() ] - Request to model failed for batch 1/1 for /Users/jmatthews/git/jwmatthews/kai/example/coolstore/src/main/java/com/redhat/coolstore/service/ShoppingCartOrderProcessor.java with exception, retrying in 10s
Can't overwrite existing cassette ('/Users/jmatthews/git/jwmatthews/kai/kai/data/vcr/coolstore/mistralai/mixtral-8x7b-instruct-v0-1/-Users-jmatthews-git-jwmatthews-kai-example-coolstore-src-main-java-com-redhat-coolstore-service-ShoppingCartOrderProcessor.java.yaml') in your current record mode ('once').
No match for the request (<Request (POST) https://bam-api.res.ibm.com/v2/text/chat_stream?version=2024-01-10>) was found.
Found 1 similar requests with 2 different matcher(s) :

1 - (<Request (POST) https://bam-api.res.ibm.com/v2/text/chat_stream?version=2024-01-10>).
Matchers succeeded : ['uri', 'method', 'scheme', 'host', 'port', 'path', 'query']
Matchers failed :
headers - assertion failure :
{'host': 'bam-api.res.ibm.com', 'accept': '*/*, text/event-stream', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'x-request-origin': 'python-sdk/2.2.0', 'user-agent': 'python-sdk/2.2.0', 'cache-control': 'no-store', 'content-length': '4826', 'content-type': 'application/json'} != {'accept': '*/*, text/event-stream', 'accept-encoding': 'gzip, deflate', 'cache-control': 'no-store', 'connection': 'keep-alive', 'content-length': '4952', 'content-type': 'application/json', 'host': 'bam-api.res.ibm.com', 'user-agent': 'python-sdk/2.2.0', 'x-request-origin': 'python-sdk/2.2.0'}
body - assertion failure :

ERROR - 2024-04-08 16:09:19,419 - [server.py:582 - get_incident_solutions_for_file() ] - /Users/jmatthews/git/jwmatthews/kai/example/coolstore/src/main/java/com/redhat/coolstore/service/ShoppingCartOrderProcessor.java failed to migrate
INFO - 2024-04-09 07:10:25,045 - [server.py:470 - get_incident_solutions_for_file() ] - START - App: 'coolstore', File: '/Users/jmatthews/git/jwmatthews/kai/example/coolstore/src/main/java/com/redhat/coolstore/service/ShoppingCartOrderProcessor.java' with 10 incidents'
INFO - 2024-04-09 07:10:25,158 - [server.py:548 - get_incident_solutions_for_file() ] - Processing incident batch 1/1 for /Users/jmatthews/git/jwmatthews/kai/example/coolstore/src/main/java/com/redhat/coolstore/service/ShoppingCartOrderProcessor.java
WARNING - 2024-04-09 07:10:25,172 - [server.py:576 - get_incident_solutions_for_file() ] - Request to model failed for batch 1/1 for /Users/jmatthews/git/jwmatthews/kai/example/coolstore/src/main/java/com/redhat/coolstore/service/ShoppingCartOrderProcessor.java with exception, retrying in 10s
Can't overwrite existing cassette ('/Users/jmatthews/git/jwmatthews/kai/kai/data/vcr/coolstore/mistralai/mixtral-8x7b-instruct-v0-1/-Users-jmatthews-git-jwmatthews-kai-example-coolstore-src-main-java-com-redhat-coolstore-service-ShoppingCartOrderProcessor.java.yaml') in your current record mode ('once').
No match for the request (<Request (POST) https://bam-api.res.ibm.com/v2/text/chat_stream?version=2024-01-10>) was found.
Found 1 similar requests with 2 different matcher(s) :

1 - (<Request (POST) https://bam-api.res.ibm.com/v2/text/chat_stream?version=2024-01-10>).
Matchers succeeded : ['uri', 'method', 'scheme', 'host', 'port', 'path', 'query']
Matchers failed :
headers - assertion failure :
{'host': 'bam-api.res.ibm.com', 'accept': '*/*, text/event-stream', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'x-request-origin': 'python-sdk/2.2.0', 'user-agent': 'python-sdk/2.2.0', 'cache-control': 'no-store', 'content-length': '4826', 'content-type': 'application/json'} != {'accept': '*/*, text/event-stream', 'accept-encoding': 'gzip, deflate', 'cache-control': 'no-store', 'connection': 'keep-alive', 'content-length': '4952', 'content-type': 'application/json', 'host': 'bam-api.res.ibm.com', 'user-agent': 'python-sdk/2.2.0', 'x-request-origin': 'python-sdk/2.2.0'}
body - assertion failure :

WARNING - 2024-04-09 07:10:35,222 - [server.py:576 - get_incident_solutions_for_file() ] - Request to model failed for batch 1/1 for /Users/jmatthews/git/jwmatthews/kai/example/coolstore/src/main/java/com/redhat/coolstore/service/ShoppingCartOrderProcessor.java with exception, retrying in 10s
Can't overwrite existing cassette ('/Users/jmatthews/git/jwmatthews/kai/kai/data/vcr/coolstore/mistralai/mixtral-8x7b-instruct-v0-1/-Users-jmatthews-git-jwmatthews-kai-example-coolstore-src-main-java-com-redhat-coolstore-service-ShoppingCartOrderProcessor.java.yaml') in your current record mode ('once').
No match for the request (<Request (POST) https://bam-api.res.ibm.com/v2/text/chat_stream?version=2024-01-10>) was found.
Found 1 similar requests with 2 different matcher(s) :

1 - (<Request (POST) https://bam-api.res.ibm.com/v2/text/chat_stream?version=2024-01-10>).
Matchers succeeded : ['uri', 'method', 'scheme', 'host', 'port', 'path', 'query']
Matchers failed :
headers - assertion failure :
{'host': 'bam-api.res.ibm.com', 'accept': '*/*, text/event-stream', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'x-request-origin': 'python-sdk/2.2.0', 'user-agent': 'python-sdk/2.2.0', 'cache-control': 'no-store', 'content-length': '4826', 'content-type': 'application/json'} != {'accept': '*/*, text/event-stream', 'accept-encoding': 'gzip, deflate', 'cache-control': 'no-store', 'connection': 'keep-alive', 'content-length': '4952', 'content-type': 'application/json', 'host': 'bam-api.res.ibm.com', 'user-agent': 'python-sdk/2.2.0', 'x-request-origin': 'python-sdk/2.2.0'}
body - assertion failure :

WARNING - 2024-04-09 07:10:45,250 - [server.py:576 - get_incident_solutions_for_file() ] - Request to model failed for batch 1/1 for /Users/jmatthews/git/jwmatthews/kai/example/coolstore/src/main/java/com/redhat/coolstore/service/ShoppingCartOrderProcessor.java with exception, retrying in 10s
Can't overwrite existing cassette ('/Users/jmatthews/git/jwmatthews/kai/kai/data/vcr/coolstore/mistralai/mixtral-8x7b-instruct-v0-1/-Users-jmatthews-git-jwmatthews-kai-example-coolstore-src-main-java-com-redhat-coolstore-service-ShoppingCartOrderProcessor.java.yaml') in your current record mode ('once').
No match for the request (<Request (POST) https://bam-api.res.ibm.com/v2/text/chat_stream?version=2024-01-10>) was found.
Found 1 similar requests with 2 different matcher(s) :

1 - (<Request (POST) https://bam-api.res.ibm.com/v2/text/chat_stream?version=2024-01-10>).
Matchers succeeded : ['uri', 'method', 'scheme', 'host', 'port', 'path', 'query']
Matchers failed :
headers - assertion failure :
{'host': 'bam-api.res.ibm.com', 'accept': '*/*, text/event-stream', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'x-request-origin': 'python-sdk/2.2.0', 'user-agent': 'python-sdk/2.2.0', 'cache-control': 'no-store', 'content-length': '4826', 'content-type': 'application/json'} != {'accept': '*/*, text/event-stream', 'accept-encoding': 'gzip, deflate', 'cache-control': 'no-store', 'connection': 'keep-alive', 'content-length': '4952', 'content-type': 'application/json', 'host': 'bam-api.res.ibm.com', 'user-agent': 'python-sdk/2.2.0', 'x-request-origin': 'python-sdk/2.2.0'}
body - assertion failure :

jwmatthews commented 2 months ago

To work around above, I needed to delete the locally cached file and retry request from IDE.

cd kai/data/vcr/coolstore/mistralai/mixtral-8x7b-instruct-v0-1

$ ls

    -Users-jmatthews-git-jwmatthews-kai-example-coolstore-src-main-java-com-redhat-coolstore-service-OrderServiceMDB.java.yaml
    -Users-jmatthews-git-jwmatthews-kai-example-coolstore-src-main-java-com-redhat-coolstore-service-ShoppingCartOrderProcessor.java.yaml
    pom.xml.yaml
    src-main-java-com-redhat-coolstore-model-Order.java.yaml
    src-main-java-com-redhat-coolstore-model-OrderItem.java.yaml
    src-main-java-com-redhat-coolstore-model-ShoppingCart.java.yaml
    src-main-java-com-redhat-coolstore-persistence-Resources.java.yaml
    src-main-java-com-redhat-coolstore-rest-CartEndpoint.java.yaml
    src-main-java-com-redhat-coolstore-rest-OrderEndpoint.java.yaml
    src-main-java-com-redhat-coolstore-rest-ProductEndpoint.java.yaml
    src-main-java-com-redhat-coolstore-rest-RestApplication.java.yaml
    src-main-java-com-redhat-coolstore-service-CatalogService.java.yaml
    src-main-java-com-redhat-coolstore-service-InventoryNotificationMDB.java.yaml
    src-main-java-com-redhat-coolstore-service-OrderService.java.yaml
    src-main-java-com-redhat-coolstore-service-OrderServiceMDB.java.yaml
    src-main-java-com-redhat-coolstore-service-ProductService.java.yaml
    src-main-java-com-redhat-coolstore-service-PromoService.java.yaml
    src-main-java-com-redhat-coolstore-service-ShippingService.java.yaml
    src-main-java-com-redhat-coolstore-service-ShoppingCartOrderProcessor.java.yaml
    src-main-java-com-redhat-coolstore-service-ShoppingCartService.java.yaml
    src-main-java-com-redhat-coolstore-utils-DataBaseMigrationStartup.java.yaml
    src-main-java-com-redhat-coolstore-utils-Producers.java.yaml
    src-main-java-com-redhat-coolstore-utils-StartupListener.java.yaml
    src-main-java-com-redhat-coolstore-utils-Transformers.java.yaml
    src-main-resources-META-INF-persistence.xml.yaml
    src-main-webapp-WEB-INF-beans.xml.yaml
    src-main-webapp-WEB-INF-web.xml.yaml

rm -- "-Users-jmatthews-git-jwmatthews-kai-example-coolstore-src-main-java-com-redhat-coolstore-service-ShoppingCartOrderProcessor.java.yaml"

Notice that until we address #150 the cached responses are indexed as full file names from the IDE side

fabianvf commented 2 months ago

These errors:

WARNING - 2024-04-09 07:10:35,222 - [server.py:576 - get_incident_solutions_for_file() ] - Request to model failed for batch 1/1 for /Users/jmatthews/git/jwmatthews/kai/example/coolstore/src/main/java/com/redhat/coolstore/service/ShoppingCartOrderProcessor.java with exception, retrying in 10s
Can't overwrite existing cassette ('/Users/jmatthews/git/jwmatthews/kai/kai/data/vcr/coolstore/mistralai/mixtral-8x7b-instruct-v0-1/-Users-jmatthews-git-jwmatthews-kai-example-coolstore-src-main-java-com-redhat-coolstore-service-ShoppingCartOrderProcessor.java.yaml') in your current record mode ('once').
No match for the request (<Request (POST) https://bam-api.res.ibm.com/v2/text/chat_stream?version=2024-01-10>) was found.
Found 1 similar requests with 2 different matcher(s) :

1 - (<Request (POST) https://bam-api.res.ibm.com/v2/text/chat_stream?version=2024-01-10>).
Matchers succeeded : ['uri', 'method', 'scheme', 'host', 'port', 'path', 'query']
Matchers failed :
headers - assertion failure :
{'host': 'bam-api.res.ibm.com', 'accept': '*/*, text/event-stream', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive', 'x-request-origin': 'python-sdk/2.2.0', 'user-agent': 'python-sdk/2.2.0', 'cache-control': 'no-store', 'content-length': '4826', 'content-type': 'application/json'} != {'accept': '*/*, text/event-stream', 'accept-encoding': 'gzip, deflate', 'cache-control': 'no-store', 'connection': 'keep-alive', 'content-length': '4952', 'content-type': 'application/json', 'host': 'bam-api.res.ibm.com', 'user-agent': 'python-sdk/2.2.0', 'x-request-origin': 'python-sdk/2.2.0'}
body - assertion failure :

mean the request was detected as a duplicate request with a different request body, which errors out by default. In #159 I've made it so that it no longer matches on request body, so it should always return a cached response. This does mean we need to be more deliberate about re-recording our responses as the prompts change

fabianvf commented 2 months ago

@jwmatthews can we close this one?

konveyor-ecosystem / kai

Ensure DEMO_MODE is able to cope with LLM server side errors #161