Vespa Deploy fails for application package with model files

rharige commented 5 months ago

Describe the bug When models are included in the application package, the vespa deploy fails with either of the below two errors:

Uploading application package... failed
Error: invalid application package (400 Bad Request)
{
    "error-code": "BAD_REQUEST",
    "message": "Unable to decompress application stream: Truncated ZIP file"
}

OR

Uploading application package... failed
Error: error from deploy API at 127.0.0.1:19071 (503 Idle timeout expired: 180005/180000 ms):
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 503</title>
</head>
<body>
<h2>HTTP ERROR: 503</h2>
<p>Problem accessing /application/v2/tenant/default/prepareandactivate. Reason:
<pre>    Idle timeout expired: 180005/180000 ms</pre></p>
<hr/>
</body>
</html>

To Reproduce The issue can be reproduced via before / after introducing model files in application package. Before //

Clone and deploy sample app: $vespa clone multi-vector-indexing my-app && cd my-app - this works because it seems the model files are being read from Vespa hosted URLs (see service.xml supplied in the sample app) and not manually created using optimum-cli

After // We will continue to use the same sample app, but introduce model files in it.

Create model files and delete unnecessary files from the command $ optimum-cli export onnx --task sentence-similarity -m intfloat/multilingual-e5-small multilingual-e5-small-onnx
Move the model files under my-app\models and make refer to the model files in service.xml
Retry vespa deploy --wait 600 -- this is where the error occurs

Expected behavior Vespa Deploy to succeed.

Screenshots If applicable, add screenshots to help explain your problem. Adding Vespa Logs snippet:

INFO    configserver     Container.com.yahoo.vespa.config.server.http.HttpErrorResponse Returning response with response code 400, error-code:BAD_REQUEST, message=Unable to decompress application stream: Truncated ZIP file

Environment (please complete the following information):

OS: MacOS 14.0 (23A344)
Infrastructure: self-hosted
Docker: Container memory usage: 3.95GB / 7.48GB

Docker: Container CPU usage: 11.13% / 1200% (12 CPUs available)


% docker version
Client:
Cloud integration: v1.0.35+desktop.11
Version:           25.0.3
API version:       1.44
Go version:        go1.21.6
Git commit:        4debf41
Built:             Tue Feb  6 21:13:26 2024
OS/Arch:           darwin/arm64
Context:           desktop-linux

Server: Docker Desktop 4.28.0 (139021) Engine: Version: 25.0.3 API version: 1.44 (minimum version 1.24) Go version: go1.21.6 Git commit: f417435 Built: Tue Feb 6 21:14:22 2024 OS/Arch: linux/arm64 Experimental: false containerd: Version: 1.6.28 GitCommit: ae07eda36dd25f8a1b98dfbf587313b99c0190bb runc: Version: 1.1.12 GitCommit: v1.1.12-0-g51d5e94 docker-init: Version: 0.19.0 GitCommit: de40ad0



**Vespa version**
Include Vespa version(s) used.
- Vespa version: 8.325.46
- Vespa Docker Image aee351eb05b1b2f54c6541b9cd38eff68ba30a0d1d290e996a290049c2942992
- Vespa CLI version 8.323.45 compiled with go1.22.1 on darwin/arm64

**Additional context**
Add any other context about the problem here.

thomasht86 commented 5 months ago

Hi! I was able to (almost) reproduce your issue, encountering

Uploading application package... failed
Error: invalid application package (400 Bad Request)
Invalid application:
Could not import model 'sentencepiece_bpe':
Could not import a Vespa model from '/opt/vespa/var/db/vespa/config_server/serverdb/tenants/default/sessions/13/.preprocessed/models/sentencepiece.bpe.model'

It seems like the issue stems from the file sentencepiece.bpe.model, which you probably have located in the models-directory. The naming of this file likely causes some unexpected behavior, and removing or renaming it will hopefully solve your problem. We will follow up internally for potential fix/doc improvement.

thomasht86 commented 5 months ago

rharige commented 5 months ago

Hi @thomasht86 - As noted in the steps to reproduce:

Create model files and delete unnecessary files from the command

The sentencepiece.bpe.model file is already removed from my my-app\models directory, it contains only onnx file and tokenizer.json. The issue still occurs, and I can reproduce this issue consistently - hence I raised the bug here.

vespa-engine / vespa

Vespa Deploy fails for application package with model files #30813