search

Kaggle / kagglehub

Python library to access Kaggle resources
Apache License 2.0
34 stars 6 forks source link

The service storage has thrown an exception #121

Open lewtun opened 2 months ago

lewtun commented 2 months ago

I am getting an error with pushing a 7B model to the Kaggle Hub that wasn't happening previously. I am using kagglehub==0.2.4 and my script to reproduce the error is


import kagglehub
from huggingface_hub import snapshot_download

"""
Script to upload a Transformers model from the Hugging Face Hub to Kaggle.

To push to your Kaggle account, generate a `kaggle.json` file with your Kaggle API credentials and store it in ~/.kaggle/kaggle.json
See: https://github.com/Kaggle/kagglehub?tab=readme-ov-file#authenticate

Usage:

python upload_model.py \
    --model_id ORG/MODEL_ID \
    --revision REV
"""

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--model_id", type=str, help="Name of repository on the Hub in '<ORG>/<NAME>' format.")
    parser.add_argument(
        "--revision", type=str, default="main", help="Name of branch in repository to save experiments."
    )
    parser.add_argument(
        "--kaggle_handle",
        type=str,
        default=None,
        help="Kaggle handle to upload the model to. Should be in the format <KAGGLE_USERNAME>/<MODEL>/<FRAMEWORK>/<VARIATION>. Defaults to <KAGGLE_USERNAME>/{model_id}/transformers/{revision}",
    )
    args = parser.parse_args()

    # Download repo
    model_name = args.model_id.split("/")[-1]
    local_dir = Path(f"data/{model_name}-{args.revision}")
    local_dir.mkdir(parents=True, exist_ok=True)
    snapshot_download(
        repo_id=args.model_id,
        revision=args.revision,
        local_dir=local_dir,
        ignore_patterns=[
            "checkpoint-*",
            "pytorch_model*",
            ".git*",
            "*_results.json",
            "*.bin",
            "trainer_*",
        ],  # Kaggle doesn't allow uploads >25 files, so we need to heavily filter: https://github.com/Kaggle/kagglehub/issues/116
    )

    # If no handle is provided, default to <KAGGLE_USERNAME>/{model_id}/pyTorch/{revision}
    if args.kaggle_handle is None:
        kaggle_username = kagglehub.config.get_kaggle_credentials().username
        kaggle_handle = f"{kaggle_username}/{model_name}/transformers/{args.revision}"

    print(f"Pushing to Kaggle Hub with handle {kaggle_handle} ...")
    kagglehub.model_upload(kaggle_handle, local_dir)

    print("Done!")

Here is the full stack trace:

Traceback (most recent call last):
  File "/fsx/lewis/git/hf/h4/scripts/deployment/aimo/upload_model.py", line 84, in <module>
    main()
  File "/fsx/lewis/git/hf/h4/scripts/deployment/aimo/upload_model.py", line 75, in main
    kagglehub.model_upload(kaggle_handle, local_dir)
  File "/fsx/lewis/miniconda3/envs/h4/lib/python3.10/site-packages/kagglehub/models.py", line 53, in model_upload
    create_model_instance_or_version(h, tokens, license_name, version_notes)
  File "/fsx/lewis/miniconda3/envs/h4/lib/python3.10/site-packages/kagglehub/models_helpers.py", line 67, in create_model_instance_or_version
    raise (e)
  File "/fsx/lewis/miniconda3/envs/h4/lib/python3.10/site-packages/kagglehub/models_helpers.py", line 61, in create_model_instance_or_version
    _create_model_instance(model_handle, files, license_name)
  File "/fsx/lewis/miniconda3/envs/h4/lib/python3.10/site-packages/kagglehub/models_helpers.py", line 34, in _create_model_instance
    api_client.post(f"/models/{model_handle.owner}/{model_handle.model}/create/instance", data)
  File "/fsx/lewis/miniconda3/envs/h4/lib/python3.10/site-packages/kagglehub/clients.py", line 122, in post
    process_post_response(response_dict)
  File "/fsx/lewis/miniconda3/envs/h4/lib/python3.10/site-packages/kagglehub/exceptions.py", line 115, in process_post_response
    raise BackendError(response["error"], error_code)
kagglehub.exceptions.BackendError: The service storage has thrown an exception. HttpStatusCode is NotFound. No such object: kaggle-models-data/inbox/1002070/534392bb61a8dd2cdbab22e064199089/generation_config.json.lock
neshdev commented 2 months ago

We have released version 0.2.5. Can you try using the new version and see if it is still an issue? https://pypi.org/project/kagglehub/0.2.5/

neshdev commented 2 months ago

Can you also provide is with the kaggle username and the handle used while publishing?

neshdev commented 2 months ago

I assume that the details are as follows: username: lewtun handles: lewtun/deepseek-math-7b-sft/transformers/aimo_v05.26 lewtun/deepseek-math-7b-sft/transformers/aimo_v05.12

lewtun commented 2 months ago

Hello @neshdev I upgraded to 0.2.5 but hit the same error. I realised that snapshot_download() was creating a hidden .huggingface cache folder with many files - removing that enabled the push to work without error.

neshdev commented 2 months ago

Even though the user has closed the issue, there seems to be something wrong with the upload. I am able to reproduce the issue.

Minimum code to recreate the issue:

import kagglehub
from huggingface_hub import snapshot_download
import huggingface_hub
import os

hf_api_key = "TODO"
huggingface_hub.login(hf_api_key)

local_dir = "TODO"
snapshot_download(
        repo_id="google/gemma-2b",
        local_dir=local_dir,
        ignore_patterns=[
            "checkpoint-*",
            "pytorch_model*",
            ".git*",
            "*_results.json",
            "*.bin",
            "trainer_*",
        ],
    )

os.environ["KAGGLE_USERNAME"] = "TODO"
os.environ["KAGGLE_KEY"] = "TODO"

kagglehub.model_upload("neshdev/my_gemma_model/transformers/init", local_dir)

Env os: Windows 11 python 3.12.1 installed using python -m venv .env kagglehub==0.2.5 huggingface-hub==0.23.0

The following exception is thrown on the first run:

kagglehub.exceptions.BackendError: The service storage has thrown an exception. HttpStatusCode is NotFound. No such object: kaggle-models-data/inbox/418540/ada3e4dff2c0c3483ed56c590d524ece/gemma-2b.gguf.lock

The following files are downloaded (trimmed large files):

-rw-r--r-- 1 neshd 197611 1 May 17 01:45 ./.huggingface/.gitignore
-rw-r--r-- 1 neshd 197611 104 May 17 01:45 ./.huggingface/download/config.json.metadata
-rw-r--r-- 1 neshd 197611 230M May 17 01:52 ./.huggingface/download/gemma-2b.gguf.084eb1abf609fe1735c2631f818bd24e93d29428fd2ba096852012152e37d14b.incomplete
-rw-r--r-- 1 neshd 197611 0 May 17 01:52 ./.huggingface/download/gemma-2b.gguf.lock
-rw-r--r-- 1 neshd 197611 128 May 17 01:47 ./.huggingface/download/gemma-2b.gguf.metadata
-rw-r--r-- 1 neshd 197611 104 May 17 01:45 ./.huggingface/download/generation_config.json.metadata
-rw-r--r-- 1 neshd 197611 127 May 17 01:46 ./.huggingface/download/model-00001-of-00002.safetensors.metadata        
-rw-r--r-- 1 neshd 197611 128 May 17 01:45 ./.huggingface/download/model-00002-of-00002.safetensors.metadata
-rw-r--r-- 1 neshd 197611 104 May 17 01:45 ./.huggingface/download/model.safetensors.index.json.metadata
-rw-r--r-- 1 neshd 197611 103 May 17 01:45 ./.huggingface/download/README.md.metadata
-rw-r--r-- 1 neshd 197611 104 May 17 01:45 ./.huggingface/download/special_tokens_map.json.metadata
-rw-r--r-- 1 neshd 197611 128 May 17 01:45 ./.huggingface/download/tokenizer.json.metadata
-rw-r--r-- 1 neshd 197611 128 May 17 01:45 ./.huggingface/download/tokenizer.model.metadata
-rw-r--r-- 1 neshd 197611 104 May 17 01:45 ./.huggingface/download/tokenizer_config.json.metadata
-rw-r--r-- 1 neshd 197611 627 May 17 01:45 ./config.json
-rw-r--r-- 1 neshd 197611 137 May 17 01:45 ./generation_config.json
-rw-r--r-- 1 neshd 197611 14K May 17 01:45 ./model.safetensors.index.json
-rw-r--r-- 1 neshd 197611 21K May 17 01:45 ./README.md
-rw-r--r-- 1 neshd 197611 636 May 17 01:45 ./special_tokens_map.json
-rw-r--r-- 1 neshd 197611 17M May 17 01:45 ./tokenizer.json
-rw-r--r-- 1 neshd 197611 4.1M May 17 01:45 ./tokenizer.model
-rw-r--r-- 1 neshd 197611 33K May 17 01:45 ./tokenizer_config.json

The file gemma-2b.gguf.lock is 0 bytes.

Subsequent runs also raise the same issue:

kagglehub.exceptions.BackendError: The service storage has thrown an exception. HttpStatusCode is NotFound. No such object: kaggle-models-data/inbox/418540/ada3e4dff2c0c3483ed56c590d524ece/gemma-2b.gguf.lock

Partial uploads are restarted from 0 bytes. Files are not resumed from last stopped location.

neshdev commented 2 months ago

Spamming the upload for the same handle also exhibits strange behavior. Version 3 was processed before v2 and v1. V1 is stuck in processing state and V2 is stuck in "files processing". See https://www.kaggle.com/models/neshdev/my_gemma_model.

Version 3 uploaded successfully, but doesn't show up in the ui.

neshdev commented 2 months ago

Below is the trace log

2024-05-17 02:35:37,325 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): huggingface.co:443    
2024-05-17 02:35:37,544 - urllib3.connectionpool - DEBUG - https://huggingface.co:443 "GET /api/whoami-v2 HTTP/1.1" 200 340
Token is valid (permission: read).
Login successful
2024-05-17 02:35:37,550 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443    
2024-05-17 02:35:37,923 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "GET /api/v1/models/neshdev/my_gemma_model/get HTTP/1.1" 200 None
Starting upload for file config.json
2024-05-17 02:35:37,931 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443    
2024-05-17 02:35:38,233 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "POST /api/v1/blobs/upload HTTP/1.1" 200 None
Uploading:   0%|                                                                         | 0.00/627 [00:00<?, ?B/s]2024-05-17 02:35:38,247 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.googleapis.com:443 
Uploading: 100%|██████████████████████████████████████████████████████████████████| 627/627 [00:00<00:00, 3.32kB/s]2024-05-17 02:35:38,671 - urllib3.connectionpool - DEBUG - https://www.googleapis.com:443 "PUT /upload/storage/v1/b/kaggle-models-data/o?uploadType=resumable&upload_id=ABPtcPqzVITY22kvVaFd9TZrTtwA8H7VZFtxiXk65dBdQFy1jQItqAYL_s5KBun3iz__Mqfxg7x7YLzgyVzz9EWRptTxaNyvA5nKQG3fD-NP0UTL HTTP/1.1" 200 925
Uploading: 100%|██████████████████████████████████████████████████████████████████| 627/627 [00:00<00:00, 1.46kB/s] 
Upload successful: config.json (627B)
Starting upload for file generation_config.json
2024-05-17 02:35:38,679 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443    
2024-05-17 02:35:39,005 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "POST /api/v1/blobs/upload HTTP/1.1" 200 None
Uploading:   0%|                                                                         | 0.00/137 [00:00<?, ?B/s]2024-05-17 02:35:39,009 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.googleapis.com:443 
Uploading: 100%|████████████████████████████████████████████████████████████████████| 137/137 [00:00<00:00, 697B/s]2024-05-17 02:35:39,460 - urllib3.connectionpool - DEBUG - https://www.googleapis.com:443 "PUT /upload/storage/v1/b/kaggle-models-data/o?uploadType=resumable&upload_id=ABPtcPqpW9SxJ1vsCcMZmlp--1ZcDE4zYVKzZ1vfUD1zjyO0MwbvFZaOjweli5a9FVSDHxixIzOXvkpQlG7p3FsTDSkA_hc1sNoRb7f3N2_C3BwpFg HTTP/1.1" 200 969
Uploading: 100%|████████████████████████████████████████████████████████████████████| 137/137 [00:00<00:00, 301B/s] 
Upload successful: generation_config.json (137B)
Starting upload for file model.safetensors.index.json
2024-05-17 02:35:39,467 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443    
2024-05-17 02:35:39,766 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "POST /api/v1/blobs/upload HTTP/1.1" 200 None
Uploading:   0%|                                                                       | 0.00/13.5k [00:00<?, ?B/s]2024-05-17 02:35:39,770 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.googleapis.com:443 
Uploading: 100%|██████████████████████████████████████████████████████████████| 13.5k/13.5k [00:00<00:00, 71.4kB/s]2024-05-17 02:35:40,206 - urllib3.connectionpool - DEBUG - https://www.googleapis.com:443 "PUT /upload/storage/v1/b/kaggle-models-data/o?uploadType=resumable&upload_id=ABPtcPpyexG4AIMuZg3KB_zJr8VibZ-7Ua1A1_PytXx2cAiChnCkQyfv2ebAvY06E3NZTyMXMLZFqxSGrV1j6Y3WyH7HudZ9JbqE-w8X6uznD7A0 HTTP/1.1" 200 995
Uploading: 100%|██████████████████████████████████████████████████████████████| 13.5k/13.5k [00:00<00:00, 30.7kB/s] 
Upload successful: model.safetensors.index.json (13KB)
Starting upload for file README.md
2024-05-17 02:35:40,210 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443    
2024-05-17 02:35:40,486 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "POST /api/v1/blobs/upload HTTP/1.1" 200 None
Uploading:   0%|                                                                       | 0.00/21.4k [00:00<?, ?B/s]2024-05-17 02:35:40,499 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.googleapis.com:443 
Uploading:  76%|███████████████████████████████████████████████▍              | 16.4k/21.4k [00:00<00:00, 86.6kB/s]2024-05-17 02:35:40,982 - urllib3.connectionpool - DEBUG - https://www.googleapis.com:443 "PUT /upload/storage/v1/b/kaggle-models-data/o?uploadType=resumable&upload_id=ABPtcPrcJeOZYjJEU2FvyqxUzMmIQny3NBMjp37HcEFRiGMCi9YC5cOed_AC_GRWJgvwjX8YvGlC50wyXZL5MqekHYGr1-KAyP3eKpThY142q_Xv HTTP/1.1" 200 919
Uploading: 100%|██████████████████████████████████████████████████████████████| 21.4k/21.4k [00:00<00:00, 43.7kB/s] 
Upload successful: README.md (21KB)
Starting upload for file special_tokens_map.json
2024-05-17 02:35:40,994 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443    
2024-05-17 02:35:41,264 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "POST /api/v1/blobs/upload HTTP/1.1" 200 None
Uploading:   0%|                                                                         | 0.00/636 [00:00<?, ?B/s]2024-05-17 02:35:41,278 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.googleapis.com:443 
Uploading: 100%|██████████████████████████████████████████████████████████████████| 636/636 [00:00<00:00, 3.33kB/s]2024-05-17 02:35:41,728 - urllib3.connectionpool - DEBUG - https://www.googleapis.com:443 "PUT /upload/storage/v1/b/kaggle-models-data/o?uploadType=resumable&upload_id=ABPtcPoBWIVCNGQQ6iWpPgiO-GqHby_G12Mgqpb1WN5lbbfk36RPRllrkUy5bhvhjYAaFlU5_6JW8BDbQdcBtw-F1ZQ97RCNL1wTnZeS9T4Ah4KJ HTTP/1.1" 200 973
Uploading: 100%|██████████████████████████████████████████████████████████████████| 636/636 [00:00<00:00, 1.39kB/s] 
Upload successful: special_tokens_map.json (636B)
Starting upload for file tokenizer.json
2024-05-17 02:35:41,737 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443    
2024-05-17 02:35:42,033 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "POST /api/v1/blobs/upload HTTP/1.1" 200 None
Uploading:   0%|                                                                       | 0.00/17.5M [00:00<?, ?B/s]2024-05-17 02:35:42,043 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.googleapis.com:443 
Uploading:  97%|████████████████████████████████████████████████████████████▎ | 17.1M/17.5M [00:02<00:00, 9.15MB/s]2024-05-17 02:35:44,417 - urllib3.connectionpool - DEBUG - https://www.googleapis.com:443 "PUT /upload/storage/v1/b/kaggle-models-data/o?uploadType=resumable&upload_id=ABPtcPoAH3SugQR1-XCOcrfQHWomzyJoNSeQhWLHpI5ZgXlPJxLK456BDYN2izSrR_MbI7AvNOqxqDb32YRI7I0cVW97Ss-ESdVXFmesQ9fjDKI7 HTTP/1.1" 200 942
Uploading: 100%|██████████████████████████████████████████████████████████████| 17.5M/17.5M [00:02<00:00, 7.36MB/s] 
Upload successful: tokenizer.json (17MB)
Starting upload for file tokenizer.model
2024-05-17 02:35:44,428 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443    
2024-05-17 02:35:44,745 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "POST /api/v1/blobs/upload HTTP/1.1" 200 None
Uploading:   0%|                                                                       | 0.00/4.24M [00:00<?, ?B/s]2024-05-17 02:35:44,753 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.googleapis.com:443 
Uploading:  79%|█████████████████████████████████████████████████             | 3.36M/4.24M [00:00<00:00, 7.39MB/s]2024-05-17 02:35:45,675 - urllib3.connectionpool - DEBUG - https://www.googleapis.com:443 "PUT /upload/storage/v1/b/kaggle-models-data/o?uploadType=resumable&upload_id=ABPtcPpqNNpZNsIpEtWJlDaPUYe-NDja0EBLfDqglueFRv7kxHkicksxC5Hc695f7CJ-XfwjeQrsiZ-eR2-1biIBNu2ju2beI5JWbOCBU1aZ7x6g HTTP/1.1" 200 945
Uploading: 100%|██████████████████████████████████████████████████████████████| 4.24M/4.24M [00:00<00:00, 4.57MB/s] 
Upload successful: tokenizer.model (4MB)
Starting upload for file tokenizer_config.json
2024-05-17 02:35:45,687 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443
2024-05-17 02:35:45,956 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "POST /api/v1/blobs/upload HTTP/1.1" 200 None
Uploading:   0%|                                                                       | 0.00/33.6k [00:00<?, ?B/s]2024-05-17 02:35:45,969 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.googleapis.com:443 
Uploading:  49%|██████████████████████████████▎                               | 16.4k/33.6k [00:00<00:00, 84.8kB/s]2024-05-17 02:35:46,428 - urllib3.connectionpool - DEBUG - https://www.googleapis.com:443 "PUT /upload/storage/v1/b/kaggle-models-data/o?uploadType=resumable&upload_id=ABPtcPrXeegtT-dTiSMY3sJpwEKJBtPMjJplnK_TkTDT6ctAZfbcdnheOucAom_wIjPkzqhh3yWxEvrD4soLbCvqr5oMD_KtSqaE9KfPfZ9yn_ZPUg HTTP/1.1" 200 967
Uploading: 100%|██████████████████████████████████████████████████████████████| 33.6k/33.6k [00:00<00:00, 72.3kB/s] 
Upload successful: tokenizer_config.json (33KB)
Starting upload for file .gitignore
2024-05-17 02:35:46,440 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443    
2024-05-17 02:35:46,707 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "POST /api/v1/blobs/upload HTTP/1.1" 200 None
Uploading:   0%|                                                                        | 0.00/1.00 [00:00<?, ?B/s]2024-05-17 02:35:46,720 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.googleapis.com:443 
Uploading: 100%|█████████████████████████████████████████████████████████████████| 1.00/1.00 [00:00<00:00, 5.37B/s]2024-05-17 02:35:47,159 - urllib3.connectionpool - DEBUG - https://www.googleapis.com:443 "PUT /upload/storage/v1/b/kaggle-models-data/o?uploadType=resumable&upload_id=ABPtcPqICf2fFXlizC3glX1z1CWSjNczV01bYbqRz-t1irBs9RMRdHdCeEYffcPWo08paQgXVqDcsskomUm1tOrALYTaLmWDAA34vlALWSS5m63t HTTP/1.1" 200 919
Uploading: 100%|█████████████████████████████████████████████████████████████████| 1.00/1.00 [00:00<00:00, 2.24B/s] 
Upload successful: .gitignore (1B)
Starting upload for file config.json.metadata
2024-05-17 02:35:47,171 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443    
2024-05-17 02:35:47,438 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "POST /api/v1/blobs/upload HTTP/1.1" 200 None
Uploading:   0%|                                                                         | 0.00/104 [00:00<?, ?B/s]2024-05-17 02:35:47,448 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.googleapis.com:443 
Uploading: 100%|████████████████████████████████████████████████████████████████████| 104/104 [00:00<00:00, 560B/s]2024-05-17 02:35:47,882 - urllib3.connectionpool - DEBUG - https://www.googleapis.com:443 "PUT /upload/storage/v1/b/kaggle-models-data/o?uploadType=resumable&upload_id=ABPtcPoyT74IhgrlX__uV3sHrRbUlFS6f2AlXvAZyYSA2Rpxn0Z4tGIgfC3gVMoHIKfd86bciLeu8dZg71XwXRlXkzTieqOgvMI9VzROEpt0dVWb HTTP/1.1" 200 961
Uploading: 100%|████████████████████████████████████████████████████████████████████| 104/104 [00:00<00:00, 236B/s] 
Upload successful: config.json.metadata (104B)
Starting upload for file gemma-2b.gguf.084eb1abf609fe1735c2631f818bd24e93d29428fd2ba096852012152e37d14b.incomplete  
2024-05-17 02:35:47,892 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443    
2024-05-17 02:35:48,174 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "POST /api/v1/blobs/upload HTTP/1.1" 200 None
Uploading:   0%|                                                                        | 0.00/241M [00:00<?, ?B/s]2024-05-17 02:35:48,183 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.googleapis.com:443 
Uploading: 100%|███████████████████████████████████████████████████████████████▊| 241M/241M [00:24<00:00, 9.86MB/s]2024-05-17 02:36:13,088 - urllib3.connectionpool - DEBUG - https://www.googleapis.com:443 "PUT /upload/storage/v1/b/kaggle-models-data/o?uploadType=resumable&upload_id=ABPtcPq2g6BJ5FxbfW3YVV2CU0SP3-bAiNo6ezoy0G3NP8GLojOuy6Y1NBXjN2fSTC2QTFg6jbtVaJPR6NOd8RMCoXz8mddR6Wn-7FdjzudwmJfJdA HTTP/1.1" 200 1243
Uploading: 100%|████████████████████████████████████████████████████████████████| 241M/241M [00:24<00:00, 9.68MB/s] 
Upload successful: gemma-2b.gguf.084eb1abf609fe1735c2631f818bd24e93d29428fd2ba096852012152e37d14b.incomplete (230MB)
Starting upload for file gemma-2b.gguf.lock
2024-05-17 02:36:13,092 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443    
2024-05-17 02:36:13,362 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "POST /api/v1/blobs/upload HTTP/1.1" 200 None
Uploading: 0.00B [00:00, ?B/s]
Upload successful: gemma-2b.gguf.lock (0B)
Starting upload for file gemma-2b.gguf.metadata
2024-05-17 02:36:13,370 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443    
2024-05-17 02:36:13,636 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "POST /api/v1/blobs/upload HTTP/1.1" 200 None
Uploading:   0%|                                                                         | 0.00/128 [00:00<?, ?B/s]2024-05-17 02:36:13,644 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.googleapis.com:443 
Uploading: 100%|████████████████████████████████████████████████████████████████████| 128/128 [00:00<00:00, 692B/s]2024-05-17 02:36:14,088 - urllib3.connectionpool - DEBUG - https://www.googleapis.com:443 "PUT /upload/storage/v1/b/kaggle-models-data/o?uploadType=resumable&upload_id=ABPtcPp7FZrtVmQ2AotTT3bckcYwjzuRionL0QiWjCBzSc9UQSFzdwhJ7DRmA2E0U9Y2t5hr8nQLQqrgmJPLWzoDJOEhULtEyTq66HH98wBeIt0JYw HTTP/1.1" 200 969
Uploading: 100%|████████████████████████████████████████████████████████████████████| 128/128 [00:00<00:00, 285B/s] 
Upload successful: gemma-2b.gguf.metadata (128B)
Starting upload for file generation_config.json.metadata
2024-05-17 02:36:14,097 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443    
2024-05-17 02:36:14,361 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "POST /api/v1/blobs/upload HTTP/1.1" 200 None
Uploading:   0%|                                                                         | 0.00/104 [00:00<?, ?B/s]2024-05-17 02:36:14,371 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.googleapis.com:443 
Uploading: 100%|████████████████████████████████████████████████████████████████████| 104/104 [00:00<00:00, 560B/s]2024-05-17 02:36:14,823 - urllib3.connectionpool - DEBUG - https://www.googleapis.com:443 "PUT /upload/storage/v1/b/kaggle-models-data/o?uploadType=resumable&upload_id=ABPtcPr-1cVodMyHAX8fNMHblYK-Su3IbiSMBvbrwHrUyyJyq9SnweihwVlejxQcG8fAP5C78sepwGMYwtQqYd_EgXE5YQf_oLXq6ISGwGTmUY8_tA HTTP/1.1" 200 1005
Uploading: 100%|████████████████████████████████████████████████████████████████████| 104/104 [00:00<00:00, 228B/s] 
Upload successful: generation_config.json.metadata (104B)
Starting upload for file model-00001-of-00002.safetensors.metadata
2024-05-17 02:36:14,826 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443    
2024-05-17 02:36:15,131 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "POST /api/v1/blobs/upload HTTP/1.1" 200 None
Uploading:   0%|                                                                         | 0.00/127 [00:00<?, ?B/s]2024-05-17 02:36:15,139 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.googleapis.com:443 
Uploading: 100%|████████████████████████████████████████████████████████████████████| 127/127 [00:00<00:00, 689B/s]2024-05-17 02:36:15,571 - urllib3.connectionpool - DEBUG - https://www.googleapis.com:443 "PUT /upload/storage/v1/b/kaggle-models-data/o?uploadType=resumable&upload_id=ABPtcPppE9YVQQBoCjaDHzlOBO0NLdtbD2UL-dIPRuBToBKSpoBFFLEwPU1rHrRLW9Dwyxa9H7yhx6kcBy0EKPU2ogmpnM3DZGEwjNngy6EbziWw1g HTTP/1.1" 200 1045
Uploading: 100%|████████████████████████████████████████████████████████████████████| 127/127 [00:00<00:00, 290B/s] 
Upload successful: model-00001-of-00002.safetensors.metadata (127B)
Starting upload for file model-00002-of-00002.safetensors.metadata
2024-05-17 02:36:15,581 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443    
2024-05-17 02:36:15,848 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "POST /api/v1/blobs/upload HTTP/1.1" 200 None
Uploading:   0%|                                                                         | 0.00/128 [00:00<?, ?B/s]2024-05-17 02:36:15,853 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.googleapis.com:443 
Uploading: 100%|████████████████████████████████████████████████████████████████████| 128/128 [00:00<00:00, 697B/s]2024-05-17 02:36:16,293 - urllib3.connectionpool - DEBUG - https://www.googleapis.com:443 "PUT /upload/storage/v1/b/kaggle-models-data/o?uploadType=resumable&upload_id=ABPtcPp_D3lQ55b7l_sRRQgSIuwFpBe3WKQBtyTaiFl1yEbbzklE989zb3a6bv51h7YHiForkuBjOjfQ5vTuMdPBdH6oEidOpsLqoZ5Va0g4sxA_Bw HTTP/1.1" 200 1045
Uploading: 100%|████████████████████████████████████████████████████████████████████| 128/128 [00:00<00:00, 289B/s] 
Upload successful: model-00002-of-00002.safetensors.metadata (128B)
Starting upload for file model.safetensors.index.json.metadata
2024-05-17 02:36:16,305 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443
2024-05-17 02:36:16,592 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "POST /api/v1/blobs/upload HTTP/1.1" 200 None
Uploading:   0%|                                                                         | 0.00/104 [00:00<?, ?B/s]2024-05-17 02:36:16,596 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.googleapis.com:443 
Uploading: 100%|████████████████████████████████████████████████████████████████████| 104/104 [00:00<00:00, 544B/s]2024-05-17 02:36:17,021 - urllib3.connectionpool - DEBUG - https://www.googleapis.com:443 "PUT /upload/storage/v1/b/kaggle-models-data/o?uploadType=resumable&upload_id=ABPtcPoXuSCEWeoFY6LhddUU0f0dzfGldkM5ECnOMkT3iB8OVEV9tPEVulSO-EQLx-P70ef1A9rNlYlu5v7wsVy_Y-2cSitdFvIZ2HVZNkcp0BOmFA HTTP/1.1" 200 1029
Uploading: 100%|████████████████████████████████████████████████████████████████████| 104/104 [00:00<00:00, 242B/s] 
Upload successful: model.safetensors.index.json.metadata (104B)
Starting upload for file README.md.metadata
2024-05-17 02:36:17,035 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443    
2024-05-17 02:36:17,304 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "POST /api/v1/blobs/upload HTTP/1.1" 200 None
Uploading:   0%|                                                                         | 0.00/103 [00:00<?, ?B/s]2024-05-17 02:36:17,308 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.googleapis.com:443 
Uploading: 100%|████████████████████████████████████████████████████████████████████| 103/103 [00:00<00:00, 551B/s]2024-05-17 02:36:17,820 - urllib3.connectionpool - DEBUG - https://www.googleapis.com:443 "PUT /upload/storage/v1/b/kaggle-models-data/o?uploadType=resumable&upload_id=ABPtcPqsBeQ5i-Y1NOAseRc87K7BR33uMomsVDypuOndS9WwRaobrNtAIfn8-BOvb9akUBkfzhlDRC5LTBXJWprpIPB_fXT0sCLAFJAf1E1spStr HTTP/1.1" 200 953
Uploading: 100%|████████████████████████████████████████████████████████████████████| 103/103 [00:00<00:00, 200B/s] 
Upload successful: README.md.metadata (103B)
Starting upload for file special_tokens_map.json.metadata
2024-05-17 02:36:17,824 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443    
2024-05-17 02:36:18,165 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "POST /api/v1/blobs/upload HTTP/1.1" 200 None
Uploading:   0%|                                                                         | 0.00/104 [00:00<?, ?B/s]2024-05-17 02:36:18,175 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.googleapis.com:443 
Uploading: 100%|████████████████████████████████████████████████████████████████████| 104/104 [00:00<00:00, 538B/s]2024-05-17 02:36:18,632 - urllib3.connectionpool - DEBUG - https://www.googleapis.com:443 "PUT /upload/storage/v1/b/kaggle-models-data/o?uploadType=resumable&upload_id=ABPtcPrcJtkr1GnLsHWhy2LhPOY-Wp0V82sVE4b_HZyObH9iaN-cK_LSkos01KqEuvhNByV_IoLTdnIOoxq_Db3Qn0hWj45meHjItQsaiTJjLmvhqQ HTTP/1.1" 200 1009
Uploading: 100%|████████████████████████████████████████████████████████████████████| 104/104 [00:00<00:00, 225B/s] 
Upload successful: special_tokens_map.json.metadata (104B)
Starting upload for file tokenizer.json.metadata
2024-05-17 02:36:18,642 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443    
2024-05-17 02:36:18,937 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "POST /api/v1/blobs/upload HTTP/1.1" 200 None
Uploading:   0%|                                                                         | 0.00/128 [00:00<?, ?B/s]2024-05-17 02:36:18,942 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.googleapis.com:443 
Uploading: 100%|████████████████████████████████████████████████████████████████████| 128/128 [00:00<00:00, 697B/s]2024-05-17 02:36:19,421 - urllib3.connectionpool - DEBUG - https://www.googleapis.com:443 "PUT /upload/storage/v1/b/kaggle-models-data/o?uploadType=resumable&upload_id=ABPtcPr0KLwY7EqnRMHcHhdduek7q1w2mjSgBvKJCyDPpJDiYwe81hL88D7cZkq5ijmvCHdxW6e7QIapq-WuHxRjbr394s3QiDeyYodfpWw7O3yx HTTP/1.1" 200 973
Uploading: 100%|████████████████████████████████████████████████████████████████████| 128/128 [00:00<00:00, 266B/s] 
Upload successful: tokenizer.json.metadata (128B)
Starting upload for file tokenizer.model.metadata
2024-05-17 02:36:19,425 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443    
Uploading: 100%|████████████████████████████████████████████████████████████████████| 128/128 [00:00<00:00, 684B/s]2024-05-17 02:36:20,136 - urllib3.connectionpool - DEBUG - https://www.googleapis.com:443 "PUT /upload/storage/v1/b/kaggle-models-data/o?uploadType=resumable&upload_id=ABPtcPqz8nloz1In2gXSBjkHGKSqiPA2PKAHwiOVjutdsySKnSsaoOm1_0pByuRIIOUEZjusuOAO1kp5TnoYcbu_b4Mh5-z0qfFzxU5HFNNcyps8bw HTTP/1.1" 200 977
Uploading: 100%|████████████████████████████████████████████████████████████████████| 128/128 [00:00<00:00, 293B/s]
Upload successful: tokenizer.model.metadata (128B)
Starting upload for file tokenizer_config.json.metadata
2024-05-17 02:36:20,148 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443
2024-05-17 02:36:20,412 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "POST /api/v1/blobs/upload HTTP/1.1" 200 None
Uploading:   0%|                                                                         | 0.00/104 [00:00<?, ?B/s]2024-05-17 02:36:20,421 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.googleapis.com:443
Uploading: 100%|████████████████████████████████████████████████████████████████████| 104/104 [00:00<00:00, 550B/s]2024-05-17 02:36:20,876 - urllib3.connectionpool - DEBUG - https://www.googleapis.com:443 "PUT /upload/storage/v1/b/kaggle-models-data/o?uploadType=resumable&upload_id=ABPtcPotGdJ-OwjRaqC8SI_tmEdh0OU3gbIghgRFPZb3ibzUuGSN-tADi-SVjGoc2OAZyRsvu9zWuf2SzIjKOrgxOCWX0NF-vx3jKoWr7nvPB5iBYg HTTP/1.1" 200 1001
Uploading: 100%|████████████████████████████████████████████████████████████████████| 104/104 [00:00<00:00, 226B/s]
Upload successful: tokenizer_config.json.metadata (104B)
2024-05-17 02:36:20,886 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443
2024-05-17 02:36:21,367 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "POST /api/v1/models/neshdev/my_gemma_model/create/instance HTTP/1.1" 200 None
2024-05-17 02:36:21,376 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): www.kaggle.com:443
2024-05-17 02:36:24,303 - urllib3.connectionpool - DEBUG - https://www.kaggle.com:443 "POST /api/v1/models/neshdev/my_gemma_model/transformers/init/create/version HTTP/1.1" 200 None
Traceback (most recent call last):
...redacted...
kagglehub.exceptions.BackendError: The requested slug "init" is already used by another model instance.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
...redacted...
kagglehub.exceptions.BackendError: The service storage has thrown an exception. HttpStatusCode is NotFound. No such object: kaggle-models-data/inbox/418540/dc4f81c1be3539a01a140dda44952389/gemma-2b.gguf.lock

It seems that the true error is being subsided due to a secondary exception. All orchestration endpoints are called even when the process is failing for the empty file.