Weaviate always fails with "No documents to query (for chatting with LLM, pick Resources->Collections->LLM)."

Fantastic project! Working perfectly for my use case with one exception. I can't seem to get h2ogpt to work with a Weaviate server.

h2ogpt version - main@338fa54

Start Weaviate:

    docker run \
        --rm --init -it \
        -v /etc/passwd:/etc/passwd:ro \
        -v /etc/group:/etc/group:ro \
        -u `id -u`:`id -g` \
        -e QUERY_DEFAULTS_LIMIT=100 \
        -e AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED=true \
        -e PERSISTENCE_DATA_PATH=/var/lib/weaviate \
        -e DEFAULT_VECTORIZER_MODULE=none \
        -e ENABLE_MODULES="" \
        -e CLUSTER_HOSTNAME=node1 \
        -v my_weave_dir:/var/lib/weaviate \
        -p 8080:8080 -p 50051:50051 \
        semitechnologies/weaviate:1.22.7 \
        --host 0.0.0.0 --port 8080 --scheme http

Make DB:

WEAVIATE_URL=http://my_weave_ip:8080
docker run \
        --gpus '"device=0"' \
        --runtime=nvidia \
        --shm-size=64g \
        --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
        --rm --init -it \
        -v /etc/passwd:/etc/passwd:ro \
        -v /etc/group:/etc/group:ro \
        -e WEAVIATE_URL \
        -u `id -u`:`id -g` \
        -v "${CACHE}":/workspace/.cache \
        -v "${MODELS}":/workspace/models \
        -v "${WORKSPACE}"/save:/workspace/save \
        -v "${WORKSPACE}"/data:/workspace/user_path \
        -v "${WORKSPACE}"/db:/workspace/db_dir_UserData \
        -v "${WORKSPACE}"/users:/workspace/users \
        -v "${WORKSPACE}"/db_nonusers:/workspace/db_nonusers \
            h2ogpt:latest /workspace/src/make_db.py \
                --verbose=True --add_if_exists=False --db_type=weaviate --hf_embedding_model=hkunlp/instructor-large --n_jobs=16

I can confirm doc chunks are added to Weaviate. For example:

curl 'http://localhost:8080/v1/objects/035b93dc-b3bd-452f-8cb1-934047bbcecf?include=vector' | jq

{
  "class": "Userdata",
  "creationTimeUnix": 1702913601148,
  "id": "035b93dc-b3bd-452f-8cb1-934047bbcecf",
  "lastUpdateTimeUnix": 1702913601148,
  "properties": {
    "chunk_id": 945,
    "date": "2023-12-18 15:32:07.666947",
    "doc_hash": "bce053d9-7",
    "file_id": 35,
    "hashid": "8eab393a-1cb9-932d-eb1e-898b88df5b39",
    "head": "W.E. Pete Peterson - Almost Perfect\nglad I had lef",
    "input_type": ".pdf",
    "order_id": 113,
    "page": 113,
    "parser": "PyPDFLoader",
    "source": "user_path/almostperfect.pdf",
    "text": "principles and follow their own agenda is bound to get in trouble. There is simply no place to hide\nin a flat, well-run organization, when those who run the company are so close to those who do the\nwork. There is also too much competition for a tall, poorly-run organization to offer a permanent\nrefuge for anyone, because that type of company will go out of business very quickly.\nAlthough I have been critical of some of their decisions, I believe very strongly that Bruce and Alan",
    "time": 1702913527.666948
  },
  "vector": [
    -0.035825904,
    -0.020826744,
    -0.013280412,
    0.013559588,
    0.06382231,
    0.018423686,
    -0.0062506837,
    0.038484592,
    -0.018907575,
    0.007586224,
    0.034014445,
    -0.0076624937,
    0.04309403,
    0.030826557,
    -0.060705226,
    -0.06251694,
    -0.031693194,
    0.013293381,
    -0.04476161,
    -0.006503874,
    0.052231003,
    0.00767843,
    0.0085253045,
    0.035037797,
    -0.0017888997,
    -0.008855045,
    -0.004251446,
    0.032390602,
    0.037113868,
    -0.06064838,
    0.026789308,
    -0.04464679,
    0.0018866265,
    -0.019423136,
    -0.03185469,
    0.01826507,
    0.025431784,
    0.02505813,
    0.038323384,
    0.040634897,
    -0.0030897958,
    0.0027550547,
    -0.0065918937,
    -0.03747724,
    0.032882236,
    -0.026114104,
    -0.0028645126,
    -0.033679266,
    0.0006252241,
    -0.025274893,
    -0.0528347,
    -0.061520528,
    0.0041837078,
    0.007669316,
    0.0033311767,
    -0.031728555,
    -0.061639875,
    0.025546294,
    0.052301604,
    -0.054527793,
    -0.045156475,
    0.009299992,
    -0.067731,
    0.03273777,
    0.043395996,
    -0.0929896,
    0.025037698,
    -0.017028317,
    0.05102154,
    -0.011709846,
    0.019950962,
    -0.054428596,
    0.037003707,
    -0.054361228,
    -0.01697939,
    -0.025216287,
    0.030666677,
    0.050221305,
    -0.029356832,
    0.039114807,
    0.025292257,
    0.01727226,
    0.034638688,
    -0.021693185,
    -0.0116154,
    0.016076732,
    -0.021771869,
    0.006391694,
    0.02152134,
    0.04650752,
    -0.042839874,
    -0.037801284,
    0.02890248,
    -0.018692013,
    0.010399034,
    0.009197637,
    0.05940138,
    0.03419825,
    0.0018091673,
    0.027178371,
    0.042454146,
    0.02833309,
    -0.023761556,
    0.035742074,
    -0.022463411,
    -0.029320212,
    0.018046083,
    -0.0070817266,
    -0.07362251,
    -0.0021369888,
    -0.014326214,
    -0.03455341,
    -0.041064486,
    0.0049499003,
    0.027699776,
    -0.019808482,
    0.031780984,
    -0.05697329,
    -0.047862496,
    -0.007415396,
    0.02542459,
    0.018390872,
    -0.045597825,
    0.0024204783,
    -0.01831901,
    0.013533528,
    -0.022870684,
    -0.024524137,
    -0.028389877,
    -0.011804357,
    0.006842089,
    0.072607495,
    0.023313392,
    -0.06553435,
    0.04303325,
    -0.013154763,
    -0.08939623,
    0.011255063,
    -0.0028467448,
    0.081278816,
    -0.06314491,
    -0.076239705,
    -0.0057883956,
    0.026591364,
    -0.0045225816,
    0.006013776,
    0.023927126,
    -0.053220537,
    -0.049238183,
    -0.024017954,
    0.039599665,
    -0.03540302,
    -0.019012813,
    0.056038357,
    -0.015707867,
    0.03892195,
    -0.046784673,
    0.0066612964,
    0.03386639,
    -0.0039303713,
    -0.04461374,
    0.046662085,
    -0.04244535,
    -0.02724471,
    -0.08033674,
    -0.01245705,
    0.02174396,
    0.07692413,
    -0.020813912,
    -0.010984508,
    -0.06952585,
    0.023459487,
    0.029072927,
    0.030563708,
    -0.014456682,
    0.06243406,
    0.033072356,
    -0.0010286336,
    -0.032505948,
    0.04778214,
    0.0215706,
    -0.024190672,
    -0.034284692,
    0.04044089,
    0.044047717,
    -0.026138967,
    0.00913486,
    0.039754156,
    0.04967719,
    -0.051371742,
    -0.017677108,
    0.0072148354,
    -0.012000443,
    0.022612672,
    0.025301987,
    -0.017805053,
    -0.024669144,
    -0.0044893245,
    -0.02242858,
    -0.0039678803,
    -0.007854908,
    -0.04682132,
    0.03862709,
    0.007140116,
    0.011725553,
    0.057333156,
    -0.009936699,
    0.08331832,
    -0.03146058,
    -0.01750092,
    -0.032493565,
    -0.033052396,
    -0.0098520005,
    0.059618793,
    0.049584277,
    0.03936321,
    -0.04772693,
    -0.06860082,
    0.03874262,
    0.017048607,
    0.027630463,
    0.012441231,
    -0.04836066,
    -0.011941276,
    0.016482174,
    0.028953573,
    0.026317284,
    -0.042484395,
    0.009744475,
    0.066074096,
    -0.03494684,
    0.011239088,
    -0.029741054,
    0.007864451,
    -0.03471012,
    -0.06800124,
    0.033739697,
    0.04043679,
    -0.030073514,
    0.02529906,
    -0.029756926,
    0.0178945,
    0.020726614,
    0.017180642,
    0.020680223,
    -0.05115241,
    -0.014638605,
    -0.011886717,
    0.06922008,
    -0.043066163,
    0.08688656,
    0.025052795,
    0.005027658,
    -0.0033915855,
    -0.015628483,
    0.035848275,
    0.083247125,
    -0.015011733,
    -0.022617305,
    0.068557136,
    0.05283267,
    0.031319402,
    0.051786333,
    -0.011634409,
    0.07171446,
    -0.015613623,
    0.009324993,
    0.05303662,
    -0.014329611,
    0.007542547,
    0.053972792,
    0.033227466,
    0.0041515017,
    -0.030453166,
    0.031274587,
    -0.0622263,
    0.045885265,
    -0.020371033,
    0.02122941,
    -0.02369468,
    -0.044096794,
    0.10544211,
    0.034065776,
    -0.05396358,
    0.00978338,
    -0.02074438,
    0.0015281395,
    0.060529195,
    -0.001103059,
    0.030037927,
    -0.02044579,
    -0.02409927,
    -0.0022760422,
    -0.078156695,
    0.0023389282,
    -0.0066180527,
    -0.032537222,
    0.008061571,
    -0.08632621,
    -0.0008486972,
    -0.07239449,
    -0.011308337,
    0.09606003,
    0.024206242,
    0.039745077,
    -0.046474494,
    0.024012186,
    -0.036892597,
    -0.032495953,
    -0.07310399,
    -0.02017437,
    -0.037712228,
    -0.058540646,
    -0.013719124,
    0.0392493,
    0.040338177,
    0.011056709,
    -0.01592603,
    -0.030052122,
    -0.0011387627,
    0.021364182,
    0.017227614,
    -0.019633634,
    -0.05359042,
    0.013536101,
    -0.008377934,
    0.015444792,
    0.012415995,
    -0.043590605,
    0.051052198,
    0.097904935,
    0.024053168,
    0.015091155,
    -0.00041161542,
    -0.030464567,
    -0.018141324,
    -0.013936589,
    -0.068193205,
    -0.049612835,
    -0.049860366,
    -0.039358594,
    -0.003997409,
    -0.035670765,
    -0.022818388,
    -0.0129037015,
    -0.00991389,
    0.037193336,
    0.06545648,
    0.019969763,
    0.00744502,
    -0.029187473,
    -0.056463122,
    0.05111185,
    -0.01727158,
    0.0010100901,
    -0.07838244,
    -0.04147593,
    0.052881625,
    0.0017803328,
    0.014243788,
    0.0057915077,
    0.06528837,
    0.016800022,
    0.008546928,
    0.016142564,
    -0.01518965,
    -0.007543113,
    -0.0115017155,
    0.02166927,
    -0.05553618,
    -0.01834256,
    -0.009410144,
    -0.033541657,
    0.037520967,
    -0.04627259,
    -0.009701261,
    -0.029178763,
    0.01898933,
    0.05483773,
    -0.003883861,
    0.0011028973,
    -0.009044029,
    -0.07641082,
    -0.016415145,
    -0.027899545,
    -0.017913498,
    0.0230269,
    0.03789936,
    -0.009032509,
    -0.04348928,
    0.030562837,
    0.038798224,
    0.03311992,
    0.013939277,
    -0.037778437,
    0.017515965,
    -0.022381483,
    0.016766438,
    -0.013932857,
    -0.024422633,
    -0.04640263,
    -0.09156362,
    -0.026740152,
    0.019162236,
    -0.030057792,
    -0.013500247,
    0.026158713,
    -0.009881496,
    -0.01499394,
    -0.005999174,
    0.028844716,
    0.04475389,
    -0.024418145,
    0.06466156,
    -0.06853237,
    0.0130461175,
    -0.025043797,
    0.006321119,
    -0.030584408,
    -0.0035245696,
    0.03128829,
    0.05918257,
    -0.009684299,
    0.02928151,
    0.044381794,
    0.05540523,
    -0.044894766,
    0.010987039,
    0.0505694,
    -0.035947148,
    -0.040536273,
    0.033974074,
    -0.032306623,
    -0.047195062,
    0.004875281,
    -0.011875557,
    0.0032856646,
    0.031875774,
    -0.00577194,
    0.0012931386,
    0.017979326,
    -0.013103386,
    -0.023797262,
    -0.029979419,
    0.024753265,
    0.025344392,
    0.0087085245,
    0.017813424,
    0.0629225,
    0.0055718906,
    0.05934364,
    -0.0033487584,
    -0.053811584,
    -0.057075802,
    0.037742537,
    0.007753702,
    -0.04973736,
    -0.06031906,
    -0.035245508,
    0.043353792,
    0.037719265,
    -0.01275575,
    -0.08191482,
    0.02115517,
    0.04769021,
    -0.061631728,
    0.032769147,
    0.015856951,
    0.028215988,
    0.053645432,
    0.006584387,
    0.030382168,
    -0.026753658,
    -0.045113463,
    0.016404968,
    0.017965136,
    -0.039135244,
    -0.023734154,
    -0.008877847,
    0.017898548,
    0.043441474,
    0.02858644,
    0.03971154,
    -0.038959544,
    -0.0035146405,
    -0.03259116,
    0.040192,
    -0.020722998,
    0.025702944,
    0.005452315,
    0.010834591,
    -0.021701774,
    -0.050631996,
    0.009713784,
    0.011605867,
    0.027849194,
    -0.042524457,
    -0.032095194,
    -0.047097668,
    0.03142822,
    0.0055976566,
    -0.039100386,
    -0.07850763,
    0.016957833,
    -0.024753474,
    0.0014829914,
    0.028371967,
    0.0045579006,
    0.051284462,
    0.068110555,
    -0.00766921,
    -0.006175907,
    -0.06913391,
    0.051646046,
    -0.0058871675,
    -0.034864545,
    0.017039334,
    -0.05677532,
    -0.010250116,
    -0.027617956,
    -0.036765665,
    -0.030854061,
    -0.02491608,
    0.04120486,
    0.015551623,
    0.037103854,
    -0.019753544,
    0.0111658815,
    0.025050683,
    -0.043020323,
    -0.0148706855,
    -0.03345115,
    -0.07131602,
    -0.005200439,
    0.04429456,
    0.018276911,
    0.03056576,
    -0.0032877873,
    0.047901776,
    0.030389618,
    0.021848114,
    -0.056339227,
    0.03822373,
    0.01263276,
    -0.033253487,
    -0.016023487,
    -0.030709365,
    0.0038350278,
    0.0041781706,
    -0.04544541,
    0.026705949,
    -0.0071811597,
    -0.016563134,
    -0.074786656,
    0.06233863,
    0.024153117,
    -0.049871303,
    -0.039915577,
    0.004602492,
    0.019463196,
    0.02820294,
    -0.04040034,
    -0.012107242,
    0.032396786,
    0.0355029,
    0.04091984,
    0.0232024,
    -0.012538408,
    0.012715641,
    -0.023447694,
    0.04372398,
    -0.06538383,
    0.02954878,
    -0.018982079,
    0.043185405,
    -0.011914549,
    0.0066580568,
    -0.061743803,
    0.023532305,
    -0.037457056,
    -0.059899107,
    0.0022556433,
    0.002961522,
    -0.029104505,
    0.012761885,
    -0.015658904,
    0.0065910923,
    -0.044080958,
    0.010889096,
    -0.0125452755,
    0.00079866487,
    -0.012654456,
    -0.006171835,
    -0.022154141,
    -0.023391265,
    -0.021971473,
    -0.020104688,
    0.022016037,
    0.0268332,
    -0.014018056,
    0.018292975,
    -0.02211383,
    -0.03807756,
    0.02366987,
    -0.048889402,
    -0.044673637,
    -0.008599694,
    -0.0011834329,
    -0.00583732,
    0.010658196,
    0.07433053,
    -0.04561803,
    -0.02760198,
    0.036203075,
    0.05283323,
    0.03298888,
    -0.003197056,
    0.009678805,
    0.02075905,
    0.03502982,
    -0.034594182,
    0.013463389,
    -0.030225974,
    0.0031703527,
    0.03666177,
    -0.021377737,
    0.022654906,
    0.0254881,
    -0.03696737,
    -0.011404308,
    0.07667297,
    0.045001965,
    0.010576677,
    -0.048117064,
    -0.049920596,
    0.028093869,
    0.0067782695,
    -0.04019697,
    0.052119628,
    0.021582903,
    -0.038833115,
    0.06273665,
    -0.012165391,
    -0.0044484898,
    0.05514537,
    0.032292392,
    -0.009440333,
    -0.0045174044,
    -0.020699676,
    0.023264663,
    0.03997296,
    -0.03733635,
    -0.0127846785,
    -0.07126947,
    0.040437233,
    -0.06350291,
    0.070963636,
    0.02499556,
    0.0006237771,
    0.0519935,
    -0.008977147,
    -0.07454968,
    0.0041089896,
    -0.03883883,
    0.03396261,
    -0.017992597,
    -0.056954797,
    0.04408982,
    0.010289226,
    -0.05428206,
    0.028288687,
    0.001820993,
    0.03329508,
    0.049902998,
    0.029611727,
    -0.026087334,
    -0.0012499939,
    -0.053305067,
    0.017710011,
    -0.06558314,
    -0.031129343,
    -0.0018004591,
    -0.027848633,
    -0.03103644,
    0.037627432,
    -0.032352187,
    0.028725361,
    0.037644453,
    -0.032834634,
    -0.05089936,
    0.014644936,
    0.0452455,
    -0.008121471,
    -0.0059090033,
    -0.016194826,
    0.024839073,
    0.017164467,
    0.040163517,
    0.028542027,
    0.005008586,
    0.00578705,
    -0.05883658,
    -0.033635475,
    0.033474643,
    0.0149030825,
    -0.013750335,
    0.025765872,
    0.011026678,
    -0.011103728,
    -0.025747271,
    -0.018508568,
    0.010330306,
    -0.038681265,
    -0.04548684,
    0.010935305,
    0.0032382864,
    0.04854592,
    -0.027237978,
    0.022536397,
    0.04277655,
    0.007913257,
    0.013997842,
    -0.0032902875,
    -0.047779247,
    0.048379045,
    -0.040531166,
    0.03622777,
    -0.023685087,
    -0.0899258,
    -0.01517377,
    -0.019668886,
    -0.009395231,
    0.00820079,
    -0.015113831,
    -0.020741878,
    0.034904357,
    0.033053536,
    -0.064145766,
    -0.041914903,
    -0.016502826,
    -0.043401945,
    0.017398663,
    0.033031106,
    -0.031720825,
    0.013127126,
    -0.047040325,
    -0.048647635,
    0.05156953,
    -0.03396366,
    -0.049392153,
    0.038324755,
    0.049014483,
    -0.02323626,
    0.0043069078,
    0.043413628,
    -0.06190807,
    0.028593268,
    0.027817471,
    0.045160558,
    0.04916084,
    0.027471915,
    -0.042546578,
    -0.0068003163,
    -0.044413242,
    -0.054522943,
    -0.0411069,
    0.011989367,
    0.0747375
  ],
  "vectorWeights": null
}

Start h2ogpt:

WEAVIATE_URL=http://my_weave_ip:8080
    docker run \
        --gpus '"device=0"' \
        --runtime=nvidia \
        --shm-size=64g \
        --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
        -p $PORT:7860 \
        --rm --init -it \
        --name "h2ogpt" \
        -v /etc/passwd:/etc/passwd:ro \
        -v /etc/group:/etc/group:ro \
        -u `id -u`:`id -g` \
        -e OPENAI_API_KEY="EMPTY" \
        -e WEAVIATE_URL \
        -v "${CACHE}":/workspace/.cache \
        -v "${MODELS}":/workspace/models \
        -v "${WORKSPACE}"/save:/workspace/save \
        -v "${WORKSPACE}"/data:/workspace/user_path \
        -v "${WORKSPACE}"/users:/workspace/users \
           h2ogpt:latest /workspace/generate.py --use_safetensors=True --base_model=TheBloke/DPOpenHermes-7B-v2-AWQ --load_awq=model --prompt_type=mptchat --save_dir=/workspace/save/ --use_gpu_id=False --score_model=None --max_max_new_tokens=4096 --max_new_tokens=2048 --min_max_new_tokens=2048 --enable_stt=False --enable_tts=False --avatars=False --h2ocolors=False --db_type=weaviate --verbose=True --debug=True --use_llm_if_no_docs=False --langchain_modes=['UserData'] --visible_langchain_actions=['Query'] --hf_embedding_model=hkunlp/instructor-large --max_seq_len=16384 --do_sample=True --top_p=0.95 --top_k=50 --temperature=0.2 --repetition_penalty=0.01 --truncation_generation=True --memory_restriction_level=0 --early_stopping=True --max_time=30 --render_markdown=True --keep_sources_in_context=False

In h2ogpt UI:

"Who wrote Almost Perfect?"

Response:

No documents to query (for chatting with LLM, pick Resources->Collections->LLM).

h2ogpt console output:

prompt: <|im_start|>system

<|im_end|><|im_start|>user
Who wrote Almost Perfect?<|im_end|><|im_start|>assistant

prompt: <|im_start|>system

<|im_end|><|im_start|>user
Who wrote Almost Perfect?<|im_end|><|im_start|>assistant

Did not generate db for UserData since no sources
prompt: <|im_start|>system

<|im_end|><|im_start|>user
Who wrote Almost Perfect?<|im_end|><|im_start|>assistant

prompt: <|im_start|>system

<|im_end|><|im_start|>user
Who wrote Almost Perfect?<|im_end|><|im_start|>assistant

Post-Generate Langchain: 2023-12-18 16:30:30.492297 decoded_output: 80

I've tried every possible combination of Weaviate server versions, various h2ogpt releases, different models, load model directly/vLLM/HF TGI, different sentence embedding models, etc but I just can't seem to get this to work. I also tried with embedded Weaviate and verified /workspace/db_dir_UserData is populated correctly. Same result from gen.py.

Chroma works perfectly when set with db_type=chroma.

What am I missing?

h2oai / h2ogpt

Weaviate always fails with "No documents to query (for chatting with LLM, pick Resources->Collections->LLM)." #1220