search

AI-Northstar-Tech / vector-io

The only Vector tooling you'll need. Star the repo and look out for an email to try out a brand new Vector Data Exploration demo! Use the universal VDF format for vector datasets to easily export and import data from all vector databases, and re-embed it using any model
https://tryvector.io
Apache License 2.0
198 stars 26 forks source link

attempt to normalize pandas df to json for qdrant upsert of wit dataset #59

Closed dhruv-anand-aintech closed 7 months ago

dhruv-anand-aintech commented 7 months ago

Try:

import_vdf \
        --id_column metadata_url \
        --max_num_rows 200 \
        --hf_dataset wikimedia/wit_base \
        --vector_columns embedding \
        qdrant \
        --url https://d57afa86-62c8-4b17-9eb5-6143a248abb7.us-east4-0.gcp.cloud.qdrant.io

JSON columns are failing qdrant upsert

cc @Anush008 I'm stuck on this. Could you help out?

Error:

:Not supported json value: [True True True True True True True True True True True True True True[00:00<?, ?it/s]
 True True True True True True True True True True True True True]
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/parallel_processor.py", line 61, in _worker
    for processed_item in worker.process(input_queue_iterable()):
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/uploader/grpc_uploader.py", line 109, in process
    yield from self.process_upload(items)
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/uploader/grpc_uploader.py", line 99, in process_upload
    yield upload_batch_grpc(
          ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/uploader/grpc_uploader.py", line 27, in upload_batch_grpc
    points = [
             ^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/uploader/grpc_uploader.py", line 31, in <listcomp>
    payload=payload_to_grpc(payload or {}),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/conversions/conversion.py", line 90, in payload_to_grpc
    return dict((key, json_to_value(val)) for key, val in payload.items())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/conversions/conversion.py", line 90, in <genexpr>
    return dict((key, json_to_value(val)) for key, val in payload.items())
                      ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/conversions/conversion.py", line 52, in json_to_value
    struct_value=Struct(fields=dict((k, json_to_value(v)) for k, v in payload.items()))
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/conversions/conversion.py", line 52, in <genexpr>
    struct_value=Struct(fields=dict((k, json_to_value(v)) for k, v in payload.items()))
                                        ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/conversions/conversion.py", line 54, in json_to_value
    raise ValueError(f"Not supported json value: {payload}")  # pragma: no cover
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Not supported json value: [True True True True True True True True True True True True True True
 True True True True True True True True True True True True True]
ERROR:root:Not supported json value: [True]
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/parallel_processor.py", line 61, in _worker
    for processed_item in worker.process(input_queue_iterable()):
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/uploader/grpc_uploader.py", line 109, in process
    yield from self.process_upload(items)
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/uploader/grpc_uploader.py", line 99, in process_upload
    yield upload_batch_grpc(
          ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/uploader/grpc_uploader.py", line 27, in upload_batch_grpc
    points = [
             ^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/uploader/grpc_uploader.py", line 31, in <listcomp>
    payload=payload_to_grpc(payload or {}),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/conversions/conversion.py", line 90, in payload_to_grpc
    return dict((key, json_to_value(val)) for key, val in payload.items())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/conversions/conversion.py", line 90, in <genexpr>
    return dict((key, json_to_value(val)) for key, val in payload.items())
                      ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/conversions/conversion.py", line 52, in json_to_value
    struct_value=Struct(fields=dict((k, json_to_value(v)) for k, v in payload.items()))
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/conversions/conversion.py", line 52, in <genexpr>
    struct_value=Struct(fields=dict((k, json_to_value(v)) for k, v in payload.items()))
                                        ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/conversions/conversion.py", line 54, in json_to_value
    raise ValueError(f"Not supported json value: {payload}")  # pragma: no cover
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Not supported json value: [True]
ERROR:root:Not supported json value: [True True True True]
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/parallel_processor.py", line 61, in _worker
    for processed_item in worker.process(input_queue_iterable()):
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/uploader/grpc_uploader.py", line 109, in process
    yield from self.process_upload(items)
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/uploader/grpc_uploader.py", line 99, in process_upload
    yield upload_batch_grpc(
          ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/uploader/grpc_uploader.py", line 27, in upload_batch_grpc
    points = [
             ^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/uploader/grpc_uploader.py", line 31, in <listcomp>
    payload=payload_to_grpc(payload or {}),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/conversions/conversion.py", line 90, in payload_to_grpc
    return dict((key, json_to_value(val)) for key, val in payload.items())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/conversions/conversion.py", line 90, in <genexpr>
    return dict((key, json_to_value(val)) for key, val in payload.items())
                      ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/conversions/conversion.py", line 52, in json_to_value
    struct_value=Struct(fields=dict((k, json_to_value(v)) for k, v in payload.items()))
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/conversions/conversion.py", line 52, in <genexpr>
    struct_value=Struct(fields=dict((k, json_to_value(v)) for k, v in payload.items()))
                                        ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/conversions/conversion.py", line 54, in json_to_value
    raise ValueError(f"Not supported json value: {payload}")  # pragma: no cover
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Not supported json value: [True True True True]
ERROR:root:Not supported json value: [True True]
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/parallel_processor.py", line 61, in _worker
    for processed_item in worker.process(input_queue_iterable()):
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/uploader/grpc_uploader.py", line 109, in process
    yield from self.process_upload(items)
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/uploader/grpc_uploader.py", line 99, in process_upload
    yield upload_batch_grpc(
          ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/uploader/grpc_uploader.py", line 27, in upload_batch_grpc
    points = [
             ^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/uploader/grpc_uploader.py", line 31, in <listcomp>
    payload=payload_to_grpc(payload or {}),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/conversions/conversion.py", line 90, in payload_to_grpc
    return dict((key, json_to_value(val)) for key, val in payload.items())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/conversions/conversion.py", line 90, in <genexpr>
    return dict((key, json_to_value(val)) for key, val in payload.items())
                      ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/conversions/conversion.py", line 52, in json_to_value
    struct_value=Struct(fields=dict((k, json_to_value(v)) for k, v in payload.items()))
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/conversions/conversion.py", line 52, in <genexpr>
    struct_value=Struct(fields=dict((k, json_to_value(v)) for k, v in payload.items()))
                                        ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/qdrant_client/conversions/conversion.py", line 54, in json_to_value
    raise ValueError(f"Not supported json value: {payload}")  # pragma: no cover
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Not supported json value: [True True]
Iterating parquet files:   0%|         | 0/1 [00:03<?, ?it/s]
Importing namespaces:   0%|            | 0/1 [00:49<?, ?it/s]
Importing indexes:   0%|               | 0/1 [00:49<?, ?it/s]
Error: Thread unexpectedly terminated

Example payload:

{
    "language": [
        "ro",
        "vi",
        "sk",
        "zh-TW",
        "ar",
        "es",
        "cs",
        "hu",
        "ba",
        "ru",
        "pt",
        "pl",
        "zh",
        "ru",
        "id",
        "it",
        "pt",
        "ca",
        "en",
        "fr",
        "uk",
        "ru",
        "ru",
        "iw",
        "nl",
        "th",
        "lv"
    ],
    "page_url": [
        "https://ro.wikipedia.org/wiki/Scolopendra_gigantea",
        "https://vi.wikipedia.org/wiki/Scolopendra_gigantea",
        "https://sk.wikipedia.org/wiki/Scolopendra_gigantea",
        "https://zh.wikipedia.org/wiki/%E7%A7%98%E9%AD%AF%E5%B7%A8%E4%BA%BA%E8%9C%88%E8%9A%A3",
        "https://ar.wikipedia.org/wiki/%D8%AD%D8%B1%D9%8A%D8%B4_%D8%A7%D9%84%D8%A3%D9%85%D8%A7%D8%B2%D9%88%D9%86_%D8%A7%D9%84%D8%B9%D9%85%D9%84%D8%A7%D9%82",
        "https://es.wikipedia.org/wiki/Scolopendra_gigantea",
        "https://cs.wikipedia.org/wiki/Stonoha_obrovsk%C3%A1",
        "https://hu.wikipedia.org/wiki/Scolopendra_gigantea",
        "https://ba.wikipedia.org/wiki/%D0%9E%D0%B3%D0%B8%D0%B1_%D1%8B%D2%99%D0%B0%D0%BD%D1%8B",
        "https://ru.wikipedia.org/wiki/%D0%9D%D0%B0%D1%81%D1%82%D0%BE%D1%8F%D1%89%D0%B8%D0%B5_%D1%81%D0%BA%D0%BE%D0%BB%D0%BE%D0%BF%D0%B5%D0%BD%D0%B4%D1%80%D1%8B",
        "https://pt.wikipedia.org/wiki/Scolopendra_gigantea",
        "https://pl.wikipedia.org/wiki/Skolopendra_olbrzymia",
        "https://zh.wikipedia.org/zh-cn/%E7%A7%98%E9%AD%AF%E5%B7%A8%E4%BA%BA%E8%9C%88%E8%9A%A3",
        "https://ru.wikipedia.org/wiki/%D0%93%D0%B8%D0%B3%D0%B0%D0%BD%D1%82%D1%81%D0%BA%D0%B0%D1%8F_%D1%81%D0%BA%D0%BE%D0%BB%D0%BE%D0%BF%D0%B5%D0%BD%D0%B4%D1%80%D0%B0",
        "https://id.wikipedia.org/wiki/Scolopendra_gigantea",
        "https://it.wikipedia.org/wiki/Scolopendra_gigantea",
        "https://pt.wikipedia.org/wiki/Scolopendra",
        "https://ca.wikipedia.org/wiki/Scolopendra_gigantea",
        "https://en.wikipedia.org/wiki/Scolopendra_gigantea",
        "https://fr.wikipedia.org/wiki/Scolopendra_gigantea",
        "https://uk.wikipedia.org/wiki/%D0%A1%D0%BA%D0%BE%D0%BB%D0%BE%D0%BF%D0%B5%D0%BD%D0%B4%D1%80%D0%B0_%D0%B3%D1%96%D0%B3%D0%B0%D0%BD%D1%82%D1%81%D1%8C%D0%BA%D0%B0",
        "https://ru.wikipedia.org/wiki/%D0%A3%D1%80%D0%BE%D1%87%D0%B8%D1%89%D0%B5_%D0%9E%D0%B3%D0%B8%D0%B1",
        "https://ru.wikipedia.org/wiki/%D0%A1%D0%BA%D0%BE%D0%BB%D0%BE%D0%BF%D0%B5%D0%BD%D0%B4%D1%80%D1%8B",
        "https://he.wikipedia.org/wiki/%D7%A8%D7%91%D7%A0%D7%93%D7%9C_%D7%90%D7%9E%D7%96%D7%95%D7%A0%D7%90%D7%A1_%D7%A2%D7%A0%D7%A7",
        "https://nl.wikipedia.org/wiki/Scolopendra_gigantea",
        "https://th.wikipedia.org/wiki/%E0%B8%95%E0%B8%B0%E0%B8%82%E0%B8%B2%E0%B8%9A%E0%B8%A2%E0%B8%B1%E0%B8%81%E0%B8%A9%E0%B9%8C%E0%B8%82%E0%B8%B2%E0%B9%80%E0%B8%AB%E0%B8%A5%E0%B8%B7%E0%B8%AD%E0%B8%87%E0%B9%80%E0%B8%9B%E0%B8%A3%E0%B8%B9",
        "https://lv.wikipedia.org/wiki/Skolopendru_dzimta"
    ],
    "attribution_passes_lang_id": [
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true
    ],
    "caption_alt_text_description": [
        null,
        null,
        null,
        null,
        null,
        null,
        "alternativní popis obrázku chybí",
        "Mindegyik szelvényén egy-egy lábpár ül",
        null,
        null,
        null,
        "Ilustracja",
        null,
        null,
        null,
        null,
        "Scolopendra gigantea",
        null,
        null,
        null,
        "Гігантська сколопендра. Пуерто-Рико",
        null,
        "Гигантская сколопендра (Scolopendra gigantea)",
        null,
        "Scolopendra gigantea",
        null,
        "Milzu skolopendra (Scolopendra gigantea)"
    ],
    "caption_reference_description": [
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        "Mindegyik szelvényén egy-egy lábpár ül",
        "Сколопендра",
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        "Сколопендра",
        null,
        null,
        null,
        null,
        "Milzu skolopendra (Scolopendra gigantea)"
    ],
    "caption_title_and_reference_description": [
        null,
        "Scolopendra gigantea [SEP] ",
        null,
        "秘魯巨人蜈蚣 [SEP] ",
        "حريش الأمازون العملاق [SEP] ",
        null,
        "Stonoha obrovská [SEP] ",
        "Scolopendra gigantea [SEP] Mindegyik szelvényén egy-egy lábpár ül",
        null,
        null,
        "Scolopendra gigantea [SEP] ",
        null,
        "秘魯巨人蜈蚣 [SEP] ",
        "Гигантская сколопендра [SEP] ",
        "Scolopendra gigantea [SEP] ",
        "Scolopendra gigantea [SEP] ",
        "Scolopendra [SEP] ",
        "Scolopendra gigantea [SEP] ",
        "Scolopendra gigantea [SEP] ",
        null,
        "Сколопендра гігантська [SEP] ",
        null,
        "Сколопендры [SEP] ",
        null,
        "Scolopendra gigantea [SEP] ",
        null,
        "Skolopendru dzimta [SEP] Milzu skolopendra (Scolopendra gigantea)"
    ],
    "context_page_description": [
        "Scolopendra gigantea este un miriapod din clasa Chilopoda, fiind cel mai mare reprezentant al genului Scolopendra. Adultul poate atinge o lungime de 26 cm, uneori depășind 30 cm. Această specie habitează în regiunile de nord și de vest a Americii de Sud, pe insulele Trinidad, insulele Virgine, Jamaica Hispaniola ș.a. Localnicii denumesc scolopendra chilopodul gigant galben și chilopodul gigant amazonian.",
        "Scolopendra gigantea là đại diện lớn nhất của chi Scolopendra nói riêng và cả lớp rết nói chung, thường đạt độ dài 26 cm và có thể vượt quá 30 cm. Sinh sống ở khu vực phía bắc và tây của Nam Mỹ và các đảo Trinidad, Puerto Rico, Saint Thomas, U.S. Virgin Islands, Jamaica, và Hispaniola.",
        "Scolopendra gigantea, starší slovenský nazov: štípavica veľká, je živočích z rodu Scolopendra, s veľkosťou do 30 cm.",
        "祕魯巨人蜈蚣祕魯巨人蜈蚣是世界上最大的蜈蚣。棲息於南美洲與加勒比地區，平均長度為30公分，有的個體甚至能長到38 cm。以節肢動物、兩棲類、哺乳動物與爬蟲類為食。",
        "حريش الأمازون العملاق و المعروفة أيضا بحريش البيرو أصفر الساق هي واحدة من أكبر ممثلي جنس العقربان بطول يصل إلى 30 سم. ويمكن العثور عليها في أماكن مختلفة من أمريكا الجنوبية ومنطقة البحر الكاريبي، حيث يفترس مجموعة كبيرة ومتنوعة من الحيوانات، بما في ذلك المفصليات الأخرى، والبرمائيات والثدييات والزواحف. ومن المعروف أنها عدوانية جدا.",
        "La escolopendra gigante es una especie de miriápodo que se encuentra en las tierras bajas de Venezuela, Costa Rica, Colombia, República Dominicana, México y las islas de Trinidad, Jamaica, Aruba, Nicaragua, Puerto Rico y Curazao.​",
        "Stonoha obrovská je druh stonožky žijící v pralesích tropické Jižní Ameriky a na antilských ostrovech. Dosahuje délky 25 až 30 centimetrů.",
        "A Scolopendra gigantea a százlábúak osztályának Scolopendromorpha rendjébe, ezen belül a Scolopendridae családjába tartozó faj.",
        "Огиб ыҙаны— төбәк әһәмиәтендәге комплекслы тәбиғәт һәйкәле. Ростов өлкәһе Усть-Донецкий районында урынлашҡан. Тоҙло тау биттәре Ростов өлкәһе Хөкүмәте 349 Ҡарары нигеҙендә тәбиғәт һәйкәле статусын алған.",
        "Настоящие сколопендры — семейство губоногих многоножек, включающее около 20 родов и более 400 видов, в том числе сколопендр.",
        "Scolopendra giganteá, também conhecida como centopeia gigante peruana de perna amarela ou centopeia gigante amazônica, é uma das maiores centopeias do gênero Scolopendra, com comprimento de até 30 centímetros. Essa espécie é encontrada em vários lugares da América do Sul e do Caribe, onde caça uma grande variedade de animais, incluindo outros artrópodes, anfíbios, mamíferos e répteis consideráveis.",
        "Skolopendra olbrzymia – gatunek południowoamerykańskiego wija z rodzaju Scolopendra.",
        "秘鲁巨人蜈蚣秘鲁巨人蜈蚣是世界上最大的蜈蚣。栖息于南美洲与加勒比地区，平均长度为30厘米，有的个体甚至能长到38 cm。以节肢动物、两栖类、哺乳动物与爬虫类为食。",
        "Гигантская сколопендра — вид губоногих многоножек из семейства настоящих сколопендр, обитающих на севере и западе Южной Америки и на островах Ямайка и Тринидад. Известны случаи нападения гигантской сколопендры на птиц, ящериц, мышей и жаб. Особенно ядовитыми являются самки. Ошибочно считается, что укус гигантской сколопендры смертелен и для человека.",
        "Scolopendra gigantea, dikenal juga sebagi lipan kaki kuning raksasa peru atau lipan raksasa amazon, adalah salah satu lipan terbesar dari genus Scolopendra dengan panjang hingga 30 sentimeter. Spesies ini ditemukan di berbagai tempat di Amerika Selatan dan Karibia, di mana ia memangsa berbagai macam hewan, termasuk artropoda lain, amfibi, mamalia dan reptil.",
        "Il centopiedi amazzonico gigante, noto anche come centopiedi peruviano dalle zampe gialle, è uno dei più grandi centopiedi del genere Scolopendra, misurando in lunghezza fino a 30 cm. Questa specie si rinviene in varie zone del Sud America e dei Caraibi, dove preda una grande varietà di animali, includendo altri considerevoli artropodi, anfibi, mammiferi e rettili.",
        "Scolopendra é um gênero de lacraias",
        "Scolopendra gigantea és una espècie de miriàpode de la classe dels quilòpodes. Viu a Veneçuela, Costa Rica, Colòmbia, República Dominicana, Mèxic i les illes de Trinidad, Jamaica, Aruba, Nicaragua, Puerto Rico i Curazao. És el major representant del gènere Scolopendra, ja que pot arribar a longituds de 26 cm i pot superar 30 cm.",
        "Scolopendra gigantea, also known as the Peruvian giant yellow-leg centipede or Amazonian giant centipede, is one of the largest centipedes of the genus Scolopendra with a length up to 30 centimetres. This species is found in various places in South America and the Caribbean, where it preys on a wide variety of animals, including other sizable arthropods, amphibians, mammals and reptiles.",
        "Scolopendra gigantea est la plus grande scolopendre au monde, elle mesure entre 25 et 30 centimètres, mais les spécimens âgés peuvent atteindre 40 centimètres. Elle vit dans les zones chaudes et humides, et il lui arrive de s'infiltrer dans les maisons. Venimeuse, elle est très agressive, sa morsure, si elle est très douloureuse, est rarement mortelle, sauf pour un enfant ou un adulte fragilisé.",
        "Сколопендра гігантська — вид багатоніжок з родини Scolopendromorpha, одна з найдовших багатоніжок на Землі.",
        "Урочище Огиб — комплексный памятник природы регионального значения. Расположен в Усть-Донецком районе Ростовской области. Статус природного памятника Тузловские склоны получило согласно Постановление правительства Ростовской области от 15.05.2014 №349.",
        "Сколопе́ндры — род губоногих многоножек из отряда сколопендровых.",
        "רבנדל אמזונאס ענק הוא מין הנדל הגדול ביותר בעולם. הוא מצוי בטבע באמריקה הדרומית ובמספר איים קרובים כמו האיים הקריביים.",
        "Scolopendra gigantea is een tijgerduizendpoot uit Zuid-Amerika. De soort jaagt onder andere op grote geleedpotigen, amfibieën, reptielen en kleine zoogdieren. Het is voor zover bekend de grootste niet uitgestorven duizendpoot ter wereld.",
        "ตะขาบยักษ์ขาเหลืองเปรู หรือ ตะขาบยักษ์อเมซอน เป็นตะขาบชนิดที่มีขนาดใหญ่ที่สุดในสกุล Scolopendra โดยปกติเมื่อโตเต็มที่จะยาว 26 เซนติเมตร แต่บางครั้งก็สามารถโตได้ถึง 30 เซนติเมตร ตะขาบชนิดนี้อาศัยอยู่ทางแถบเหนือและตะวันตกของทวีปอเมริกาใต้ และตามเกาะแก่งของประเทศตรินิแดดและจาไมกา เป็นสัตว์กินเนื้อ โดยกินจิ้งจก, กบ, นก, หนู และแม้แต่ค้างคาวเป็นอาหาร และขึ้นชื่อในเรื่องความดุร้าย",
        "Skolpendru dzimta pieder pie simtkāju kārtas. Ap 400 dzimtas sugas sastopamas visā pasaulē, īpaši subtropu un tropu apgabalos. Mitinās augsnē, nobirušās lapās, plaisās, spraugās."
    ],
    "context_section_description": [
        null,
        "Scolopendra gigantea (còn được gọi là Rết chân vàng khổng lồ Peru và Rết khổng lồ Amazon) là đại diện lớn nhất của chi Scolopendra nói riêng và cả lớp rết nói chung, thường đạt độ dài 26 cm (10 in) và có thể vượt quá 30 cm (12 in). Sinh sống ở khu vực phía bắc và tây của Nam Mỹ và các đảo Trinidad, Puerto Rico, Saint Thomas, U.S. Virgin Islands, Jamaica, và Hispaniola.",
        null,
        "祕魯巨人蜈蚣（學名：Scolopendra gigantea）祕魯巨人蜈蚣是世界上最大的蜈蚣。棲息於南美洲與加勒比地區，平均長度為30公分（12英寸），有的個體甚至能長到38 cm（15英寸）。以節肢動物、兩棲類、哺乳動物與爬蟲類為食。",
        "حريش الأمازون العملاق و المعروفة أيضا بحريش البيرو أصفر الساق (الاسم العلمي: Scolopendra gigantea) هي واحدة من أكبر ممثلي جنس العقربان بطول يصل إلى 30 سم (12 بوصة). ويمكن العثور عليها في أماكن مختلفة من أمريكا الجنوبية ومنطقة البحر الكاريبي، حيث يفترس مجموعة كبيرة ومتنوعة من الحيوانات، بما في ذلك المفصليات الأخرى، والبرمائيات والثدييات والزواحف. ومن المعروف أنها عدوانية جدا.",
        null,
        "Stonoha obrovská (Scolopendra gigantea) je druh stonožky žijící v pralesích tropické Jižní Ameriky a na antilských ostrovech. Dosahuje délky 25 až 30 centimetrů.",
        "A Scolopendra gigantea a százlábúak (Chilopoda) osztályának Scolopendromorpha rendjébe, ezen belül a Scolopendridae családjába tartozó faj.",
        "Огиб ыҙаны — төбәк әһәмиәтендәге комплекслы тәбиғәт һәйкәле. Ул Ростов өлкәһе Усть-Донецкий районында урынлашҡан. Тоҙло тау биттәре тәбиғәт һәйкәле статусын Ростов өлкәһе Хөкүмәте 349 Ҡарары (15.05.2014 йыл) нигеҙендә ала.",
        null,
        "Scolopendra giganteá, também conhecida como centopeia gigante peruana de perna amarela ou centopeia gigante amazônica, é uma das maiores centopeias do gênero Scolopendra, com comprimento de até 30 centímetros. Essa espécie é encontrada em vários lugares da América do Sul e do Caribe, onde caça uma grande variedade de animais, incluindo outros artrópodes, anfíbios, mamíferos e répteis consideráveis.",
        null,
        "秘鲁巨人蜈蚣（学名：Scolopendra gigantea）秘鲁巨人蜈蚣是世界上最大的蜈蚣。栖息于南美洲与加勒比地区，平均长度为30厘米（12英寸），有的个体甚至能长到38 cm（15英寸）。以节肢动物、两栖类、哺乳动物与爬虫类为食。",
        null,
        "Scolopendra gigantea, dikenal juga sebagi lipan kaki kuning raksasa peru atau lipan raksasa amazon, adalah salah satu lipan terbesar dari genus Scolopendra dengan panjang hingga 30 sentimeter (12 in). Spesies ini ditemukan di berbagai tempat di Amerika Selatan dan Karibia, di mana ia memangsa berbagai macam hewan, termasuk artropoda lain, amfibi, mamalia dan reptil.",
        "Il centopiedi amazzonico gigante (Scolopendra gigantea (Linnaeus, 1758)), noto anche come centopiedi peruviano dalle zampe gialle, è uno dei più grandi centopiedi del genere Scolopendra, misurando in lunghezza fino a 30 cm. Questa specie si rinviene in varie zone del Sud America e dei Caraibi, dove preda una grande varietà di animali, includendo altri considerevoli artropodi, anfibi, mammiferi e rettili.",
        "Scolopendra é um gênero de lacraias (família Scolopendridae)",
        "Scolopendra gigantea és una espècie de miriàpode de la classe dels quilòpodes. Viu a Veneçuela (inclosa l'illa de Margarita), Costa Rica, Colòmbia, República Dominicana, Mèxic i les illes de Trinidad, Jamaica, Aruba, Nicaragua, Puerto Rico i Curazao. És el major representant del gènere Scolopendra, ja que pot arribar a longituds de 26 cm i pot superar 30 cm.",
        "Scolopendra gigantea, also known as the Peruvian giant yellow-leg centipede or Amazonian giant centipede, is one of the largest centipedes of the genus Scolopendra with a length up to 30 centimetres (12 in). This species is found in various places in South America and the Caribbean, where it preys on a wide variety of animals, including other sizable arthropods, amphibians, mammals and reptiles.",
        null,
        "Сколопендра гігантська (Scolopendra gigantea) — вид багатоніжок з родини Scolopendromorpha, одна з найдовших багатоніжок на Землі.",
        "Урочище Огиб является комплексным памятником природы регионального значения. Находится он в Усть-Донецком районе Ростовской области. Статус природного памятника Тузловские склоны получило согласно Постановление правительства Ростовской области от 15.05.2014 №349. ",
        "Сколопе́ндры (лат. Scolopendra) — род губоногих многоножек из отряда сколопендровых (Scolopendromorpha).",
        null,
        "Scolopendra gigantea is een tijgerduizendpoot uit Zuid-Amerika. De soort jaagt onder andere op grote geleedpotigen, amfibieën, reptielen en kleine zoogdieren. Het is voor zover bekend de grootste niet uitgestorven duizendpoot ter wereld.",
        null,
        "Skolpendru dzimta (Scolopendridae) pieder pie simtkāju kārtas. Ap 400 dzimtas sugas sastopamas visā pasaulē, īpaši subtropu un tropu apgabalos. Mitinās augsnē, nobirušās lapās, plaisās, spraugās."
    ],
    "hierarchical_section_title": [
        "Scolopendra gigantea",
        "Scolopendra gigantea",
        "Scolopendra gigantea",
        "秘魯巨人蜈蚣",
        "حريش الأمازون العملاق",
        "Scolopendra gigantea",
        "Stonoha obrovská",
        "Scolopendra gigantea",
        "Огиб ыҙаны / Һүрәтләү",
        "Настоящие сколопендры",
        "Scolopendra gigantea",
        "Skolopendra olbrzymia",
        "秘魯巨人蜈蚣",
        "Гигантская сколопендра / Фото",
        "Scolopendra gigantea",
        "Scolopendra gigantea",
        "Scolopendra",
        "Scolopendra gigantea",
        "Scolopendra gigantea",
        "Scolopendra gigantea / Galerie",
        "Сколопендра гігантська",
        "Урочище Огиб / Описание",
        "Сколопендры",
        "רבנדל אמזונאס ענק",
        "Scolopendra gigantea",
        "ตะขาบยักษ์ขาเหลืองเปรู",
        "Skolopendru dzimta"
    ],
    "is_main_image": [
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true
    ],
    "page_changed_recently": [
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true,
        true
    ],
    "page_title": [
        "Scolopendra gigantea",
        "Scolopendra gigantea",
        "Scolopendra gigantea",
        "秘魯巨人蜈蚣",
        "حريش الأمازون العملاق",
        "Scolopendra gigantea",
        "Stonoha obrovská",
        "Scolopendra gigantea",
        "Огиб ыҙаны",
        "Настоящие сколопендры",
        "Scolopendra gigantea",
        "Skolopendra olbrzymia",
        "秘魯巨人蜈蚣",
        "Гигантская сколопендра",
        "Scolopendra gigantea",
        "Scolopendra gigantea",
        "Scolopendra",
        "Scolopendra gigantea",
        "Scolopendra gigantea",
        "Scolopendra gigantea",
        "Сколопендра гігантська",
        "Урочище Огиб",
        "Сколопендры",
        "רבנדל אמזונאס ענק",
        "Scolopendra gigantea",
        "ตะขาบยักษ์ขาเหลืองเปรู",
        "Skolopendru dzimta"
    ],
    "section_title": [
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        null,
        "Һүрәтләү",
        null,
        null,
        null,
        null,
        "Фото",
        null,
        null,
        null,
        null,
        null,
        "Galerie",
        null,
        "Описание",
        null,
        null,
        null,
        null,
        null
    ]
}
dhruv-anand-aintech commented 7 months ago 
ERROR:root:<_InactiveRpcError of RPC that terminated with:
        status = StatusCode.INVALID_ARGUMENT
        details = "cannot convert to json number"
        debug_error_string = "UNKNOWN:Error received from peer ipv4:35.245.15.233:6334 {created_time:"2024-02-23T14:37:30.563961+05:30", grpc_status:3, grpc_message:"cannot convert to json number"}"

This is the error I'm getting right now

Anush008 commented 7 months ago

@dhruv-anand-aintech, can you try REST instead of GRPC for debugging? The errors will be clearer.

dhruv-anand-aintech commented 7 months ago

found something related in https://github.com/qdrant/qdrant/issues/3389 Trying to convert the NaNs to 0 now,

Edit: Worked. Merging