Open DiegoAlfa7 opened 4 months ago
In your gist example, payload data passed to the request does not match the required format. You can see info about the use of /bulk
here: https://manual.manticoresearch.com/dev/Data_creation_and_modification/Adding_documents_to_a_table/Adding_documents_to_a_real-time_table#Bulk-adding-documents
I've updated the bug script, forget the gist.
Heres a screenshot that shows the bug when executing the script and ngreping the API port:
Hi, It appears the request from the updated script still has a few errors:
PAYLOAD
data you encode already has encoded unicode sequences which leads to double encoding and an error when Manticore tries to decode it, so the request needs unicode-decoding as well (something like PAYLOAD.encode('utf-8').decode('unicode-escape')
should help)null
values, unvalid in the JSON schemasentiment
feeds
field is defined as multi64
, but string
values are passed for it in the data which is not allowed in Manticoreid
field passed twice for each data row, as a separate field and as a field in the doc
object, which cannot be correctly handled by Manticore (also, take note that the document id must be a number, not a string)Can you please fix them and repeat your tests?
Im into it, anyway, if the payload contains such errors (wich are definetly not supposed to be there) shouldn't manticore respond only one time with the corresponding 400 / 500 Response code?
I mean, the issue in the MRE is still on our side, but seems odd that the API responds 200 - OK to python requests, that could (an did, in my case) lead to silent errors and unexpected missing data.
Im still convinced that the issue has something to do with the unicode chars. I will solve all issues except for that one in a specific test and then i will test it again with the payload encoded in utf8 and show you both results.
Sorry for the issues with the MRE. Thanks for your time.
Bug Description:
As the title says, the JSON HTTP API returns 2 responses for a single
bulk
request (sometimes)Here's the MRE to reproduce the bug (once the enviroment is set).
Running the above script:
python bug.py
2 out of 5 runs, it fails and searchd returns two HTTP responses 🤯
Manticore version used:
Manticore 6.2.13 f6c433af8@24052107 dev (columnar 2.2.5 a915428@24051706) (secondary 2.2.5 a915428@24051706) (knn 2.2.5 a915428@24051706)
See this capture with
ngrep -d lo -W byline port 9308
:The script is being executed in the host, whose running a docker-compose with the
manticore:dev
tag running as a service namedmanticore
. The manticore service default ports are redirected to the host machine.Manticore Search Version:
Manticore 6.2.13 f6c433af8@24052107 dev (columnar 2.2.5 a915428@24051706) (secondary 2.2.5 a915428@24051706) (knn 2.2.5 a915428@24051706)
Operating System Version:
Distributor ID: Ubuntu Description: Ubuntu 20.04.6 LTS Release: 20.04 Codename: focal
Have you tried the latest development version?
Yes
Internal Checklist:
To be completed by the assignee. Check off tasks that have been completed or are not applicable.