Open danieljue opened 4 years ago
Hi @danieljue thank you for reporting, you've probably hit RedisGraph parser buffer size limit. Let me check what are the consequences of enlarging this buffer.
Thanks! I looked on github and maybe there are some ideas that can be borrowed from here:
Their actual bulk insert library doesn't work well for my use case, because the data is not in a CSV type of format, but I saw some code in there relating to buffer size.
Perhaps there's a way to allow an "unsafe" commit for the edges, where the code doesn't check for the existence of the nodes. In this way we could insert nodes in smaller batches, and insert edges later on, putting the responsibility of those nodes' existence on the developer.
I'm running the current docker image in Windows. I discovered this problem when my program encountered a large book and tried to ingest sentences from it. I was able to reproduce the problem with a more simple example in a jupyter notebook.
If the range is only 9040, no errors. If I increase it to 9041 or higher, I get this error:
Error ResponseError: errMsg: Invalid input 'J': expected '.', AND, OR, XOR, NOT, '=~', '=', '<>', '+', '-', '*', '/', '%', '^', IN, CONTAINS, STARTS WITH, ENDS WITH, '<=', '>=', '<', '>', IS NULL, IS NOT NULL, '[', '{', a label, ',' or '}' line: 1, column: 1048604, offset: 1048603 errCtx: ...e{w:"GVNjJCREATE (pkuvboenoi:name{w:"Jean Piaget"}),(tabnluexcb:name{w:"Ye... errCtxOffset: 40
Depending on your machine or the size of the nodes, the number could differ (like 10000 or 20000). I was also able to cause this problem with only one node that had a large property.
Expected behavior: Tell me if the content of the commit is too large Or Handle it gracefully.