Boltex.Bolt.generate_chunks fails with too much data

[adri]

Hi, first of all thank you very much for this project!

When running a query with parameters I encountered the following error message:

(FunctionClauseError) no function clause matching in Boltex.Bolt.generate_chunks/3
    (bolt_sips) lib/boltex/bolt.ex:115: Boltex.Bolt.generate_chunks([<<178, 16, 209, 7, 244, 77, 69, 82, 71, 69, 32, 40, 103, 105, 116, 104, 117, 98, 58, 71, 105, 116, 104, 117, 98, 80, 114, 111, 102, 105, 108, 101, 32, 123, 32, 103, 105, 116, 104, 117, 98, 73, 100, 58, 32, 36, 117, 115, 101, ...>>, <<177, 63, 192>>], [], "")
    (bolt_sips) lib/boltex/bolt.ex:102: Boltex.Bolt.send_messages/3
    (bolt_sips) lib/boltex/bolt.ex:157: Boltex.Bolt.run_statement/4

With a larger amount of parameters it seems like generating chunks isn't working correctly. Unfortunately I wasn't able to find the exact size.

What I saw is that on this line the chunks variable is mentioned twice in the function signature. Could that be an issue?

[adri]

I just saw that this is the wrong library. I'll open an issue in the right one ;)

[florinpatrascu]

Thank you for the report! Probably not the wrong place, since I forked the Boltex, and kind of maintaining it in Bolt.Sips too?! :) Would it be possible to provide a test case for this issue? Thank you!

Edit:

• I'll watch the development on mschae/boltex/issues/13 and get the feedback from there. Thanks again.

[adri]

Since there are some tests in this project already it was easier add a test that fails in bolt_sips :) In the module Query.Test you can use:

  test "executing a Cypher query, with large set of parameters", context do
    conn = context[:conn]

    cypher = """
      MATCH (n:Person {bolt_sips: true})
      FOREACH (i IN $largeRange| SET n.test = TRUE )
    """
    case Bolt.Sips.query(conn, cypher, %{largeRange: Enum.to_list(0..1_000_000)}) do
      {:ok, stats} ->
        IO.inspect stats
        assert stats["properties-set"] > 0, "Expecting many properties set"
      {:error, reason} -> IO.puts "Error: #{reason["message"]}"
    end
  end

As far as I can see the error is related to what I posted in the issue description:

** (MatchError) no match of right hand side value: <<178, 16, 208, 83, 77, 65, 84, 67, 72, 32, 40, 110, 58, 80, 101, 114, 115, 111, 110, 32, 123, 98, 111, 108, 116, 95, 115, 105, 112, 115, 58, 32, 116, 114, 117, 101, 125, 41, 10, 32, 32, 70, 79, 82, 69, 65, 67, 72, 32, 40, ...>>
                 lib/boltex/bolt.ex:147: Boltex.Bolt.generate_chunks/3
                 lib/boltex/bolt.ex:117: Boltex.Bolt.send_messages/3

Note, that the query works if you change largeRange from 1_000_000 to 1_000.

[florinpatrascu]

Thank you, @adri! I'll add a pointer to @mschae's project, for the reference; https://github.com/mschae/boltex/issues/13. I am curious to test this too, as it seems to be a special use case. Out of pure curiosity, did you stumble upon this issue in a real scenario or it was pure experimental? Never encountered a situation where I had to transport this much data, in a single call.

[mschae]

Thank you @adri and @florinpatrascu. While I implemented multi-chunk messages sending it seems like (contrary to what I believed) I never actually tested it.

Guess what: Untested code doesn't work. Who would've thought.

Sorry about the snafu on my end, I addressed (and believe to have fixed) it in mschae/boltex#15.

[florinpatrascu]

Thank you both for reporting the issue and for the support. With Boltex working correctly now (thanks @mschae!), I was able to spot a bug at my end too: the pool was expiring prematurely and a misplaced timeout parameter. @adri - I aded your test to the test suite and it is passing w/o issues on local and remote databases; many thanks for it! I even throttled the remote db, and verified the test still passes, provided the driver configuration is properly tuned. New package published to https://hex.pm/packages/bolt_sips/0.2.1 - and thanks again, both! Cheers!