Using zkcopy can be difficult in certain cases due to the transaction / batch size..
After trying to do a few copies eventually you can face errors such as:
NIOServerCxn.Factory:0.0.0.0/0.0.0.0:1111:NIOServerCnxn@362] - Exception causing close of session 0x........ due to java.io.IOException: Len error 1634568
which points to jute.maxbuffer type errors etc..
I tried playing a guessing game where each time I get this kind of error I divide by batch size to see what possibly the average size is and adjust -b flag.. This works until it doesn't and eventually came across some special big znodes at about 500kb each which even a batchsize of 2 can be too big compared to jute.maxbuffer 1M default!
So I wonder if maybe when trying to AutoCommitTransactionWrapper.create() we could check the data length and if it appears to be reasonably big (eg: as big as 1/3 of jute.maxbuffer?) and there are pending commits to actually commit() the old and then single commit() the big transaction, or similar.
My current workaround was to use -b 1 (and potentially suppress logging if (transactionSize > 10) { Writer.logger.info("Committing transaction.... )
Using zkcopy can be difficult in certain cases due to the transaction / batch size.. After trying to do a few copies eventually you can face errors such as:
which points to
jute.maxbuffertype errors etc..I tried playing a guessing game where each time I get this kind of error I divide by batch size to see what possibly the average size is and adjust -b flag.. This works until it doesn't and eventually came across some special big znodes at about 500kb each which even a batchsize of 2 can be too big compared to jute.maxbuffer 1M default!
So I wonder if maybe when trying to
AutoCommitTransactionWrapper.create()we could check the data length and if it appears to be reasonably big (eg: as big as 1/3 of jute.maxbuffer?) and there are pending commits to actually commit() the old and then single commit() the big transaction, or similar.My current workaround was to use -b 1 (and potentially suppress logging
if (transactionSize > 10) { Writer.logger.info("Committing transaction....)