How to increase the buffer size of the builder

[VEDANTDOKANIA]

I am getting very large data from the plugin side and i am getting only half data because of length of buffer. How can I increase the length of the buffer in stdout and stderr

[bturner]

@VEDANTDOKANIA,

You can't increase the buffer sizes; they're hard-coded. The expectation for large output (or for sending in large input) is that you'll receive multiple callbacks. It's not scalable for NuProcess to assume it's safe to try and load the full stdout or stderr and hand it to the callback in a single buffer. If you're getting >64K in output, you'll need to do something with it between callbacks. (Note that you cannot leave it in the buffer NuProcess passes to onStdout or onStderr, as doing so will cause processing to fail. You need to return the buffer to NuProcess with at least one byte available for writing.) One option is to allocate your own buffer (be it a ByteBuffer, ByteArrayOutputStream or whatever) and use that to aggregate data across onStdout/onStderr callbacks (where you'd probably need 2 buffers if you're using both).

[VEDANTDOKANIA]

Hello brett

Thank you for the answer. I will definetly try the same. I have one small question regarding Nu process builder and vertx. When you use Nu process builder in your function and function returns future and you call the same future inside execute blocking the nu process builder handler runs in process queue instead of worker thread.

My function :-

public static Future checkAvailability(JsonObject credential) { var errors = new ArrayList(); Promise promise = Promise.promise(); if ((!credential.containsKey("ip")) || credential.getString("ip") == null) { errors.add("IP address is null in check availability"); LOGGER.error("IP address is null in check availability"); } else { var processBuilder = new NuProcessBuilder(Arrays.asList("fping", "-c", "3", "-t", "1000", "-q", credential.getString("ip"))); var handler = new ProcessHandler(); processBuilder.setProcessListener(handler); var process = processBuilder.start(); try{ handler.onStart(process); process.waitFor(4000, TimeUnit.MILLISECONDS); var result = handler.output(); if (result == null) { errors.add("Request time out occurred"); } else { var pattern = Pattern.compile("\d+.\d+.\d+.\d+\s+\:\s+\w+/\w+/%\w+\s+\=\s+\d+/\d+/(\d+)%"); var matcher = pattern.matcher(result); if (matcher.find()) { if (!matcher.group(1).equals("0")) { errors.add(" packet loss percentage is :" + matcher.group(1)); } } } }catch (Exception exception){ errors.add(exception.getCause().getMessage()); }finally { process.destroy(true); } if (errors.isEmpty()) { promise.complete(credential); } else { promise.fail(errors.toString()); }} return promise.future(); } I am calling function like this vertx.executeBlocking(blockingHandler ->{ Utils.checkAvailability(handler.body()). compose(future ->Utils.checkPort(handler.body())). compose(future -> Utils.spwanProcess(handler.body())) .onComplete( completeHandler->{ if(completeHandler.succeeded() && completeHandler.result().getJsonObject(RESULT).getString(STATUS).equals(SUCCESS)){ handler.reply(completeHandler.result().getJsonObject(RESULT)); }else{ handler.fail(-1, completeHandler.cause().getMessage()); } blockingHandler.complete(); }); }); But instead of running in worker thread the nu process handler Executes in normal process queue. As specific reason for the same? Thanking you Vedant Dokania

---

From: Bryan Turner @.> Sent: Thursday, May 26, 2022 4:39:38 AM To: brettwooldridge/NuProcess @.> Cc: Vedant Dokania @.>; Mention @.> Subject: Re: [brettwooldridge/NuProcess] How to increase the buffer size of the builder (Issue #140)

@VEDANTDOKANIAhttps://github.com/VEDANTDOKANIA,

You can't increase the buffer sizes; they're hard-coded. The expectation for large output (or for sending in large input) is that you'll receive multiple callbacks. It's not scalable for NuProcess to assume it's safe to try and load the full stdout or stderr and hand it to the callback in a single buffer. If you're getting >64K in output, you'll need to do something with it between callbacks. (Note that you cannot leave it in the buffer NuProcess passes to onStdout or onStderr, as doing so will cause processing to fail. You need to return the buffer to NuProcess with at least one byte available for writing.) One option is to allocate your own buffer (be it a ByteBuffer, ByteArrayOutputStream or whatever) and use that to aggregate data across onStdout/onStderr callbacks (where you'd probably need 2 buffers if you're using both).

— Reply to this email directly, view it on GitHubhttps://github.com/brettwooldridge/NuProcess/issues/140#issuecomment-1137933293, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AOUIT4DAEYE5XIGAUPMTENLVL2XLFANCNFSM5W4BKZFA. You are receiving this because you were mentioned.Message ID: @.***>

[bturner]

@VEDANTDOKANIA,

I'm not entirely sure what you're trying to ask, or what your code is trying to show. (One thing I can tell you for sure, you shouldn't call handler.onStart(process); NuProcess will make all the handler callbacks itself automatically, in the correct places for the process's lifecycle. It will call onStart after onPreStart at some point after you call processBuilder.start().)

I don't have any direct familiarity with VertX, so I can't really offer any advice/insight into anything that involves it. That said, if you use NuProcessBuilder.start to start your process, you're essentially telling NuProcess to run your process asynchronously, and NuProcess handles the threading. That can be very useful if your application starts a number of long-running processes that sporadically produce output, since a large number of processes can efficiently be pumped by a small number of threads. In that model, all of your NuProcessHandler.onStdin, onStdout and onStderr callbacks will happen from a NuProcess-managed thread (and it will always be the same NuProcess thread), regardless of your surrounding environment. NuProcess always calls onPreStart and onStart from the thread where you call NuProcessBuilder.start. Your handler's onExit callback could be invoked from either thread--the starting thread or a NuProcess thread--depending on how the process exits.

If you always want to block and wait for the process to complete, you may want to consider using NuProcessBuilder.run instead. If you use run, all NuProcessHandler callbacks will always be made on the thread you called run on, since it's used to do the actual process handling. If your goal is for NuProcess's processing to happen on some work thread you control, run should do it.

[VEDANTDOKANIA]

Thank you for the answer and spending your valuable time on the same.

Vedant Dokania

---

From: Bryan Turner @.> Sent: Thursday, May 26, 2022 7:19:01 AM To: brettwooldridge/NuProcess @.> Cc: Vedant Dokania @.>; Mention @.> Subject: Re: [brettwooldridge/NuProcess] How to increase the buffer size of the builder (Issue #140)

@VEDANTDOKANIAhttps://github.com/VEDANTDOKANIA,

I'm not entirely sure what you're trying to ask, or what your code is trying to show. (One thing I can tell you for sure, you shouldn't call handler.onStart(process); NuProcess will make all the handler callbacks itself automatically, in the correct places for the process's lifecycle. It will call onStart after onPreStart at some point after you call processBuilder.start().)

I don't have any direct familiarity with VertX, so I can't really offer any advice/insight into anything that involves it. That said, if you use NuProcessBuilder.start to start your process, you're essentially telling NuProcess to run your process asynchronously, and NuProcess handles the threading. That can be very useful if your application starts a number of long-running processes that sporadically produce output, since a large number of processes can efficiently be pumped by a small number of threads. In that model, all of your NuProcessHandler.onStdin, onStdout and onStderr callbacks will happen from a NuProcess-managed thread (and it will always be the same NuProcess thread), regardless of your surrounding environment. NuProcess always calls onPreStart and onStart from the thread where you call NuProcessBuilder.start. Your handler's onExit callback could be invoked from either thread--the starting thread or a NuProcess thread--depending on how the process exits.

If you always want to block and wait for the process to complete, you may want to consider using NuProcessBuilder.run instead. If you use run, all NuProcessHandler callbacks will always be made on the thread you called run on, since it's used to do the actual process handling. If your goal is for NuProcess's processing to happen on some work thread you control, run should do it.

— Reply to this email directly, view it on GitHubhttps://github.com/brettwooldridge/NuProcess/issues/140#issuecomment-1138056130, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AOUIT4GU37TOIAKZRQM2HM3VL3KA3ANCNFSM5W4BKZFA. You are receiving this because you were mentioned.Message ID: @.***>