Status code 500 when command exceeds a (short) length with basic auth supplied

Hi,

I am trying to connect to a local winrm listener and execute a powershell script that has more than about 1.7K characters. Unfortunately, the call produces a 500 result. To reproduce the issue I tested code like:

try (ShellCommand sh = winrmClient.createShell()) {
    String ps = "echo " + Base64.getEncoder().encodeToString(bytes);
    int exitCode = sh.execute(ps, out, err);
}

where "bytes" is from the script content. As soon as "ps" exceeds a length of approx. 1.7K, the following error is returned:

[main] INFO org.apache.cxf.services.WinRm.RESP_IN - RESP_IN Address: http://localhost:5985/wsman ResponseCode: 500 ExchangeId: 9d619610-617a-410a-b2ab-d7a288982c2a ServiceName: WinRmService PortName: WinRmPort PortTypeName: WinRm Headers: {Server=Microsoft-HTTPAPI/2.0, Connection=close, Content-Length=0, Date=Tue, 05 Oct 2021 11:18:05 GMT}

Stack trace:

org.apache.cxf.binding.soap.SoapFault: Error reading XMLStreamReader: Unexpected EOF in prolog at [row,col {unknown-source}]: [1,0] at org.apache.cxf.binding.soap.interceptor.ReadHeadersInterceptor.handleMessage(ReadHeadersInterceptor.java:304) at org.apache.cxf.binding.soap.interceptor.ReadHeadersInterceptor.handleMessage(ReadHeadersInterceptor.java:70) at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) at org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:829) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1696) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1570) at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1371) at org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit$AsyncWrappedOutputStream.close(AsyncHTTPConduit.java:429) at io.cloudsoft.winrm4j.client.encryption.SignAndEncryptOutInterceptor$EncryptAndSignOutputStream.processAndShip(SignAndEncryptOutInterceptor.java:148) at io.cloudsoft.winrm4j.client.encryption.SignAndEncryptOutInterceptor$EncryptAndSignOutputStream.close(SignAndEncryptOutInterceptor.java:84) at org.apache.cxf.ext.logging.LoggingOutputStream.postClose(LoggingOutputStream.java:53) at org.apache.cxf.io.CachedOutputStream.close(CachedOutputStream.java:228) at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:671) at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:63) at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:530) at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:441) at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:356) at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:314) at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96) at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:140) at com.sun.proxy.$Proxy48.command(Unknown Source) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at io.cloudsoft.winrm4j.client.RetryingProxyHandler.invoke(RetryingProxyHandler.java:29) at com.sun.proxy.$Proxy49.command(Unknown Source) at io.cloudsoft.winrm4j.client.ShellCommand.execute(ShellCommand.java:94)

Original request was:

[main] INFO org.apache.cxf.services.WinRm.REQ_OUT - REQ_OUT
Address: http://localhost:5985/wsman
HttpMethod: POST
Content-Type: application/soap+xml; action="http://schemas.microsoft.com/wbem/wsman/1/windows/shell/Command"
ExchangeId: 9d619610-617a-410a-b2ab-d7a288982c2a
ServiceName: WinRmService
PortName: WinRmPort
PortTypeName: WinRm
Headers: {Authorization=Basic sanitized, Accept=*/*}
Payload: <soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope">
<soap:Header>
<Action xmlns="http://schemas.xmlsoap.org/ws/2004/08/addressing">http://schemas.microsoft.com/wbem/wsman/1/windows/shell/Command</Action>
<MessageID xmlns="http://schemas.xmlsoap.org/ws/2004/08/addressing">urn:uuid:f9553fbe-b3c0-409b-8c17-f08fe67f07cb</MessageID>
<To xmlns="http://schemas.xmlsoap.org/ws/2004/08/addressing">http://localhost:5985/wsman</To>
<ReplyTo xmlns="http://schemas.xmlsoap.org/ws/2004/08/addressing">
  <Address>http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous</Address>
</ReplyTo>
<ResourceURI xmlns="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd" xmlns:ns2="http://schemas.microsoft.com/wbem/wsman/1/windows/shell" xmlns:ns3="http://schemas.xmlsoap.org/ws/2004/09/transfer">http://schemas.microsoft.com/wbem/wsman/1/windows/shell/cmd</ResourceURI>
<MaxEnvelopeSize xmlns="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd" xmlns:ns2="http://schemas.microsoft.com/wbem/wsman/1/windows/shell" xmlns:ns3="http://schemas.xmlsoap.org/ws/2004/09/transfer">153600</MaxEnvelopeSize>
<OperationTimeout xmlns="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd" xmlns:ns2="http://schemas.microsoft.com/wbem/wsman/1/windows/shell" xmlns:ns3="http://schemas.xmlsoap.org/ws/2004/09/transfer">PT60S</OperationTimeout>
<Locale xmlns="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd" xmlns:ns2="http://schemas.microsoft.com/wbem/wsman/1/windows/shell" xmlns:ns3="http://schemas.xmlsoap.org/ws/2004/09/transfer" xml:lang="en-US"/>
<SelectorSet xmlns="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd" xmlns:ns2="http://schemas.microsoft.com/wbem/wsman/1/windows/shell" xmlns:ns3="http://schemas.xmlsoap.org/ws/2004/09/transfer">
  <Selector Name="ShellId">906ADFF7-6B34-42D9-B713-52C6763AE3ED</Selector>
</SelectorSet>
<OptionSet xmlns="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd" xmlns:ns2="http://schemas.microsoft.com/wbem/wsman/1/windows/shell" xmlns:ns3="http://schemas.xmlsoap.org/ws/2004/09/transfer">
  <Option Name="WINRS_CONSOLEMODE_STDIN">TRUE</Option>
  <Option Name="WINRS_SKIP_CMD_SHELL">FALSE</Option>
</OptionSet>
</soap:Header>
<soap:Body>
<ns2:CommandLine xmlns="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd" xmlns:ns2="http://schemas.microsoft.com/wbem/wsman/1/windows/shell" xmlns:ns3="http://schemas.xmlsoap.org/ws/2004/09/transfer">
  <ns2:Command>echo "sanitized base64 encoded script with length ~1.7K"</ns2:Command>
</ns2:CommandLine>
</soap:Body>
</soap:Envelope>

The 500 http response doesn't provide additional error information. Also, with a slightly shorter command, it is working as expected. However, the max envelope sizes are a lot higher than 2kb. There is no error event in Windows winrm event log.

Do you have an idea what's going on here?

Best regards!

These bits:

ResponseCode: 500

Connection=close, Content-Length=0

Unexpected EOF in prolog at [row,col {unknown-source}]: [1,0]

Makes me think the remote side is sending back an empty response. Would be nice if it told us why but seems you've looked in the obvious places both client-side and server-side.

Could you try doing this using the windows winrm cli or the python or ruby winrm tools, to see if it is on our side or the windows side? If those work, can you compare captures of the wire traffic (eg with ethereal) to see what we are doing differently?

I know windows has an 8k command-length limit in some places which as you say we're not near ... but looks like there is a smaller limit somewhere, maybe around winrm shell soap objects. Possibly there is some automatic chunking strategy that should kick in.

What we do when we have a large file -- which might be a workaround for you -- is to chunk it before invoking the library. Seems we use 1k chunks (size before base64-encoding, so approx 1.4k post) which would be consistent with the 1.7k limit you observe (as I wouldn't expect normally to see such a small chunk size):

https://github.com/apache/brooklyn-server/blob/master/software/winrm/src/main/java/org/apache/brooklyn/util/core/internal/winrm/winrm4j/Winrm4jTool.java#L142

Thanks for the quick response. Using winrs it worked, by

winrs -r:localhost powershell -EncodedCommand ...

and

winrs -r:my computer name powershell -EncodedCommand ...

The powershell script that is producing the error with winrm4j can be run with winrs without problems. So I sniffed the loopback traffic with wireshark and observed 2 differences:

winrm4j is using only <Command> whereas winrs is using <Command> and multiple <Arguments>
winrm4j is shunking http when basic auth is supplied

I cloned the winrm4j git repo and changed these things to be the same as for winrs. The first difference did not change anything but might be useful some day, so here's the patch:

Index: client/src/main/java/io/cloudsoft/winrm4j/client/ShellCommand.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/client/src/main/java/io/cloudsoft/winrm4j/client/ShellCommand.java b/client/src/main/java/io/cloudsoft/winrm4j/client/ShellCommand.java
--- a/client/src/main/java/io/cloudsoft/winrm4j/client/ShellCommand.java    (revision 41dae8289e6733577d9ffc71aa6473e3311ec74e)
+++ b/client/src/main/java/io/cloudsoft/winrm4j/client/ShellCommand.java    (date 1633502896050)
@@ -2,6 +2,7 @@

 import java.io.IOException;
 import java.io.Writer;
+import java.util.Collections;
 import java.util.List;
 import java.util.function.Predicate;

@@ -75,10 +76,15 @@
     }

     public int execute(String cmd, Writer out, Writer err) {
+        return execute(cmd, Collections.emptyList(), out, err);
+    }
+
+    public int execute(String cmd, List<String> arguments, Writer out, Writer err) {
         WinRmClient.checkNotNull(cmd, "command");

         final CommandLine cmdLine = new CommandLine();
         cmdLine.setCommand(cmd);
+        cmdLine.getArguments().addAll(arguments);
         final OptionSetType optSetCmd = new OptionSetType();
         OptionType optConsolemodeStdin = new OptionType();
         optConsolemodeStdin.setName("WINRS_CONSOLEMODE_STDIN");

However, the second difference, disabling chunking actually worked. Longer scripts are accepted without error. In initializeClientAndService in WinRmClient class I added at the bottom:

        else {
            AsyncHTTPConduit httpClient = (AsyncHTTPConduit) client.getConduit();
            httpClient.getClient().setAllowChunking(false);
        }

Of course this is just for testing. Can I disable chunking without modifying the code?

Thanks! Good intel.

I'm not sure if chunking will work for all the auth/encryption schemes but if it works with Basic is it worth exposing an option to allow that to be configurable.

I've also heard anecdotally that setting WINRS_SKIP_CMD_SHELL true (at https://github.com/cloudsoft/winrm4j/blob/c1fbbbd85ffcb9157bb761501e2d13d57a046067/client/src/main/java/io/cloudsoft/winrm4j/client/ShellCommand.java#L89 ) can help with long scripts.

If I get some spare cycles I'll add that option and do some experiments myself. (If you beat me to it and raise a PR, even better!)

I didn't understand the two differences you noticed. Can you elaborate?

winrm4j is using only whereas winrs is using and multiple

What does this mean?

I also didn't understand the purpose of your patch which I think is related to this. It looks like effectively does a cmdLine.getArguments().addAll(Collections.emptyList()) but I'd expect that to be no-op and not affect the wire traffic.

winrm4j is shunking http when basic auth is supplied

Do you mean to say winrm4j is not chunking? Or chunking only when using Basic auth is used? (I agree allowing chunking might be an answer.)

@ahgittin Sorry, I updated my comment above, the < and > letters were recognized as tag I guess. Now it should make sense :)

What does this mean?

This was just an observation, not affecting the chunking. The patch will allow <Command> and <Arguments> tags to be sent. But this is just an enhancement and I don't know if it is useful ;)

Do you mean to say winrm4j is not chunking?

Winrm4j is chunking for Basic auth. It is not chunking for the other authentication methods. I tried with basic auth and I had the problem that even short command lines were rejected. I think it would be nice to have a configuration to disable http chunking also for basic auth. Without chunking, everything works as expected. For example, when I use Kerberos, it works fine.

Thanks - makes sense. I will make chunking false the default for all modes, and have an option to enable.

(I think http chunking probably doesn't work so not yet a useful option, but it is a more sensible default for basic!)

cloudsoft / winrm4j

Status code 500 when command exceeds a (short) length with basic auth supplied #144