Exception in TIdTcpServer : error:1409E10F:SSL routines:ssl3_write_bytes:bad length

[macc2010]

Hello,

I have an application that uses TIdTCPServer to listen for tcp connections made in Delphi 10.4.2 and a client in Android that uses TIdThreadComponent, TIdTCPClient and TIdSSLIOHandlerSocketOpenSSL, also compiled with Delphi 10.4.2.

During the write of a file ( 13429719 bytes ), the server is raising 2 exceptions, first this :

EIdOpenSSLAPISSLError : error:00000000:lib(0):func(0):reason(0)

And after few milliseconds, this exception is raised : error:1409E10F:SSL routines:ssl3_write_bytes:bad length.

In the server part, I have a certificate, I am using in the server and in the client sslvTLSv1_2 and I have set PassThrough := False.

I had this applications working ok in Delphi 10.2.3 and I had not observed these exceptions, although, in Delphi 10.4.2, I can see these exceptions.

I have tried to use the last sources from github, and the server part compiles ok, but the FMX client application for Android does not compiles with an error, so in the FMX client application I am using the Indy version that Delphi 10.4.2 does install.

What can I do to try to solve the problem?.

Thank you.

[macc2010]

I am testing the Android application in 2 devices, one with Android 6 and the other with Android 10, the problem looks always to be in Android 10.

[macc2010]

I am using in the Android application, libcrypto.so 1.892.328 bytes from 17/05/2016 and libssl.so 396.984 from 17/05/2016 extracted in the past from AndroidOpenssl1.0.2h.zip.

[rlebeau]

You say the OpenSSL error occurs during the write of a file, is that the 1st write on a new connection? Are other writes succeeding, or also failing? Can you show your actual code for both sides?

What compiler error are you getting? Delphi 10.4.2 ships with a fairly recent version of Indy from GitHub. Although there have been checkins made after 10.4.2's release, most of them shouldn't affect what you are doing. Though, one big change made just a few months ago was to introduce iconv support under Android, but that was for FreePascal, is that what you are failing on in Delphi, by chance?

[macc2010]

I have found a solution. The problem occurs in the Android part ( FMX ). I have seen that in Delphi 10.4.2, it has been modified the procedure TIdIOHandler.ReadStream to avoid side-effects of an FMX issue with catching and reraising exceptions here during TIdFTP data transfers on iOS. If I use ReadStream from Delphi 10.4.2 I got the exception, but if I use ReadStream from Delphi 10.2.3 ( is very similar except in this part ), the server does not raises any exception, so it looks as if this is the problem.

[macc2010]

My code is the following :

In the server I write to the IOHandler :

        Fs := TFileStream.Create( FileDownloadZip, fmOpenRead or fmShareExclusive );
        try
          // send file size
          FIdContext.Connection.IOHandler.Write( Int64( Fs.Size ) ); 
          // send file content
          FIdContext.Connection.IOHandler.SendBufferSize := 1024 * 1024;
          FIdContext.Connection.IOHandler.Write( Fs, 0, False );
        finally
          Fs.Free;
        end;

And in the client part :

   nSize := FIdTCPClient1.IOHandler.ReadInt64;
   //
    Fs := TFileStream.Create( sFileZip, fmCreate );
    try
      FIdTCPClient1.IOHandler.RecvBufferSize := 1024 * 1024;
      try
        FIdTCPClient1.IOHandler.ReadStream( Fs, nSize ); 
      finally
        FIdTCPClient1.IOHandler.RecvBufferSize := GRecvBufferSizeDefault;
      end;
    finally
      Fs.Free;
    end;

To solve the problem, I have replaced the call to FIdTCPClient1.IOHandler.ReadStream( Fs, nSize ) by a owned procedure called FIdTCPClient1_ReadStream( Fs, nSize ) that contains the code of ReadStream that was in Delphi 10.2.3 ( similar to 10.3..3 ).

Thank you.

[macc2010]

What compiler error are you getting? Delphi 10.4.2 ships with a fairly recent version of Indy from GitHub. Although there have been checkins made after 10.4.2's release, most of them shouldn't affect what you are doing. Though, one big change made just a few months ago was to introduce iconv support under Android, but that was for FreePascal, is that what you are failing on in Delphi, by chance?

Try to compile a FMX android application with github sources in Delphi 10.4.2 and you will see that the compiler defines about MEMORY LEAK from fastmm is used in the FMX android compilation and it output an error in the compiler.

[rlebeau]

I have found a solution. The problem occurs in the Android part ( FMX ). I have seen that in Delphi 10.4.2, it has been modified the procedure TIdIOHandler.ReadStream to avoid side-effects of an FMX issue with catching and reraising exceptions here during TIdFTP data transfers on iOS. If I use ReadStream from Delphi 10.4.2 I got the exception, but if I use ReadStream from Delphi 10.2.3 ( is very similar except in this part ), the server does not raises any exception, so it looks as if this is the problem.

But, what is the ACTUAL PROBLEM with that new logic? I see nothing in the new code that should be causing OpenSSL errors. So, can you please be more specific?

In the server I write to the IOHandler ... And in the client part ...

Is there a reason why you are sending/reading the file size manually, instead of letting Write()/ReadStream() handle that for you?

Fs := TFileStream.Create( FileDownloadZip, fmOpenRead or fmShareExclusive );
try
  // send file size and content
  FIdContext.Connection.IOHandler.SendBufferSize := 1024 * 1024;
  FIdContext.Connection.IOHandler.LargeStream := True;
  FIdContext.Connection.IOHandler.Write( Fs, 0, True );
finally
  Fs.Free;
end;

Fs := TFileStream.Create( sFileZip, fmCreate );
try
  FIdTCPClient1.IOHandler.RecvBufferSize := 1024 * 1024;
  FIdTCPClient1.IOHandler.LargeStream := True;
  FIdTCPClient1.IOHandler.ReadStream( Fs, -1, False ); 
finally
  Fs.Free;
end;

To solve the problem, I have replaced the call to FIdTCPClient1.IOHandler.ReadStream( Fs, nSize ) by a owned procedure called FIdTCPClient1_ReadStream( Fs, nSize ) that contains the code of ReadStream that was in Delphi 10.2.3 ( similar to 10.3..3 ).

That should not be necessary. I would rather like to see the new code be debugged to find out why it is failing so it can be fixed properly, as Indy will not be reverting back to the old code.

Try to compile a FMX android application with github sources in Delphi 10.4.2 and you will see that the compiler defines about MEMORY LEAK from fastmm is used in the FMX android compilation and it output an error in the compiler.

That is not an option for me right now (I don't have 10.4.2 installed), which is why I asked you to describe the error you are seeing. Can you please provide specific details?

[macc2010]

But, what is the ACTUAL PROBLEM with that new logic? I see nothing in the new code that should be causing OpenSSL errors. So, can you please be more specific?

No, I have solved the problem by logic, I thought that in Delphi 10.2.3 the code was working well, and first I compare openssl units with old ones, and after that I thought that the communication between server and client works ok except in the part of ReadStream, so I compared this pas file with the old of Delphi 10.2.3 and I saw a lot of changes, so I decided to test with a old version of ReadStream and the problem has gone. I have not stopped more to see what is the problem, Delphi 10.4.2 is giving me a lot of problems, and I have had to make a lot of patchs to it ( none in Indy until now ).

Is there a reason why you are sending/reading the file size manually, instead of letting Write()/ReadStream() handle that for you?

Yes, the reason is that until now, my server application was compiled in Delphi 5 and there is an error in that routines in Delphi 5. My code is doing the same work that you are suggesting me.

That is not an option for me right now (I don't have 10.4.2 installed), which is why I asked you to describe the error you are seeing. Can you please provide specific details?

I can prepare a demo reproducing the problem, but you will need Delphi 10.4.2 installed with last patches ( there is one that adresses a problem with the compiler in non windows platform that ignores the finally statements, and other issues, so these patches are very important ). I think that you will need also an Android 10 device because I was testing with an Android 6 and the problem was more reproducible in Android 10 device.

I do not know if already exists a demo, I think that yes, it would be a server writing a file and a client receiving it, both in threads and the client with BeginWork ... EndWork to show the progress of the download operation.

Thank you.

[macc2010]

I have done a demo application, is this :

Indy.zip

You only need to compile the server application with Delphi 10.4.2 ( win32 ) and put a file named download.zip with whatever you want ( I have done the tests with a file of 70mb ) in the server.exe directory, and compile the cliente application with Delphi 10.4.2 ( android 32 bits ) in a device with Android 10.

In the server, sometimes, the exception : EIdOpenSSLAPISSLError : error:00000000:lib(0):func(0):reason(0) is raised.

But I have not seen this : error:1409E10F:SSL routines:ssl3_write_bytes:bad length in the server, but this one I have seen only once in the client ( Android ).

Thank you.

[macc2010]

I do not know what's the difference with my real application where this error is raised in the server : error:1409E10F:SSL routines:ssl3_write_bytes:bad length, but the demo has in common with my real application that the EIdOpenSSLAPISSLError exception : error:00000000:lib(0):func(0):reason(0) is raised in the server.

Thank you.

[rlebeau]

You only need to compile ... with Delphi 10.4.2

As I told you earlier, I can't do that right now.

Off hand, I don't see anything in your demo code that should be causing any OpenSSL issues. So, if you are getting errors, you are just going to have to debug them yourself for now.

On the other hand, I do see plenty of things in your code that should be cleaned up and/or re-written. For instance, getting rid of the use of TThread.Synchronize() in code that already runs in the main UI thread. Getting rid of ALL calls to Application.ProcessMessages(), they are not needed at all. Getting rid of your InputBuffer handling in the server OnExecute event and letting the call to IOHandler.ReadLn() block normally. Etc.

[macc2010]

Sorry by some dirty code, I wrote it so much quickly, I know that Synchronize must not be called in the main thread, and the Application.Processmessages was inherited from Delphi 10.2.3 in that some controls did not update without a call to it, I agree that InputBuffer handlling before ReadLn is not neccesary.

At this moment, I have not knowledge, nor time to debug this bug, as I told you, Delphi 10.4.2 is giving me a lot of problems, and I cannot stop more in this problem, I know that if I call the ReadStream from Delphi 10.2.3, the problem dissapear, so for now I have the solution, but the problem in Indy is there.

When I will have time, I will try to debug the problem, but I think that it need an expert like you.

Thank you very much.

[macc2010]

FYI, I think that I have not told you that when the server raises the exception, the server close the client connection, and the client stop receiving the file, but the client does not raises any exception. I need to stop the download in the client ( clicking the ButtonStop ) to close the connection.

[WeberAndre]

I think are hit by the same problem like me. Does the first exception happen after 30s or so after the last write? Does your EIdOpenSSLAPISSLError contain an RetCode 10060? The problem at my site is a very slow connection, so I get a write timeout. See also #483

[macc2010]

I think are hit by the same problem like me. Does the first exception happen after 30s or so after the last write? Does your EIdOpenSSLAPISSLError contain an RetCode 10060? The problem at my site is a very slow connection, so I get a write timeout. See also #483

The solution that I give to this problem was to create a copy of the ReadStream of Delphi 10.2.3 in Delphi 10.2.4, in the Android TCP client application.

This is my code :

TCrClientSocket = class( TComponent )
  private
    FIdTCPClient1 : TIdTCPClient;
  ...
  end;

procedure TCrClientSocket.FIdTCPClient1_ReadStream( AStream : TStream; const AByteCount : Int64 );
var
  i: Integer;
  LBuf: TIdBytes;
  LByteCount : Int64;
const
  cSizeUnknown = -1;
begin
  LByteCount := AByteCount;
  // Presize stream if we know the size - this reduces memory/disk allocations to one time
  AStream.Size := LByteCount;
  AStream.Position := 0;
  FIdTCPClient1.IOHandler.BeginWork(wmRead, LByteCount);
  try
    // If data already exists in the buffer, write it out first.
    // should this loop for all data in buffer up to workcount? not just one block?
    if FIdTCPClient1.IOHandler.InputBuffer.Size > 0 then
    begin
      i := IndyMin(FIdTCPClient1.IOHandler.InputBuffer.Size, LByteCount);
      Dec(LByteCount, i);
      if AStream <> nil then
        FIdTCPClient1.IOHandler.InputBuffer.ExtractToStream(AStream, i)
      else
        FIdTCPClient1.IOHandler.InputBuffer.Remove(i);
    end;
    // RLebeau - don't call Connected() here!  ReadBytes() already does that internally. Calling Connected() here can cause an
    // EIdConnClosedGracefully exception that breaks the loop prematurely and thus leave unread bytes in the InputBuffer.
    // Let the loop catch the exception before exiting...
    SetLength(LBuf, FIdTCPClient1.IOHandler.RecvBufferSize); // preallocate the buffer
    repeat
      i := IndyMin(LByteCount, Length(LBuf));
      if i < 1 then
        Break;
      try
        try
          FIdTCPClient1.IOHandler.ReadBytes(LBuf, i, False);
        except
          on E: Exception do
          begin
            // RLebeau - ReadFromSource() inside of ReadBytes() could have filled the InputBuffer with more bytes than actually requested, so don't extract too many bytes here...
            i := IndyMin(i, FIdTCPClient1.IOHandler.InputBuffer.Size);
            FIdTCPClient1.IOHandler.InputBuffer.ExtractToBytes(LBuf, i, False);
            raise;
          end;
        end;
      finally
        if i > 0 then
        begin
          if AStream <> nil then
            TIdStreamHelper.Write(AStream, LBuf, i);
          Dec(LByteCount, i);
        end;
      end;
    until False;
  finally
    FIdTCPClient1.IOHandler.EndWork(wmRead);
    if AStream <> nil then
    begin
      if AStream.Size > AStream.Position then
        AStream.Size := AStream.Position;
    end;
    LBuf := nil;
  end;
end;

And an example of usage is :

    Fs := TFileStream.Create( sFileZip, fmCreate );
    try
      FIdTCPClient1.IOHandler.RecvBufferSize := 1024 * 1024;
      try
        //FIdTCPClient1.IOHandler.ReadStream( Fs, nSize ); // -> do not use in Delphi 10.4.2, a bug?
        FIdTCPClient1_ReadStream( Fs, nSize ); // I will use FIdTCPClient1_ReadStream that is inspired in Delphi 10.2.3 
      finally
        FIdTCPClient1.IOHandler.RecvBufferSize := GRecvBufferSizeDefault;
      end;
    finally
      Fs.Free;
    end;

Try it, it solved me the problem.

I hope that this can solve your problem.

Regards.

[WeberAndre]

Hello, thanks for your response. May be that we have different problems, or your new code ReadStream changes the timing. At my situation I got the EIdOpenSSLAPISSLError durring IoHandler.FlushBuffer, and the next call to a writer operation raises SSL Error "routines:ssl3_write_bytes:bad length" exception. Did you ever look into the EIdOpenSSLAPISSLError what's the value of the RetCode property? May be enlighten what's going on. In my case this was the error code from the windows socket call, which failed inside openssl.