How to use GetContainerLogsAsync

[vbisbest]

I cannot find a good simple example of how to pull logs for a container. I have tried GetContainerLogsAsync but I dont understand the encoding. I looked at MultiplexedStream on AttachContainerAsync but it shows nothing or I am not using it properly.

Is there a simple example on pulling logs for a currently running container?
Docker v2.0.0 on Windows. Visual Studio 2017 using C#.

[zegroz]

Just try it out!

// Default Docker Engine on Windows
[Docker for Windows](https://github.com/microsoft/Docker.DotNet)
using Docker.DotNet;

try
{

    UTF8Encoding utf8 = new UTF8Encoding(false);
    DockerClient client = new DockerClientConfiguration(new Uri("npipe://./pipe/docker_engine")).CreateClient();

    IList<ContainerListResponse> containers = await client?.Containers?.ListContainersAsync(new ContainersListParameters()
    {
        Limit = 10
    });
    if (containers != null && containers.Any())
    {
        var parameters = new ContainerLogsParameters
        {
            ShowStdout = true,
            ShowStderr = true
        };
        foreach (var container in containers)
        {
            if (client != null)
            {
                var logStream = await client?.Containers?.GetContainerLogsAsync(container.ID, parameters, default);
                if (logStream != null)
                {
                    using (var reader = new StreamReader(logStream, utf8))
                    {
                        string log = reader.ReadToEnd();
                    }
                }
            }

        }
}catch (Exception ex)
{
    //throw 
}

[cfauchere]

I am also getting weird encoding. Following steps from @zegroz I get the following:
Found 1 file(s)
/dataset/foo.txt I was not expecting those special characters (SOH DLF and SOH DC1) I am using nuget version 3.125.2 and docker 2.1.0.5 on Windows

[zegroz]

@cfauchere see here: https://github.com/microsoft/Docker.DotNet/issues/359 and https://github.com/microsoft/Docker.DotNet/pull/360

[sjbeskur]

I am struggling with the same issue. @zegroz the links you provided no longer seem available. I am trying to just read the last N lines from a containers log however the api returns a MultiplexStream with seemingly no access to the underlying Stream.

[sjbeskur]

The following solves the last N issue but still no good solution for the unprintable chars. Any assistance would be hugely appreciated.

var logStream = await _client.Containers.GetContainerLogsAsync(containerId, true, new ContainerLogsParameters() { ShowStdout = true, ShowStderr = true, Tail = "10", Timestamps = true });

[Bert1974]

the data I read with GetContainerLogsAsync contains 8 byte headers, with length of the segment in the last. and after that 8 bytes header, the data in, I guess, UTF-8. Somtimes multiples headers+data are read at once.

[rabyjaycie14]

Adding onto this thread, is anyone able to provide a sample for getting the logs from the multiplexed stream, when a tty is not used?

I.e. this method https://github.com/dotnet/Docker.DotNet/blob/49c0dafbf43b98839830c57fcdb2b9cca09aedaa/src/Docker.DotNet/Endpoints/IContainerOperations.cs#L116-L125

Here's what I have so far, but am not even sure where to begin on reading from the stream.

public async Task WriteLogsAsync()
{
    using DockerClientConfiguration clientConfiguration = GetClientConfiguration();

    using DockerClient client = clientConfiguration.CreateClient();

    IList<ContainerListResponse> containers = await client.Containers.ListContainersAsync(
        new ContainersListParameters
        {
            Limit = 10,
        });

    foreach (var container in containers)
    {
        var parameters = new ContainerLogsParameters
        {
            ShowStdout = true,
            ShowStderr = true,
            Timestamps = true,
        };

        MultiplexedStream logs = await client.Containers.GetContainerLogsAsync(container.ID, false, parameters, CancellationToken.None);

        // read logs from stream
    }
}

private static DockerClientConfiguration GetClientConfiguration()
{
    string dockerHost = ConfigurationManager.AppSettings["DockerHost"];

    return string.IsNullOrEmpty(dockerHost) ? new DockerClientConfiguration() : new DockerClientConfiguration(new Uri(dockerHost));
}

TIA

[rabyjaycie14]

Update

Was able to get a working sample, thanks to ClemensOesterle, and their Gist:

• https://gist.github.com/ClemensOesterle/fc7612671c376164b2eef47802cba4c6

Note: there's an issue where some logs are cut off in the middle of a line, but this is the closest working sample I could get for right now

public async Task WriteLogsAsync()
{
    using DockerClientConfiguration clientConfiguration = GetClientConfiguration();

    using DockerClient client = clientConfiguration.CreateClient();

    IList<ContainerListResponse> containers = await client.Containers.ListContainersAsync(
        new ContainersListParameters
        {
            Limit = 10,
        });

    foreach (var container in containers)
    {
        var parameters = new ContainerLogsParameters
        {
            ShowStdout = true,
            ShowStderr = true,
            Tail = "100",
            Timestamps = true,
        };

        MultiplexedStream logStream = await client.Containers.GetContainerLogsAsync(container.ID, false, parameters, CancellationToken.None);

        await ReadOutputAsync(logStream);
    }
}

private static async Task ReadOutputAsync(MultiplexedStream multiplexedStream, CancellationToken cancellationToken = default)
{
    byte[] buffer = System.Buffers.ArrayPool<byte>.Shared.Rent(81920);

    while (true)
    {
        Array.Clear(buffer, 0, buffer.Length);

        MultiplexedStream.ReadResult readResult = await multiplexedStream.ReadOutputAsync(buffer, 0, buffer.Length, cancellationToken);

        if (readResult.EOF)
        {
            break;
        }

        if (readResult.Count > 0)
        {
            var responseLine = Encoding.UTF8.GetString(buffer, 0, readResult.Count);

            Console.WriteLine(responseLine.Trim());
        }
        else
        {
            break;
        }
    }

    System.Buffers.ArrayPool<byte>.Shared.Return(buffer);
}

private static DockerClientConfiguration GetClientConfiguration()
{
    string dockerHost = ConfigurationManager.AppSettings["DockerHost"];

    return string.IsNullOrEmpty(dockerHost) ? new DockerClientConfiguration() : new DockerClientConfiguration(new Uri(dockerHost));
}

[Bert1974]

It are packages which are read.. probably depending on the kind of logger docker uses.. but not found yet the documentation on that.. but in buffer i could find, (not sure what the other bytes in the header are, may be timestamp and more length bytes):

    if  (readResult.Count   >  0)
    {
        var type=buffer[0];
                     var len=(buffer[6] << 8) + buffer[7];
        var  responseLine  =  Encoding.UTF8.GetString(buffer,8, len);

        //followed by the next package in buffer or next read until EOF

    }

On 4/12/2022 5:53 PM, Jaycie Hill wrote:

Update

Was able to get a working sample, thanks to ClemensOesterle https://gist.github.com/ClemensOesterle, and their Gist:

• https://gist.github.com/ClemensOesterle/fc7612671c376164b2eef47802cba4c6

Note: there's an issue where some logs are cut off in the middle of a line, but this is the closest working sample I could get for right now

public async Task WriteLogsAsync() { using DockerClientConfiguration clientConfiguration = GetClientConfiguration();

using DockerClient client = clientConfiguration.CreateClient();

IList containers = await client.Containers.ListContainersAsync( new ContainersListParameters { Limit = 10, });

foreach (var container in containers) { var parameters = new ContainerLogsParameters { ShowStdout = true, ShowStderr = true, Tail = "100", Timestamps = true, };

  MultiplexedStream  logStream  =  await  client.Containers.GetContainerLogsAsync(container.ID,false,parameters,CancellationToken.None);

  await  ReadOutputAsync(logStream);

} }

private static async Task ReadOutputAsync(MultiplexedStream multiplexedStream,CancellationToken cancellationToken = default) { byte[]buffer = System.Buffers.ArrayPool.Shared.Rent(81920);

while (true) { Array.Clear(buffer,0,buffer.Length);

  MultiplexedStream.ReadResult  readResult  =  await  multiplexedStream.ReadOutputAsync(buffer,0,buffer.Length,cancellationToken);

  if  (readResult.EOF)
  {
      break;
  }

  if  (readResult.Count  >  0)
  {
      var  responseLine  =  Encoding.UTF8.GetString(buffer,0,readResult.Count);

      Console.WriteLine(responseLine.Trim());
  }
  else
  {
      break;
  }

}

System.Buffers.ArrayPool.Shared.Return(buffer); }

private static DockerClientConfiguration GetClientConfiguration() { string dockerHost = ConfigurationManager.AppSettings["DockerHost"];

return string.IsNullOrEmpty(dockerHost)? new DockerClientConfiguration(): new DockerClientConfiguration(new Uri(dockerHost)); }

— Reply to this email directly, view it on GitHub https://github.com/dotnet/Docker.DotNet/issues/379#issuecomment-1096904814, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCUHTOYSYWFH6DIXNS5XJLVEWL6RANCNFSM4G5NTKDQ. You are receiving this because you commented.Message ID: @.***>

[Carl-Hellberg-Eriksson]

Update

Was able to get a working sample, thanks to ClemensOesterle, and their Gist:

• https://gist.github.com/ClemensOesterle/fc7612671c376164b2eef47802cba4c6

Note: there's an issue where some logs are cut off in the middle of a line, but this is the closest working sample I could get for right now

public async Task WriteLogsAsync()
{
  using DockerClientConfiguration clientConfiguration = GetClientConfiguration();

  using DockerClient client = clientConfiguration.CreateClient();

  IList<ContainerListResponse> containers = await client.Containers.ListContainersAsync(
      new ContainersListParameters
      {
          Limit = 10,
      });

  foreach (var container in containers)
  {
      var parameters = new ContainerLogsParameters
      {
          ShowStdout = true,
          ShowStderr = true,
          Tail = "100",
          Timestamps = true,
      };

      MultiplexedStream logStream = await client.Containers.GetContainerLogsAsync(container.ID, false, parameters, CancellationToken.None);

      await ReadOutputAsync(logStream);
  }
}

private static async Task ReadOutputAsync(MultiplexedStream multiplexedStream, CancellationToken cancellationToken = default)
{
  byte[] buffer = System.Buffers.ArrayPool<byte>.Shared.Rent(81920);

  while (true)
  {
      Array.Clear(buffer, 0, buffer.Length);

      MultiplexedStream.ReadResult readResult = await multiplexedStream.ReadOutputAsync(buffer, 0, buffer.Length, cancellationToken);

      if (readResult.EOF)
      {
          break;
      }

      if (readResult.Count > 0)
      {
          var responseLine = Encoding.UTF8.GetString(buffer, 0, readResult.Count);

          Console.WriteLine(responseLine.Trim());
      }
      else
      {
          break;
      }
  }

  System.Buffers.ArrayPool<byte>.Shared.Return(buffer);
}

private static DockerClientConfiguration GetClientConfiguration()
{
  string dockerHost = ConfigurationManager.AppSettings["DockerHost"];

  return string.IsNullOrEmpty(dockerHost) ? new DockerClientConfiguration() : new DockerClientConfiguration(new Uri(dockerHost));
}

The issue with the log randomly cut of might be because you are using "WriteLine" instead of "Write".