::Docker::Image.build_from_dir does not stream when passed a block

[ghost]

Passing a block to build_from_dir does not stream the output, also results in a JSON parse error.

docker-api 1.28.0

Repro

Dir.mktmpdir do |dir|
        dockerfile = <<EOF
FROM debian:jessie
LABEL image.test=true
RUN sleep 5
RUN sleep 5
EOF

  File.write(File.join(dir, 'Dockerfile'), dockerfile)

  ::Docker::Image.build_from_dir(docker_build_path) do |v|
    puts('*** LOG STATEMENT ***')
    puts(v)

    if (log = JSON.parse(v)) && log.has_key?('stream')
      $stdout.puts log['stream']
    end
  end
end

Output:

*** LOG STATEMENT ***
{"stream":"Step 1/4 : FROM debian:jessie\n"}
{"stream":" ---\u003e 40aa6d4339d4\n"}
{"stream":"Step 2/4 : LABEL image.test true\n"}
{"stream":" ---\u003e Using cache\n"}
{"stream":" ---\u003e 2d49e905e1a9\n"}
{"stream":"Step 3/4 : RUN sleep 5\n"}
{"stream":" ---\u003e Running in 70444a679077\n"}
{"stream":" ---\u003e 211887f948b9\n"}
{"stream":"Removing intermediate container 70444a679077\n"}
{"stream":"Step 4/4 : RUN sleep 5\n"}
{"stream":" ---\u003e Running in 19772dfb215c\n"}
{"stream":" ---\u003e 02b54986a8e0\n"}
{"stream":"Removing intermediate container 19772dfb215c\n"}
{"stream":"Successfully built 02b54986a8e0\n"}

First, I expect to see *** LOG STATEMENT *** once per line or on a regular basis.

Second, this JSON is invalid. and I get JSON::ParserError:

[tlunter]

It seems your chunk size is much larger than per line. Try using v.each_line { } to get the per line JSON.

[ghost]

oh! thanks

[ghost]

also, how do I change the chunk size?

[ghost]

so, does the readme need to be updated? cause I started with a copy/paste and it resulted in this error.

[tlunter]

Actually, looking at one of our internal tools, it looks like we do exactly what you're doing and it works fine. I'll pull out the bits of code:

        ...
        ::Docker::Image.build_from_tar(archive, options, @connection) do |line|
          log_build_line(line)
        end
        ...

then:

      def log_build_line(line)
        stream = JSON.parse(line)['stream']
        stream.each_line { |msg| debug "[build] #{msg.chomp}" } if stream
      rescue
        debug "[build] #{line}"
      end

You can reduce the chunk size by creating a new docker connection object with additional parameters:

        ::Docker::Connection.new(
          ::Docker.url,
          ::Docker.env_options.merge(chunk_size: 1)
        )

[ghost]

thank you very much for the quick response!

[hayfever]

I was still running into issues getting a normal looking stream from docker after trying the above, using the yajl-ruby gem got me what I was looking for:

parser = Yajl::Parser.new
parser.on_parse_complete = ->(obj) { print obj['stream'] }

# In context of a rails app, change to your dockerfile dir
image = Docker::Image.build_from_dir(Rails.root.to_s) { |line| parser.parse(line) }

You should see stdout identical to running docker build from your shell.

[rucker]

It looks like log streaming example shown here in the README does still result in the second issue OP reported (the JSON::ParserError) when lines of output contain a newline character (e.g. {"stream":"\n"}).

Here's a small example, build.rb:

#!/usr/local/bin/ruby -w

require 'docker'

Docker::Image.build_from_dir('.') do |v|
  if (log = JSON.parse(v)) && log.has_key?("stream")
    $stdout.puts log["stream"]
  end
end

$ ./build.rb
 (Excon::Error::Socket)s/2.4.0/gems/json-2.1.0/lib/json/common.rb:156:in `parse': 765: unexpected token at '{"stream":"\n"}
' (JSON::ParserError)
        from /usr/local/lib/ruby/gems/2.4.0/gems/json-2.1.0/lib/json/common.rb:156:in `parse'
        from ./build.rb:6:in `block in <main>'
        from /usr/local/lib/ruby/gems/2.4.0/gems/docker-api-1.34.2/lib/docker/image.rb:345:in `block in response_block'
        from /usr/local/lib/ruby/gems/2.4.0/gems/excon-0.62.0/lib/excon/response.rb:119:in `parse'
        from /usr/local/lib/ruby/gems/2.4.0/gems/excon-0.62.0/lib/excon/middlewares/response_parser.rb:7:in `response_call'
        from /usr/local/lib/ruby/gems/2.4.0/gems/docker-api-1.34.2/lib/excon/middlewares/hijack.rb:45:in `response_call'
        from /usr/local/lib/ruby/gems/2.4.0/gems/excon-0.62.0/lib/excon/connection.rb:414:in `response'
        from /usr/local/lib/ruby/gems/2.4.0/gems/excon-0.62.0/lib/excon/connection.rb:263:in `request'
        from /usr/local/lib/ruby/gems/2.4.0/gems/docker-api-1.34.2/lib/docker/connection.rb:40:in `request'
        from /usr/local/lib/ruby/gems/2.4.0/gems/docker-api-1.34.2/lib/docker/connection.rb:65:in `block (2 levels) in <class:Connection>'
        from /usr/local/lib/ruby/gems/2.4.0/gems/docker-api-1.34.2/lib/docker/image.rb:274:in `build_from_tar'
        from /usr/local/lib/ruby/gems/2.4.0/gems/docker-api-1.34.2/lib/docker/image.rb:293:in `build_from_dir'
        from ./build.rb:5:in `<main>'

Explicitly changing my chunk_size to 1 gets me further, but I do fail later on with other output from Docker:

(Excon::Error::Socket)s/2.4.0/gems/json-2.1.0/lib/json/common.rb:156:in `parse': 765: unexpected token at '{"stream":" ---\u003e 8e5bf52abd1a\n"}
{"stream":"Step 7/7 : RUN locale-gen"}
{"stream":"\n"}

The yajl-ruby solution posted above by @hayfever does work beautifully, however.