Add logs to a cloudflare zone

simongareste commented 5 years ago

Hi,

I needed to add the logs functionality, so that we can get them from CloudFlare and store them in our premises.

I did this commit, which has a (important) caveat grearding the result. The cloudflare doc states that they do not, for the logs, send back one JSON, but many lines of JSON, so basically something of the like:

{"ClientIP":"1.2.3.4","ClientRequestHost":"fqdn.domain.name","ClientRequestMethod":"GET","ClientRequestURI":"/file1.png"}
{"ClientIP":"2.3.4.5","ClientRequestHost":"fqdn.domain.name","ClientRequestMethod":"GET","ClientRequestURI":"/file1.png"}
{"ClientIP":"4.5.6.7","ClientRequestHost":"fqdn.domain.name","ClientRequestMethod":"GET","ClientRequestURI":"/other_file.png"}

As you can see, each line is a valid JSON, but the result itself is not. So what I did was adding options so that we can force text/plain instead of application/json, and in those case stop the parsing and split it instead. The JSON parsing is then done by the Logs class, as it should. But if we get an error from CloudFlare, I do not know how I can figure it out. Given that in the connection.rb file, you set (in the initializer)

            # Convert HTTP API responses to our own internal response class:
            super(url, headers: headers, accept: 'application/json', **options) do |response|
                Response.new(response.request.url, response.body, **options)
            end

Because of this, the response.code (which would help) is not accessible. But changing the Response initializer signature is IMO an important change. I'm uncertain on which way would be best to continue, so your input is very welcom.