Octet Counting

[drusellers]

https://tools.ietf.org/html/rfc6587#section-3.4.1

I'm trying to parse syslog messages from Heroku but I think I'm getting an error based on the octet counting aspect of the messages.

https://devcenter.heroku.com/articles/log-drains

Example

83 <40>1 2012-11-30T06:45:29+00:00 host app web.3 - State changed from starting to up
119 <40>1 2012-11-30T06:45:26+00:00 host app web.3 - Starting process with command `bundle exec rackup config.ru -p 24405`

[ottobackwards]

What is the error you are getting?

[ottobackwards]

ok, I see. simple-syslog-5424 ( and simple-syslog which it has been rolled into ) supports parsing the SYSLOG_MSG part ( in the right format ), not the octet counted RFC 6587 prefix. If you want to consume these messages you will have to parse out the size prefix and pass the SYSLOG_MSG part in.

I would have to understand how you are reading the data better to think about what it would take to support it properly. I would want to see a simple code example of how you are using it and reading things.

[ottobackwards]

and how you are getting the data etc

[ottobackwards]

palindromicity.slack.com

there is a channel for it there, if you want to chat it out, maybe tomorrow.

[drusellers]

Error

com.github.palindromicity.syslog.dsl.ParseException: Syntax error @ 1:0 missing '<' at '8'

Code

package io.omc.hindy.resources;

import com.github.palindromicity.syslog.SyslogParser;
import com.github.palindromicity.syslog.SyslogParserBuilder;
import io.omc.hindy.HindyFactory;

import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import java.io.BufferedReader;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;

@Path("/syslog")
@Produces(MediaType.TEXT_PLAIN)
public class SyslogResource extends ApiResource {

    public SyslogResource(HindyFactory factory) {
        super(factory);
    }

    @POST
    public Response post(InputStream stream) throws Throwable {

        List<Map<String,Object>> syslogMapList = new ArrayList<>();
        SyslogParser parser = new SyslogParserBuilder()
                .build();

            try (BufferedReader reader = new BufferedReader(new InputStreamReader(stream))) {
                syslogMapList = parser.parseLines(reader);
            }

        // For LogPlex we must return a 'Content-Length: 0' empty body
        return Response.ok().build();
    }

}

[drusellers]

palindromicity.slack.com

there is a channel for it there, if you want to chat it out, maybe tomorrow.

Couldn't find the signup button. :)

[ottobackwards]

OK, So those syslog lines are not valid 5424, they are missing a '-' for the structured data. if you change the line to

<40>1 2012-11-30T06:45:29+00:00 host app web.3 - - State changed from starting to up then it will parse without the prefix. I have a fix for supporting this, but I'm going to do it in the simple-syslog library, since that library supports different specifications already. Do you have any _real_ examples of these logs?

[ottobackwards]

This issue is addressed, with support for RFC_6587_5424 and RFC_6587_3164 added in the simple-syslog 0.0.4 release Please switch to using that library. https://github.com/palindromicity/simple-syslog

[drusellers]

Do you have any real examples of these logs?

As I get further along, I will come back and give you real data - thank you good sir.

[drusellers]

@ottobackwards

207 <190>1 2020-08-31T13:46:15.962186+00:00 host app web.1 - I, [2020-08-31T13:46:15.962087 #4]  INFO -- : [74cbefb1-a35c-4378-b635-9a81a1fe8c94] Started GET \"/\" for 162.196.139.145 at 2020-08-31 13:46:15 +0000
190 <190>1 2020-08-31T13:46:15.963098+00:00 host app web.1 - I, [2020-08-31T13:46:15.963030 #4]  INFO -- : [74cbefb1-a35c-4378-b635-9a81a1fe8c94] Processing by DashboardController#index as HTML
305 <190>1 2020-08-31T13:46:15.967676+00:00 host app web.1 - D, [2020-08-31T13:46:15.967598 #4] DEBUG -- : [74cbefb1-a35c-4378-b635-9a81a1fe8c94]   \u001b[1m\u001b[36mUser Load (1.4ms)\u001b[0m  \u001b[1m\u001b[34mSELECT \"users\".* FROM \"users\" WHERE \"users\".\"id\" = $1 ORDER BY \"users\".\"id\" ASC LIMIT $2\u001b[0m  [[\"id\", XX], [\"LIMIT\", XX]]
206 <190>1 2020-08-31T13:46:15.995834+00:00 host app web.1 - I, [2020-08-31T13:46:15.995748 #4]  INFO -- : [74cbefb1-a35c-4378-b635-9a81a1fe8c94]   Rendering dashboard/index.html.erb within layouts/application
243 <190>1 2020-08-31T13:46:16.003722+00:00 host app web.1 - I, [2020-08-31T13:46:16.003642 #4]  INFO -- : [74cbefb1-a35c-4378-b635-9a81a1fe8c94]   Rendered dashboard/index.html.erb within layouts/application (Duration: 7.8ms | Allocations: 1974)
217 <190>1 2020-08-31T13:46:16.004925+00:00 host app web.1 - I, [2020-08-31T13:46:16.004845 #4]  INFO -- : [74cbefb1-a35c-4378-b635-9a81a1fe8c94]   Rendered application/_crumbs.html.erb (Duration: 0.0ms | Allocations: 8)
219 <190>1 2020-08-31T13:46:16.005138+00:00 host app web.1 - I, [2020-08-31T13:46:16.005063 #4]  INFO -- : [74cbefb1-a35c-4378-b635-9a81a1fe8c94]   Rendered application/_actions.html.erb (Duration: 0.0ms | Allocations: 14)
225 <190>1 2020-08-31T13:46:16.005555+00:00 host app web.1 - I, [2020-08-31T13:46:16.005485 #4]  INFO -- : [74cbefb1-a35c-4378-b635-9a81a1fe8c94] Completed 200 OK in 42ms (Views: 6.4ms | ActiveRecord: 14.9ms | Allocations: 9020)

I'm happy to move this issue to the simple-syslog repository as well. Thank you for all of your help.

[ottobackwards]

Super, if you try the other library with the new specification settings and it doesn't work, then I think you can open the issue :)

[ottobackwards]

@drusellers just to let you know, that syslog is not valid 5424 and will continue to fail.

octet count	pri	version	timestamp	host	app name	proc id	msg id	structured data	msg
225	<190>	1	2020-08-31T13:46:16.005555+00:00	host	app	web.1	-		I, [2020-08-31T13:46:16.005485 #4] INFO -- : [74cbefb1-a35c-4378-b635-9a81a1fe8c94] Completed 200 OK in 42ms (Views: 6.4ms	ActiveRecord: 14.9ms	Allocations: 9020)

[ottobackwards]

structured data must have a '-' if not present

[ottobackwards]

https://devcenter.heroku.com/articles/log-drains#https-drains

So the feature request in simple-syslog would be to support missing structured data.