Logs masking support

[shashank-44]

Hello Team,

I am trying to externalize service logs. I am looking for a way to mask the logs before pushing them to an external server. Since the Syslog appender does not support encoder, is there a way to perform masking on the logs?

I will be happy to provide any other details as required. Thanks & Regards, Shashank

[davidelang]

look at mmanon to mask IP addresses, what else are you trying to modify in them?

David Lang

On Wed, 23 Aug 2023, shashank-44 wrote:

Hello Team,

I am trying to externalize service logs. I am looking for a way to mask the logs before pushing them to an external server. Since the Syslog appender does not support encoder, is there a way to perform masking on the logs?

I will be happy to provide any other details as required. Thanks & Regards, Shashank

[shashank-44]

Hello David, Thanks for the prompt response! Apologies for the late reply.

I am trying to mask specific fields in the request/response paylods sent into logs. The solution currently implemented is making using a regex.

The issue that I understood is rsyslog works with the syslog appender which doesn't support an encoder for masking. Any help on this will be appreciated.

Thanks & Regards, Shashank

[davidelang]

I think that what you want is to write a message modification module that will do the masking that you are looking for.

There is mmexternal that lets you invoke an external program to do the modifications

see https://www.rsyslog.com/doc/master/configuration/modules/mmexternal.html https://github.com/rsyslog/rsyslog/blob/master/plugins/external/INTERFACE.md

@rainer, I don't see mmexternal handling batched messages, is that correct? If it's not correct, I think we need a little more documentation explaining it (and I think that the explination at the bottom of INTERFACE.md should be added to the mmexternal.html page if it really is as simple as 'pass it json, get back json listing what needs to be modified' on a 1:1 basis

David Lang

On Sun, 27 Aug 2023, shashank-44 wrote:

Hello David, Thanks for the prompt response! Apologies for the late reply.

I am trying to mask specific fields in the request/response paylods sent into logs. The solution currently implemented is making using a regex.

The issue that I understood is rsyslog works with the syslog appended which doesn't support an encoder for masking. Any help on this will be appreciated.

Thanks & Regards, Shashank

[Rainer]

Hello David,

thank you for including me in this conversation. I haven't used that software in a long time, so I really don't know anymore about its issues.

Kind regards,

Rainer

On Sun, Aug 27, 2023 at 3:06 PM David Lang @.***> wrote:

I think that what you want is to write a message modification module that will do the masking that you are looking for.

There is mmexternal that lets you invoke an external program to do the modifications

see https://www.rsyslog.com/doc/master/configuration/modules/mmexternal.html

https://github.com/rsyslog/rsyslog/blob/master/plugins/external/INTERFACE.md

@rainer, I don't see mmexternal handling batched messages, is that correct? If it's not correct, I think we need a little more documentation explaining it (and I think that the explination at the bottom of INTERFACE.md should be added to the mmexternal.html page if it really is as simple as 'pass it json, get back json listing what needs to be modified' on a 1:1 basis

David Lang

On Sun, 27 Aug 2023, shashank-44 wrote:

Hello David, Thanks for the prompt response! Apologies for the late reply.

I am trying to mask specific fields in the request/response paylods sent into logs. The solution currently implemented is making using a regex.

The issue that I understood is rsyslog works with the syslog appended which doesn't support an encoder for masking. Any help on this will be appreciated.

Thanks & Regards, Shashank

— Reply to this email directly, view it on GitHub https://github.com/rsyslog/rsyslog-doc/issues/1015#issuecomment-1694664733, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAHA5SA4JJT5HEMPIERMRLXXNA6PANCNFSM6AAAAAA33JQCCM . You are receiving this because you were mentioned.Message ID: @.***>

--

---

Rainer Wolf

Dipl.-Informatiker FHBachgrabenweg 7 63517 Rodenbach Tel +49 (0)177 / 2 95 83 10 Email @.***

---

[rgerhards]

I don't see mmexternal handling batched messages, is that correct?

It does. It uses the old-style action interface, but rsyslog core knows that (it queries when loading the module) and so does the (new style) batch handling transparent for the module.

Note that even the old-style interface did handle batches. We just changed the way they are processed to make the flow more intuitive in some border cases.

The doc is talking about batches internally in mmexternal. It has the capability to optionally feed multiple messages into the external program. That is a further optimization.

If it's not correct, I think we need a little more documentation explaining it (and I think that the explination at the bottom of INTERFACE.md should be added to the mmexternal.html page if it really is as simple as 'pass it json, get back '> json listing what needs to be modified' on a 1:1 basis

Actually, mmexternal and omprog received notable extended functionality due to contributions. I have not worked on it for quite a long time. Among others, this means I need to read the full doc and source myself to create that doc. So it's not high on my priority list, but I'll see what I can do.

Rainer Gerhards

El dom, 27 ago 2023 a las 15:06, David Lang @.***>) escribió:

I think that what you want is to write a message modification module that will do the masking that you are looking for.

There is mmexternal that lets you invoke an external program to do the modifications

see https://www.rsyslog.com/doc/master/configuration/modules/mmexternal.html https://github.com/rsyslog/rsyslog/blob/master/plugins/external/INTERFACE.md

@rainer, I don't see mmexternal handling batched messages, is that correct? If it's not correct, I think we need a little more documentation explaining it (and I think that the explination at the bottom of INTERFACE.md should be added to the mmexternal.html page if it really is as simple as 'pass it json, get back json listing what needs to be modified' on a 1:1 basis

David Lang

On Sun, 27 Aug 2023, shashank-44 wrote:

Hello David, Thanks for the prompt response! Apologies for the late reply.

I am trying to mask specific fields in the request/response paylods sent into logs. The solution currently implemented is making using a regex.

The issue that I understood is rsyslog works with the syslog appended which doesn't support an encoder for masking. Any help on this will be appreciated.

Thanks & Regards, Shashank

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>