Remote rsyslog forwarding

[daniew01]

Hi

Can someone please assist me in indicating what role I can use to configure it on the appliance and then also how to add sources to it

Kind Regards DJW

[sygilber]

Hi DJW

Can you share which log types you are interested in to externalize ?

And there are various options.

A) One can use '- role: add_system_alerts_rsyslog' for the Appliance Event log

B) For ISAM PS 'deamon' server/audit logs, it is possible to configure them directly from the ivmgrd.conf/...

    - role: set_runtime_conf
      tags: ["ISAM_PS", "baseline", "ivmgrd.conf"]
      set_runtime_conf_entries:
        - resource_id: "ivmgrd.conf"
          stanza_id: "aznapi-configuration"
          entries: >
            [
            ['logcfg','audit.authn:rsyslog server={{ ifc_rsyslog_server }},port={{ ifc_rsyslog_port_isamaudit }},log_id={{ inventory_hostname.split('.')[0] }}-audit,{{ ifc_rsyslog_logcfg_tunings }}'],
            ['logcfg','audit.azn:rsyslog server={{ ifc_rsyslog_server }},port={{ ifc_rsyslog_port_isamaudit }},log_id={{ inventory_hostname.split('.')[0] }}-audit,{{ ifc_rsyslog_logcfg_tunings }}'],
            ['logcfg','audit.mgmt:rsyslog server={{ ifc_rsyslog_server }},port={{ ifc_rsyslog_port_isamaudit }},log_id={{ inventory_hostname.split('.')[0] }}-audit,{{ ifc_rsyslog_logcfg_tunings }}']
            ]

C) For ISAM RP 'deamon' server/audit/http logs, it is possible to configure them directly from the webseald.conf/...

    - role: update_reverseproxy_conf
      tags: ["ISAM_RP", "baseline", "general"]
      update_reverseproxy_conf_entries:
        - { stanza_id: "logging", entry_id: "server-log-cfg", value_id: "rsyslog server={{ ifc_rsyslog_server }},port={{ ifc_rsyslog_port_isamrpserver }},log_id={{ inventory_hostname.split('.')[0] }}-{{ set_reverseproxy_conf_reverseproxy_id }}" }

    - role: set_reverseproxy_conf
      tags: ["ISAM_RP", "baseline", "general"]
      set_reverseproxy_conf_entries:
        - stanza_id: "aznapi-configuration"
          entries: >
            [
              ['logcfg','audit.authn:rsyslog server={{ ifc_rsyslog_server }},port={{ ifc_rsyslog_port_isamaudit }},log_id={{ inventory_hostname.split('.')[0] }}-{{ set_reverseproxy_conf_reverseproxy_id }}-audit,{{ ifc_rsyslog_logcfg_tunings }}'],
              ['logcfg','http.clf:rsyslog server={{ ifc_rsyslog_server }},port={{ ifc_rsyslog_port_isamrphttp }},log_id={{ inventory_hostname.split('.')[0] }}-{{ set_reverseproxy_conf_reverseproxy_id }}-http,{{ ifc_rsyslog_logcfg_tunings }}']
            ]

But some may use the rsyslog forwarder for ISAM PS/RP 'deamon' logs but we have not opted for that simply because ISAM 'deamons" are rsyslog-capable already.

D) For AAC related logs, the rsyslog forwader is handy (and the only mean of doing it)

    - role: set_rsyslog_forwarder
      tags: ["ISAM_RTE", "rsyslog", "forwarder"]
    - role: set_rsyslog_forwarder_sources
      tags: ["ISAM_RTE", "rsyslog", "forwarder", "sources"]

with some inventory as follow:

set_rsyslog_forwarder_server: "{{ ifc_rsyslog_server }}"
set_rsyslog_forwarder_protocol: "udp"

set_rsyslog_forwarder :
  - server: "{{ set_rsyslog_forwarder_server }}"
    port: "{{ ifc_rsyslog_port_isamrteserver }}"
    protocol: "{{ set_rsyslog_forwarder_protocol }}"
  - server: "{{ set_rsyslog_forwarder_server }}"
    port: "{{ ifc_rsyslog_port_isamrtetrace }}"
    protocol: "{{ set_rsyslog_forwarder_protocol }}"

set_rsyslog_forwarder_sources :
  - server: "{{ set_rsyslog_forwarder_server }}"
    port: "{{ ifc_rsyslog_port_isamrteserver }}"
    protocol: "{{ set_rsyslog_forwarder_protocol }}"
    name: "Runtime Messages"
    tag: "ISAM_RTE_MSG"
    facility: "syslog"
    severity: "debug"
  - server: "{{ set_rsyslog_forwarder_server }}"
    port: "{{ ifc_rsyslog_port_isamrtetrace }}"
    protocol: "{{ set_rsyslog_forwarder_protocol }}"
    name: "Runtime Trace"
    tag: "ISAM_RTE_TRC"
    facility: "syslog"
    severity: "debug"

But I just realize that for the above 2 roles (set_rsyslog_forwarder/set_rsyslog_forwarder_sources) we kinda forgotten to pull request them for sharing. Will do so today if time allows.

Let me know if you need more info.

[daniew01]

WOW ... so many options ... I would need to look at all :-)

I am trying to configure the Monitor -> Analysis and Diagnostics -> Remote Syslog Forwarding config. To be honest I am not sure which this relates to your comments.

I would need to look at what you suggested

Kind Regards Danie (DJW)

[daniew01]

Hi

This I gave done already : role: add_system_alerts_rsyslog

[daniew01]

Hi

The option C) For ISAM RP 'deamon' server/audit/http logs, it is possible to configure them directly from the webseald.conf/... is an interresting one.

We moved away from this option to log from the RP config to a syslog and just enabled normal RP logging to log to the local files and then via my inital question "Remote Syslog Forwarding" we forward the request / msg_ files to a syslog.

We have gone this route after consultation with IBM resources as there is no best practises way of configuring Appliance base/PDACLd/RP for syslog. The challenge with this is that the different components log sin different formats and different times, so getting the golden thread to see what transactions flows through the system and/or components becomes very difficult.

I think the your option D is actually what I am refering to ... I will check

Do you perhaps have got the best practises / ideas for configuring syslog for all the different components?

[sygilber]

Hi again

You are bringing an interesting point about what method to choose from.

First, yes, most logs have different log format. ISAM 'deamons' (PS,RP, ACLD) tend to have similar log format. Appliance Event log is some other format. AAC Runtime/Audit are running inside Liberty Runtime so this is "WebSphere" server logs format .... AAC Audit is some other XML format. So I would say analyse your requirements: support, auditing, compliance, and get the logs out of that Appliance that you need. Now, no you can't route them all onto the same rsyslog port so one can route them differently on distinct ports dedicated for each "isam" log type.

Note that not all logs sent by the Appliance insert the originating IP in the event message so sorting everything out can be messy: Appliance Event Log.

I will be at IBM Think next week. We can discuss about it, and I will see how I can address this item in upcoming presentation next week.

https://myibm.ibm.com/events/think/all-sessions/session/2358A

[daniew01]

Thanx

I will investigate the different distinct port an option Will wait for an update on the role you mention - if time permits

Have a nice Trip

DJW

[daniew01]

Hi According to the ISAM documentation the Monitor->Audit Configurations is for AAC/FED audits and this can also be configured to a remote syslog.

The PS|RP|ACLD you need to configure by updating the [anzapi] stanza to point to remote rsyslog

Appliance adit events is forwarded to remote syslog by updating the system alerts config

All have different formats. as per previous comments in this trail

Please clarify this for me,

I can opt to configure each component [appliance base|PS|RP|ACLD|FED|AAC] on its own to forward to remote syslog? With this option there is more config items/steps to follow but having all the audit logs pointing to different ports, this will be a much cleaner. I also do not need to use the Monitor->Remote Syslog Forwarding config

OR

I can configure all Components to log to local files an then use Monitor->Remote Syslfog Forwarding to forward all to a remote syslog. The problem here is. I needs to look at the complexity to un-clutter all these logs on the remote systems and try to distinguish between operational and audit logging.

Last question : will all the different configs make use of the same SSL store and certificates as TLS delivery would be the best?

[ram-ibm]

Writing logs to disk and then using the remote syslog forwarder ensures two things:

1. The writing to disk removes pressure from real time transactions.
2. Remote syslog forwarder can forward using tcp/udp or tls over tcp - many more options than other syslog options.

Looks like you are already exploring this option of writing to disk and using remote syslog forwarder. You can tag each log forwarder you configure - so that on the receiving syslog end you can separate out the logs.

Simply re-use the same SSL keystore and certs to ensure it is the same everywhere. You could use different ones if you desire.

[sygilber]

Thank you Ram for merging your own provided code to us !

[sygilber]

This issue could now be closed

[daniew01]

Hi Ram So what is the best way? Best Practices if you may?

I tried both - Both is working or can I take it from your suggestion earlier - writing to local disk and then to remote sysylog is the better preferred option ?

DJW

[ram-ibm]

I would recommend using the remote syslog forwarder. Has more options and does not impact the real time processing of a request.