LG-265: Add custom access log fields to JSON logs and logstashCollector

[donbourne]

There are different ways that this could be done:

• use the logFormat from the httpAccessLogging or accessLogging element
• provide an explicit configuration for logging to indicate the fields desired from accessLogging

The first way seems preferable since having access logging related config in the logging element would be odd.

One design consideration is that there can be multiple httpAccessLogging and accessLogging elements in a server, thus multiple logFormats. Should the fields used in JSON logging / logstashCollector for access logs be different depending on which http endpoint the request comes from? For backward compatibility we probably also want any fields specified in the logFormat to be in addition to the fields already collected by JSON logging / logstashCollector from access logs.

Related RFEs: http://www.ibm.com/developerworks/rfe/execute?use_case=viewChangeRequest&CR_ID=102441 http://www.ibm.com/developerworks/rfe/execute?use_case=viewChangeRequest&CR_ID=98106

UFO: https://ibm.box.com/s/6hic7jrc9wx53f9o363tict4qeooi9ra

[Yushan-Lin]

Issues: UFO: #10616 Implementation: #10617 ID: #10618

[Yushan-Lin]

UFO: https://ibm.box.com/s/6hic7jrc9wx53f9o363tict4qeooi9ra

[atosak]

---

List of Steps to complete or get approvals / sign-offs for Onboarding to the Liberty release (GM date)

Instructions:

• Do the actions below and mark them complete in the checklist when they are done.
• Make sure all feature readiness approvers put the appropriate tag on the epic to indicate their approval.

---

TARGET COMPLETION DATE Before Development Starts or 8 weeks before Onboarding

• [ ] POC Design / WAD Review Scheduled (David Chang) or N/A.

• [ ] POC Design / WAD Reviewed (Feature Owner) or N/A.

• [ ] Complete any follow-ons from the POC Review.

• [ ] Design / WAD Approval (Alasdair Nottingham) or N/A.

• [ ] No Design / No WAD Approval (Arthur De Magalhaes - cloud / Alasdair Nottingham - server) or N/A.

• [ ] SVT Requirements identified. (Epic owner / Feature owner with SVT focal point)

• [ ] ID Requirements identified. (Epic owner / Feature owner with ID focal point)

• [ ] Create a child task of the epic entitled "FAT Approval Test Summary". Add and fill in the template as described here: https://github.ibm.com/was-liberty/WS-CD-Open/wiki/Feature-Review-(Feature-Test-Summary-Process)

• ---

  TARGET COMPLETION DATE 3 weeks before Onboarding

• [x] Identify all open source libraries that are changing or are new. Work with Legal Release Services (Cass Tucker or Release PM) to get open source cleared and approved. Or N/A. (Epic Owner). New or changed open source impacts license and Certificate of Originality.

• ---

  TARGET COMPLETION DATE 3 weeks before Onboarding

• [x] All new or changed PII messages are checked into the integration branch, before the last translation shipment out. (Epic Owner)

• ---

  TARGET COMPLETION DATE 2 weeks before Onboarding

• [x] Implementation complete. (Epic owner / Feature owner)

• [x] All function tests complete. Ready for FAT Approval. (Epic owner / Feature owner)

• [ ] Review all known issues for Stop Ship. (Epic owner / Feature owner / PM)

• ---

  APPROVALS with TARGET COMPLETION DATE 2 to 1 week before Onboarding

  Prereq: You must have the Design Approved or No Design Approved label on the GitHub Epic.

• [x] Accessibility - (G Scott Johnston). Accessibility testing is complete or N/A. Approver adds label focalApproved:accessibility to the Epic in Github.

• [x] FAT Liberty SOE - (Kevin Smith). SOE FATS are running successfully or N/A . Approver adds label focalApproved:fat to the Epic in Github.

• [x] Globalization (Sam Wong - Liberty / Simy Cheeran - tWAS). Translation is complete or N/A. TVT - complete or N/A. Approver adds label focalApproved:globalization to the Epic in Github.

• [x] ID - (Kareen Deen). Documentation work is complete or N/A . Approver adds label focalApproved:id to the Epic in Github.

• [x] Performance - (Jared Anderson). Performance testing is complete with no high severity defects or N/A . Approver adds label focalApproved:performance to the Epic in Github.

• [x] Serviceability - (Don Bourne). Serviceability has been addressed.

• [x] STE - (Swati Kasundra). STE chart deck is complete or N/A . Approver adds label focalApproved:ste to the Epic in Github.

• [x] SVT - (Greg Ecock - Cloud, Brian Hanczaryk- APS). SVT is complete or N/A . Approver adds label focalApproved:svt to the Epic in Github.

• [x] Demo - (Liberty only - Tom Evans or Chuck Bridgham). Demo is scheduled for an upcoming EOI. Approver adds label focalApproved:demo to the Epic in Github.

• ---

  TARGET COMPLETION DATE 1 week before Onboarding

• [ ] No Stop Ship issues for the feature. (Epic owner / Feature owner / Release PM)

• [ ] Ship Readiness Review and Release Notes completed (Epic owner / Feature owner / Release PM)

• [ ] Github Epic and Epic's issues are closed / complete. All PRs are committed to the master branch. (Epic owner / Feature owner / Backlog Subtribe PM)

• ---

  NOT REQUIRED FOR A FEATURE

• [ ] OL Guides - (Yee-Kang Chang). Assessment for OL Guides is complete or N/A.

• [ ] WDT - (Leonard Theivendra). WDT work complete or N/A.

• ---

  Related Deliverables TARGET COMPLETION DATE General Availability

• [ ] Blog article writeup (Epic owner / Feature owner / Laura Cowen)

[gscottj]

This feature helps enable an end user get their log files into a log analysis solution of their own choosing. Therefore, it will be the end user who decides how to get their logs rendered in a user interface. This feature does not prevent users from taking advantage of accessibility features provided by log analysis solutions or which are available in assistive technologies. This feature does not introduce any new user interfaces or significantly modify any existing user interfaces. Accessibility focal approval granted.

[oliver-steinbrecher]

When are you planning to deliver this feature ? :)

[donbourne]

@oliver-steinbrecher , we have a PR being actively worked for it https://github.com/OpenLiberty/open-liberty/pull/11189 . While we can't commit to future delivery dates (or even that something will ever get delivered) I'm hoping we could beta this in 20.0.0.6 timeframe.

If you don't mind sharing -- what's your use case for using this?

[donbourne]

@samwatibm I think this should be beta:20600 -- I'll change the tag... please let me know if you have questions.

[samwatibm]

@donbourne Sorry my mistake; thanks for catching. I was specifically told this was NOT in beta:20500 but tagged it incorrectly anyways!

[jennifer-c]

@chirp1 ID Content has been provided in the ID issue #10618. May I get ID approval? @skasund Do you require the STE deck to be filled out? Please let me know. @gecock SVT is not required for this feature. May I get SVT approval? @tevans78 @cbridgha Demo was done in EOI call for 20.12. May I get demo approval? @jhanders34 We are not expecting a performance hit with this feature. Please assess.

[gecock]

@jennifer-c I am no longer the SVT focal. Please reach out to @hanczaryk.

[jennifer-c]

@hanczaryk SVT is not required for this feature. May I get SVT approval?

[jennifer-c]

@sabolo Feature Test Summary is provided in https://github.com/OpenLiberty/open-liberty/issues/12803

[jennifer-c]

@skasund STE deck is filled out here: https://ibm.box.com/s/suzwawbsx4vt11erlehqlwumdl3ud0l9

[jennifer-c]

@donbourne I have answered the following questions, may I have serviceability approval?

1. WAD -- does the WAD identify the most likely problems customers will see and identify how the feature will enable them to diagnose and solve those problems without resorting to raising a PMR? Have these issues been addressed in the implementation? Yes, the WAD addresses problems customers might encounter. Warning messages were implemented and provides information about issues customers might encounter.

2. Test and Demo -- As part of the serviceability process we're asking feature teams to test and analyze common problem paths for serviceability and demo those problem paths to someone not involved in the development of the feature (eg. L2, test team, or another development team). a) What problem paths were tested and demonstrated?

• User provides an invalid logFormat token
• User provides an invalid value for the jsonAccessLogFields attribute
• User does not provide header name for %o and %i tokens

b) Who did you demo to? Tanya Kulik and Rumana Haque c) Do the people you demo'd to agree that the serviceability of the demonstrated problem scenarios is sufficient to avoid PMRs for any problems customers are likely to encounter, or that L2 should be able to quickly address those problems without need to engage L3? Yes

3. SVT -- SVT team is often the first team to try new features and often encounters problems setting up and using them. Note that we're not expecting SVT to do full serviceability testing -- just to sign-off on the serviceability of the problem paths they encountered. a) Who conducted SVT tests for this feature? Tanya Kulik and Rumana Haque b) Do they agree that the serviceability of the problems they encountered is sufficient to avoid PMRs, or that L2 should be able to quickly address those problems without need to engage L3? Yes

4. Which L2 / L3 queues will handle PMRs for this feature? Ensure they are present in the contact reference file and in the queue contact summary, and that the respective L2/L3 teams know they are supporting it. Ask Don Bourne if you need links or more info. Salesforce: WAS L3: Liberty Log Analytics Retain: LL3ANA, 103

[Yushan-Lin]

@skasund STE slides are uploaded here: https://ibm.ent.box.com/file/691892257869 May I have STE approval?

[chirp1]

Approving. The info in the docs issue looks OK. As I discussed with Yushan, the eGA blog needs enough info so that customers can use the new function. The docs might not be updated by eGA as OL writers are working down a prioritized backlog.

[skasund]

@Yushan-Lin Thanks for the STE slides. I've approved the feature.

[jennifer-c]

@samwatibm Translations have been merged and I have verified that everything is there. May I have globalization approval?