What Services to Ship Logs From

[cmharlow]

What is the list of logs to capture, from where, in top to bottom priority order. Starting from list gathered here:

• SURI Prod: sul-lyberservices-[test|dev|prod].stanford.edu:/usr/share/tomcat6/webapps/suri2/WEB-INF/suri2.log
• Dor Services App Prod: dor-services-prod.stanford.edu:/opt/app/dor_services/dor-services-app/current/logs
• Dor Workflow Service Prod:
• Stacks Prod: sul-stacks-prod-[a|b|c].stanford.edu:/opt/app/stacks/stacks/current/log/*.log
• Argo Prod:
• Purl Fetcher Prod:
• Dor-Indexing-App:
• Discovery Dispatcher:
• Robot-Master Prod
• Hydrus Prod
• Hydrus Test
• ETDs Prod
• WAS Prod
• (any robots?)

[cmharlow]

Preference for test only sources as well.

[cmharlow]

Include details on proposed filtering & Apache vs application logs, i.e.:

• Lyberservices offers Apache error, access & ssl-access logs, but they are not separated by application. Uncertain the error logs contain much information of value to this project, but worth aggregating.
• For access logs, we can readily filter for application-specific logs by leveraging regex or string matching on the requested endpoint contained in the access log information. Does not appear to be a similar pattern to work off of for the error logs.

[cmharlow]

What I've got so far. Have PR in to get access to dor services machines to be able to fill out ; will also ask tomorrow for SDR-PC access so can fill that in.

This seems like a good place to start in priority of interest + not effecting current log users. @eefahy let me know if this is helpful + even if i can help start integrating these into that Puppet PR work.

Service, Application, or Machine	Apache Logs Location	Application Logs
Lyberservices Test (sul-lyberservices-test.stanford.edu)	Access: /var/log/httpd/access_log[-YYYYMMDD] Error: /var/log/httpd/error_log[-YYYYMMDD]	--
SURI Test (sul-lyberservices-test.stanford.edu)	-- (lyberservices)	/usr/share/tomcat6/webapps/suri2/WEB-INF/suri2.log[.#]
Dor Workflow Service Test (sul-lyberservices-prod.stanford.edu)	-- (lyberservices…??)	?? (on dor staging?)
Workflow Service Test (sul-lyberservices-test.stanford.edu)	-- (lyberservices)	/home/lyberservices/apps/workflow/home/logs/wfservice.log[.#]
SDR Pres Core Test (??)	Access: /var/log/httpd/access_log[-YYYYMMDD] Error: /var/log/httpd/error_log[-YYYYMMDD]	--
SDR Pres Core Dev (??)	--	??
SDR Pres Core Prod (??)	--	??
Lyberservices Dev (sul-lyberservices-dev.stanford.edu)	Access: /var/log/httpd/access_log[-YYYYMMDD] Error: /var/log/httpd/error_log[-YYYYMMDD]	--
SURI Dev (sul-lyberservices-dev.stanford.edu)	-- (lyberservices)	/usr/share/tomcat6/webapps/suri2/WEB-INF/suri2.log
Dor Workflow Service Dev (sul-lyberservices-dev.stanford.edu)	-- (lyberservices…??)	?? (on dor staging?)
Workflow Service Test (sul-lyberservices-test.stanford.edu)	-- (lyberservices)	/home/lyberservices/apps/workflow/home/logs/wfservice.log[.#]
Lyberservices Prod (sul-lyberservices-prod.stanford.edu)	Access: /var/log/httpd/access_log[-YYYYMMDD]/var/log/httpd/ssl_access_log[-YYYYMMDD] Error: /var/log/httpd/error_log[-YYYYMMDD]/var/log/httpd/ssl_error_log[-YYYYMMDD]	--
SURI Prod (sul-lyberservices-prod.stanford.edu)	-- (lyberservices)	/usr/share/tomcat6/webapps/suri2/WEB-INF/suri2.log[.#]
Dor Workflow Service Prod (sul-lyberservices-prod.stanford.edu)	-- (lyberservices)	/home/lyberadmin/dor-services-app/current/log/workflow_service.log[.YYYYMMDD][.#]
Workflow Service Prod (sul-lyberservices-prod.stanford.edu)	-- (lyberservices)	/home/lyberadmin/apps/workflow/home/logs/wfservice.log

[cmharlow]

Mapped here for now, bc needs more work to get this mess straight: https://docs.google.com/spreadsheets/d/1_2L2fxUccPImcjhbjIQEj2VPUy4hZNUthAEKfFGws20/edit#gid=0

[cmharlow]

Need to add the SDR PC Replication sprint (PCC) details here with the current work in #26

[cmharlow]

• Lyberservices Test (machine) Apache Logs (Error & Access)
• Lyberservices Dev (machine) Apache Logs (Error & Access)
• Lyberservices Prod (machine) Apache Logs (Error & Access & SSL_Error & SSL_Access)
• SURI Test (application) Application Logs (on lyberservices machine)
• SURI Dev (application) Application Logs (on lyberservices machine)
• SURI Prod (application) Application Logs (on lyberservices machine)
• Workflow Dev (machine) Apache Logs (Error & Access)
• Workflow Test (machine) Apache Logs (Error & Access)
• Workflow Service Dev (application) Application Logs (on workflow machine)
• Workflow Service Test (application) Application Logs (on workflow machine)
• SDR Preservation Core Test (machine) Apache Logs (Error & Access)
• SDR Preservation Core Dev (machine) Apache Logs (Error & Access)

Pulled from this proof of concept work for now:

• Dor Workflow Service (application) is spread among applications that use the gem, so pulling this off
• SDR Pres Core Prod (machine) Apache Logs

Splitting out ticket for setting up cloudwatch streaming of logs for SDR PCC machines.