Cleaned up the aws cloud agent log collection list so that the list contains only the file paths actually relevant to the specific verdi worker. Prior to this change, every verdi worker aws cloud agent was looking through ~40 file paths and incurred up to 200% CPU utilization.
Proposed Changes
Parameterized ASG launch template user data creation so that it only lists files paths relevant to the verdi worker type.
Issues
993
Testing
Ran all SCIFLO and most query and download jobs. Verified that all logs were being collected into cloudwatch as before this change.
DISP-S1 SCIFLO jobs seems to run faster which is expected. Small sample size of 3 suggests 20% improvement in performance but will need to do a larger, more controlled study. Performing such study now.
Purpose
Proposed Changes
Issues
993
Testing