GSA-TTS / cg-logshipper

Sample code for draining Cloud Foundry logs to S3 and New Relic using FluentBit
Other
6 stars 2 forks source link

Update .profile to find tags instead of specific service names #11

Closed asteel-gsa closed 9 months ago

asteel-gsa commented 9 months ago

Following up from PR #29 on the 18F terraform modules, which add tag support to the s3 bucket (and other modules), the following changes are proposed so that we can define a specific tag to get the values in the profile, instead of the name of the service.

This allows us to define a specific tag for the three required resources and keep the names of the services unique, while keeping the same functionality. Ex: logshipper-s3 logshipper-creds & newrelic-creds

The usecase for this proposal, is that we already have a service name newrelic-creds bound to our running application, and we cannot have 2 services sharing the same name. The terraform was already updated to add the credential NEW_RELIC_LOGS_ENDPOINT, so I left the tag generic to tag it as newrelic. The newrelic-creds that a consumer of the cg-logshipper may use by default does not implicitly mean it is used for logshipper. The same logic would be applied for an s3 module, where the s3 bucket may be tagged with s3 & logshipper-s3, but since the s3 is dedicated to the consumption of the logs, rather than newrelic-creds service (which could be used globally) this seemed like the logical approach.

vcap@04f6127a-7e6a-4c5a-7b60-5f2b:~$ echo "$VCAP_SERVICES" | jq --raw-output '."user-provided" | .[] | select(.tags[] | contains("newrelic-creds"))  | .credentials.NEW_RELIC_LICENSE_KEY'
b*************
vcap@04f6127a-7e6a-4c5a-7b60-5f2b:~$ echo "$VCAP_SERVICES" | jq --raw-output '."user-provided" | .[] | select(.tags[] | contains("newrelic-creds")) | .credentials.NEW_RELIC_LOGS_ENDPOINT'
https://gov-log-api.newrelic.com/log/v1
vcap@04f6127a-7e6a-4c5a-7b60-5f2b:~$ echo "$VCAP_SERVICES" | jq --raw-output '."s3" | .[] | select(.tags[] | contains("logshipper-s3")) | .credentials.bucket'
c*************
vcap@04f6127a-7e6a-4c5a-7b60-5f2b:~$ echo "$VCAP_SERVICES" | jq --raw-output '."s3" | .[] | select(.tags[] | contains("logshipper-s3")) | .credentials.region'
us-gov-west-1
vcap@04f6127a-7e6a-4c5a-7b60-5f2b:~$ echo "$VCAP_SERVICES" | jq --raw-output '."s3" | .[] | select(.tags[] | contains("logshipper-s3")) | .credentials.access_key_id'
A*************
vcap@04f6127a-7e6a-4c5a-7b60-5f2b:~$ echo "$VCAP_SERVICES" | jq --raw-output '."s3" | .[] | select(.tags[] | contains("logshipper-s3")) | .credentials.secret_access_key'
ns*************
vcap@04f6127a-7e6a-4c5a-7b60-5f2b:~$ echo "$VCAP_SERVICES" | jq --raw-output '."user-provided" | .[] | select(.tags[] | contains("logshipper-creds")) | .credentials.HTTP_USER'
56*************
vcap@04f6127a-7e6a-4c5a-7b60-5f2b:~$ echo "$VCAP_SERVICES" | jq --raw-output '."user-provided" | .[] | select(.tags[] | contains("logshipper-creds")) | .credentials.HTTP_PASS'
6**********

I am waiting for the FAC's preview environment to be free, so I can validate that tags are applied to the s3 resource via the 18F module, but will post those results when I can and move out of draft

asteel-gsa commented 9 months ago

@akf Since this is coming from a fork, I am unable to add a reviewer, but this is largely set.

I did change to unique names, since when having all of them use logshipper it will get 2 values for the ."user-provided" services, so making them unique. This should (for now) remove the dependency on the service-name and a future enhancement can modify the designated tags, or enhance the jq to look at both ."user-provided" creds and filter based on the tags to not return a null value