search

geofffranks / spruce

A BOSH template merge tool
MIT License
426 stars 77 forks source link

Possible bug or advice needed #376

Open vulturm opened 1 year ago

vulturm commented 1 year ago

Hello,

First of all thank you for this tool. I've integrated it in our workflow and I stumbled upon the following issue, possible yaml unmarshal bug.

I've tried various ways such as, not specifying the yaml key:

# cat qwe.yaml
---
fetched_from_secrets: (( awssecret "/sre/non-prod-config" ))

or

---
fetched_from_secrets:
  (( awssecret "/sre/non-prod-config" ))

# spruce merge qwe.yaml
fetched_from_secrets: |-
  options:
    labelsValidations:
      Owner:
      Manufacturer:
        - value1
        - value2
        - value3
        - value4

However, this way the value is presented as multiline string/block, and it can't be further parsed by subsequent (( join "/" fetched_from_secrets.labelsValidations.Manufacturer )) which we would like.

Any suggestion that would allow me to achieve the Expected outcome ?

Thank you in advance!

geofffranks commented 1 year ago

Just making sure I understand the ask - you have yaml formated text in your AWS secret that you would like to retrieve and be able to manipulate with spruce. However when spruce retrieves it, it comes back as a multiline string, rather than a datastructure. Correct?You may be able to modify the awssecret operator to parse this out into a datastructure, but I'm not sure if it will get pulled in soon enough to  be referenced  by all the other operators since there are multiple phases of the merge process.You might be able to use some of the cherry pick flags to isolate the value being returned, save it to a file, and then run a second merge process though. if that doesn't work you could probably send the aws secret merge output to yq -r and then pull that info into a second merge. It's been a while but you may also be able to pull in datastructures from environment variables, so if you're concerned about writing secrets to disk temporarily, you could store them as an env var. there might be code related to parsing that out into datastructures you can reference for the aws operator. Sent from my iPhoneOn Apr 28, 2023, at 11:48 AM, Mihai Vultur @.***> wrote: Hello, First of all thank you for this tool. I've integrated it in our workflow and I stumbled upon the following issue, possible yaml unmarshal bug.

Spruce version: v1.30.2

Description: I would like to store a yaml file in AWS Secrets manager with plaintext option. Then use the awssecret DSL to fetch the content and be interpreted as yaml.

Steps to reproduce: Store the following yaml content in aws secrets manager:

options: labelsValidations: Owner: Manufacturer:

Then use the following file to retrieve it as a YAML structure: cat qwe.yaml

fetched_from_secrets: (( awssecret "/sre/non-prod-config?key=options" ))

Expected outcome:

spruce merge qwe.yaml

fetched_from_secrets: labelsValidations: Owner: Manufacturer:

Actual outcome:

spruce merge qwe.yaml fetched_from_secrets: map[labelsValidations:map[Manufacturer:[value1 value2 value3 value4] Owner:]]

I've tried various ways such as, not specifying the yaml key:

cat qwe.yaml

fetched_from_secrets: (( awssecret "/sre/non-prod-config" ))

or

fetched_from_secrets: (( awssecret "/sre/non-prod-config" ))

spruce merge qwe.yaml

fetched_from_secrets: |- options: labelsValidations: Owner: Manufacturer:

However, this way the value is presented as multiline string/block, and it can't be further parsed by subsequent (( join "/" fetched_from_secrets.labelsValidations.Manufacturer )) which we would like. Any suggestion that would allow me to achieve the Expected outcome ? Thank you in advance!

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>

vulturm commented 1 year ago

Hi Geoff,

Just making sure I understand the ask - you have yaml formated text in your AWS secret that you would like to retrieve and be able to manipulate with spruce. However when spruce retrieves it, it comes back as a multiline string, rather than a datastructure. Correct?

Yes, that is correct. Is that expected or a bug?

You may be able to modify the awssecret operator to parse this out into a datastructure, but I'm not sure if it will get pulled in soon enough to be referenced by all the other operators since there are multiple phases of the merge process.You might be able to use some of the cherry pick flags to isolate the value being returned, save it to a file, and then run a second merge process though. if that doesn't work you could probably send the aws secret merge output to yq -r and then pull that info into a second merge. It's been a while but you may also be able to pull in datastructures from environment variables, so if you're concerned about writing secrets to disk temporarily, you could store them as an env var. there might be code related to parsing that out into datastructures you can reference for the aws operator.

Actually, I was thinking to defer the processing of the rest of the operators until the next run and run spruce in multiple phases. Something like:

fetched_from_secrets: (( awssecret "/sre/non-prod-config?key=options" ))

ManufacturerSlashSeparated: (( defer join "/" fetched_from_secrets.labelsValidations.Manufacturer ))

Thanks!