search

herrjulz / aviator

Merge YAML/JSON files in a in a convenient fashion based on a configuration file called aviator.yml
MIT License
59 stars 7 forks source link

Possible Bug: merge is overwriting instead of appending #50

Open HackXIt opened 1 year ago

HackXIt commented 1 year ago

Hello,

I'm using aviator (version 1.9.0) to create a complex, or rather lengthy, pipeline. The pipeline consists of over a dozen resources, and it would be hard to maintain or add to the pipeline if created by hand.

My aviator file is supposed to create an all_resources.yml, but it's not working as intended. The file contains the full list of resources with their respective repo-names and repo-urls, and additionally it also contains a job, where all those resources are fetched with get as input to a compilation task. I managed to get the part of the resources: section working and made a really identical part for the plan: section, but SOMEHOW, the - get lines overwrite each other, which ends up with only the last resource in the final file.

I'm really at a dead end here, because I can't see anything I'm doing wrong. Can you help me out here?

This is my aviator.yml file:

spruce:
  # First we generate all resources to use in the pipeline
  - base: factory/meta-resource-stub.yml
    prune:
      - meta
    for_each:
      in: meta-resources/
      except:
        - example-resource.yml
      regexp: ".*.(yml)"
    to_dir: temp/resources/
  # Merge all resources into one file
  - base: factory/all-resources-stub.yml
    merge:
      - with_all_in: temp/resources/
        regexp: ".*.(yml)"
    to: temp/all-resources.yml

And these are the stubs: meta-resource.stub.yml


---
jobs:
  - get: (( grab meta.git.repo-name ))
    trigger: (( grab meta.git.trigger ))
    params:
      depth: 1

resources:
  - name: (( grab meta.git.repo-name ))
    type: git
    source:
      uri: (( grab meta.git.http ))
      username: bot-tt
      password: ((sesource.bot-tt-pw))
      branch: (( grab meta.git.branch ))

all-resources-stub.yml

---
jobs:
  -
resources:
  -

When run, it correctly produces the all-resources.yml but inside, it failed to add each resource in the jobs section, instead it overwrites itself, so only the last resource was added. The resources section correctly produces ALL generated resources.

I don't know what's wrong, since the two sections are near-identical And also, I know that the resulting file currently is not correct in terms of concourse syntax, but I reduced it to a minimal example to figure out what's wrong.

HackXIt commented 1 year ago

I think I figured it out.

According to https://github.com/geofffranks/spruce/blob/main/doc/array-merging.md the identifier 'name' is special and in the default behavior of merge, it will merge existing items based on the value of the key.

I would need the ability to use the (( merge in <key> )) function inside aviator, to tell it to merge on the key get, so that I can add multiple inputs.