search

garden-io / garden

Automation for Kubernetes development and testing. Spin up production-like environments for development, testing, and CI on demand. Use the same configuration and workflows at every step of the process. Speed up your builds and test runs via shared result caching
https://garden.io
Mozilla Public License 2.0
3.32k stars 268 forks source link

Could not parse .gitignore in directory #2516

Open MarkMurphy opened 3 years ago

MarkMurphy commented 3 years ago

Bug

Current Behavior

Attempting to run garden plugins terraform plan-root --env=dev

Could not parse .gitignore in directory /Users/mark/project/analysis/.gitignore as valid YAML: unidentified alias "*~" at line 1, column 4:
    **~
       ^

Expected behavior

Not that.

Reproducible example

The .gitignore in question has this:

**~
*.env
*.docx
.DS_Store

Workaround

unknown

Suggested solution(s)

I'm not entirely sure why you're trying to parse this file as yaml or at all for that matter. I've specified in my project.garden.yml to use only .gardenignore ie.

dotIgnoreFiles:
  - .gardenignore

And in that file I've even tried ignoring the submodule that it's reading that .gitignore from

Your environment

garden version 0.12.24

MarkMurphy commented 3 years ago

This is being picked up in a submodule's submodule if that's worth anything.

I tried deleting the .gitignore file it's choking on and now it's complaining about "Invalid configuration found in README.md" of that nested submodule, which is even more strange.

edvald commented 3 years ago

Not that.

😆 fair enough!

Hmm, this is an odd one.

I honestly have no initial idea on what shenanigans might be going on there, so we might need some more info... Can you share A) how your terraform provider is configured, and B) perhaps run the command with --log-level=silly to see what's going on leading up the error?

MarkMurphy commented 3 years ago

Can you share A) how your terraform provider is configured

environments:
  - name: development
    variables:
      gcp_project: example-project-development
      gcp_region: us-central1
      hostname: ${git.branch}.dev.example.com
      workspace: ${git.branch}

providers:
  - name: terraform
    environments:
      - development
    initRoot: "./terraform/${environment.name}"
    variables:
      gcp_project: ${var.gcp_project}
      gcp_region: ${var.gcp_region}
      hostname: ${var.hostname}
    version: 0.13.3
    workspace: ${var.workspace}

B) perhaps run the command with --log-level=silly to see what's going on leading up the error?

$ garden plugins terraform plan-root --env=development --log-level=silly
Plugins ⚙️ 

Initializing enterprise API client.
Enterprise domain and/or project ID missing. Aborting.
Calling git with args 'rev-parse --abbrev-ref HEAD' in /Users/mark/curv/pose-stack
BufferedEventStream: Connected
Calling git with args 'config --get remote.origin.url' in /Users/mark/curv/pose-stack
Scanning for configs
Scanning for configs in /Users/mark/curv/pose-stack
Scanning project root at /Users/mark/curv/pose-stack
Includes: **/*garden.y*ml
Excludes:.garden/**/*,.git,.gitmodules,.garden/**/*,debug-info*/**
   Calling git with args 'rev-parse --show-toplevel' in /Users/mark/curv/pose-stack
   Calling git with args 'diff-index --name-only HEAD /Users/mark/curv/pose-stack' in /Users/mark/curv/pose-stack
   Calling git with args 'ls-files --ignored --cached --exclude /Users/mark/curv/pose-stack/.garden --exclude-per-directory .gardenignore' in /Users/mark/curv/pose-stack
   Submodules listed at api/api-server (git@github.com:rep-ai/api-server.git), analysis/analysis-server (git@github.com:rep-ai/analysis-server.git), grafana (git@github.com:rep-ai/grafana.git), schemas (git@github.com:rep-ai/schemas.git)
   Calling git with args 'ls-files -s --others --cached --exclude /Users/mark/curv/pose-stack/.garden --exclude-per-directory .gardenignore :(glob)**/*garden.y*ml' in /Users/mark/curv/pose-stack
   Scanning submodule at /Users/mark/curv/pose-stack/api/api-server
Includes: undefined
Excludes:
      Calling git with args 'rev-parse --show-toplevel' in /Users/mark/curv/pose-stack/api/api-server
   Scanning submodule at /Users/mark/curv/pose-stack/analysis/analysis-server
Includes: undefined
Excludes:
      Calling git with args 'rev-parse --show-toplevel' in /Users/mark/curv/pose-stack/analysis/analysis-server
   Scanning submodule at /Users/mark/curv/pose-stack/grafana
Includes: undefined
Excludes:
      Calling git with args 'rev-parse --show-toplevel' in /Users/mark/curv/pose-stack/grafana
   Scanning submodule at /Users/mark/curv/pose-stack/schemas
Includes: undefined
Excludes:
      Calling git with args 'rev-parse --show-toplevel' in /Users/mark/curv/pose-stack/schemas
      Calling git with args 'diff-index --name-only HEAD /Users/mark/curv/pose-stack/api/api-server' in /Users/mark/curv/pose-stack/api/api-server
      Calling git with args 'diff-index --name-only HEAD /Users/mark/curv/pose-stack/analysis/analysis-server' in /Users/mark/curv/pose-stack/analysis/analysis-server
      Calling git with args 'diff-index --name-only HEAD /Users/mark/curv/pose-stack/grafana' in /Users/mark/curv/pose-stack/grafana
      Calling git with args 'diff-index --name-only HEAD /Users/mark/curv/pose-stack/schemas' in /Users/mark/curv/pose-stack/schemas
      Calling git with args 'ls-files --ignored --cached --exclude /Users/mark/curv/pose-stack/.garden --exclude-per-directory .gardenignore' in /Users/mark/curv/pose-stack/api/api-server
      Calling git with args 'ls-files --ignored --cached --exclude /Users/mark/curv/pose-stack/.garden --exclude-per-directory .gardenignore' in /Users/mark/curv/pose-stack/analysis/analysis-server
      Calling git with args 'ls-files --ignored --cached --exclude /Users/mark/curv/pose-stack/.garden --exclude-per-directory .gardenignore' in /Users/mark/curv/pose-stack/grafana
      Calling git with args 'ls-files --ignored --cached --exclude /Users/mark/curv/pose-stack/.garden --exclude-per-directory .gardenignore' in /Users/mark/curv/pose-stack/schemas
      Calling git with args 'ls-files -s --others --cached --exclude /Users/mark/curv/pose-stack/.garden --exclude-per-directory .gardenignore' in /Users/mark/curv/pose-stack/api/api-server
      Calling git with args 'ls-files -s --others --cached --exclude /Users/mark/curv/pose-stack/.garden --exclude-per-directory .gardenignore' in /Users/mark/curv/pose-stack/grafana
      Calling git with args 'ls-files -s --others --cached --exclude /Users/mark/curv/pose-stack/.garden --exclude-per-directory .gardenignore' in /Users/mark/curv/pose-stack/schemas
      Submodules listed at AnalysisServer/schema/assessment/AssessmentSchemas (git@github.com:rep-ai/AssessmentSchemas.git)
      Calling git with args 'ls-files -s --others --cached --exclude /Users/mark/curv/pose-stack/.garden --exclude-per-directory .gardenignore' in /Users/mark/curv/pose-stack/analysis/analysis-server
      Found 1 files in submodule /Users/mark/curv/pose-stack/grafana
      Scanning submodule at /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas
Includes: undefined
Excludes:
         Calling git with args 'rev-parse --show-toplevel' in /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas
         Calling git with args 'diff-index --name-only HEAD /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas' in /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas
      Found 1 files in submodule /Users/mark/curv/pose-stack/api/api-server
         Calling git with args 'ls-files --ignored --cached --exclude /Users/mark/curv/pose-stack/.garden --exclude-per-directory .gardenignore' in /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas
      Found 1 files in submodule /Users/mark/curv/pose-stack/schemas
         Calling git with args 'ls-files -s --others --cached --exclude /Users/mark/curv/pose-stack/.garden --exclude-per-directory .gardenignore' in /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas
         Found 25 files in submodule /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas
      Found 26 files in submodule /Users/mark/curv/pose-stack/analysis/analysis-server
   Found 36 files in project root /Users/mark/curv/pose-stack
Load module and workflow configs from /Users/mark/curv/pose-stack/project.garden.yml
Load module and workflow configs from /Users/mark/curv/pose-stack/analytics/garden.yml
Load module and workflow configs from /Users/mark/curv/pose-stack/broker/garden.yml
Load module and workflow configs from /Users/mark/curv/pose-stack/db/garden.yml
Load module and workflow configs from /Users/mark/curv/pose-stack/faktory/garden.yml
Load module and workflow configs from /Users/mark/curv/pose-stack/prometheus/garden.yml
Load module and workflow configs from /Users/mark/curv/pose-stack/storage/garden.yml
Load module and workflow configs from /Users/mark/curv/pose-stack/grafana/garden.yml
Load module and workflow configs from /Users/mark/curv/pose-stack/api/api-server/garden.yml
Load module and workflow configs from /Users/mark/curv/pose-stack/schemas/garden.yml
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/garden.yml
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/.gitignore
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/README.md
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/schemas/agility.json
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/schemas/ballistic_pushup.json
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/schemas/body_scan.json
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/schemas/broad_jump.json
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/schemas/curv_freeform.json
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/schemas/gait_posterior.json
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/schemas/gait_sagittal.json
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/schemas/glute_bridge.json
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/schemas/l_drill.json
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/schemas/ninety_ninety_straight_leg_raise.json
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/schemas/repeated_max_vert_jump.json
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/schemas/sample.json
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/schemas/shoulder_ab_ad.json
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/schemas/shoulder_flex_ext.json
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/schemas/shoulder_int_ext.json
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/schemas/sit_to_stand.json
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/schemas/ten_yard_split.json
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/schemas/two_leg_squat_front.json
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/schemas/two_leg_squat_side.json
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/schemas/vert_jump.json
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/scripts/change_model.py
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/scripts/convert_buffer_frames_to_time.py
Load module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/scripts/update_staging.py
Loaded module and workflow configs from /Users/mark/curv/pose-stack/analytics/garden.yml
Loaded module and workflow configs from /Users/mark/curv/pose-stack/broker/garden.yml
Loaded module and workflow configs from /Users/mark/curv/pose-stack/db/garden.yml
Loaded module and workflow configs from /Users/mark/curv/pose-stack/faktory/garden.yml
Loaded module and workflow configs from /Users/mark/curv/pose-stack/storage/garden.yml
Loaded module and workflow configs from /Users/mark/curv/pose-stack/prometheus/garden.yml
Loaded module and workflow configs from /Users/mark/curv/pose-stack/grafana/garden.yml
Loaded module and workflow configs from /Users/mark/curv/pose-stack/api/api-server/garden.yml
Loaded module and workflow configs from /Users/mark/curv/pose-stack/schemas/garden.yml
Loaded module and workflow configs from /Users/mark/curv/pose-stack/analysis/analysis-server/garden.yml
Loaded module and workflow configs from /Users/mark/curv/pose-stack/project.garden.yml
Done flushing all events and log entries.
Done flushing all events and log entries.
Could not parse .gitignore in directory /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/.gitignore as valid YAML: unidentified alias "*~" at line 1, column 4:
    **~
       ^
Could not parse .gitignore in directory /Users/mark/curv/pose-stack/analysis/analysis-server/AnalysisServer/schema/assessment/AssessmentSchemas/.gitignore as valid YAML: unidentified alias "*~" at line 1, column 4:
    **~
       ^

See error.log for detailed error message

This issue seems to go away when I remove the offending submodule's entry from the .gitmodules file.

MarkMurphy commented 3 years ago

The same thing happens running garden validate so it's definetly not terraform or provider related

MarkMurphy commented 3 years ago

ok so, the issue seems to be related to nested submodules. It doesn't appear that any form of .ignore or module.exclude applies to anything contained within a nested submodule.

The hierarchy of the project looks something like this:

project/
├─ project.garden.yaml
├─ submodule-1a/
│  ├─ submodule-2a/
│  │  ├─ submodule-3a/
├─ submodule-1b/
├─ submodule-1c/

Configuring .gardenignore at the project root level to ignore submodule/**/* seems to have no effect. Similarly, placing a .gardenignore in a submodule works at that level but will not allow me to ignore anything in the nested submodules.

I've discovered I can work around this by placing a .gardenignore in the submodules with **/* as its contents.

However, I feel like I'm just addressing the symptoms of a larger issue i.e. why would garden try to parse a .gitignore as yaml or a README.md for that matter. 😵 Something very strange going on there.

edvald commented 3 years ago

Yeah this is certainly something we need to fix! Thanks for narrowing this down, should be easy enough to build a test case around this. Glad you have a workaround but we'll look into this.

MarkMurphy commented 3 years ago

Would the .gardenignore file effect what files/folders are available to docker for building the container image?

I'm seeing an error saying that a directory we're trying to COPY in our Dockerfile doesn't exist, though it does in the source location, it appears it's not present in the .garden/build/<name> folder where garden ran the docker build command.

I'm wondering if my workaround has created a different problem.

eysi09 commented 3 years ago

Yeah, files in .gardenignore will not be staged for builds in the .garden/build directory which explains the build error. We essentially run docker build in that directory so the build context won't include ignored or excluded files.

So I guess we're back to square one, right?

langfors commented 1 year ago

Note: This behavior also occurs in a singly nested git submodule.