Support recursive glob **/*.y*ml

[HariSekhon]

Please support recursive globs:

$ datree test **/*.yaml

stat **/*.yaml: no such file or directory

$ datree test *.yaml

<works>

Relates to https://github.com/datreeio/action-datree/issues/3 where the documentation link below says this recursive glob should work:

https://github.com/datreeio/action-datree#examples

[eyarz]

I ran Datree on this dir and I'm not able to reproduce this bug:

which version of Datree (datree version) and OS do you use when you see this error?

[HariSekhon]

I'm using datree 1.4.10 on macOS and running against this latest master checkout:

https://github.com/HariSekhon/Kubernetes-configs

$ datree version
1.4.10

[eyarz]

🤔 interesting. I'm still not able to reproduce (also running now on macOS):

I don't think this is the case, but just to verify, can you please update to version 1.4.13 and check again?

[HariSekhon]

I upgraded to the latest 1.4.26 and it still occurs:

$ datree version
1.4.26

$ datree test **/*.yaml
stat **/*.yaml: no such file or directory

$ datree test '**/*.yaml'
stat **/*.yaml: no such file or directory

[eyarz]

is this happening when running in github actions? are you able to reproduce it locally?

[HariSekhon]

Yes this happened in GitHub Actions originally, which is why I then reproduced it locally for this ticket.

I've actually reproduced this on 2 different versions of macOS.

I tried this on Ubuntu 20.04 in Docker for comparison and sort of that works, although the spinner goes up the screen:

$ docker run --rm -ti -e DATREE_TOKEN ubuntu:20.04

$ apt update && apt install git curl zip -y

$ curl https://get.datree.io | /bin/bash

$ git clone https://github.com/HariSekhon/Kubernetes-configs k8s

$ cd k8s/

$ datree test **.yaml
...
WWWWWWWWWWWWW                                                                                                                                                                                                                                                                                                                                                               WWWWWWWWWWWWW
<continues to overwrite all the lines above on the screen with WWWW lines, probably an errant backspace>
....

File: /k8s/argocd-disable-autosync.jsonpatch.yaml WWWWWWWWWWWWW [X] YAML validation WWWWWWWWWWWWW ❌ yaml validation error: yaml: unmarshal errors: line 29: cannot unmarshal !!seq into map[string]interface {} WWWWWWWWWWWWW WWWWWWWWWWWWW [?] Kubernetes schema validation didn't run for this file [?] Policy check didn't run for this file WWWWWWWWWWWWW >> File: /k8s/argocd-proj-default.jsonpatch.yaml WWWWWWWWWWWWW [X] YAML validation WWWWWWWWWWWWW ❌ yaml validation error: yaml: unmarshal errors: line 32: cannot unmarshal !!seq into map[string]interface {} WWWWWWWWWWWWW WWWWWWWWWWWWW [?] Kubernetes schema validation didn't run for this file [?] Policy check didn't run for this file WWWWWWWWWWWWW >> File: /k8s/argocd-resources.argocd-application-controller.jsonpatch.yaml WWWWWWWWWWWWW [X] YAML validation WWWWWWWWWWWWW

A copy paste of the WWWW lines isn't rendering the rest of them in the code block so I've taken them out of the code block so you can see them interspersed with the expected output.

Edit: adding --output simple gets rid of the messed up output above, and does indeed work with **.yaml on Ubuntu in Docker, so perhaps this **.yaml problem is only an issue on Macs?

[HariSekhon]

It looks like the latest version of datree and action-datree does work with **.yaml on GitHub Actions now.

But the latest version still fails with **.yaml on Mac as you can see above.

[eyarz]

we tried to reproduce this on several MacOS machines w/o a success. I will close this issue as won't fix for now.

[HariSekhon]

Ok, the latest github action works so that's good enough for now...

[vaibhav-fx]

the latest github action was breaking for me. I had to change it to

 - name: Run Datree Policy Check
    uses: datreeio/action-datree@main
     with:
       path: '**/**/*.yaml'
       cliArguments: '--only-k8s-files'

[adifayer]

@vaibhav-fx Few questions to be able to drilldown the issue- When you say breaking, what do you mean? Did you get the same error? Please provide a screenshot :)

[hadar-co]

I tested locally and was able to reproduce. It seems that using */.yaml only looks 1 level down in the dir structure, so files that are deeper down than that will not be tested. Reopening the issue

[romanlab]

@hadar-co can you test it with **.yaml ? It seems that there's a behavior that when a / is present it limits the recursion to only the directory and its direct subdirectories

[hadar-co]

@romanlab tested, still the same result.

[vaibhav-fx]

there is also something funny basis on the path and recursive / which are given to it this is my structure

.
├── README.md
├── auth
│   └── ap-south-1
│       ├── cluster-roles.yaml
│       ├── roles.yaml
│       └── users.yaml
└── cluster
    └── ap-south-1
        └── namespaces.yaml

4 directories, 5 files

when I run datree test **/*.yaml

(Summary)

- Passing YAML validation: 4/4

- Passing Kubernetes (1.20.15) schema validation: 4/4

- Passing policy check: 4/4

when I run datree test **/**/*.yaml

(Summary)

- Passing YAML validation: 12/12

- Passing Kubernetes (1.20.15) schema validation: 12/12

- Passing policy check: 12/12

when I run **/**/**/*.yaml I get

(Summary)

- Passing YAML validation: 24/24

- Passing Kubernetes (1.20.15) schema validation: 24/24

- Passing policy check: 24/24

the recursive digging into the files and number of validations reported increases, but the number of files are always same

[vaibhav-fx]

also the problem I had with **/*.yaml was with github actions. In my local **/*.yaml works fine

[hadar-co]

After further investigation, it seems that the issue is with the shell being used. Bash does not support recursive globs, while zsh and others do. @vaibhav-fx In your local environment - are you using zsh or some other shell?

[vaibhav-fx]

@hadar-co I use zsh in my local. the github actions is based on an ubuntu image, so I guess bash there

[HariSekhon]

If '**/*.yaml' is passed to datree in single quotes, can't datree do the recursion in code to avoid any shell environment issue?

[hadar-co]

Starting with bash version 4, a new command was introduced that allows recursive globbing.

The github action has been fixed using this command, and now it fully supports recursive globbing.

When using the CLI locally:

• recursive globbing will work with zsh
• recursive globbing will work with bash version 4 and up, with the globstar feature enabled
• recursive globbing will not work with bash version 3 or lower

[eyarz]

@hadar-co so if you updated the docs and fixed action, can we close this issue?

[hadar-co]

Yes, closing

[HariSekhon]

Btw Macs are stuck with Bash version 3 due to licensing issues, so expect this to be a persisting problem for Macs running datree locally.

I suggest a feature be added to do a glob recursion in code if passed '**' single-quoted.

[eyarz]

@HariSekhon thank you for this insight! We think it should be handled by the user (use Bash 4 or zsh) and not by the CLI (code).

We added this info to our docs:

[HariSekhon]

I can appreciate not wanting to complicate the code by adding this logic.

Perhaps add a workaround note for people using older versions of Bash on Mac or even Bourne shells that they can achieve a similar effect using a workaround like this:

find . -name '*.yaml' -print0 | xargs -0 datree test

[hadar-co]

@HariSekhon Thanks for the note, we added this workaround to the docs:)

[vaibhav-fx]

I use zsh in my local. the github actions is based on an ubuntu image, so I guess bash there

On Thu, Jun 30, 2022 at 1:11 PM hadar-co @.***> wrote:

After further investigation, it seems that the issue is with the shell being used. Bash does not support recursive globs, while zsh and others do. @vaibhav-fx https://github.com/vaibhav-fx In your local environment - are you using zsh or some other shell?

— Reply to this email directly, view it on GitHub https://github.com/datreeio/datree/issues/618#issuecomment-1170878221, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZRGPDKGV3MSJBTXNOOJNPTVRVFULANCNFSM5VY5XRSA . You are receiving this because you were mentioned.Message ID: @.***>

--

Vaibhav Krishna

DevOps

https://calendly.com/

https://www.linkedin.com/ http://twitter.com/

FalconX Insights https://blog.falconx.io/

Bloomberg https://www.bloomberg.com/news/articles/2021-08-10/crypto-platform-falconx-quintuples-valuation-to-3-75-billion | Business Wire https://www.businesswire.com/news/home/20210810005701/en/FalconX-Raises-210M-Following-30x-YoY-Revenue-Growth-Now-Valued-at-3.75-Billion | TechCrunch https://techcrunch.com/2020/05/13/falconx-raises-17m-to-power-its-crypto-trading-service/