The reason for this task are further performance improvements. As described in the PR:
...especially that reading yamls from workspace (which is a part of validation) takes, like 10 - 20 seconds (see below).
(...) What is used now for scanning for manifests is native workspace.findeFiles. It takes into account files ignored in VSC via files.exclude but does not look on .gitignore.
Now measuring times for monokle-saas repo, it seems not finding files but parsing them takes most time (...) interestingly we got 50 resources from 837 files, which means most files are not K8s resources (I quickly check and just by ignoring node_modules we get to 137 files and less than 5 seconds).
For context, scanning took less than 500ms while parsing more than 7 second.
Possible solutions
Adjust workspace.findeFiles to take into account .gitignores
There is no build in support for this and would need to be done manually. And this is a bit tricky because:
This method accepts only single glob pattern as exclude pattern. So all gitignore entries needs to be glued correctly (like {line1,line2} AFAIU) also making sure that ** are added if needed.
There can be multiple gitignores (in monokle-saas there are 17) which needs to be found first, parsed and glued to exclude glob.
And also while initially reading gitignores, you will get ones in gitignored folders (chicken and egg problem).
As an alternative, we could scan the same way as now and then just filter out ignored files (still requires parsing all gitgnores and glob filtering on resulted files list).
What is important here, I did a bit of benchmarking, and this method is pretty efficient (in relation to other approaches). For monokle-saas repo it took less than 500ms, and found 807 yamls.
Use 3rd-party library
I tested this with globby. It has a build-in option to take gitignore files automatically into account. And it is considered one of the fastest JS implementations from what I see.
However, there are 2 issues I see - the results are quite different - for build-in method I got 807 files while with globby only 78 (even with gitignores not taken into account). Also scanning entire repo takes globby around 15 seconds on my machine (while VSC build in is less than 500ms).
Others
It might be worth looking into how other extension do this maybe 🤔 I did quite searching in VSCode docs and outside for a build-in way or quick solution to do this in VSCode extension but haven't found anything worth mentioning here.
Extracted from https://github.com/kubeshop/vscode-monokle/pull/68#issue-2030890818. Initially I thought it could be the part of mentioned PR but since it's more complex and the PR itself resolves initial issue, it will be better handled separately.
The reason for this task are further performance improvements. As described in the PR:
For context, scanning took less than 500ms while parsing more than 7 second.
Possible solutions
Adjust
workspace.findeFilesto take into account.gitignoresThere is no build in support for this and would need to be done manually. And this is a bit tricky because:
{line1,line2}AFAIU) also making sure that**are added if needed.monokle-saasthere are 17) which needs to be found first, parsed and glued to exclude glob.As an alternative, we could scan the same way as now and then just filter out ignored files (still requires parsing all gitgnores and glob filtering on resulted files list).
What is important here, I did a bit of benchmarking, and this method is pretty efficient (in relation to other approaches). For
monokle-saasrepo it took less than 500ms, and found 807 yamls.Use 3rd-party library
I tested this with
globby. It has a build-in option to take gitignore files automatically into account. And it is considered one of the fastest JS implementations from what I see.However, there are 2 issues I see - the results are quite different - for build-in method I got
807files while withglobbyonly78(even with gitignores not taken into account). Also scanning entire repo takesglobbyaround 15 seconds on my machine (while VSC build in is less than 500ms).Others
It might be worth looking into how other extension do this maybe 🤔 I did quite searching in VSCode docs and outside for a build-in way or quick solution to do this in VSCode extension but haven't found anything worth mentioning here.