Tool gets stuck when defining glob filepattern on big repo

[kostyay]

Hey This is my config:

aliases:
  - type: kebabcase
  - type: filepattern
    paths:
      - '**/manifest.yaml'
    patterns:
      - 'featureFlag: FLAG_KEY'
      - 'feature_flag: FLAG_KEY'

Running the tool like this:

ld-find-code-refs -t KEY -d ./ -p default -r REPO --debug -b master -T github -B master --dryRun -C 0 -l 1

Get the following output and the tool gets stuck:

INFO: 2023/08/10 20:15:58 coderefs.go:27: absolute directory path: XXXX
INFO: 2023/08/10 20:15:58 git.go:48: git branch: master
DEBUG: 2023/08/10 20:15:58 git.go:169: identified head sha: XXXXX

Could it be that I got rate limited because I was running it a bunch of times? There are 317 files matching that pattern in the repo.

Sometimes it works then I get this output: dry run found 1113 code references across 720 flags and 844 files

However its still not finding everything I'd expect to be found. Is there a more verbose debugging mode?

[jazanne]

@kostyay unfortunately, it is a known issue that the doublestar pattern does not perform very well for alias searching. We recommend using a more limited file pattern if possible. Given that you have 317 patterns to match in the repo, i understand that could be difficult.

There isn't more verbose debugging at this time for alias searching.

[kostyay]

Ive tried using xxx/*/*/manifest.yaml and it didnt help either

EDIT: I realized that patterns must contain a regex group, otherwise they won't work anyway. This is what its supposed to look like:

    patterns:
      - 'featureFlag: (FLAG_KEY)'
      - 'feature_flag: (FLAG_KEY)'

[kostyay]

@jazanne You can significantly speed up the initial parsing by caching the results returned by this function: https://github.com/launchdarkly/ld-find-code-refs/blob/68b13f09c742e421f552fbe3c085a6df2f68bdd1/aliases/alias.go#L79

Example:

var filepathGlobCache = map[string][]string{}

func filepathGlob(absGlob string) ([]string, error) {
    cachedFilePaths, ok := filepathGlobCache[absGlob]
    if ok {
        return cachedFilePaths, nil
    }

    matches, err := doublestar.FilepathGlob(absGlob)
    filepathGlobCache[absGlob] = matches

    return matches, err
}