B1NARY-GR0UP / nwa

A More Powerful License Header Management Tool
Apache License 2.0
6 stars 2 forks source link

Fatal error when handling a lot of files #5

Open clemlatz opened 1 week ago

clemlatz commented 1 week ago

Describe the bug

When running nwa on my project for php files, I get a fatal error. I guess this is caused by the vendor directory which contains the dependencies (where I do not want to add the header) and a lot (~9500) files.

fatal error: all goroutines are asleep - deadlock!

goroutine 1 [chan send]:
github.com/B1NARY-GR0UP/nwa/util.prepareAdd({0x5554fff713d2, 0x46}, {0x7ef9d0?, 0xc0004f4910}, {0xc000194400, 0x28f, 0x400}, 0x0)
        /app/util/file.go:175 +0xfb
github.com/B1NARY-GR0UP/nwa/util.PrepareTasks.walkDir.func1({0x5554fff713d2, 0x46}, {0x7ef9d0, 0xc0004f4910}, {0x0?, 0x0?})
        /app/util/file.go:57 +0x29a
path/filepath.walkDir({0x5554fff713d2, 0x46}, {0x7ef9d0, 0xc0004f4910}, 0xc00010db00)
        /usr/local/go/src/path/filepath/path.go:442 +0x50
path/filepath.WalkDir({0x5554fff713d2, 0x46}, 0xc00010db00)
        /usr/local/go/src/path/filepath/path.go:532 +0xaf
github.com/B1NARY-GR0UP/nwa/util.walkDir(...)
        /app/util/file.go:30
github.com/B1NARY-GR0UP/nwa/util.PrepareTasks({0xc00026e000?, 0x27c6, 0x0?}, {0xc00012c000, 0x262, 0x280}, {0x754bb9, 0x3}, {0xc000182130, 0x1, ...}, ...)
        /app/util/task.go:26 +0x105
github.com/B1NARY-GR0UP/nwa/cmd.executeCommonCmd(0x0?, {0xc00026e000, 0x27c6, 0x27cc}, {0x0, {0x5554fff66ad6, 0x11}, {0xc0001ae898, 0x4}, {0x5554fff66af2, ...}, ...}, ...)
        /app/cmd/root.go:150 +0x33b
github.com/B1NARY-GR0UP/nwa/cmd.glob..func1(0xc00022c100?, {0xc00026e000?, 0x4?, 0x754f2c?})
        /app/cmd/add.go:32 +0x53
github.com/spf13/cobra.(*Command).execute(0xa00d00, {0xc000472000, 0x27cc, 0x2800})
        /go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:944 +0x863
github.com/spf13/cobra.(*Command).ExecuteC(0xa00a20)
        /go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5
github.com/spf13/cobra.(*Command).Execute(...)
        /go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992
github.com/B1NARY-GR0UP/nwa/cmd.Execute()
        /app/cmd/root.go:51 +0x1a
main.main()
        /app/main.go:21 +0xf

Additionally, I've tried to used the --skip argument with vendor but it does not seems to have an effect.

To Reproduce

Steps to reproduce the behavior:

1. Clone my project

git clone https://github.com/biblys/biblys.git

2. Install dependencies

eg. using docker

docker run --rm --interactive --tty \
  --volume $PWD:/app \
  composer install

3. Run nwa on php files

docker run -it -v /Users/clement/Developer/biblys/biblys:/src ghcr.io/b1nary-gr0up/nwa:main add \
  --copyright "Clément Latzarus" \
  --license agpl-3.0-only \
  **/*.php 

3. Run nwa on php files with the --skip arg

docker run -it -v /Users/clement/Developer/biblys/biblys:/src ghcr.io/b1nary-gr0up/nwa:main add \
  --copyright "Clément Latzarus" \
  --license agpl-3.0-only \
  --skip vendor
  **/*.php 

Expected behavior

Either successfully add the header or display an explicit error that there is too much files (if this is indeed the issue).

Additionally, effectively ignore the vendor directory when using the --skip argument (or improve documentation on this feature if I've used it the incorrectly).

Environment:

Provided docker image

Additional context

I've worked around this issue by deleting the vendor folder so there is absolutely no emergency for me. But I'd thought it would be nice to fix this for future users.

By the way, thank you for this awesome tool that saved me a lot of time and for being so reactive with my previous pull requests 🫶

justlorain commented 1 week ago

Thanks so much for your report!

  1. Fatal Error: This is indeed a hidden bug. The current version triggers it when processing more than 1000 files at once. I have fixed this issue and removed the 1000-file limit.

  2. Skip Option Not Working: This is more of a usage issue. I tested the skip option, and it works as expected. I will add documentation on this feature soon.

    You can set any pattern supported by doublestar for the --skip option. For example, if you want to skip the vendor directory, you can use the following command:

    docker run -it -v /Users/clement/Developer/biblys/biblys:/src ghcr.io/b1nary-gr0up/nwa:main add \
     --copyright "Clément Latzarus" \
     --license agpl-3.0-only \
     --skip "vendor/**" \
     **/*.php

    or

    docker run -it -v /Users/clement/Developer/biblys/biblys:/src ghcr.io/b1nary-gr0up/nwa:main add \
     --copyright "Clément Latzarus" \
     --license agpl-3.0-only \
     --skip=vendor/** \
     **/*.php

    The use of double quotes ("") or the equals sign (=) is because some shells interpret ** in the command, which can cause it to fail. Using quotes and the equals sign avoids this problem.

It's truly a great joy to see my tool being able to help others. If you hadn't used it and submitted these pull requests, I might not have had the motivation to improve it further. Thank you for using it and for your pull requests ❤️