Grep finds result, ack doesn't. Problem with large directories?

[jamischarles]

I love ack, want to use it for everything. But I ran into this issue.

I have a pretty large node_modules directory.

Using ack yields nothing:

$ ack -i --noEnv   'buyerUserAgreement' node_modules

Using grep does:

$ grep -R buyerUserAgreement node_modules

node_modules/dummySomething-config/config/links/links.json:     "buyerUserAgreement": {
node_modules/dummySomething-dummyB/models/fergus/someOtherModel.js: somelinks.buyerUserAgreement = {};

I even tried commenting out this in my .ackrc file

# Node modules created by npm
#--ignore-directory=is:node_modules

Any thoughts?

[petdance]

The first rule for "why can't it find X" is "instead of searching for X, use ack -f to see what files it is looking in." The -f flag is "show me what files I would search, but don't actually search."

So do that and see if the problem is that ack is not looking in the file you expect it to, or something else.

[hoelzro]

Hi, thanks for the report! Would you mind providing a little more information, namely:

• What version of ack you are using
• The output of ack - - dump

[jamischarles]

@petdance I've verified that it does search the file:

ack -f node_modules | grep node_modules/dummySomething-config/config/links/links.json

node_modules/dummySomething-config/config/links/links.json

[jamischarles]

@hoelzro I installed ack via homebrew

ack --version
ack 2.14
Running under Perl 5.18.2 at /usr/bin/perl

→ ack --dump
/Users/jacharles/dev/project/.ackrc
===================================
  --ignore-directory=is:.build
  --ignore-directory=is:.bzr
  --ignore-directory=is:.cabal-sandbox
  --ignore-directory=is:.cdv
  --ignore-directory=is:.git
  --ignore-directory=is:.hg
  --ignore-directory=is:.metadata
  --ignore-directory=is:.pc
  --ignore-directory=is:.svn
  --ignore-directory=is:CMakeFiles
  --ignore-directory=is:CVS
  --ignore-directory=is:RCS
  --ignore-directory=is:SCCS
  --ignore-directory=is:_MTN
  --ignore-directory=is:_build
  --ignore-directory=is:_darcs
  --ignore-directory=is:_sgbak
  --ignore-directory=is:autom4te.cache
  --ignore-directory=is:blib
  --ignore-directory=is:cover_db
  --ignore-directory=is:tests
  --ignore-directory=is:~.dep
  --ignore-directory=is:~.dot
  --ignore-directory=is:~.nib
  --ignore-directory=is:~.plst
  --ignore-file=ext:bak
  --ignore-file=ext:gif,jpg,jpeg,png
  --ignore-file=ext:pdf
  --ignore-file=match:/[.-]min[.]js$/
  --ignore-file=match:/[.]css[.]map$/
  --ignore-file=match:/[.]css[.]min$/
  --ignore-file=match:/[.]js[.]map$/
  --ignore-file=match:/[.]js[.]min$/
  --ignore-file=match:/[.]min[.]css$/
  --ignore-file=match:/[._].*\.swp$/
  --ignore-file=match:/^#.+#$/
  --ignore-file=match:/core\.\d+$/
  --ignore-file=match:/~$/
  --type-add=actionscript:ext:as,mxml
  --type-add=ada:ext:ada,adb,ads
  --type-add=asm:ext:asm,s
  --type-add=asp:ext:asp
  --type-add=aspx:ext:master,ascx,asmx,aspx,svc
  --type-add=batch:ext:bat,cmd
  --type-add=cc:ext:c,h,xs
  --type-add=cfmx:ext:cfc,cfm,cfml
  --type-add=clojure:ext:clj
  --type-add=cmake:ext:cmake
  --type-add=cmake:is:CMakeLists.txt
  --type-add=coffeescript:ext:coffee
  --type-add=cpp:ext:cpp,cc,cxx,m,hpp,hh,h,hxx
  --type-add=csharp:ext:cs
  --type-add=css:ext:css
  --type-add=dart:ext:dart
  --type-add=delphi:ext:pas,int,dfm,nfm,dof,dpk,dproj,groupproj,bdsgroup,bdsproj
  --type-add=dust:ext:dust
  --type-add=elisp:ext:el
  --type-add=elixir:ext:ex,exs
  --type-add=erlang:ext:erl,hrl
  --type-add=fortran:ext:f,f77,f90,f95,f03,for,ftn,fpp
  --type-add=go:ext:go
  --type-add=groovy:ext:groovy,gtmpl,gpp,grunit,gradle
  --type-add=haskell:ext:hs,lhs
  --type-add=hh:ext:h
  --type-add=html:ext:htm,html
  --type-add=jade:ext:jade
  --type-add=java:ext:java,properties
  --type-add=js:ext:js
  --type-add=json:ext:json
  --type-add=jsp:ext:jsp,jspx,jhtm,jhtml
  --type-add=less:ext:less
  --type-add=lisp:ext:lisp,lsp
  --type-add=lua:ext:lua
  --type-add=lua:firstlinematch:/^#!.*\blua(jit)?/
  --type-add=make:ext:mak
  --type-add=make:ext:mk
  --type-add=make:is:Makefile
  --type-add=make:is:Makefile.Debug
  --type-add=make:is:Makefile.Release
  --type-add=make:is:makefile
  --type-add=matlab:ext:m
  --type-add=objc:ext:m,h
  --type-add=objcpp:ext:mm,h
  --type-add=ocaml:ext:ml,mli
  --type-add=parrot:ext:pir,pasm,pmc,ops,pod,pg,tg
  --type-add=perl:ext:pl,pm,pod,t,psgi
  --type-add=perl:firstlinematch:/^#!.*\bperl/
  --type-add=perltest:ext:t
  --type-add=php:ext:php,phpt,php3,php4,php5,phtml
  --type-add=php:firstlinematch:/^#!.*\bphp/
  --type-add=plone:ext:pt,cpt,metadata,cpy,py
  --type-add=python:ext:py
  --type-add=python:firstlinematch:/^#!.*\bpython/
  --type-add=rake:is:Rakefile
  --type-add=rr:ext:R
  --type-add=rst:ext:rst
  --type-add=ruby:ext:rb,rhtml,rjs,rxml,erb,rake,spec
  --type-add=ruby:firstlinematch:/^#!.*\bruby/
  --type-add=ruby:is:Rakefile
  --type-add=rust:ext:rs
  --type-add=sass:ext:sass,scss
  --type-add=scala:ext:scala
  --type-add=scheme:ext:scm,ss
  --type-add=shell:ext:sh,bash,csh,tcsh,ksh,zsh,fish
  --type-add=shell:firstlinematch:/^#!.*\b(?:ba|t?c|k|z|fi)?sh\b/
  --type-add=smalltalk:ext:st
  --type-add=smarty:ext:tpl
  --type-add=sql:ext:sql,ctl
  --type-add=stylus:ext:styl
  --type-add=tcl:ext:tcl,itcl,itk
  --type-add=tex:ext:tex,cls,sty
  --type-add=tt:ext:tt,tt2,ttml
  --type-add=vb:ext:bas,cls,frm,ctl,vb,resx
  --type-add=verilog:ext:v,vh,sv
  --type-add=vhdl:ext:vhd,vhdl
  --type-add=vim:ext:vim
  --type-add=xml:ext:xml,dtd,xsl,xslt,ent
  --type-add=xml:firstlinematch:/<[?]xml/
  --type-add=yaml:ext:yaml,yml

[petdance]

Is this JSON file one massive long line by any chance?

[jamischarles]

@petdance No. Appears to be a normally structured json file. ~ 1300 lines.

[jamischarles]

Grep also finds results in several .js files.

[petdance]

Does the file in question have linefeeds or CR/LFs?

[petdance]

Would you be willing to share the file, as-is unmodified?

[jamischarles]

I can't share the exact file, but I'll make a copy, change the values (try to leave the hidden chars intact) and upload to a gist. Does that work? I'll verify that it still can't be found first.

[petdance]

You can try, but I'll bet a nickel it won't fail if you gist it and we download it, but it can't hurt to try, no?

[jamischarles]

So I discovered something interesting:

When I comment out the .ackrc line to ignore node_modules, this works: $ ack -C 5 'buyerUserAgreement' node_modules

but this doesn't: ack --noenv -C 5 'buyerUserAgreement' node_modules

I assume this is what's happening: My .ackrc file has the 'ignore node_modules' line commented out, so it doesn't ignore that folder. When I flip the --noenv flag, it ignores my override, and loads the default which ignores node_modules.

Is this accurate?

[petdance]

Aha. --noenv ignores your ~/.ackrc but it also ignores other .ackrc files, like .ackrc in your project, or the /etc/ackrc for the machine.

But then again, if there was an .ackrc other than the one in your project we would have seen it in the ack --dump.

[jamischarles]

Any chance we can have passing the folder as the last param override the setting to ignore that folder?

Or am I stuck doing something like this then?

$ ack --noignore-dir=node_modules  -C 5   'buyerUserAgreement' node_modules

Here's my use case: I first want to search my normal folders. If I can't find it, then I want to quickly flip and search the node_modules as well.

[jamischarles]

So based on what you're saying, with the --noenv it shouldn't have ignored the node_modules folder? Or does it even load the ack defaults with that on?

[petdance]

With --noenv it only uses the default settings.

Do an ack --noenv --dump to see what they are.

[jamischarles]

Ah. There it is. So these won't be affected by --noenv. That flag simply removes overrides and falls back to defaults. Makes sense.

ack --noenv --dump
Defaults
========
  [...]
  --ignore-directory=is:autom4te.cache
  --ignore-directory=is:blib
  --ignore-directory=is:cover_db
  --ignore-directory=is:node_modules
  --ignore-directory=is:~.dep
  --ignore-directory=is:~.dot
  [...]

[jamischarles]

Thank you for your help. Last question: Any chance we can have passing the folder as the last param override the setting to ignore that folder?

Or am I stuck doing something this?

$ ack --noignore-dir=node_modules  -C 5   'buyerUserAgreement' node_modules

I would expect that passing the folder name would override an ignore setting since I am doing so explicitly

[petdance]

Anything on the command line overrides anything in an .ackrc, or that came before it on the command line.

[jamischarles]

I would expect this to work, since I am being so explicit with the folder name: $ ack 'buyerUserAgreement' node_modules

instead I have to do this $ ack --noignore-dir=node_modules 'buyerUserAgreement' node_modules

Any chance I can convince you to add that? :)

[petdance]

Yes, you are right. Explicitly specifying node_modules as a starting point should override the ignore-dir. What if you do it as node_modules/? Does that do what you expect?

[jamischarles]

Same result.

[petdance]

Well, etiher way I think it's a bug. Someone needs to write up a very simple test case and make a new issue out of it.

[jamischarles]

I could take a stab at it, though I'm not very perl savvy. Do you have an existing test I could use as a starting point?

[petdance]

It's fine, we've put you through enough as is. :-) All the .t files are in the repo if you're feeling brave.

The big thing is that it should be easily reproducible. It sounds to me like this should fail

mkdir temp
echo 'Bongo!' > temp/dir.txt
ack --ignore-dir=temp Bongo temp

and that should turn up nothing even though you specified temp on the command line. And similiarly, ack -f --ignore-dir=temp temp should return dir.txt but doesn't.

[jamischarles]

agreed. That currently fails. I just tested it.

[petdance]

So please start a new issue based on that. Give it a meaningful title. And then we can close this one (but point back to it).