Problems searching large codebase

[chrismatheson]

I have what i imagine to be a reasonably large & complex project.

i can't use searching or quick open in my project, however this is a reasonably new development.

i can open each sub directory in turn as its own project and finding seems to be working fine in isolation.

ill try and dump as much info as possible now so thanks for reading & please let me know if i can do anything else to help

<ChrisMatheson>  hey, has anyone had any problems with search & quick open on large codebases ??
[15:24:58]  <@redmunds>  we've been trying to improve that
[15:25:20]  <@redmunds>  which version are you on?
[15:25:51]  <ChrisMatheson>  latest i think
[15:25:53]  <ChrisMatheson>  40 ?
[15:26:08]  <ChrisMatheson>  trying to do a count now to give you an idea of my project size :)
[15:26:21]  <@redmunds>  have you tried excluding files for Find in Files?
[15:26:24]  <ChrisMatheson>  my searching etc was fine (little slow) up until about a week or so ago
[15:26:28]  <ChrisMatheson>  yeah
[15:26:38]  <ChrisMatheson>  but the count want even show how many are left :P
[15:26:40]  <ChrisMatheson>  ooo eerrrr
[15:26:41]  <ChrisMatheson>  lol
[15:26:41]  <@redmunds>  it actually gives you a count inside the exclusion dialog
[15:26:47]  <@redmunds>  really?
[15:27:01]  <ChrisMatheson>  yeah
[15:27:09]  <@redmunds>  the count field is filled in asynchronously
[15:27:10]  <ChrisMatheson>  think my DIR has 15757 files (recursive)
[15:27:18]  <ChrisMatheson>  based on find . -type f -print | wc -l
[15:27:51]  <@redmunds>  exclusions don't yet apply to quick open, but that's coming soon
[15:27:52]  <hareth>     with 38 k I was able to quick open
[15:27:53]   vertexclique (~vertexcli@78.173.94.72) joined the channel.
[15:28:05]  <hareth>     (after just increasing the maxFiles number in FileSystem)
[15:28:13]  <hareth>     and still had quickopen working
[15:28:28]  <ChrisMatheson>  I'm assuming that the .git folder is excluded and its not trying to traverse that ?
[15:28:39]  <hareth>     everything was just a little slow, and a directory containing 10K+ files could not be opent in the navbar
[15:28:43]  <@redmunds>  should be excluded
[15:28:47]  <ChrisMatheson>  hareth yeah it seemed to be working fine a little while ago.
[15:29:04]  <ChrisMatheson>  i could try re-installing sprint 38 and see if it still works in that older version ?
[15:29:17]  <@redmunds>  maybe a super-large log file?
[15:29:43]  <ChrisMatheson>  mmmm don't think i have any log files
[15:29:51]  <@redmunds>  not sure why that would mess up just counting files though
[15:30:09]  <ChrisMatheson>  possibly its trying to search my built output dir which will have some huge minified files
[15:30:28]   bootsWitDaFur (~Adium@pool-96-246-59-137.nycmny.east.verizon.net) joined the channel.
[15:30:29]  <ChrisMatheson>  yeah i just see " counting files ...." and it never gives a number
[15:30:34]  <@redmunds>  yeah, try excluding that folder
[15:31:28]  <ChrisMatheson>  yeah its excluded
[15:31:35]  <ChrisMatheson>  but count still no go :(
[15:31:41]  <ChrisMatheson>  ill try delting it . .
[15:31:54]  <@redmunds>  any messages in console?
[15:32:50]  <ChrisMatheson>  not when i edit the find filters section
[15:33:21]  <ChrisMatheson>  none on search but the search just hangs forever have to do a hard reload to get it back
[15:33:50]  <@redmunds>  using File > Open Folder... sets the "project" in brackets
[15:34:12]  <@redmunds>  maybe trying on subfolders of your project to try to isolate problem
[15:34:19]  <@redmunds>  we'd love to fix that
[15:34:37]  <hareth>     btw redmunds since you are here, brackets team never answered my messages : https://groups.google.com/forum/#!searchin/brackets-dev/epic-linter/brackets-dev/C9RsE9XJlqE/RywKdPyVVHcJ
[15:34:43]  <ChrisMatheson>  if i open the src as a project everything is hunky dory
[15:35:05]  <@redmunds>  what was the "project" before?
[15:35:41]  <ChrisMatheson>  have node_modules, src, bower_components, dist
[15:35:46]  <hareth>     I’m in the middle of adding tick mark on right and I wanted to know if I should do it in extension or work directly in the source :o
[15:35:54]  <ChrisMatheson>  also a .vagrant folder
[15:36:05]  <ChrisMatheson>  i think most of that is self explanatory ? ;)
[15:36:10]  <ChrisMatheson>  oh and a test folder
[15:36:37]  <ChrisMatheson>  so right now the problem is that i can't really work in just src because i need to access bower_compoenets & test cir's on a regular basis
[15:36:55]  <ChrisMatheson>  but in the top level i run into these search problems :(
[15:37:30]  <@redmunds>  are you saying that you can change project to each of those subfolders and it works, but not if you use main folder?
[15:38:09]  <ChrisMatheson>  only tried src just now
[15:38:26]  <ChrisMatheson>  i would imagine that node_modules would overload brackets quite easily :P
[15:38:29]  <ChrisMatheson>  but ill try now . . .
[15:38:50]  <@redmunds>  node_modules is one of the first folders I exclude :)
[15:39:13]  <@redmunds>  because I'm not usually working with those files
[15:39:21]  <ChrisMatheson>  bower & node modules work but starting to get slow (id say over 500ms isn)
[15:39:31]  <ChrisMatheson>  same ;)
[15:42:15]  <ChrisMatheson>  each dir seems to work fine on its own
[15:42:51]  <ChrisMatheson>  I'm happy to do a file list with sizes or something like that if it would help investigations or whatever on your end ? (if you fancy investigating)
[15:42:56]  <@redmunds>  maybe a file in root? you can right-click in project tree to do Find in... for a single file
[15:43:30]  <ChrisMatheson>  tried a find in src, just hangs :(
[15:43:48]  <ChrisMatheson>  ooo just noticed . . . Error parsing preference file: /Users/chrismatheson/Code/dealer/.jscodehints ScopeManager.js:143
[15:43:48]  <ChrisMatheson>  Unexpected string ScopeManager.js:145
[15:43:48]  <ChrisMatheson>  loaded panel template main.js:85
[15:43:48]  <ChrisMatheson>  Error parsing preference file: /Users/chrismatheson/Code/dealer/.jscodehints ScopeManager.js:143
[15:43:48]  <ChrisMatheson>  Unexpected string
[15:43:57]  <ChrisMatheson>  could this be linked?
[15:44:09]   vertexclique (~vertexcli@78.173.94.72) left IRC. (Quit: Lingo - http://www.lingoirc.com)
[15:44:45]  <@redmunds>  sounds like it could be related
[15:45:15]   busykai (~aikinzha@134.134.137.75) left the channel.
[15:45:18]  <ChrisMatheson>  just fixed it, it was trying to exlude a file that does not exist
[15:45:18]  <@redmunds>  please open an issue with that info
[15:45:44]  <ChrisMatheson>  no luck :(
[15:45:51]  <@redmunds>  please open an issue -- that shouldn't break search
[15:45:57]   lmclister (~lmclister@mobile-166-137-176-108.mycingular.net) joined the channel.
[15:46:06]  <ChrisMatheson>  sure will do. which info would you like  the file list or just what we have discussed ??

aprox 15k files in my project

Build 40, OSX 10.9

[chrismatheson]

node deps

 "devDependencies": {
    "azure": "^0.8.1",
    "bower": "~1.2.8",
    "event-stream": "~3.1.0",
    "gulp": "^3.5.6",
    "gulp-angular-templatecache": "~1.1.0",
    "gulp-concat": "~2.2.0",
    "gulp-filter": "~0.4.0",
    "gulp-html-replace": "~0.2.0",
    "gulp-jsdoc": "~0.1.0",
    "gulp-jshint": "~1.5.0",
    "gulp-karma": "0.0.4",
    "gulp-less": "~1.2.1",
    "gulp-recess": "~0.2.0",
    "gulp-rename": "~1.1.0",
    "gulp-uglifyjs": "^0.2.0",
    "gulp-util": "~2.2.14",
    "ink-docstrap": "~0.3.0-0",
    "jasmine-reporters": "^0.4.0",
    "karma": "0.12.1",
    "karma-chrome-launcher": "^0.1.3",
    "karma-jasmine": "~0.2.0",
    "karma-ng-html2js-preprocessor": "~0.1.0",
    "karma-phantomjs-launcher": "~0.1.2",
    "karma-teamcity-reporter": "vvzz/karma-teamcity-reporter",
    "lodash": "^2.4.1",
    "mime": "^1.2.11",
    "protractor": "^0.24.1"
  }

[chrismatheson]

Bower deps

"dependencies": {
    "gmap3": "5.1.1",
    "momentjs": "2.0.0",
    "node-uuid": "~1.4.1",
    "lodash": "~2.4.0",
    "angular": "1.2.16",
    "angular-route": "~1.2.16",
    "angular-mocks": "~1.2.16",
    "ngMediaFilter": "~0.1.0",
    "angular-bootstrap": "git@github.com:benaghaeipour/bootstrap-bower.git#0.10.1",
    "angular-animate": "~1.2.16",
    "bootstrap": "~3.0.3",
    "jquery": "~1.9.1",
    "component-lib": "git@github.com:iVendi/component-lib.git#0.1.8",
    "iv-utils": "~0.0.6",
    "n3-charts.line-chart": "git@github.com:benaghaeipour/line-chart.git#master",
    "angular-mediator": "~0.0.2",
    "ngInfiniteScroll": "1.1.1",
    "html5-history-api": "~4.1.0"
  },
  "devDependencies": {
    "angular-mocks": "~1.2.16"
  },

[redmunds]

More info: @chrismatheson said that excluding node and bower modules solved the problem, so he provided list so maybe we can isolate problem.

Another piece of info -- event the file count in exclusion dialog never completes.

[peterflynn]

This looks like a duplicate of #6962, but that should only occur when there are > 30,000 files + folders in the project. @chrismatheson are you sure there are only 15k files in your project, if you include all your node_modules folders, Bower dependencies, build output, etc.? Are there any symlinks in your project tree? (Brackets will follow symlinks, so all those files will be included in the count too).

[chrismatheson]

i removed a few extensions and seem to have search back now (all be it a little slow) is there a chance a third party plugin was increasing the overhead of searching and causing problems?

id love to give a list of the plugins i removed but i had quite a few and went on a bit of rampage so ive forgoten :frowning: sorry

[chrismatheson]

@peterflynn to answer your question, there are definitely symlinks in my project to other libs (npm link) & (bower link) and also there is a .vagrant .git which im not sure if they have large tree's below them.

as @redmunds mentioned, after removing node_modules and bower_modules completely from the project the search returned. i then re-installed node_modules and search worked, then installed bower_components also and it failed, same visa- versa. so this could be just a case of a "hidden" file count that is 30k+

recap for clarity

project setup	working search
extensions + src	:+1:
extensions + src + bower_components	:+1:
extensions + src + node_modules	:+1:
extensions + src + node_modules + bower_components	:-1:
src + node_modules + bower_components	:+1:

[peterflynn]

Hmm, it sure sounds like a dupe of #6962 then, except for the part about uninstalling extensions helping. But was that a red herring? From the part where you say "then installed bower_components also and it failed", it sounds like the problem came back even after you'd uninstalled some of your extensions...