Feature requests: Anti-long refresh tools

[Ariane-B]

Sometimes, I do something horribly wrong with my project while CodeKit is watching and I forgot to pause file-watching. A Samba connection may have gone unstable; I may have created an invalid/circular symbolic link; someone else may be messing with a Samba shared project while I still have it enabled in CodeKit; I may have made huge changes to my folder by accident...

Now, I know using CodeKit through Samba isn't the best idea, but sometimes we're reduced to doing that.

All kinds of things happen that cause CodeKit to go on a seemingly endless refreshing spree. It can then be tempting to restart the app... only to be greeted by an even longer all-project refresh instead. Then you can fix everything with a safe-start, but doing so makes you lose every CodeKit project you had, which is a bummer.

I can't expect you to plan and deal with all sorts of accidents and unintended uses. But it would be nice if we had some tools to deal with the effects of some of those.

• Situation: I restart CodeKit or my computer and realize I have a 9999 enabled projects. A very long refresh is initiated, and I don't really have any options other than curse my past self for not disabling projects I was no longer working on, or perform a safe start.
  • Solution: Allow me to stop the all-project refresh, uncheck the currently irrelevant projects and restart the refresh manually.
  • Other solution: Allow me to uncheck projects during the refresh process, and dynamically adapt to my changes. That might be more complicated though.
• Situation: I add a new, big project. It has tons of subfolders, a lot of which are useless to CodeKit. I'll surely select and right click > skip those folders later, but for now, I'm waiting.
  • Solution: I'm not sure. Maybe add a step on project-add, where only the directory structure is analyzed (which I'm guessing would be faster), and an interface offers me to select folders to skip, and only after that would it do the full refresh. Of course, this step isn't relevant to everyone, especially those with small projects and efficient SSDs, so a handy "Never ask me again" or "Only ask me for complex projects" checkbox might be cool.
  • Other solution: Maybe a list of default skipped folders could also be specified in the app. When you make a lot of websites, it's to be expected that most of them will have a similar structure.
  • Other solution: CodeKit could automatically detect some popular frameworks/CMS and suggest skipped folders accordingly. "This looks like a Laravel project. Would you like to skip the following folders?"
• Situation: I messed up. I click-dragged wrong and made a big directory change, created a weird symlink; anything. Either way, I did something wrong and I know what it is. Now CodeKit is refreshing and shows no sign of stopping.
  • Solution: Allow me to stop the refresh, pause file-watching, fix my shit, and then restart the refresh and resume file watching
• Situation: CodeKit is refreshing endlessly and I don't understand why. Restarting CodeKit only brings more endless refreshing. I have to safe-start, and maybe delete my CodeKit config file. That may or may not work. I'm pretty sure I did something wrong, but can't figure out what. Talking to you for support will probably solve the problem eventually, but it's a bit frustrating not to know where to start.
  • Solution: It would be nice if we could access a user-friendly log of what CodeKit is doing behind the scenes, separate from the normal compile log. It might help make it clearer where the problem is.
  • Other solution: When an operation is taking a longer time than it should, CodeKit could automatically warn the user and make suggestions to fix the issue. For instance, "Your project has 9999 files and 999 directories in 99 levels. This is a lot, and it may slow CodeKit down. You may want to skip folders that CodeKit shouldn't watch." I'm pretty sure that would be quite a challenge to implement though.

[bdkjones]

Yea, I hear you. There are several issues:

Disabled Projects

The "disabled" projects are, behind the scenes, no different than enabled projects. CodeKit is still indexing them and watching for changes, it just doesn't respond to the changes when they occur.

Big Folders

The way to avoid big folder problems when adding a new project is to use the Edit Defaults For New Projects option to add common problem folder names to the Skipped Folders list. Do that before adding the project. Showing a UI on every project-add that asks which folders should be indexed isn't very elegant. In general, indexing is very fast as long as you don't have a massively nested folder structure (think node_modules).

Logging

The overhead involved in printing a log of each action CodeKit is taking while re-indexing a project has a non-trivial performance cost. The steps are fairly straightforward:

1. CodeKit walks the file tree in the project looking for any files that have been moved, renamed, deleted, or added. It creates the appropriate model objects for those files or updates existing model objects as appropriate.

2. CodeKit flattens the file tree to an array of hashes and then determines dependency links between files—which files create which other files and which files import which other files. This is extremely fast for relatively shallow file trees (a few dozen layers). It can slow down when the file tree is something like a node_modules folder, which involves many, many recursive descents. That's why folders like node_modules and bower_components are not indexed by default.

Additionally, scanning for import links involves opening and reading EVERY file that could potentially contain links: Sass, Less, JS, CoffeeScript, Markdown, etc. (JavaScript developers just LOVE their little 14-line files. They split everything up into so many tiny little pieces that it's ridiculous.) This obviously has a big penalty when scanning for links between files—there is more overhead involved in opening 14,000 tiny files than 140 big files.

The import-linking algorithms in CodeKit are written in pure C using pointer arithmetic. It's as low-level as you can get without dropping to Assembly. They are MASSIVELY fast as a result, but there is simply overhead involved in tearing through a node_modules folder or similar because there's so much filesystem access for so many little, pointless files.

[bdkjones]

This is a problem that other tools, such as Grunt, Gulp, and Webpack don't have. Precisely because they don't need to model the entire filesystem under a project—there is no UI to display.

They rely on YOU to specify what files are where and to what location they should be output by making you write massive config files. And god help you if you move items around without updating that config file.

CodeKit DOES have to construct a model of the filesystem in order to show you the project and decide what options to present when you select an item (is it a Sass file? Is it a Markdown file? etc.) And CodeKit is massively intelligent about tracking changes. If you create a Sass file in /some/folder/myFile.sass and you customize the settings for that file, then quit CodeKit and completely rename that file and relocate it to new/folder/way/down/here/baz.sass, the next time CodeKit runs it will know that the file has been moved and where it's been moved and it will keep those custom settings you applied—it will NOT simply create a new model object and go back to the Project-Level defaults for Sass files. That sort of attention to detail is the hallmark of really great Mac apps and you never even notice it because it operates so seamlessly. However, that does add some overhead to the re-indexing process for very large, very nested projects.

[Ariane-B]

I don't doubt that your program is efficient. It's gotten very fast with the more recent versions, and I understand the challenges involved in that efficiency. I thank you for that effort.

But, to clarify, what would be feasible? Because no matter how much you optimize things, there will inevitably be the occasional exception (such as deeply nested projects) that will cause long reloads.

Would an option to stop the current refresh/project adding process and unlocking the UI be doable? If CodeKit's done analyzing the directory structure (or already had previous structure data) at that point, it can display it without the files. If not, the project will remain un-analyzed. Then we could go around with CodeKit file watching paused and ignore directories, fix broken structures, disable irrelevant projects, etc. and re-refresh when things are OK.

This feature alone would probably solve 90 % of annoying cases on its own.

Also, as for disabled projects, is it really necessary to keep listening to changes? What would it imply to just shut the whole watching process down for disabled projects?