Enhancement - generate project tags internally and hide from user

[mshenfield]

While symbols-view is an awesome plugin, it is still very much an expert user experience which requires knowledge of ctags to generate tag files for symbols-view:toggle-project-symbols. To enhance the experience as an user, it would be nice if ctags was entirely hidden, and symbols-view smartly generated tags when the user requested them through one of the symbols-view methods.

Some of the mechanism required for this already exist in the TagGenerator class.

I think a discussion around when symbols-view would generate and cache the project tags file would be valuable. Some possibilities:

• the user manually triggers tag generations through a symbols-view:generate-project-symbols command
• symbols-view only generates tags only when it has none in memory, usually the first time symbols are needed after a project is added
• like the previous, but file watchers are added that trigger symbol regeneration when a file is changed

I also am interested in the best way to notify users of these changes. If a tag file is found in their project root, a subtle notification could inform them that symbols-view now generates the tag files internally. The option to turn this feature off and read tag files from the project root as in previous versions could also be helpful. This could be accomplished by implementing #127 and allowing the user to specify a project-specific tag source in the notification.

[mshenfield]

Tagging @nathansobo

[nathansobo]

I really like the idea of generating project-wide tags automatically in the background.

• An auto-generated tags file should probably be located somewhere in ~/.atom so we don't spam the user's project directories.
• What about language-specific configuration of the tags generator? Is there a way this could be provided to the symbols view via a service interface by language bundles? My only concern here is a bunch of ctags-specific configuration going into the bundles, but I suppose it's probably fine.
• Watching is going to be tricky with the path-watcher library because it requires a file descriptor for each file you watch. It definitely seems like a good idea, but you may want to explore a newer library such as nsfw that can watch a whole directory tree.
• As for notifying the expert users that we can now generate tags files automatically, I'd actually think we can wait on that. If they're generating tags themselves then it probably works for them, and they could discover it's automatic via social channels.
• Are you imagining shelling out to a ctags process or somehow generating the tags in JS?

[mshenfield]

An auto-generated tags file should probably be located somewhere in ~/.atom so we don't spam the user's project directories

:+1:

What about language-specific configuration of the tags generator? Is there a way this could be provided to the symbols view via a service interface by language bundles? My only concern here is a bunch of ctags-specific configuration going into the bundles, but I suppose it's probably fine.

I like that idea, and language specific tag configuration seems like a popular request. This could happen w/ or w/out project level tagging, so I think filing a separate issue might be warranted.

Watching is going to be tricky with the path-watcher library because it requires a file descriptor for each file you watch. It definitely seems like a good idea, but you may want to explore a newer library such as nsfw that can watch a whole directory tree

nsfw looks perfect for the job.

As for notifying the expert users that we can now generate tags files automatically, I'd actually think we can wait on that. If they're generating tags themselves then it probably works for them, and they could discover it's automatic via social channels.

:+1:

Are you imagining shelling out to a ctags process or somehow generating the tags in JS?

ctags is the safe bet. The symbol-gen package already generates tag files in an atom process. It could be re-used and combined with a file watcher like nsfw to provide the needed functionality.

An alternative is to harness the languages bundles to generate symbols through a js process. ctags has some great parsers (see universal-ctags php parser) and some not so great (see the Clojure parser). Atom has consistently good language support, so I think it would be a fair trade-off, and I like the idea of symbols-view growing with Atom. That said it's definitely the riskier choice. Generating symbols with the language bundles might be best suited as a separate package or experiment.

[lee-dohm]

Isn't ctags kind of an all or nothing thing? You can use it to generate tags for an entire project or a single file, but if you only need to update the tags for a single file in an entire project then you have to either run the entire project again or manually prune and merge? Couldn't this be slow for large projects? I'm concerned about things like the fuzzy-finder perpetually indexing on large projects.

[nathansobo]

You can use it to generate tags for an entire project or a single file, but if you only need to update the tags for a single file in an entire project then you have to either run the entire project again or manually prune and merge?

I don't know enough about ctags. What would pruning and merging look like? Kind of amazing it isn't incremental. Makes me wonder if it's worth considering other approaches or formats.

[lee-dohm]

The tag file format specifies that it is a list of lines. Each line is a record. It is sorted on tag name so that binary searches are possible for speed.

What this sounds like to me, if I am correct in my recollection that ctags doesn't support incremental updates, is that pruning would be going through and stripping out records for the file in question, then merging in the new records for the file in question. I suppose you could update, insert or delete records instead ... but it mostly amounts to the same thing?

I was looking into this a few months ago and I was thinking that using a SQLite database might be easier and faster for some use cases. (Like the aforementioned pruning and merging for incremental updates.)

[nathansobo]

Looks like it would be a good idea to store the tags as objects in Chrome's built-in IndexedDB facility. It's basically a key-value store that can store arbitrary JS objects (anything compatible with structured clone) and supports indexes on their properties. I'd prefer to avoid WebSQL because it's deprecated and won't ever become standard.

We could invoke ctags on a per-file basis, then use an index on the file path to delete old entries for a file and refresh them when files change. Using JS objects gives us a lot of flexibility for including all sorts of metadata in our database as things evolve.

@mshenfield @lee-dohm What do you think?

[lee-dohm]

Sounds like a good approach to me. Flexible and plenty of room to grow.

[mshenfield]

That sounds good.

Side note - goto is an existing package that generates symbols using the language bundles . It could use some :heart:, but cool implementation and exposes some potential shortcomings of the approach, for example v3ss0n/goto#4.

[riadd]

Sorry for the resurrecting this thread. Has this been implemented? I'm trying to understand how symbols-view works in my project event though I don't think I have the atom-ctags package installed. Does it already auto build the ctags for my project? If yes how to do I add my own ctags definition that gets included automatically?

[mshenfield]

No worries. @riadd symbols-view comes packaged with it's own ctags binaries. For file specific symbols-view commands, it calls one of these on the fly and slurps the standard output into memory.

[riadd]

Thank you! That helps me a lot. As far as I understand I should be able to add the custom ctags declaration for my own scripting language to $HOME/.ctags and it should work. Since I eventually want to distribute my plugin it would be nice if there would a way to tell symbols-view about my ctags declaration and having if live in my plugin folder instead of the user home folder. I'll investigate if I see a way of doing that.

[smithtim]

@riadd, were you able to find a way to do this? Or does anyone else know how? I also have a custom language, and I would like the language package I made to work with symbols-view.