Question: Spotlight-internals & Open-source alternative to spotlight

[Centril]

I'm a Linux/Win user who'd like to develop an open source version of spotlight for all platforms using either C++ or Rust. The "project" is in the idea stage: https://github.com/Centril/open-spotlight/wiki/Requirements-&-Ideas

I'm wondering what you have figured out about the spotlight internals...

• Do you understand anything about how the indexer works? As far as I understand, spotlight is just a GUI on top of the filesystem indexer (for the FS part at least...) which runs as a separate daemon/service.
• Is all content that spotlight displays HTML & Javascript, i.e, is it running in a browser? (Then perhaps I should run a local Servo or something). Or is the browser something that flashlight adds?
• How much of the API that you expose? As far as I understand it, the only thing you expose in the JS side is the bash command, what do you expose on the python side?

Any insights are most welcome!

[nate-parrott]

This is a really cool idea!

To answer your questions:

• Flashlight doesn't really get deeper than the UI layer of Spotlight. You're right that the Spotlight process is just a fronted for the metadata system, which has a couple auxiliary processes (mds, mdworker, etc). Those services observe changes to the process, create inverted indexes, and serve queries to them. The special stuff Spotlight offers (like unit conversions or calculations) are done by the UI process.
• Spotlight's UI is native — Flashlight processes use a web view because it's more convenient than having plugin authors write Objective-C
• We don't really expose anything — we just invoke plugins during search, and when a user clicks or presses Enter on a search result.

[nate-parrott]

I see you're considering to use Flashlight plugins as this system's plugin interface — if you need any help, I'd be glad to lend a hand. There's a bit of Mac-specific stuff in many plugins, but the majority of plugins would probably work fine. There's a lot of Objective-C code in Flashlight core itself (FlashlightKit), but it actually might be possible to build in a cross-platform way, with something like GNUStep.

[Centril]

Glad you think so =) If you have any ideas you'd like to see or so, please add them!

• Hmmh... Changes to the Spotlight process or the file-system "process"? I'm guessing somewhere the mds & mdworker processes are listening in to files being created and destroyed to run the indexing, right?
• Is it possible to access the indexing data of metadata? There's Windows Search Indexer or something like that - maybe I can use that so that the users can use their native indexer and fallback to a custom one for those that don't have one...?
• HTML-driven interface would have made stuff so much easier for me =) But I guess I'll have to use GTK+ or Qt or WxWidgets or something like it. I'd really like to use the OS X L&F as default but let users have the standard system L&F if they wish.
• Have to taken any look at http://www.rust-lang.org/ ? Servo, Mozilla's new rendering engine for browsers puts Web-Kit & Gecko to shame, it seems to be super great at multi-core stuff, and indexing and search results is something you can parallelize a lot! I haven't written a line of ObjC in my life, it feels really weird to me ;) Rust is kinda new, and bindings to widget libraries are not super mature yet, but while I'm living on the edge, I might as well try it?
• Maybe it would be a good idea to use Flashlights existing plugin interface as one version and a more extensible one, given that I can control things more exactly, at the same time giving users both the benefit of an existing community and more stuff?

[nate-parrott]

• Oops, I wrote "process" when I meant "filesystem."
• Probably possible to access the underlying index, but definitely not documented. There is a documented API for querying the md system, but if you're going cross-platform, I think it would make sense to build your own indexing system. SQLite has full-text indexing — you could build an indexer using that.
• HTML seems like the way to go. Spotlight uses native UIs, but you could definitely match the look-and-feel using HTML/CSS/JS. I wouldn't worry too much about making the Linux version look like OS X.
• Rust seems really cool. Using servo for the UI layer does seem kind of crazy, but if you can get it to work...
• agreed. Flashlight does a lot of NLP stuff on queries, which you'd need to port or rewrite, but I do think the plugin architecture works pretty well.

[Centril]

• Ye, makes sense with that correction ;)
• Making my own indexer is a given, can't expect all platforms to even have an indexer... but using your own indexer without disabling the native OS one (might not even be possible to disable...) means double the work. But I guess that initially, I should focus on my own indexer and then maybe later on specialize if it makes sense.
• Linux UIs normally look so badly done - like no graphic designers have touched it at all, making it have the nice L&F of spotlight on OS X is def a priority for my own eyes (and pride ^^).
• Servo is really crazy fast, http://www.phoronix.com/scan.php?page=news_item&px=MTgzNDA I guess I just have to figure out how to ditch the window-borders and chrome to make it seem like it's not really HTML, if I take that route.
• Agreed =)