FR: Custom indexing

[MrMugame]

Currently indexing is fully implemented in the index plug. This limits the ability to index custom stuff like csv files (#877), RSS feeds, a weather api or something even more creative. I think decentralizing the indexing, meaning that any plug could provide indexed data, would make sense. I'm unsure on how this would be implemented tho, the main questions here being:

• When does a plug index data? For attachments, the plug should probably just listen for a change like a new attachment file and check if it's indexable and if so, index it. But I'm unsure with stuff like RSS feeds? Should these just run behind a cron job, or maybe it should be lazily generated?
• How does the data get from one plug to the index plug. One idea would be the event system, which maybe just sends out an event like index:indexed with the relevant data/information. This idea would have the issue, that the index plug could never know if the data still takes time to index, or there just isn't anyone indexing this data, or some other issue is present.

Any input on the implementation problems would be welcome, but I think the idea itself would make silverbullet much more "hackable".

[zefhemel]

This is already how it works, if I'm understanding you correctly.

Although it happens to to be the case that all indexers are implemented in the index plug, any plug can listen to the page:index event (which is triggered any time a page has been updated) and act appropriately. As you can see in the code, this is also what all existing indexers do. Search the code base for page:index and page:index_text to see how and when this is triggered.

Recently a new attachment:index was introduced by @onespaceman which allows for indexing of non markdown files, and could (I'm pretty sure) be used to index e.g. CSV files too, or RSS feeds (if they're files in your space).

Note this is all architected to index content inside your space, not arbitrary outside sources. So if you'd want to use it to e.g. index RSS feeds, you'd have to pull in those files into your space separately.

[MrMugame]

Oh, so a plug would be supposed to set the values in the datastore? Oh, I kind of just assumed that that's scoped per plug ....

[zefhemel]

It's recommended you use the index APIs to index objects (if that's what you want to do) here: https://github.com/silverbulletmd/silverbullet/blob/main/plugs/index/plug_api.ts (there's api.ts for use within the index plug, and plug_api.ts for use from other plugs).