How do I insert/update a json document the format of which you don't know ahead of time

[prabhatsharma]

How to index an unknown json document? I could do that in bleve using index.index("id", doc)

I could not find a mechanism to do that in bluge as I have to add every field manually using doc.AddField

[mschoch]

Indexing an unknown JSON document is not a supported operation in Bluge.

The reason is that there is no single correct way to index the data. A user or application must make decisions about how they want to index a particular JSON document. Bleve supported this by defining a "mapping", which is just another data-structure to support a declarative set of rules to map arbitrary JSON into a Bleve document. Over time we learned that having this declarative mapping inside the Bleve library was problematic. Basically for the reasons that declarative configuration always breaks down:

• config complexity increases over time due to one-off additions
• changing the config itself is problematic because the most convenient thing to do with the mapping is serialize it, but now applications need backwards compatibility as well
• in the end, all users want the config modified to support their rules, and no users want the config to keep changing because they want backwards compatibility

So Bluge has decided not to include this functionality. For application developers, I think this will actually make the library easier to use (once you get past the fact that you have to build the bluge document itself). Now you don't have to learn about a mapping at all, you simply write Go code to explicitly build exactly the index you want. If you need to support arbitrary end-user JSON, then you will obviously also need to write some reflection code to walk the unknown JSON, and apply your own index-building logic.

The biggest downside is that if you still need some sort of serializable mapping, to share between users or nodes in a cluster, etc, then you may have to build your own mapping. But again, I think that is a better place for it. Whenever you want to change the mapping, you can just do it directly, instead of trying to get the library itself to change.

Indirectly, I hope it lead to fewer confused beginners. Having an accidentally faulty mapping is the root cause of a significant number of bleve issues (anecdotal, I don't have data to support this)