HouzuoGuo / tiedot

A rudimentary implementation of a basic document (NoSQL) database in Go
BSD 2-Clause "Simplified" License
2.72k stars 257 forks source link

Indexing an empty collection voids the index #132

Closed PlkMarudny closed 7 years ago

PlkMarudny commented 7 years ago

If I create a database, a collection and an index (on the empty collection), later on ther index is not updated, i.e. inserted documents are not indexed.

PlkMarudny commented 7 years ago

This happens when index has a longer path, for example: a,b,c

HouzuoGuo commented 7 years ago

Hello, are you sure that the document you're placing into the collection has an attribute located in the path? If you are indexing a,b,c, the value it looks for is located in {"a": {"b": {"c": "value here"}}}

PlkMarudny commented 7 years ago

Hi, I am suspecting so. This is the begining of the document inserted into the db: { "Updated": false, "126847110970121761": { "TwitterTrend": { "AsOf": "2017-06-06T12:56:36Z", "CreatedAt": "2017-06-06T12:53:54Z", "Locations": [ { "Name": "Kuwait", "Woeid": 23424870 } ], .... (removed)

Indexes are defined as follows:

[ [ "Updated" ], [ "TwitterTrend", "Locations", "Woeid" ] ]

The simple one on "Updated" field works. The composite one, so to say, does not. Until I delete it and reindex the db with data inserted.

PlkMarudny commented 7 years ago

But when a create a simple example with only one composite index, it works.

PlkMarudny commented 7 years ago

And when I create the test case using http, it works... in code, no.

HouzuoGuo commented 7 years ago

I see, there was a misunderstanding. tiedot does not handle composite index, an index only works on a set of attribute names that lead to an attribute value (or an array of values).

To find the "woeid" attribute value, the index path should be simply ["Updated", "TwitterTrend", "Locations", "Woeid"]. When the document is placed into the collection, the value 23424870 will be indexed even though its object is an array element.

PlkMarudny commented 7 years ago

I was not clear enough; one index is on the field "Updated", another one on ["TwitterTrend", "Locations", "Woeid"] that leads to "Woeid" I am interested in. The document (shortened, it has a plenty of fields) looks like below, so "Updated" field requires its own index (that works), but to index "Woeid" I need another one: TwitterTrend,Locations,Woeid.

"4645211266929063441": { "TwitterTrend": { "AsOf": "2017-06-07T09:26:50Z", "CreatedAt": "2017-06-07T09:24:18Z", "Locations": [ { "Name": "Kuwait", "Woeid": 23424870 } ], "Updated": true }

What I do (in code) is creating a collection, two indexes, inserting some data and trying to look for documents. Index on "Updated" works, the ["TwitterTrend", "Locations", "Woeid"] does not, until I delete it and reindex again (with data already inserted).

HouzuoGuo commented 7 years ago

Oh, I wonder if you were hit by a Golang quirk in JSON serialisation of float.

When you query the records, do you deserialise a prepared query from JSON string into a structure, and then feed the structure to query function? Or do you construct a map[string]interface{} that represents a query manually? If you are doing it using the latter method, remember to convert an integer to float64, or the query processor may not match attribute data that is always float64.

HouzuoGuo commented 7 years ago

I hope it eventually worked well for you. Feel free to come back with more questions.