search

HouzuoGuo / tiedot

A rudimentary implementation of a basic document (NoSQL) database in Go
BSD 2-Clause "Simplified" License
2.72k stars 258 forks source link

Question about result caching #162

Closed frednomoon closed 6 years ago

frednomoon commented 6 years ago

Hi there, I just thought I would ask a question relating to a strange problem I have with my application.

I have 2 services attempting to read/write to the same tiedot collection (although I realise this is not an ideal way to approach an application!). One services (call it A) is basically responsible for doing all the work, and the other service (call it B) simply does a quick check every few seconds to see if anything has been added.

I find that when a new document is added by service A, service B does not recognise that the document has been added - instead it appears to query the database in whatever state it had when the service was initialised. If however I restart service B, it will immediately see the new document added by service A.

Does this sound like expected behaviour? For example if there is some kind of clever caching mechanism in which service B assumes there are no new records unless it adds them itself?

If not there is something rather strange happening in my application!

agoalofalife commented 6 years ago

@frednomoon to record information on the disk is used mmap therefore entry does not occur instantly, perhaps this is the reason

quote

At the moment tiedot does not use a journal file, therefore it relies on operating system to periodically synchronize mapped file buffer with underlying storage device; this means, that in case of a system crash, you may lose several most recent document updates.

but maybe I'm wrong

frednomoon commented 6 years ago

@agoalofalife thanks for the quick response Ilya!

I'll expand on my explanation to say why I don't think that is the problem... In the situation I describe, service A adds a new Document which it is almost immediately able to read/ update etc. Despite service A being able to see the new document it has added, service B cannot see it. This situation persists until I restart service B (I have seen this happen for >5 minutes).

Also to be more specific in what I mean by "restart". I mean that the tiedot function openDB needs to be called, which I usually would do once upon starting the application.

I have solved the problem for now in my application by simply calling openDB again when I need service B to check for new information. But I am still interested to find out what is happening in the background!

HouzuoGuo commented 6 years ago

Hello @frednomoon. I am sorry to say that this use case is not well supported. OpenDB function is very expensive to call, it may take up to several seconds to finish for a large collection, and there cannot be two DB instances operating under the same data directory.

For your use case, please consider running an HTTP service on the server A and let server B use HTTP client to query for the new document.

agoalofalife commented 6 years ago

I have not read the original code I can try to guess again maybe something is stored in memory, because the main use of using REST API I think the addresses of the documents are stored in the buffer you can write a function to fill or update the buffer more details probably will answer @HouzuoGuo In any case, good luck!

frednomoon commented 6 years ago

Thanks for the responses!