search

orientechnologies / orientdb

OrientDB is the most versatile DBMS supporting Graph, Document, Reactive, Full-Text and Geospatial models in one Multi-Model product. OrientDB can run distributed (Multi-Master), supports SQL, ACID Transactions, Full-Text indexing and Reactive Queries.
https://orientdb.dev
Apache License 2.0
4.75k stars 871 forks source link

Improve current storage durability #1404

Closed andrii0lomakin closed 9 years ago

andrii0lomakin commented 11 years ago

Current storage uses MMAP as disk buffer which means that management of disk pages relies upon OS not DB itself as result we can control cache behavior in limited manner and this also makes storage not safe in case of power failure to improve given situation following is proposed:

  1. Replaced MMAP by our own 2Q cache which was designed for usage in DBs as disk cache + write cache which will contain records are going to be flushed sorted by position which will minimize random IO operations.
    1. Implement paginated storage so we can treat pages as atomic units of data, as side effect we can remove storage global lock and replace 2 disk seeks IO by one disk seek IO for retrieval of record by id
    2. Implement WAL inside of paginated storage.
    3. Use direct memory for page cache to avoid GC overhead (Unsafe + JNA fallback).
lvca commented 11 years ago

WDYT about creating a compression factory with Snappy as implementation and a Nothing as default?

Something like OCompressionFactory{getCompression(String name);register(String name, OCompression algo);} OCompression{compress();uncompress();}

In this way we could use it in different places even OStorageLocal where compression.compress() do nothing by default.

WDYT?

lvca commented 11 years ago

I'm going to open a new issue...

andrii0lomakin commented 11 years ago

Yes, Sure why not.

On Mon, Apr 1, 2013 at 11:51 AM, Luca Garulli notifications@github.comwrote:

I'm going to open a new issue...

— Reply to this email directly or view it on GitHubhttps://github.com/nuvolabase/orientdb/issues/1404#issuecomment-15708782 .

With best regards, Andrey Lomakin

lvca commented 11 years ago

It already exists: issue #208

andrii0lomakin commented 11 years ago

plocal is stable now to release but we need to do some improvements to be sure that restore procedure is ready on 100% so we move it to 2.0

andrii0lomakin commented 11 years ago

I added several sub issues when they will be finished this issue can be completed. https://github.com/orientechnologies/orientdb/issues/1608 https://github.com/orientechnologies/orientdb/issues/1607 https://github.com/orientechnologies/orientdb/issues/1606 https://github.com/orientechnologies/orientdb/issues/1605 https://github.com/orientechnologies/orientdb/issues/1604 https://github.com/orientechnologies/orientdb/issues/1603 https://github.com/orientechnologies/orientdb/issues/1602 https://github.com/orientechnologies/orientdb/issues/1626

lvca commented 10 years ago

@laa Is this fixed?

andrii0lomakin commented 10 years ago

No, till we fix issues listed in https://github.com/orientechnologies/orientdb/issues/1404#issuecomment-22461169

lvca commented 9 years ago

@laa What's the status of this?

andrii0lomakin commented 9 years ago

we may close this issue