search

griddb / griddb

GridDB is a next-generation open source database that makes time series IoT and big data fast,and easy.
https://griddb.org/
GNU Affero General Public License v3.0
2.37k stars 4.95k forks source link

Provide a camparison with Elasticsearch #266

Open BigGitWorld opened 4 years ago

BigGitWorld commented 4 years ago

I'd like to use the GridDB in an IoT company to store millions of sensors data. I searched a lot in the Google and find that the ELK stack (Elasticsearch, Logstash, Kibana) is very popular in the IoT collecting data. I read the GridDB method. it seems is very nice and smart. But I doubt to start it in a real world IoT project with thousands of IoT sensors. Can you please provide a camparison between the GridDB and the Elasticsearch?

BigGitWorld commented 4 years ago

No Answer? I think you should be up to date. it seems the griddb is an archived database and I scare to use it in my own company!

AnggaSuherman commented 4 years ago

Hi @BigGitWorld , we have not done a direct comparison with Elasticsearch, but you can check our developer site for performance benchmark with other leading databases: griddb dev site You may want to consider GridDB if performance and petabytes scalability is of your concern.

BigGitWorld commented 4 years ago

Thanks for your reply. But the main problem is that only you have mentioned these features, but there is no practical comparison with tangible results. I think it would be great if you could put both real comparison tables and more support. For example, creating a Q&A community would be a good option.

knonomura commented 4 years ago

Sorry for late reply.

Roughly speaking, Elasticsearch is search engine and GridDB is database.

GridDB has been designed and developed as scalable database for IoT system, has the following features:

There are the following description in db-engines.com.

https://db-engines.com/en/system/Elasticsearch

Description: A distributed, RESTful modern search and analytics engine based on Apache Lucene Primary database model: Search engine Secondary database models: Document store Consistency concepts: Eventual Consistency. Get by ID may show delays up to 1 sec. Transaction concepts: no

https://db-engines.com/en/system/GridDB

Description: Scalable in-memory time series database optimized for IoT and Big Data Primary database model: Time Series DBMS Secondary database models: Key-value store, Relational DBMS Consistency concepts: Immediate consistency within container, eventual consistency across containers Transaction concepts: ACID at container level

Specially, we adopt Key-Container data model.

For Key-Container data model: https://griddb.net/en/docs/documents/3-2_key-container-model.php

You can find GridDB FAQ here: https://griddb.net/en/faq/

BigGitWorld commented 4 years ago

@knonomura Than you for your response.

Elasticsearch is not only a search engine, but also a database that works by inverted index mechanism to do fast search and query.

I'm agree with you that the gridDB key-container model is a theorically good solution for the IoT scenario. But Can you tell me please :

  1. What can your database do that Elasticsearch can't do?Please explain in a practical approach not only theory.
  2. how the search time on Hundreds of millions of data using your database is different from elasticsearch? I can query/search on elasticsearch in some milli seconds.

Therefore, although the elasticsearch does not support key-container model, but due to its fast search and aggregations, we can do different query/searches very simple.

Also, I was thinking about an architecture so that use both the elasticsearch and the gridDB simultaneously in infrastructures of our company, but could not understand why do I have to use both of them while the elasticsearch is working fine.

If you can convince me, I promise to be a great advertiser for your database.

sincerely

ostampflee commented 4 years ago

I've implemented GridDB as the database for several solutions that have hundreds of millions of records. I've only briefly used the ELK stack for log/system metric analysis and was not impressed by it's performance nor scalability for that small workload.

What can your database do that Elasticsearch can't do?Please explain in a practical approach not only theory.

GridDB will require less storage and be faster on cheaper hardware. We haven't done a direct comparison and I could not find a direct comparison of ElasticSearch with a database compared directly with GridDB but you can use the following three papers to see the relative performance advantage:

ElasticSearch and MongoDB:

MongoDB and Cassandra (Now behind a email-wall):

Cassandra and GridDB: https://griddb.net/en/docs/Fixstars_NoSQL_Benchmarks.pdf

how the search time on Hundreds of millions of data using your database is different from elasticsearch? I can query/search on elasticsearch in some milli seconds.

I have a dataset that is nearly half a billion records and my most common queries are completed in a couple of milliseconds with three 2 vCPU 8GB memory instances. Converting the data to JSON is nearly always slower than querying and fetching from GridDB.

BigGitWorld commented 4 years ago

@ostampflee Thank you so much for sharing your idea.