search

vkuznet / transfer2go

Distributed, loosely couple agent-based transferring system
MIT License
8 stars 2 forks source link

Test how much data sqlite db can hold #14

Closed vkuznet closed 7 years ago

vkuznet commented 7 years ago

We need a test how much records can single sqlite db hold which is used by transfer agents.

rishiloyola commented 7 years ago

Can you guide me to implement this testing method? How can I test this? Do I have to continuously upload data using /tfc end point? I guess the capacity of SQLite DB can vary machine to machine.

vkuznet commented 7 years ago

You don't really need Go-testing method for that. It is simple test we need to do to measure capacity of SQLiteDB and perform benchmarks for read/write.

You can write simple bash script for that and inject as many data as you want. Measure the injection and look-up times (for later we probably need indexes in DB).

The choice of SQLiteDB was simplicity of its usage and deployment, i.e. I don't need to run a dedicated database which requires a server, port, and possibly dedicated hardware. The SQLiteDB is DB in a plain file. But it is fully SQL complaint and therefore code developed against SQLiteDB can be easily ported to another DB back-end if necessary.

On 0, Rishi notifications@github.com wrote:

Can you guide me to implement this testing method? How can test this? Do I have to continuously upload data using /tfc end point?

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/vkuznet/transfer2go/issues/14#issuecomment-300832471

rishiloyola commented 7 years ago

Do we need to store data on disk? What do you think about in-memory SQL database? Or may be in memory sql-lite option?

vkuznet commented 7 years ago

I'm not sure what are you asking?

If you refer to keep database in memory it is not persistent, server can go down and with its outage the database content is gone.

We should be able to shutdown the server and start it again such that it reads the metadata from SQLiteDB (or any other DB).

But if you asking about storing real data into DB then here is my answer. There is no way you can store PB in memory, a typical CMS site will have TB-PB disk space.

Plus we're not storing data to database, the database keeps metadata info. The data itself are stored on local disk/storage.

That's why we have LFN (Logicial File Name) which is a unique alias, e.g. /file.root which points to PFN (Physical File Name) which points to real file on local file system/storage.

On 0, Rishi notifications@github.com wrote:

Do we need to store data on disk? What do you think about in-memory SQL database? Or may be in memory sql-lite option?

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/vkuznet/transfer2go/issues/14#issuecomment-300890928