search

Kinto / kinto

A generic JSON document store with sharing and synchronisation capabilities.
http://docs.kinto-storage.org/
Other
4.32k stars 421 forks source link

SQLite Support #1668

Open zeeshanabid94 opened 6 years ago

zeeshanabid94 commented 6 years ago

Would it be beneficial for kinto to have SQLite support? I am thinking of doing some part time projects and this seems like a good project for me since I have a good background in databases.

Natim commented 6 years ago

It's definitely feasible and we even considered moving the Memory backend to an in memory SQLIte database in order to support transaction with the memory backend.

I am wondering if you should start it as an external dependency though like for instance: https://github.com/Kinto/kinto-redis

zeeshanabid94 commented 6 years ago

Thanks for that link. It looks like I could develop this as an external dependency.

I will start planning and working on this in my past time, so the timeline is indefinite for this project. I will need help and suggestions about kinto and its workings. I will keep posting my questions here.

Anything I should know before starting this project apart from contributing.md?

zeeshanabid94 commented 6 years ago

@Natim @leplatrem

I have looked at kinto-redis and these are the ideas/Questions that I got from the code base of kinto and kinto-redis. Correct me if I am wrong:

1- The base classes including storagebase, cachebase and permissionbase define behavior for the derived classes. Correct? They are there to make sure a standard API is exposed. 2- Working with storage, I do not need to know how Cornice and Pyramid are working in getting the request and processing it. Correct? Although I am aware that in the future if I want to add a middleware to process a request, I will need to learn the docs of pyramid and cornice. 3- The cache in kinto-redis seems to be using redis as a default. Can I make the cache for SQLite in memory and the storage on the physical drive? This would make retrieval fast but also provide a consistent storage for testing purposes as that would mitigate the need to reload test data into the storage.

Again, Thank you for your time. I love to code, and this is my first time trying open source.

Natim commented 6 years ago
  1. Yes
  2. No you don't need to learn anything about Cornice and Pyramid for this storage implementation. For the middleware maybe a little bit
  3. You should let the user configure where they are planning to store their data. But in both case it should be possible to load it from memory or from a file.
zeeshanabid94 commented 6 years ago

Hi guys,

I am working on this. I am making progress. I have understood testing it as well. I opened an issue with one of the test cases in kinto.core.storage.testing. Have a look and let me know if I am wrong or the issue is real.

https://github.com/Kinto/kinto/issues/1711

zeeshanabid94 commented 6 years ago

@Natim @leplatrem

Hey guys. I pushed an initial version on my repo. It involves the basic methods to create, update, insert and delete records. I am working on the more complicated methods get_all, delete_all and others. I am making these methods pass the base tests first. Once they pass the base tests, I will write SQLite specific tests. Have a look, review the code, let me know your questions and validate if I am going on the right track.

Also, thank you for the patience, feels good to be contributing to an open source project.

https://github.com/zeeshanabid94/kinto-sqlite/tree/initial-code

zeeshanabid94 commented 6 years ago

I haven't heard from you guys, but here is an update. So SQLite doesn't support json blobs. It has support for simple text blobs. Hence, I serialize and deserialize on writing and reading. But now in get all, I can not filter based on fields and sub fields of the data. Any workarounds or suggestions for this?

Natim commented 6 years ago

I guess you can load the JSON in memory and use a similar mecanism to the one we use with Redis and the memory backend to do the filtering. (You might be able to inherit from the Memory Base Storage class and just use the existing functions.