Implement NoSQL Storage for ActiveRecord

[cebe]

Will implement:

redis (versions >= 2.6) (pull request: #905 merged)

• http://redis.io/topics/protocol
• https://github.com/nrk/predis/blob/v0.7/lib/Predis/Network/StreamConnection.php#L70
• https://github.com/Shumkov/Rediska
• https://github.com/ptrofimov/tinyredisclient/blob/master/src/TinyRedisClient.php
• phpnode: https://github.com/phpnode/YiiRedis/blob/master/ARedisRecord.php
• https://github.com/jdp/redisent/blob/master/src/redisent/Redis.php
• Cluster: https://github.com/colinmollenhour/credis
• [x] Redis Connection Class
• [x] Redis Active Record Class
• [x] Redis Active Query Class
• [x] Redis Extended Active Query Class (where, order, ...)
• [x] Redis Relation Concept
• [x] Redis Relation Implementation
• Redis Transaction Class (will not be implemented as retrieving values from redis is not possible in a transaction)
• [x] Redis Cache (ready on master)
• [x] Redis Session (via yii\web\CacheSession and yii\caching\RedisCache)

elastic search (pull request: #1295 merged)

• elastic search: https://github.com/phpnode/YiiBlocks/tree/master/elasticSearch
• [x] Connection Class
• [x] Active Record Class
• [x] Active Query Class
• [x] Relation Concept
• [x] Relation Implementation
• [x] Transaction Class(?)

mongodb (pull request: #1438 merged)

• http://docs.mongodb.org/manual/reference/sql-comparison/
• http://docs.mongodb.org/ecosystem/drivers/php-libraries/
• http://snmaynard.com/2012/10/17/things-i-wish-i-knew-about-mongodb-a-year-ago/
• http://www.yiiframework.com/extension/mongoyii/
• http://php.net/manual/en/class.mongoclient.php
• https://github.com/Sammaye/MongoYii
• https://github.com/canni/YiiMongoDbSuite
• [x] Connection Class
• [x] Active Record Class
• [x] Active Query Class
• [x] Relation Concept
• [x] Relation Implementation
• [ ] Transaction Class(?)

Sphinx (Pull Request merged)

http://sphinxsearch.com/

Other systems that may get implemented (suggestions welcome):

• solr: https://github.com/phpnode/YiiSolr
• neo4j: https://github.com/Haensel/Neo4Yii
• couchdb: http://couchdb.apache.org/
• orientdb
  • http://www.orientdb.org/
  • https://github.com/AntonTerekhov/OrientDB-PHP
  • http://odino.org/graph-in-php-through-orientdb/
  • https://github.com/nuvolabase/orientdb

Other resources:

• https://github.com/Haensel/ActiveResource

[cebe]

Btw the checkboxes in here are made like this: https://github.com/blog/1375-task-lists-in-gfm-issues-pulls-comments

[oherych]

Most noSQL databases have connection via REST. Will be greate have one defined interface for all Rest client. This give us able have one selected (by developer) REST library for all needs. We just say with which library it should use

[Aoseala]

Personal think elasticsearch PHP client, Elastica is very good! this is his github repository URL: https://github.com/ruflin/Elastica

[ghost]

You mentioned 2 of my yii extensions so I might share my thoughts/experiences on this issue here: The whole noSQL topic is far too complex to make a "one fits all" solution even possible (I've tried and failed :-) ). I wanted to make things easier with my ActiveResource extension but there are a lot of design flaws in there so it is a bad candidate for reviewing. My suggestion for Yii in this respect: Take a look at Guzzle which is a REST client framework. I takes care of the annoying parts (authentication, caching, asynchronous/parallel curl requests), is extensible via plugins and actively maintained. Yii could offer an interface on top of Guzzle so that there is a well defined API to develop against, maybe in combination with the repository pattern

[DaSourcerer]

As I see Guzzle being mentioned here: While Guzzle is indeed a formidable solution with facilities that make inclusion into Yii trivial, I'd like to advise caution. As mentioned before, it is not just a library but an entire framework. It would also introduce a hard dependency on libcurl (it is my impression that the original author simply wanted to spare himself from dealing with all the gory details of socket programming). Just saying :smirk:

[rawtaz]

This is almost getting a bit OT, but I have to say: +1 for the repository pattern. Been missing it in Yii a lot of times.

[dmill-bz]

I would love to see HyperDex in this list. I know this is a bit of a longshot but it's got good performance, supports ACID transactions and has strong consistency guarantees and fault tolerance to boot.

It's outperformed MongoDB, Cassandra, and I believe Redis in the YCSB (despite better consistency guarantees and fault tolerance). It's only downside really is that it's relatively young compared to other solutions.

Here are some links:

http://hyperdex.org/doc/ http://hyperdex.org/performance/ http://hyperdex.org/performance/details/ (more details regarding benchmark) https://github.com/tritondigital/php-hyperdex

Alternativly I could settle with MongoDB even though I'm uncomfortable with some aspects of the data storage (or non-storage in this case)

[klimov-paul]

Implement NoSQL Storage for ActiveRecord

I am not sure how exactly such implementation should look at the end. I can not see an universal solution for this. Different NOSQL storages use different approaches for the data storage and data structure. There are several types:

• key-value oriented (Redis)
• document oriented (MongoDb)
• graph oriented (Neo4j)

Key-value storages could match the original Yii model paradigm, which relies on architecture, when model has only (or at least mostly) scalar type attributes.

However document oriented storage does not fit the Yii model paradigm. In such storages (as MongoDb) the single storage entry is a huge nested JSON document, which can not be described by simple model with scalar attributes.

Graph oriented database (such as Neo4j) actually could fit the Yii model paradigm. However the search inside such storages is rather different from the common database. Here you are starting with some node and then go to the nodes related with it. Sometimes universal search criteria can not be composed for fetching particular node with the single query. But the main problem with such storages as Neo4j is they do not support PHP directly and there are no PHP drivers for them. Such storages can be accessed only via REST API.

[cebe]

@Haensel commented in https://github.com/yiisoft/yii2/issues/73#issuecomment-16884173: I don't know if it is still the case but if you are still planing to implement a mapper for graph databases (Neo4j, OrientDB etc.) I would suggest you to take a look at Tinkerpop Blueprints (https://github.com/tinkerpop/blueprints/wiki). From the wiki:

"Blueprints is analogous to the JDBC, but for graph databases. As such, it provides a common set of interfaces to allow developers to plug-and-play their graph database backend. Moreover, software written atop Blueprints works over all Blueprints-enabled graph databases."

Neo4j and OrientDb both support Blueprints and are the most widely used graph dbs, so it might be worth a look

[cebe]

In case you wonder where to find this: I'm on the redis branch: https://github.com/yiisoft/yii2/tree/redis

[dmill-bz]

Is there an interface or required functions for Connection class? I would like to implement support for Rexster (as per the tinkerpop comment earlier). I've already made a php client for RexPro (which part of is very similar to the Yii Connection.php). I'd like to use it as a helper as a first step even if it's redundant (because I might move on to making a php extension to support rexPro. I'm going to start now just trying to stick as much as possible to the current file/functionality.

[cebe]

Is there an interface or required functions for Connection class?

Will be there but I am not finished with that yet. Will define the interface or base class, when we have more dbms implemented so I'd be very interested in seeing your class.

[dmill-bz]

ok! I haven't gone very far yet. I've used your Connection class as a basis and have made one for Rexter. I've left out Transactions and the concept of Isolation (or non-isolation in this case which allows for queries to use bindings set by previous queries) because it will probably need a bit of discussion . I'll get onto looking into activeRecord and ActiveQuery classes next. Once I get ideas as to what to do about representation of Vertices vs Edges and routes and so forth. I'll probably post my suggestions. (I also should have a look at Neo4Yii to see how Hansael tackled it)

[dmill-bz]

ok having a look around. How would you suggest modifying get() and set() for shema-less db structures in activeRecord? This is kind of tricky as it'll catch all.

[dmill-bz]

graph DB and gremlin discussion for graph implementation here : http://www.yiiframework.com/forum/index.php/topic/43863-graph-databases-gremlin-qquery-building/ I reckon I should have something to push online by the end of the week depending on how the discussion goes. a preview version if you will

[cebe]

Sadly haven't had the time to go there, but there is some really interesting stuff at berlinbuzzwords conference this year http://berlinbuzzwords.de/ .

http://berlinbuzzwords.de/program/session-schedule/all

• http://berlinbuzzwords.de/sessions/document-relations-elasticsearch
• http://de.slideshare.net/martijnvg/document-relationsbbuz2013
• http://berlinbuzzwords.de/sessions/couchdb-implementation

[Sammaye]

I don't mean to plug my own content here but: https://github.com/Sammaye/MongoYii can handle MongoDB in a schemaless manner and I have the intention of converting it to Yii2 as soon as a stable interface for its conversion is made.

Maybe a stripped down version of something like that could be used for core?

[cebe]

It's nothing bad about showing your own content when you think it is good. Looks really interesting, will definitifely take a look at it. Thanks for sharing!

[dmill-bz]

I've been very busy recently but finally shifted towards making Yii independent libraries for graph DBs. Mostly speaking a DB client for rexster and an equivalent of a QueryBuilder for gremlin. Now I'm concentrating on integrating these into Yii2. I was wondering where the correct place for external libraries to reside was? I've been sticking them in the helpers dir for now even though that didn't seem correct. Any help would be appreciated. Thanks

[Sammaye]

Normally, for better composer integration, many have taken to adding such things to a vendors folder. This pattern is, of course, also replicated in many languages including Java.

[dmill-bz]

Yeah, I couldn't seem to find one though. Doesn't matter, I'll place them there for now and move on. I'll move them around once I get a better idea.

[jabbon]

Support for MongoDB +1 (ActiveRecord, Sessions) Support for Redis +1 (Cache, Sessions)

[cebe]

Support for Redis +1 (Cache, Sessions)

@jabbon this is already implemented and available on master branch.

Support for MongoDB +1 (ActiveRecord, Sessions)

does it really make sense to store Sessions on MongoDB?

[Sammaye]

Yes, MongoDB, if done right, can store the session completely in working set, i.e. RAM just like redis

[qiangxue]

Suggest moving /db/redis to /redis because redis is relatively independent of the other classes under db.

[cebe]

The current classes are but the AR does extend from the db\ActiveRecord I think we should make a decision when redis an elasticsearch are ready.

[qiangxue]

By having redis directly under the framework folder, it's like making it a self-contained extension with dependency on Yii. Other directories under db mainly contain drivers which are not the same as redis.

[cebe]

Agree. Will make the move soon.

[mcd-php]

I vote for Apache Cassandra inclusion - it uses SQL-like language ( CQL ), has PDO driver, i made it work for my closed-source job project.

On the other side, my piece for it may be application-biased (or have some parts of application inside it), and ignorant of some framework guidelines. It's also ridden with backward-compatibility kludges, since i converted from Yii1 and from PHPCassa to PDO - two overhauls far before start ;).

So i it may be better if core team write their own, stumble upon all lack of dependency injection, and ask me questions if stuck. Or should i go over all my code and file issues on all stumbles i had to dodge ?

Anyway, i am willing to help with this.

[qiangxue]

It would be great if you could submit a PR for this, and we can work together to polish it (unless other @yiisoft/core-developers has experience with Cassandra and willing to work on this). Thanks.

[cebe]

I think we can close this issue now. I am done with all the system I know and can support as I am going to work with them on a regular basis. Others should be provided as extensions or by other core developers. If someone wants to start working on an implementation of an AR feel free to contact me, I can help with that.

[qiangxue]

This is a wonderful piece of work! Thank you, @cebe and @klimov-paul

[cebe]

:-)

[basil-inc]

great work guys, thanks!

[Sammaye]

I will probably start work on a Solr extension in a min, have need of one and it appears there isn't one yet when I search Google.

[cebe]

We decided to use elasticsearch in favor of solr, as it is much more powerful. But that shouldn't stop you from creating a solr extension :)

[nineinchnick]

I also use Solr and would help to write/test/review such extension.

[Sammaye]

I have just started on a Solr application component which extends from the Pear PHP Solr Client I have started here since I am still quite new to PHP coding in Solr (it is normally another guy who does it, I am an ES and Sphinx guy) but I wouldn't mind someone (like you or whatever) with the knowledge taking something like: https://github.com/phpnode/YiiSolr and making something pure awesome from it.

The idea is to eventually get to that but as I said, I am still a newbie to Solr but it is a requirement of my application so I have to port it over.

[cebe]

So what do you need to get started with it?

[cebe]

You may take the elastcisearch implementeation and mark all code that needs to be adjusted with todos. After that starting to implement connection and basic querying.

[Sammaye]

Yeah I think that's a good idea actually thanks

[Sammaye]

I will probably strip the ES code and then try and copy the Pear PHP client over, though not too much since that code is old and decreped so...yeah, just take the basic requirements of Solr (escape functions etc) and then build it into the rest that exists in ES code

[rawtaz]

There's http://www.solarium-project.org/ too. I haven't looked at the code of the PEAR extension you mentioned, but given how completely awful the majority of other PEAR extensions have been through the years I ever looked at them, perhaps this one is a cleaner code base to use for your implementation (assuming the license is compatible).

[Sammaye]

Ah yeah, that looks much better I'll take a look and see what I can with it, maybe use that as the base and the Yii2 classes as abstraction

On 25 April 2014 12:11, rawtaz notifications@github.com wrote:

There's http://www.solarium-project.org/ too. I haven't looked at the code of the PEAR extension you mentioned, but given how completely awful the majority of other PEAR extensions have been through the years I ever looked at them, perhaps this one is a cleaner code base to use for your implementation (assuming the license is compatible).

— Reply to this email directly or view it on GitHubhttps://github.com/yiisoft/yii2/issues/11#issuecomment-41381636 .

[Sammaye]

Wow that Solarium is pretty much a framework in itself

[Sammaye]

I am unsure if solarium project is fully compatible with Yii2.

It uses the symfony event dispatcher ( https://github.com/symfony/EventDispatcher ) which I don't think would be callable with Yii2's own events. Not only that but it isn't the lightest piece of code.

Anything built from it would require a lot of hacking in Yii2's internals to make sure Yii2 does not interferer with it.

Basically it will be more of a jerry rig if I build from that due to how complete it is. I will still try though I think, it has all of the functions etc that are needed, but then I might also be able to do these simpler if I coded from start and used the ES base

[cebe]

They should not interferre but there is the question of the need of a yii implementation when solarium has everything you need already. What is missing from solarium what a yii extension would bring?

[Sammaye]

Well I guess it is implementation with Yiis active record etc, I mean I don't really need that, only querying but I am sure others would

[cebe]

Is solr really used as a data storage as it is the case with elasticsearch? If it is not a common usage to create update and delete records in solr it may not be worth the effort of creating AR layer for it.

[Sammaye]

We were recomended by users on the Solr group to actually use Solr as a full data store when querying it, so our entire "books" table is duplicated across MySQL and Solr.

The reason we don't need Active record is because we are usng old code that relies on cronjobs to send xml files to Solr once every so often.

Thugh Solr is more of a search tech than a database...hmmm

On 25 April 2014 13:31, Carsten Brandt notifications@github.com wrote:

Is solr really used as a data storage as it is the case with elasticsearch? If it is not a common usage to create update and delete records in solr it may not be worth the effort of creating AR layer for it.

— Reply to this email directly or view it on GitHubhttps://github.com/yiisoft/yii2/issues/11#issuecomment-41387023 .