Use a database abstraction layer

[martinheidegger]

Strider requires to have a database setup and running, unlike most other CI systems (Jenkins, Teamcity,...) this makes installation & backup more difficult. It would be awesome if this could be optional by using a database abstraction layer like waterline.js

[jaredly]

+1 I fully support abstracting out and isolating all of our db code.

[knownasilya]

What about using waterlinejs. It's pretty flexible and allows you to use many different adapters. There is already a mongodb adapter, and a disk adapter.

[niallo]

I would strongly recommend not using an ORM/ODM for this. Instead, create a data access API (a bunch of functions which do the necessary storage operations) and use that.

On Fri, Sep 12, 2014 at 11:57 AM, Ilya Radchenko notifications@github.com wrote:

What about using waterlinejs https://github.com/balderdashy/waterline. It's pretty flexible and allows you to use many different adapters? There is already a mongodb adapter, and a disk adapter.

— Reply to this email directly or view it on GitHub https://github.com/Strider-CD/strider/issues/559#issuecomment-55445992.

Niall O'Higgins W: http://niallohiggins.com E: n@niallo.me T: @niallohiggins

[knownasilya]

@niallo yeah, might be some good advise. Is that from your django days? :wink:

[niallo]

I have seen ORMs go wrong so many times :) Even Mongoose is probably overkill for Strider. Instead, use low-level driver like mongoskin and just put functions on top.

On Fri, Sep 12, 2014 at 12:42 PM, Ilya Radchenko notifications@github.com wrote:

@niallo https://github.com/niallo yeah, might be some good advise. Is that from your django days? [image: :wink:]

— Reply to this email directly or view it on GitHub https://github.com/Strider-CD/strider/issues/559#issuecomment-55451379.

Niall O'Higgins W: http://niallohiggins.com E: n@niallo.me T: @niallohiggins

[knownasilya]

Then we could use rethinkdb, which IMO is better and more friendly then mongodb.

[knownasilya]

@niallo for that to work well, we need to do a design session of what we need to access and what functions need to be written, so that we aren't writing many unnecessary API functions. So basically design the data abstraction API.

[niallo]

Yep. We would need to sketch out APIs around jobs, projects, users, etc - primarily. The Strider API is quite simple, fortunately! Not many types of objects and behaviours.

On Fri, Sep 12, 2014 at 12:49 PM, Ilya Radchenko notifications@github.com wrote:

@niallo https://github.com/niallo for that to work well, we need to do a design session of what we need to access and what functions need to be written. So we aren't writing many unnecessary API functions. So basically design the data abstraction API.

— Reply to this email directly or view it on GitHub https://github.com/Strider-CD/strider/issues/559#issuecomment-55452154.

Niall O'Higgins W: http://niallohiggins.com E: n@niallo.me T: @niallohiggins

[knownasilya]

Maybe add organizations while we're at it.

[knownasilya]

So we should use js-data.io

[niallo]

My gut feeling here is too much magic. Write a concrete data access layer specific to the application, rather than trying to use a magic layer.

For example, there might be high level functions such as:

create_user
update_user
delete_user

create_project
update_project
delete_project

etc etc. Underneath, these can use any database (MongoDB for today). Or potentially even use js-data underneath. But the key thing is to have a concrete data access API which has well defined semantics.

On Thu, Mar 12, 2015 at 10:43 AM, Ilya Radchenko notifications@github.com wrote:

So we should use js-data.io

— Reply to this email directly or view it on GitHub https://github.com/Strider-CD/strider/issues/559#issuecomment-78543544.

Niall O'Higgins W: http://niallohiggins.com E: n@niallo.me T: @niallohiggins

[knownasilya]

Sounds exactly like js-data, not too crazy and has support for adapters, mongo included.

[niallo]

There's an important different which is that js-data is an ORM (or ODM) not an API. You can do whatever you want, which will lead to performance and architectural problems in the future (similar to Mongoose now).

On Thu, Mar 12, 2015 at 10:55 AM, Ilya Radchenko notifications@github.com wrote:

Sounds exactly like js-data, not too crazy and has support for adapters, mongo included.

— Reply to this email directly or view it on GitHub https://github.com/Strider-CD/strider/issues/559#issuecomment-78547443.

Niall O'Higgins W: http://niallohiggins.com E: n@niallo.me T: @niallohiggins

[knownasilya]

Yeah, I think we are on the same page now :+1:. Time to start sketching out the model API.

[gavinengel]

I'd like to have the option to use a simpler db engine like nedb for smaller projects instead of mongodb. Would resolving this issue make that possible?

[knownasilya]

@gavinengel yes it would. Would love to see some PRs from you :+1:

[knownasilya]

I would structure it as a directory that exports all of the "models", where each model is a file that exports named functions that return promises. We could add a environment variable that would require a specific module instead of that directory and pass it the config.

[davemackintosh]

I've spent the last few months working with Waterline, it's impressive but it has it's limitations/inconsistencies.

Personal experience, avoid Mongoose at all cost except for quick prototyping. Actually, my advice would be to avoid Mongo altogether with Node projects unless you really understand how Mongo works, cursors, memory paging, sharding, blah blah since I've not yet seen a Node driver that doesn't have memory leaks or a shit-ton of weird edge case bugs.

I do like Waterline, it's stable, super fast and it allows tech choice, my main gripe with it is different adapters require different config keys, for instance; Mongo requires username but PostgreSQL requires user..

Apart from that; actually, it's really good. No memory leaks that I've found (without soak testing) and few bugs.

[knownasilya]

I would stay away from a specific ORM and just have a basic setup. People can build off of that with whatever they want. So you could make an abstraction module that uses waterline if you want or anything else. Just implement all of the models and their methods.

[davemackintosh]

So, an abstraction layer on an abstraction layer? Are we building for CI teams or developers?

[knownasilya]

I'm not building the factory factory :laughing: I said that if someone wants to do that, they can waste their time, but I don't think waterline is the solution, and I don't think that mongoose is a problem. The real issue is a tight coupling to the data layer, which is everywhere in the app. I'm for ORMs, but I've yet to see any good ones for node, and that's why I'd rather build something that isn't so tightly coupled and we don't have to worry in the future.

At the same time, this is hardly on my priority list any way, since the datalayer works just fine as is.