Modeling friendship with hasMany on this side

[ivanseidel]

I'm using MongoDB, and I have a model that looks like this (in my mind):

User: {
   name: ..., email: ..., [...],
   facebookId: ...,
   facebookFriends: [<faceId1>, <faceId2>, [...]]
}

After doing some tests, I guess that I figured out that associations are only done in the (other) side if it's a hasMany. What I need, is a hasMany on this side... Is it currently possible?

If not, is there a (nice) way of implementing this on my model, other than using MongoDB direct query for fetching all it's facebookFriends, or fetching one by one?

[danfinlay]

Oh hey Ivan, extra point: That eager-loading of nested associations only works with relational databases. If you want that kind of convenience, you should use Postgres or MySQL.

If you want a hasMany to hasMany, the traditional method (in a relational database) is to use a "join table", or a model where each entry hasOne of each of the models you want to haveMany of the other.

Example: Person has many Parties, Parties have many People.

You might have a model called PersonParty, and it hasOne('Person') and hasOne('Party'), and now you can use through associations to declare things like this:

User.hasMany('PersonParty');
User.hasMany('Parties', {through: 'PersonParty'});

If you do that on both sides of the relationship, you'll be able to say user.getParties() like you'd hope.

Sorry, that's a real short crash course on a pretty big topic, it'll probably take some trial and error but hopefully it helps.

[danfinlay]

If you just want to use Mongo, maybe serialize an array of IDs on each model? It's just JSON, so you can do whatever you want.

[ivanseidel]

I see...

You might understand my needs (why i'm doing it in such a way) in this question: http://stackoverflow.com/questions/25006643/who-has-more-related-friends-in-fb

Since Neo4j is pretty new and doesn't has good frameworks to use with, and also, my only usage would be for that specific point, I'm trying to port it to my needs with MongoDB.

Using a Join model would definitely solve it, but I would end up with lots of documents more... Adding it as an array, would give me the advantage of loading it pretty easily, and doing that computation really fast on the server...

I guess I should implement my own friendship methods on the model to load/associate with friends...

[danfinlay]

Oh, I'm not a graph database expert, so I can't really tell you the best way to do this. A lot of it will definitely depend on how big of a dataset you're using. If you aren't using a huge dataset, best practices aren't as important, and even getting lots of join table entries isn't the worst thing.

If you're doing a big data study, then picking the right tools for the job might be very worth it.

One way you could do this with Model would be using its streaming interface. Since you're just counting relationships to a user that are on your list, if you have that "whitelist" loaded, you can easily count the entries without having to hold all entries in RAM by streaming them out of the database, and just checking and counting them as they stream out.

This is another Model feature that is relational-DB only.

[ivanseidel]

That's interesting (world is conspiring to make me use Relational-DB?)!

I will save it as an array in the Model. It won't need associations, so we can only process it locally... Loading multiple Users can be achieved by adding multiples or's on a find query. Sounds like a good plan? I guess there won't be more than 20 in a single query...

[danfinlay]

No need to use many ORs, if you include an array of many values instead of a value they will all be matched for.

{name:['foo','bar']}

[danfinlay]

Relational DBs are really awesome, I would need some great reasons for a project to use something else.

That said, it looks like neo4j has a pretty mature node module. You could use this in the model file with Geddy, and hang neo4j on the geddy global object during init if you wanted to access it in any controller.

[mde]

@ivanseidel Worth considering the fact that in this case, you're looking at Friendships, which represent a relationship between your models. This is, in fact, relational data, and using a relational DB is a totally appropriate solution to the problem. In my experience, using a purely document DB for this type of data and analysis means you end up running lots of time-consuming map/reduce queries where the equivalent SQL query would actually be considerably faster.

We actually have some tests in our test suite that do something very similar to what you're talking about:

https://github.com/geddy/model/blob/master/test/integration/adapters/shared.js#L967

It's a question of right tool for the job. If you need to slice and dice the relationships between models, you should use a relational database. If you're storing totally unrelated data, a document DB is a great fit.

[ivanseidel]

@flyswatter : The gain using Neo4j would only be for that purpose of computing friendship associations. Since all other data wouldn't need such process on nodes, any database is Ok for me.

@mde : You have a valid point. All my models are linked to each other... Using Relational DB would gain me speed in querying, while using MongoDB would give me flexibility with my data...

My models are pretty connected, but I'm not sure if using Relational DB would give me freedom later. For every field I want inside a model, that would be a column...

On the other hand, looking at prices of DB's in Heroku, PostgreeSQL has a really attractive price for a low DB size (up to 10M rows) costing about 9$: https://addons.heroku.com/heroku-postgresql#hobby-basic

While MongoDB, is not expensive as well (18$), and would give 1G of data, which is more than I would ever use (at least I guess): https://addons.heroku.com/mongohq#ssd_1g_elastic

[ivanseidel]

I guess that I have a solution for all that.

There are only 3 models that requires this kind of Association:

• Publishes.users, which contains an array with User Id's
• Photo.tagged, which contains an array with Tag Id's
• User.friends, which contains an array with User Ids (might be the FacebookId)

Creating a helper such as belongsToMany, that injects those methods inside my Model. Like extending it... Would look like this:

belongsToMany.(geddy.model.Users, 'friendship', geddy.model.Users, 'facebookId');

Which would Generate (with the name passed in the second parameter):

• getFriendships, which returns the array in sync, and if a callbackfunction is assigned, it will associate and callback it.
• getFriendship(index), returns the id at index, and if callback is set, finds that model and returns it
• removeFriendship([f1, f2, f3,...]), removes those friendships, and callback

And could also, generate on the other side, the findFriendships, findPhotos...

It would work with Mongo, and also with PostgreeSQL, if in case, I switch to it later... Sounds like a good idea?

[ivanseidel]

Ok. I guess I will push this to npm also: https://gist.github.com/ivanseidel/cf2bf4d3ef405e90122f

It's pretty simple, but helps with this kind of association...

What is missing is the through association. I don't need that (yet)...

[mde]

The flexibility is great at the beginning when you're not sure what your data model looks like. It becomes a huge drawback as you write more and more code that needs to be able to have some basic assumptions about that fields a model will always have. You add a field at some point, and unless you write some sort of backfill script, none of your older documents in a non-relational DB will have that property. It's a real pain in the ass. Having empty cols in a relational DB isn't any sort of problem, and the SQL migrations in Geddy make it simple to keep your schema in sync.

We do see a common pattern of people starting with Mongo (either because they like the initial flexibility, or because the think it's cool), and migrating to Postgres later.

[ivanseidel]

Probably I will do just that, but for a while, I think it's better to stick to the MongoDB. When things get more stable in terms of ideas/implementations, we switch to PostgreeSQL.

Thanks again for the help.

[danfinlay]

@ivanseidel On the perceived drawback of "every field would be a column", and wanting flexibility later:

I've gotten very comfortable keeping my Postgres database in a state of evolution by using regular migrations. I like to think I made this process even a little bit easier with my module Smilegrate, which lets you add/remove a column from a model with a single line in the terminal.

If it doesn't work exactly how you'd want, feel free to file a request or PR. I'm partly considering taking out the geddy jake db:migrate call from it, to allow further modification of its generated migrations before the migration is called.

[ivanseidel]

That's ir really nice @flyswatter ! One of the reasons why I don't use (yet) PostgreeSQL is that it would be hard to maintain synced with my models...

I'm really new to scalable apps and also migrations. But I see that is the perfect tool for Related DB.

Is there a way, for example, to generate a migration from an existing model, pretending to be the first time it will migrate?

Thanks

[danfinlay]

I'm unclear when you would want to pretend to be the first time it will migrate, but basically yes, you could re-write your first migration to reflect the data model you ultimately decide on, although I don't see why you would bother, you ca always re-install the app, and geddy jake db:init and geddy jake db:migrate will get their database all set up.

[mde]

@flyswatter Whaaaaat? That Smilegrate is awesome! Actually Rails is pretty smart about how it lets you generate migrations from the CLI, kind of like what you have here. This is great!

[danfinlay]

Oh, thanks! Someday I'd like to learn @der-on's gens well enough to add migration generators into Geddy core. But I'm on crunch-month at the moment.

[mde]

That would definitely be ideal. No worries about timing. We're in no rush with this stuff. :)

[ivanseidel]

geddy jake db:init will create a migration file with the current model? I thought migrations were applied in sequence, like: first migration, second, third, ...

One other question (out of topic): I'm doing a REST api with all my models, and I will use a pattern for it, in such a way that all of them use the same methods for querying (find, remove, update, create). Is there anything done for this? (I want to avoid copying and pasting the same code in all models)

[danfinlay]

No, sorry I must've misunderstood your question. There are ways, they're not fun, but I'm not really sure why you'd want to flatten your migration history. Maybe make a module for it, probably easier than doing it by hand for a long history :)

To the second point, I'm actually a little unclear, haven't scaffolded in a while, but I'm pretty sure the controllers come with basic CRUD behaviors, isn't that right @mde?

[ivanseidel]

That's right @flyswatter , but they are generated... I don't really like that...

I was thinking about this, and if GeddyJs has something like this, would be amazing! Imagine:

var User = function () {
    // Responds with stuff...
    geddy.util.installRest(this, {
        update: {
            error: myMethod, // Call this in case something goes wrong
        },
        find: {
            action: 'search', // Modify the name of the action
            before: method // Before finding, call this
        },
        destroy: {
            before: myCustomMethodd, // Call it before destroying
            after: myCustomMethod // Call it after destroying model
        },
        create: false, // Don't create it
    });
}

And in the router: router.rest('User');

I'm coding a helper module to do this for me. It can even include pagination/queries on find, and in this way, all the models that use it, will have the same pattern. What do you think @mde @flyswatter ?

[danfinlay]

I like it a lot for many types of applications.

I think it's a bit funny that to avoid generated code you're writing what sounds a lot like a code generator.

I also think you should take a glimpse at loopback.io for inspiration, they have a fully declarative, JSON-config-file based REST server that I think shows a lot of vision, except that they built it on Express ;)

[ivanseidel]

@flyswatter it's not a code generator... It's just a singleton (not sure if that's the name) for manipulating/outputting model's data.

It's one of the few things Sails.js has... But they do it in the wrong way: letting it not be generic (no callbacks, no customisation...)

(let me finish it, and then you will understand better)

[danfinlay]

Cool, nice to have someone bringing over the best Sails has to offer :)

[ivanseidel]

done @mde and @flyswatter : https://github.com/ivanseidel/geddy-rest

I guess that it can be used to improve geddy as well, if I can help with anything, just let me know!

[danfinlay]

I like that! Especially since you provided before and after hooks, it would be very easy to use this as a jumping off point for further customization. I'll try using it on a personal project soon.

[ivanseidel]

That's the idea!

With it, it's also possible to do some advanced stuff, such as generic pagination on find, batch creation...

[danfinlay]

How about filtering with query params? Seems a natural fit as well!

[ivanseidel]

That's totally posible.

Could be something like pass in the query params as an object (literally). It's like: 1 line more of code, and could be in the options to enable/disable it.

[ivanseidel]

@flyswatter and @mde : I implemented a simple nested findings with REST, you want to check it out: https://github.com/ivanseidel/geddy-rest#nested-findings

In that way, it's really simple to do things like:

GET /users/:id/photos
GET /photos/:id/owner
GET /users/:id/friends
GET /users/:id/anyThing (with association)

[danfinlay]

That's really cool @ivanseidel! I'm in some deep work this week but I'll be looking forward to playing with your module!