marktani commented 6 years ago

This feature requests serves as a central place to discuss development and progress for the MongoDB connector. Please contribute in the comments below. I will update the top comment as consensus forms.

alexandrakl commented 6 years ago

I would like to work on this! Where is a good place to start looking? How to wrap head around the task and break it into its constituent parts? Please provide some guidelines if possible, would be much appreciated! Thanks for your work on this project, it is very cool, reminds me of RESTHeart ;)

cbioley commented 6 years ago

@ookangzheng Have a look at the project README file. It says "Prisma can be used for MySQL Databases out of the box" ;)

sorenbs commented 6 years ago

Hi @alexandrakl it would be great to collaborate on this!

The first step is to map out the kind of queries that are particularly well supported by Mongo as well as queries currently supported by Prisma that does not fit the document model.

Based on this we can design the API for Document-flavoured GraphQL.

Feel free to dump your thoughts in this thread so we can get the conversation going :-)

GilUpstream commented 6 years ago

@alexandrakl @sorenbs I'd love to work with you all on this.

sorenbs commented 6 years ago

Here's an outline of some things we need to consider:

[ ] align with existing Mongo-based GraphQL APIs (vulcanJs)
[ ] active/passive (does schema migration even make sense?)
[x] mutations (can we support the powerful nested mutations? relations?)
[x] queries (What are the query limitations?)
[ ] embedded types and relations
[ ] subscriptions

Take a look at https://github.com/graphcool/prisma/issues/1340 for the API spec for the MySQL conenctor

@GilUpstream @alexandrakl and anyone who want to contribute. It would be great if you could start mapping out some of these questions in this thread.

sorenbs commented 6 years ago

Top level fields

For each type with an id field, the following fields are generated:

Types without an id field are called embedded types and are stored as nested documents.

The id field is represented as _id in the mongo document per normal mongo convention. Prisma does not support having both _id and id fields.

The type User is used as an example throughout this document

Query

user returns a single user
users returns a list of users
usersConnection returns a relay style Connection of users

In addition to the generated query fields, the following field is present:

node returns a single node by id

field name generation

The user field name is generated by taking the type name and lowercasing the first letter

The users and usersConnection field names are generated by taking the type name and lowercasing the first letter and then using the evo-inflector library

Mutation

createUser create a single user
updateUser update a single user
deleteUser delete a single user
upsertUser update a single user if exists, otherwise create
updateManyUsers update multiple users
deleteManyUsers delete multiple users

Note: The API has no explicit mutations for managing relations. Instead relations are managed through nested mutations #1280

MongoDB has no concept of foreign key constraints and cascading deletes. Prisma ensures that relations stays valid and that cascading deletes are enforced.

Examples:

A -> B

Deleting B will fail if the relation is required and there is no cascading delete
Deleting B will succeed if the relation is required and there is cascading delete
Deleting B will succeed if the relation is not required and there is no cascading delete. In this case the relation field will be set to null
Deleting B will (obviously) succeed if the relation is not required and there is cascading delete

Note: MongoDB has no concept of transactions that span more than one document. As mutations that involve relations or mutate many nodes, involve multiple documents, it is possible that the mutation will be only partially applied if the server is stopped before the mutation completes.

Details

type Query {
  user(where: UserWhereUniqueInput): User

  users(
    where: UserWhereInput
    orderBy: UserOrderByInput
    skip: Int
    before: String
    after: String
    first: Int
    last: Int
  ): [User!]!

  usersConnection(
    where: UserWhereInput
    orderBy: UserOrderByInput
    skip: Int
    before: String
    after: String
    first: Int
    last: Int
  ): UserConnection!
}

type Mutation {
  createUser(data: UserCreateInput!): User!
  updateUser(data: UserUpdateInput!, where: UserWhereUniqueInput!): User
  deleteUser(where: UserWhereUniqueInput!): User
  upsertUser(where: UserWhereUniqueInput!, create: UserCreateInput!, update: UserUpdateInput!): User

  updateManyUsers(data: UserUpdateInput!, where: UserWhereInput!): BatchPayload
  deleteManyUsers(where: UserWhereInput!): BatchPayload
}

query fields

where

Same filter as for the SQL connector, except relational filters only work for embedded types. See #1890 All predicates map directly to mongo:

scalar

field: $eq
field_not: $ne
substring matching: contains, starts_with, ends_with: $regex
field_lt: $lt
field_lte: $lte
field_gt: $gt
field_gte: $gte
field_in: $id
field_not_in: $nin

relation to embedded type

field_every: $all
field_some: $elemMatch
field_none: $not $elemMatch

orderBy

Same enum as for SQL connector

orderBy: CATEGORY_ASC maps to sort: { category: 1 }

skip

Integer specifying the number of documents to skip

before

id of the document to start from, going backwards: max

after

id of the document to start from: min

first

Integer limiting the number of documents to return: limit

last

Integer limiting the number of documents to return: limit

Nested Mutations

Mongo supports arbitrarily nested documents but has no understanding of relations between documents.

Prisma supports two ways to model related data:

relations
embedded types.

relations is a great fit for relational databases and Prisma implements embedded types for SQL databases by automatically managing all the tables and foreign keys required.

In the same way, embedded types is a great fit for mongos document model. Prisma implements relations by performing multiple inserts/updates or queries under the hood.

When is a type embedded?

In Prisma a type is considered to be embedded when it has no id field. Types without an id field can only ever exist when referenced from an "owning" type. The mongo connector takes advantage of this fact by using mongos native capability to nest documents. This can result in improved performance compared to SQL.

Note: MongoDB has a storage limit of 16mb per document. This limits the aggregate size of a single node with embedded types. The SQL connector does not have this limit.

Note: The SQL connector is able to change a type from embedded to normal and back again very easily because they underlying storage mechanism is the same. This is not the case for the MongoDB connector as embedded types are embedded in the parent document.

handling of relations

todo: specify how mutations and queries are handled for embedded types

scalar lists

todo: specify how scalar lists are implemented. They should be embedded. Document how each operation maps to mongos primitives. Document storage limit

juicycleff commented 6 years ago

There needs the be a common api or interface so one can switch to different connectors.

sorenbs commented 6 years ago

That's a great point @juicycleff

We plan to support database specific APIs as well a a Core API. Most of the database specific APIs will have significant overlap with the core API making it easy to switch between connectors. For Mongo and SQL specifically we will be able to have almost 100% overlap :-)

Yogu commented 6 years ago

If you need some inspiration for a GraphQL type system of a document database, have a look at cruddl. It's a TypeScript library that serves a GraphQL API similar the one of prisma, but it uses ArangoDB as database (might also be interesting for #1645). It is built to embrace the possibilities of document databasees like embedding. We invented four kinds of object types to support different use cases of embedding and relating objects - like when ids should be auto-generated and whether omitted fields in update mutations should remove them or leave them unchanged.

Considering the similarities between cruddl and prisma, we would look forward to work towards a common "standard" GraphQL API for document databases.

DavyBello commented 6 years ago

Have a look at graphql-compose it's a toolkit for construction of complex GraphQL schemas. Provides bunch of plugins for type generation. Allows to build your own plugins/helper-functions for type construction and modification and graphql-compose-mongoose a Mongoose model converter to GraphQL types with resolvers. They also have type generator plugins for elasticsearch and aws

duytai commented 6 years ago

@sorenbs in my opinion, we should consider a model without embedded type to remove the complexity of mongodb connector. As far as I know:

Current generated Prisma api does not support embedded type
We just implement connector to support current Prisma API

sorenbs commented 6 years ago

@duytai i agree.

The term embedded type is used here only to describe a certain way to lay out your data. Prismas SQL connector has no understanding of that term and neither will the Mongo connector. The final api will be almost identical apart from less powerful filters.

0xDaksh commented 6 years ago

any updates on this?

ojongerius commented 6 years ago

Is someone working on this at the moment?

marktani commented 6 years ago

Yes, we are 🙂

We plan to start a preview for the MongoDB connector later this month. Please reach out to me in Slack (@nilan) if you're interested in participating.

lfades commented 6 years ago

An update for this note:

Note: MongoDB has no concept of transactions that span more than one document. As mutations that involve relations or mutate many nodes, involve multiple documents, it is possible that the mutation will be only partially applied if the server is stopped before the mutation completes.

Transactions will be supported in 4.0

lfades commented 6 years ago

todo: specify how mutations and queries are handled for embedded types

In Mongodb joins/relations are not recommended for cases where the related documents don't change often, better said, we have a lot more of queries than mutations, for example, suppose that our User has a books fields which can be a one-to-many relation or just an embedded field with the books instead, A Book will probably never change so we don't need a one-to-many relation here.

I believe embedded types can be handled in the same way Prisma handles relations, but every embedded document should contain an id (and the operations for update/remove/insert will be using different mongodb operators). In most cases embedded documents will contain an id, or something that will allow them to be identified inside the array (like the index), if the embedded document is just an object and not an array, all operations will be for that document.

nhuesmann commented 6 years ago

@sorenbs Have you considered including any mongoose-specific abstractions like virtuals and references/reference population (i.e. using references as an alternative to embedded documents)? I've really liked utilizing mongoose's populate operator to keep document sizes to a predictable limit, avoid arrays growing without bound, and normalize the data as much as possible. Virtuals/getters and setters are more of a nice-to-have but have still proven to be handy in certain situations, acting like Redux selectors.

I know the purpose of the mongodb connector isn't to be a mongoosejs connector, however I anticipate a large number of developers having likely interacted with mongoosejs as much if not more than the mongodb API directly, so having some familiar features might make the adoption process easier.

Also, do you plan to support GeoJSON object types for geospatial queries? They've been indispensable in a few projects and I'd really hope for them to be supported.

heysailor commented 6 years ago

+1 for GeoJSON. My primary use case for mongo is to enable efficient geospatial queries. These complex queries can be done with mongo directly, but the ability to resolve GeoJSON data through Prisma would be very handy.

debianw commented 6 years ago

hi @marktani. Do you have this MongoDB connector in a specific branch, what's the name of the branch. I would like to take a look to the code. Thanks

heysailor commented 6 years ago

...just to clarify my earlier comment re GeoJSON, the key to allowing location queries is a location field on the document. It'd be great if the Prisma SDL included a type such as GeoPoint (as it does other types, like DateTime). Then a mongo geolocation query could be done independent of Prisma, or even awesomer, through Prisma with its paging support.

ArvinGanesan commented 6 years ago

@alexandrakl,@sorenbs - I would love to contribute to this project. I hope that it's not too late. Currently, I'm studying data science at Berkeley. And I have been a fan of this project. Full Disclosure - I only just got started with Scala. I would love for this to be a learning experience.

sorenbs commented 6 years ago

@ArvinGanesan - it is never too late to contribute :-)

I think the MongoDB connector would be too big of a mouthful for you to tackle if you are new to Scala. As a first step I would recommend that you clone the repo and get to a point where you can run the tests locally.

Issues tagged good-first-issue https://github.com/graphcool/prisma/issues?q=is%3Aopen+is%3Aissue+label%3Agood-first-issue might be a good place to start. Alternatively you could find a reported issue that you find interesting and create a test case that demonstrates the issue. This will help us prioritise and fix that issue and it helps you get an understanding for the code base.

Please keep in mind that the code base is moving fast and we might not be able to merge your PR.

@debianw We will begin implementing the MongoDB connector in a few weeks and will link the branch from this issue so you can follow along. Our initial plan was to begin the implementation last month, but we had to rearrange some work tasks.

@heysailor - thanks for your input on GeoJSON. This is definitely on our radar!

ArvinGanesan commented 6 years ago

@sorenbs - I ran in to some sbt build error. I did create an issue for that - sbt build error #2349

Any help is appreciated!

jay-jlm commented 6 years ago

Is this being developed in a public branch somewhere? I'm curious on how it's progressing since the initial delivery timeframe was back in April 🙊. We have a new project rapidly approaching its prototype phase and we are asking ourselves if we should count on this being available by then.

sorenbs commented 6 years ago

@jay-jlm We are planning to release a public roadmap soon. Although our planning is not final yet, you should expect a beta release of the MongoDB connector in August.

bazaglia commented 6 years ago

@sorenbs I read in prisma blog that version 1.11 (next one) will include the beta support for mongodb. Does this roadmap still the same?

sorenbs commented 6 years ago

@bazaglia that is inaccurate - sorry about the confusion. August is the current plan :-)

dandv commented 6 years ago

Transactions will be supported in 4.0

MongoDB 4.0 was released on June 28.

https://www.mongodb.com/transactions

Also, https://www.prisma.io/features/databases/ lists OpenCRUD support for MongoDB. Is it usable already?

nolandg commented 6 years ago

https://www.prisma.io/ lists MongoDB as a selling feature but this thread seems to contradict that...am I missing something? is there another branch I can checkout?

lfades commented 6 years ago

@nolandg The mongodb connector is probably the closest connector coming out soon, it will be released in August

gianpaj commented 6 years ago

happy 10th of August 😎 🏖

do4gr commented 6 years ago

There is a related discussion about embedded types. So if someone has strong opinions around embedded types you can chime in here: https://github.com/prisma/prisma/issues/2836

jay-jlm commented 6 years ago

Happy 24th of August, evryone! Say what you want about the Prisma team, but you can't deny they know how to keep the graphql community on their toes! 😆😆

firrae commented 6 years ago

Are there any updated timelines @sorenbs as we're about to start the first week of September? The features page on the site still lists MongoDB as being somewhat supported, but there's no documentation on it with most things listed as "Soon".

do4gr commented 6 years ago

We are actively working on it. You can follow this PR https://github.com/prisma/prisma/pull/2883

ysantalla commented 6 years ago

How can I test the new functionalities???

do4gr commented 6 years ago

Once we have a public preview available we'll report back here.

Hitabis commented 6 years ago

@do4gr do you have an approximate date for a preview?

do4gr commented 6 years ago

We now have a very early prototype available for testing. We decided to implement the new feature of embedded types first to get feedback on that early on. You can head over here to check out how to try out the early preview.

iamrommel commented 6 years ago

The website says it working on mongodb but you comments says beta, prototype, in-progress, I highly suggesst better removed that "check" on mongodb because it's totally misleading.

Admit it, you just want a marketing bait!!!!!

Grow up prisma and separate yourself from those poor documented and poor company that just want attention based on non-existing products. I've been to that, I've been to the company that just sells "photoshoped" products, products that exists only on photoshop, but no really running product.

Grow up and fixed yourself!!!

marktani commented 6 years ago

Thanks for sharing your point of view, @iamrommel! 🙂You are expressing what probably many are thinking. I can feel your frustration! I am sorry to hear that you feel misled, and I offer a different perspective.

It is not our intention to use marketing terms or click bait titles to draw in more attention. In fact, we are positively overwhelmed about the attention and the passionate feedback for the MongoDB connector, and many other connectors! It makes me smile everytime there's a new feature request for a new DB connector, or someone offering their help in pushing a particular connector forward.

Driven by our excitement and the community feedback, we are working hard on building more connectors and are really excited about its potential. We have validated the MongoDB connector in thorough internal tests and many discussions and demos with customers. Hence, we are convinced that the MongoDB connector will provide a lot of value, and we want to share our excitement and conviction with the community on our homepage!

However, we don't always find the right ways and words to express this excitement, what we think about a feature, or what state it's currently in. And we have been working much longer on the MongoDB connector than anticipated. These are aspects where we can improve a lot, and I really appreciate your participation by reminding us of this.

One particular step of improvement that I can think of is to add a "beta" label whereever the MongoDB connector is mentioned on our homepage. I will bring this up to the team.

I hope you will try out the preview soon, and I'm curious to hear what you think about it. You can always find me in Slack (@nilan), if you have any questions or want to share anything else. Thanks 🙌

nolandg commented 6 years ago

Great answer. I think everyone needs to realize that making money off of open source involves walking some tricky and thin lines and it will be impossible to do it all perfectly and keep everyone happy. This must be even more difficult when you're spending someone else's $4.5M and they need to see returns. I accept commercial open source being a bit trigger happy in the marketing department as a necessity of getting this great software. Cheers to Prisma, you're making me money.

marktani commented 6 years ago

Even though there have been a lot of messages since my last post, I don't think I can add anything of value to the conversation at this point which I haven't said already. I think everyone had the chance to share their input 🙂

This thread is not right the place to continue this discussion. I would encourage everyone to start a new discussion in the feedback section of our Forum if there is further need to talk: https://www.prisma.io/forum/c/feedback. I appreciate any and all input there.

If you are interested in the MongoDB preview, head over to https://github.com/prisma/Mongo-Connector-Preview.

And again, you can always find me in Slack (@nilan) to talk about anything.

Thank you.

do4gr commented 6 years ago

We just merged Join-Relations (relations between models in different collections) into the Mongo Connector preview. If you want to try it out, I’ve updated the preview page. As always: feedback is very welcome. https://github.com/prisma/Mongo-Connector-Preview/blob/master/README.md (ed

do4gr commented 6 years ago

This is a proposal for the directives to be introduced for the Mongo connector.

We are already using connector specific directives in the Postgres Passive connector. https://www.prisma.io/docs/data-model-and-migrations/introspection-mapping-to-existing-db-soi1/#overview

Postgres Directive	Active	Passive
@pgTable(name: "X")	opt	opt	can be provided to rename tables
@pgColumn(column: "X")	opt	opt	can be provided to rename columns
@pgRelation(column: "X")	opt	req	has to be provided to identify fields with relation ids
@pgRelationTable(table: "X")	-	req	identifies a relation table

Mongo Directive
@mongoCollection(name: "X")	opt	opt	can be provided to rename collections
@mongoField(field:"X")	opt	opt	can be provided to rename fields
@mongoRelation	req	req	explicitly determine the side that stores inline relation ids
@mongoRelation(field:"X")	opt	opt	can be provided to rename inline relations and embedded types

The field argument on @mongoRelation is optional. If it is not provided Prisma will use the exact fieldName from the data model to look for as a field in the collection.

Active Connector

Bi-Directional Join Relation

type Parent{
    name: String
    child: Child @mongoRelation    # required! this side has inline ids, autogenerated field names
}

type Child{
    name: String
    parent: Parent
}

Uni-Directional Join Relation

type Parent{
    name: String
    child: Child     # optional. no directive necessary since there is only one relationField
}

type Child{
    name: String
 }

Embedded Type Relation

type Parent{
    name: String
    child: Child     # optional. embedded under fieldname
}

type Child @embedded {
    name: String
}

Embedded Type To Other Non-Embedded Type Relation

type Parent{
    name: String
    child: Child     # optional. stored under fieldname
}

type Friend{
    name: String
}

type Child @embedded {
    name: String
    friend: Friend  # optional. stored under fieldname
}

Passive Connector

Bi-Directional Join Relation

type Parent{
    name: String
    child: Child @mongoRelation    # required! field which stores ids
}

type Child{
    name: String
    parent: Parent
}

Uni-Directional Join Relation

type Parent{
    name: String
    child: Child    # optional. field which stores ids
}

type Child{
    name: String
 }

Embedded Type Relation

type Parent{
    name: String
    child: Child      # optional. field that stores inline ids
}

type Child @embedded {
    name: String
}

Embedded Type To Other Non-Embedded Type Relation

type Parent{
    name: String
    child: Child     # optional. field that stores inline ids
}

type Friend{
    name: String
}

type Child @embedded {
    name: String
    friend: Friend   # optional. field that stores inline ids
}

Open Questions:

Always use name instead of field/column/name for the optional directive argument?
Is distinction between Mongo / Postgres directives necessary?
Are relation tables in Mongo a thing?

sorenbs commented 6 years ago

While I appreciate the passion displayed by the community in the comments here, this thread is a place to discuss design decisions for the upcoming MongoDB connector. I have deleted a number of comments that were off-topic. If you are interested in these other topics, feel free to reach out to me in slack or open a conversation in the forum.

agolomoodysaada commented 6 years ago

Take a look at mongodb references. Equivalent to foreign keys in pgsql.

It was also previously mentioned that multi-document transactions is supported by mongo 4.0.

sorenbs commented 6 years ago

Thanks @agolomoodysaada!

We will most likely support multi-document transactions in the future. Keep in mind that this is a feature that currently does not support sharding, and even when implemented in MongoDB 4.2 will incur a performance penalty that might be too high for many applications.

References similarly does not support sharding, and I don't know of any public plans to support it in the future.

sorenbs commented 6 years ago

As we are getting closer to the first beta release of the Mongo connector, we are looking for early adopters to take it for a spin.

If you are planning to build an application using Prisma and MongoDB, now is a good time to get involved to make sure the MongoDB connector covers all your needs.

If you are interested, please ping @do4gr here or in the Prisma slack 🙏

prisma / prisma1

MongoDB Connector #1643

Top level fields

Query

field name generation

Mutation

Details

query fields

where

scalar

relation to embedded type

orderBy

skip

before

after

first

last

Nested Mutations

When is a type embedded?

handling of relations

scalar lists

Active Connector

Passive Connector

Open Questions: