search

dgraph-io / dgraph

The high-performance database for modern applications
https://dgraph.io
Other
20.44k stars 1.5k forks source link

Product Roadmap #1

Closed manishrjain closed 5 years ago

manishrjain commented 8 years ago

After v1.0 / Proprietary Plugins

KyleAMathews commented 8 years ago

It'd be cool if the DB could serve GraphiQL. This would be perfect for demos + internal spelunking through data.

manishrjain commented 8 years ago

I'm running a DGraph server holding Freebase Film data, which you can query like so: curl dgraph.xyz/query -XPOST -d '{}'. An example of the query is in the README.

PhilAndrew commented 8 years ago

Support tinkerpop? https://tinkerpop.apache.org

manishrjain commented 8 years ago

Tinkerpop would happen with Gremlin support. We'll do that after v1.0 gets released. This roadmap is largely focused on v1.0.

F21 commented 8 years ago

For gossip discovery, I've found hashicorp/memberlist to work quite well.

manishrjain commented 8 years ago

@F21: Instead of using Gossip protocol, we used the already implemented RAFT protocol for keeping track of membership. In terms of discovery, our cluster requires knowing at least one node in the cluster; any healthy node would do.

julsdelatierra commented 7 years ago

I suggest also support cypher for queries, to avoid the queries migration from neo4j users.

manishrjain commented 7 years ago

@jcebXD -- Yeah, Cypher is something we'll fully support.

pmualaba commented 7 years ago

Cypher support will be great indeed for the application layer. I also see that Tinkerpop support is on the Roadmap after 1.0, but I would have prefered to see SPARQL instead. Since the data model for Dgraph is already Triple/Nquad based, it would be logic and convenient to expose a SPARQL 1.1 endpoint out of the box, in addition to the GraphQL endpoint. SPARQL support is very important to fully interact with the semantic web, without the need to take a full Dgraph (triples) backup which then again needs to be imported in a separate TripleStore such as BlazeGraph to be able to run SPARQL queries against the Dgraph data.

ghost commented 7 years ago

Is there any interest in supporting mobile ? Running a dgrpah DB on mobiles using gomobile is possible. Even with the CGO aspect of rocksdb. I currently run boltDB on the servers and the clients with golang. It is nice to have that reuse.

For example i build apps for mobile, desktop using golang with this: https://github.com/therecipe/qt It uses CGO heavily, and works well.

manishrjain commented 7 years ago

@gedw99 -- We're not currently thinking about mobile. All our efforts are focused on production for large scale deployments.

pmualaba commented 7 years ago

Aggregations are not on the roadmap for 1.0, and neither after 1.0 ? The ability to perform aggregations (COUNT, SUM, AVG, MIN, MAX, GROUP BY, DISTINCT,...) is clearly one of the main reasons you store data in a database in the first place and all acient and modern databases and database languages (SQL, CYPHER, SPARQL,...) support it. Why are they missing from the roadmap for DGraph?

Also for a new product anno 2017 it also make sense to have built in support for subscriptions (changefeeds ala RethinkDB). since many of the applications we built today needs support for real-time and push features...

DGraph is a great and promising database project, but without these features one would still be forced to move to other open-source alternatives such as ArangoDB which at this moment is the only serious open-source "competitor" for DGraph.

ashwin95r commented 7 years ago

@pmualaba: We're working on aggregate functions already! #532 It was missing on the roadmap. Thanks for pointing it out. We've added it now.

pmualaba commented 7 years ago

You guys are lightning fast ;-) Great to hear that you're already working on this.

pmualaba commented 7 years ago

While playing with DGraph, there is one thing that doesn't feel quite right to me : DGraph returns JSON format (which is great!), GraphQL queries also uses JSON format (without values) which is also great! But GraphQL mutations only supports NQuad format? I would like to see mutations also to be consistent in using JSON. As a matter of fact, there is already a W3C compliant RDF serialization format called JSON-LD: http://json-ld.org/

Accepting valid JSON-LD in mutations will also greatly improve developer experience using DGraph, since application logic these days all generate or serialize into JSON before persisting to a database. Manual JSON to NQuad conversion can then be eliminated from the application layer and moved to the database layer.

I would not suggest to remove support for NQuad mutations enterily (sometimes it can come in handy), But to add additional support for JSON-LD. I believe NQuad is still the way to go for massive batch imports, but JSON-LD is preferable for use in mutation queries, i guess.

manishrjain commented 7 years ago

Thanks for the feedback, @pmualaba. JSON-LD seems like a nice format for mutations. We have @ashishnegi working on it.

joeblew99 commented 7 years ago

@manishrjain Are the proprietry plugins going to be closed source. Sorry to aks this, but ACL's for example not being part of the open source offering ? Its pretty harsh.

manishrjain commented 7 years ago

Hey @joeblew99 , others,

It is very important for us to figure out a revenue model, if we want to continue working on Dgraph three years down the lane. That's the only way we can keep supporting the salaries of people who are dedicated their lives on Dgraph. Without money, Dgraph would stop spinning (think RethinkDB).

Keeping that in mind, we have divided the project into what we consider open core and what we consider commercial. Anything that a startup needs to run their services and scale is part of the open core. This includes all of the current functionality and running Dgraph distributedly (which is not for e.g., allowed by Neo4j).

Anything that's an advanced feature would then be put into the commercial, closed-source version of Dgraph. This means: ACLs, cluster management, multi-tenancy, auto-scaling on Amazon, Google, etc.

This is a delicate balance between giving startups the same (I know it's better but I want to stay humble) level of technology built within the walls of big companies, like Google, Facebook, etc.; while also finding a way to monetization to make the company profitable and keep hiring the top engineers we can find.

Note that for small companies, which are pre-revenue, we would most likely make the enterprise version free of cost (free, but not open). But, if a company is making revenue and benefitting from Dgraph, we do expect them to pay. That's what is going to allow us to keep our engineers working for Dgraph, in a very competitive market where giants can pay up a huge sum to entice engineers. So, their salaries have to be competitive and they must feel that this company is not going to vaporize, and is going to be around 10 years from now.

foobatman commented 7 years ago

@manishrjain,

I appreciate your thought behind this. But can't dgraph have model similar to Firefox or let's many of the Apache foundation projects with some big enterprises supporting it financially. And dgraph as company can work as consultant as well as developer of product. Like something that comes quickly to my mind is the way 'typesafe' works.

This will give dgraph complete freedom to focus on building technically better graph DB than neo4j or any other competitors.

On Tue, Mar 14, 2017, 6:59 AM Manish R Jain notifications@github.com wrote:

Hey @joeblew99 https://github.com/joeblew99 , others,

It is very important for us to figure out a revenue model, if we want to continue working on Dgraph 3 years down the lane. That's the only way we can keep supporting the salaries of people who are dedicated their lives on Dgraph. Without money, Dgraph would stop spinning.

Keeping that in mind, we have divided the project into what we consider open core and what we consider commercial. Anything that a startup needs to run their services and scale is part of the open core. This includes all of the current functionality and running Dgraph distributedly (which is not for e.g., allowed by Neo4j).

Anything that's an advanced feature would then be put into the commercial, closed-source version of Dgraph. This means: ACLs, cluster management, multi-tenancy, auto-scaling on Amazon, Google, etc.

This is a delicate balance between giving startups the same (I know it's better but I want to stay humble) level of technology built within the walls of big companies, like Google, Facebook, etc.; while also finding a way to monetization to make the company profitable and keep hiring the top engineers we can find.

Note that for small companies, which are pre-revenue, we would most likely make the enterprise version free of cost (free, but not open). But, if a company is making revenue and benefitting from Dgraph, we do expect them to pay. That's what is going to allow us to keep our engineers working for Dgraph, in a very competitive market where giants can pay up a huge sum to entice engineers. So, their salaries have to be competitive.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dgraph-io/dgraph/issues/1#issuecomment-286294593, or mute the thread https://github.com/notifications/unsubscribe-auth/AB-j0Rapd8OnXghVfs4PC-kxRhVbFaXmks5rle13gaJpZM4GrLa6 .

manishrjain commented 7 years ago

One thing that's clear to me: Dgraph can only survive if we are 10x better than Neo4j and others. If we genuinely didn't believe that, we'd be instead working for some big company, 9-5 and getting a nice fat check at the end of the year. So, making Dgraph the best graph database out there is the top priority. And for that, we need to hire and retain the best engineers.

I think Firefox is a bad example. They were thriving when Google was paying (donating?) them a big sum of money; until Google decided to stop that and focus on Chrome. Since then, Firefox revenue has steadily declined, while Chrome has taken over the world. At this point, Firefox can't afford to get the top talent working for them -- while Google (/Chrome) continues to recruit the best and brightest, which in turn makes Chrome even better. http://howdoesitmakemoney.com/how-does-mozilla-firefox-make-money/

Consultancy is great, and we intend to do that, but consultancy alone is not going to make a company significantly profitable (more consultancy requires more human power). All top companies today are profitable due to selling software (Google, Microsoft, Facebook, etc.).

P.S. I don't intend this to become a thread of its own; so feel free to email me if anyone has concerns: manish, dgraph.io.

joeblew99 commented 7 years ago

@manishrjain i agree with you. Many startups that were making new database systems failed due to lack of revenue. We dont want another rethinkdb, and the open core model makes sense.

But, everyone needs ACLS. I would think about if ACLS are in the open core of not. The other stuff is for commercial for sure - so i agree with that.

But ACLS is so fundamental to any system you run always. I mean any open source project that wants to build a forum or a chat system needs ACLS.

ghost commented 7 years ago

What are your thoughts on subscriptions? Do you plan on supporting libraries for languages other than Go (and Python and Java)?

manishrjain commented 7 years ago

We have community run libraries for Python and Java. You can find their repos in dgraph-io org in Github.

Not sure what you mean about subscriptions.

ghost commented 7 years ago

I mean GraphQL Subscriptions (https://github.com/facebook/graphql/pull/267).

manishrjain commented 7 years ago

We do intend to have a real-time feed, where mutations can cause certain queries to be re-run based on some dependency chart; and the results streamed back. We just haven't yet come across the right use case to do it. But, that's definitely a feature I'd love to have in Dgraph.

So, if you have specific use case which this feature would help with, ping me.

ghost commented 7 years ago

Thanks @manishrjain, that would indeed be a great addition to an already impressive feature set. I'll be sure to do so should the need arise. Thank you.

Do you have any plans for a JavaScript client?

manishrjain commented 7 years ago

Javascript client? If there's a need for it, we can create one. But, the current flow is pretty straightforward, you query via POST, and get the results. So, hard to determine how would this client help.

TwinXu commented 7 years ago

How to deploy the distributed server? And after a 2 days running test , we found a serious memory leak. Is it caused by RAFT?

janardhan1993 commented 7 years ago

@TwinXu You can find the documentation for running distributed server at. https://docs.dgraph.io/v0.7.5/deploy/#multiple-instances

This issue is for tracking product roadmap, can you please file an issue with the details of memory leak. There shouldn't be any memory leak due to RAFT. It would be great if you could provide some more details about the memory leak.(What all you did, how much memory leak did you notice, what was the configuration with which you started the server). You might notice that the memory consumed by the server increases if your query touches new data until you hit the stw ram limit. That is not memory leak, it would be evicted eventually when needed.

TwinXu commented 7 years ago

@janardhan1993 thank you for your reply. I will new a issue for the detail of memory leak~

cfresh commented 7 years ago

Re: usefulness of a javascript/typescript client: I think that providing a light wrapper around gRPC functionality (similar to the java client) would be helpful. I would imagine that a package like this will land on NPM anyway, so it is really just a question of whether it is maintained by Dgraph or some other person.

iamcarbon commented 7 years ago

Hey guys -- am working on a Dgraph client for .NET. Is it safe to assume that the API / gRPC shape is going to remain in flux until 1.0? Do we have any milestones on when to expect the api contract to stabilize?

voxadam commented 7 years ago

No mention of Badger? Are you still planning on migrating from RocksDB to Badger by 0.8?

rodamunoz commented 7 years ago

@voxadam currently rocksdb was removed from master and now it is using badger as storage, I guess the next release will contain badger as default

manishrjain commented 7 years ago

Do we have any milestones on when to expect the api contract to stabilize?

We're now starting to actively think about v1.0, and stabilizing the APIs.

MichelDiz commented 7 years ago

About the JS client. It would be interesting to have "seemingly native" support from you. Because I believe that the JavaScript community would adopt this project if it speak their language, even though it is super easy to adopt and "Go Lang" is simple. A lot of people ignore by the simple fact of not having "native support" for the language they are used to. Did u get it?

I myself have already thought a thousand ways to reconcile this database with my project. Currently I use RedisGraph, but it takes lot of work to maintain along with Redis and GraphQL. "Typing" meaning.

In this case, the easiest would be to plan my Schema to do in my resolvers a fecthing plan and that's it. I knew this project exactly 30 minutes ago. That's what I thought of right away.

And I'm only here for the simple fact that this project promises me speed (See I use Redis), Graph and is GraphQL like. The best of both worlds. From the little that I have read and seen, I believe that this project is perfect for me.

Pls "Suport Js"

Cheers

MarcErdmann commented 7 years ago

Yes I can only agree. It would be cool if you guys support JavaScript.

manishrjain commented 7 years ago

Yeah, JS support is on our radar. We do want to support it; we're looking for an engineer with good JS skills to help us do this -- so if someone wants to help us out, here's the link:

https://boards.greenhouse.io/dgraph#.WcTDa3UjG0o

Didericis commented 7 years ago

Just to clarify, "JS client" means something like apollo, correct? I would love some sort of modified apollo client that works with GraphQL+-. Is the query language stable enough for that sort of a project?

MichelDiz commented 7 years ago

Means "any support" native - Even an Apollo like. But what I really said was to create a Javascript context-friendly support model - To create applications in JS using NodeJS for example. Do not changing the core in Go Lang, which would be impossible and meaningless.

I have already cataloged some projects that support Dgraph. However, they take a long time to update as the Dgraph team make changes and add features. It's tricky for them to keep something of the sort practically alone. I'm helping with my limitations.

But wait second, Apollo does not handle data from a GraphQL API just through "Json" results? I do not think it would be complicated to leave Dgraph compatible with Apollo. It would make an exclusive endpoint to "trick" Apollo. Making "him" think it's consuming GraphQL. It would not be possible?

Didericis commented 7 years ago

@MichelDiz Apollo looks at queries in order to determine what's cached/how to deduplicate things/how to normalize the results, so it needs to understand GraphQL. I think GraphQL+- is similar enough that a lot of Apollo could probably interpret GraphQL+- the same way without many changes, but I'm not sure.

Another solution is to convert GraphQL+- to GraphQL to expose something that Apollo could work with (the "tricky" endpoint you mentioned), which is what this does: http://dpeek.com/dgraphql/. I haven't tried hooking it up yet, but it looks like it should work. I just got excited about talk of a javascript client for Dgraph. I worry about GraphQL+- and GraphQL drifting apart and/or Apollo getting fancier to the point where it isn't easy to hook them up. It'd be cool if there was some sort of frontend client to help with caching/efficient querying specifically with GraphQL+- compatibility in mind.

joeblew99 commented 7 years ago

On another topic..

Would there be a possibility for change notification on queries ?

I use NATS for this currently btw.

Thanks for a awesome dB !!!

manishrjain commented 7 years ago

@Didericis @MichelDiz -- By JS client, I meant a Dgraph specific JS client which would talk to Dgraph over GRPC. We're adding support for transactions, and hence, all the mutable queries now need to happen via a client.

Apollo / GraphQL compatibility is something we intend to look into past v1.0, once our query language is stable, and we are clear that it fulfills most functionality required by our users. At that point, we can figure out a way to bridge the gap between GraphQL and our version of it.

@joeblew99 : There is an issue about something similar: https://github.com/dgraph-io/dgraph/issues/824

sebastianmacias commented 7 years ago

@manishrjain

Anything that's an advanced feature would then be put into the commercial, closed-source version of Dgraph. This means: ACLs, cluster management, multi-tenancy, auto-scaling on Amazon, Google, etc.

I think ACL is very important not to be considered part of the open source core.

graphcool offers ACL out of the box along with authentication/authorization, subscriptions and other features.

They offer a hosted service Graphcool Cloud and that is how the monetize; maybe a hosted service would allow you to monetize without sacrificing essential features all devs and small startups need.

Just a thought.

manishrjain commented 7 years ago

@sebastianmacias : Noted. Once we're past v1.0, we'll reconsider this.

sebastianmacias commented 7 years ago

@manishrjain sounds good. Thanks

manishrjain commented 7 years ago

@pmualaba : Dgraph v0.9 adds distributed transactions, and fully supports JSON as both input and output format. It makes interaction with Dgraph a lot easier.

joeblew99 commented 6 years ago

Can you guys write a demo example for distributed transactions. Can be simulated with dockers. Really curious how the linearizability performs. Lots of gremlins

MichelDiz commented 6 years ago

They will produce some examples soon. Follow the news in both the blog and the forum.

ghost commented 6 years ago

Cool thanks..

On Tue, 28 Nov 2017, 17:45 Michel Conrado, notifications@github.com wrote:

They will produce some examples soon. Follow the news in both the blog and the forum.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dgraph-io/dgraph/issues/1#issuecomment-347585585, or mute the thread https://github.com/notifications/unsubscribe-auth/ATuCwhyct3zeJbUG7qUAK3xGkt8AAoZ2ks5s7DipgaJpZM4GrLa6 .