Request for project update

[Bajix]

It seems like Wither has the potential to be essentially the equivalent of what mongoose is to Node, but the lack of recent development makes me somewhat concerned when considering switching off of NodeJS/mongoose and onto Rust/Wither for a production app. I get the feeling that a large reason for this being the case is simply that for this project to move forward, there had to be progress with the MongoDB driver. Now that it is this case that there's a new officially supported driver that's been released, is Wither going to resume active development again? Do you have a roadmap?

[thedodd]

Your analysis is correct, I’ve been effectively blocked by the underlying driver, but now that the new one has been released, we should be able to resume meaningful development.

I do have a roadmap, it is essentially broken up between two milestones (the only two in this repo). Between the two, it is only a handful of items. I would love to see more feature requests and the like.

Personally, I think it would be cool to add some features on top of the change streams and transactions features — but they have not yet been implemented in the driver.

[Bajix]

It might also be worth taking a look at Avacado and Diesel for inspiration. Whereas with your crate queries use bson Document types, Avacado abstracts over this and provides Query/Aggregate traits that can be implemented as to create type-safe Documents for these operations. Over on the Diesel side, they have a really nice Query DSL as well as statically typed filters (a feature Avacado lacks). None of these libraries have gotten everything spot on, and all certainly will need extensive updates especially for async await, but I think all of these projects have unique strengths and that all of the authors can/should learn from each other and collaborate.

Here's my thoughts about the larger picture of all this: if it becomes such that Rust has the best ORMs/ODMs of any language with full async and clean abstractions that provide compile-time guarantees of query safety that this would mark the beginning of Rust mass adoption in the web app space.

Also, check out async-trait; it's probably not too unreasonable to start building out an async version of this by using asnc-std/new-scheduler as to wrap the blocking MongoDB operations, and async-trait enables an abstraction as to facilitate async functions on traits until this becomes part of Rust stable so really those two are probably all that's needed to magically make this all async with maybe less-than-perfect but still awesome performance. I'm not sure how well this will time out with everything, but were the mongodb driver to release their async version prior to your release, then switching over to real async will be much easier.

[thedodd]

Yea, I've looked at Avacado before. Looks like most of their code was ripped off from my code here haha. Having worked with MongoDB in a lot of different languages, the patterns in Wither tend to be best, IMHO. Provide modelling capabilities, indexing from within the application code, and then get out of the way. Attempting to move away from BSON, even though MongoDB itself is built so closely around it ... fool's errand, IMHO. More to be said on this front, but the real question is, do you think that there are features from that project (or others) which should be implemented here?

To be sure, moving away from _id: ObjectId PKs is an anti-pattern in Mongo, and using a pattern other than BSON documents for queries — though it might be nice at times — diverges too far from Mongo norms, and often just gets in the way. Direct access to the BSON means that query building is directly translatable from the MongoDB docs and there is no guessing about it. Not going to diverge from BSON for queries.

I've used Diesel quite a lot. It is great. Query building system is good. The domain is quite different than that of the MongoDB domain though. Anything in particular which you believe should be adopted here?

As far as async is concerned, I'm not going to try to get out ahead of the MongoDB team's driver. Once they have support for async patterns, then we can adapt the code here. Otherwise we would be wasting out time with writing a lot of code, only to have the driver end up taking a different pattern. Not good.

[Figments]

If I may chime in, one of the things that Wither could look into doing in a future release is defining document schemas using RON files so that they can be mapped near 1-to-1 with how the documents would look in the database. The great thing about Mongoose schemas is that they're simply JavaScript objects at their core, and JavaScript objects map very neatly to JSON/BSON documents.

Using RON schema files could approximate this and appear more natural to work with than the current solution of mapping structs to objects, especially in the case of nested objects, where you have to define a separate struct and then use it as a type in the main struct--a move that looks and feels very unintuitive and kind of backwards, if that makes sense.

It may be something to look into as Wither goes forward.

[Bajix]

Yea, I've looked at Avacado before. Looks like most of their code was ripped off from my code here haha.

Haha yea I noticed that as well, though I can see with how different it is how this ended up as a separate project.

Not going to diverge from BSON for queries

If you take another look, Avacado never diverged from using BSON either. There are blanket implementations of all of the database operation traits on bson::Document which makes it thus that this will work with doc! syntax already, and anywhere that does use structs that implement these traits still will use a bson::Document for the actual query as that's the return type of filter. The advantage really is that when doing it this way, because you define a strongly typed query struct, that's going to add an extra level of safety, and you can do neat things like combine this with traits like juniper::GraphQLInputObject and actix::Message or even to use this as a JSON extractor.

To be sure, moving away from _id: ObjectId PKs is an anti-pattern in Mongo

Swapping off of ObjectID's is non-default and feature gated. The purpose of their wrapper type is solely to add type safety, which it does. You could simply omit the feature gated UUID behaviors and include this still as a type safety enhancer. For instance, post._id == user._id wouldn't work, but post.owner == user._id would work because post.owner would be Uid<User>.

I've used Diesel quite a lot. It is great. Query building system is good. The domain is quite different than that of the MongoDB domain though. Anything in particular which you believe should be adopted here?

I think we should take a page from Avacado, and make it so that anywhere that currently accepts a bson::Document instead accepts a struct that implements the trait respective the operation and then we use blanket implementations for bson::Document to provide for both syntax. This could be combined with a query builder; I think one of the nice aspects of Diesel is that there are type definitions for building queries that make it such that queries are compile-time checked; this could do the same, and we can borrow ideas from NodeJS/mongoose such as the ability to merge queries. There are also really nice composability advantages of query builders, which while infrequently needed are quite useful time to time.

[Bajix]

It would really be fantastic to make the aggregate pipeline more ergonomic and type safe as well. For instance, as aggregate can be used for updates but only with $addFields/$set, $project/$unset, and $replaceRoot/$replaceWith and no other stages, it could be such that using other stages cause a type error. The model derive could set up the query builders specific to that collection, and so that could server to make things like $match type checked.

Also, it doesn't look like you have any populate behavior, and that the selection behavior here is lacking. Coming from the NodeJS/Mongoose side of things, I've found the best design pattern for managing various projections is to make it such that the schema has a default set of selected fields (select: false) and then to modify this as needed with inclusion/exclusion field selects or to pass an exact projection shape. It would be nice to see more done here to make population & selection ergonomic and safe. It would be a win if this could make it easier to be correct when dealing with a number of different shapes of the same schema, including with population.

[thedodd]

Hey, wanted to drop an update here that I am in the process of cutting this system over to the new mongodb driver (0.9.2). As far as other design elements described in this issue, we can extract some of these ideas into their own issues to iterate on design.

[Figments]

Awesome! Would it be good to then separate out the suggestions raised into their own issues right now, or after the update to the latest driver is made?

[thedodd]

@Figments I'm fine either way. If you would like to open an issue for the items you've suggested, that's good with me :). Else, I will circle back to this issue once the PR for the latest driver lands.

[rumanHuq]

Hey, wanted to drop an update here that I am in the process of cutting this system over to the new mongodb driver (0.9.2).

Would you please elaborate. I didn't understand this bit.

[thedodd]

@rumanbsl for a long time, this was the main mongodb driver available in the Rust ecosystem and it is the one which the driver is currently based on: https://github.com/mongodb-labs/mongo-rust-driver-prototype

Not too long ago, the mongodb team archived that project and released a new officially supported driver: https://github.com/mongodb/mongo-rust-driver I am updating this system (Wither) to be based on the new driver.

[rumanHuq]

I thought you are giving up on the project. Thanks for clearing it up

[thedodd]

@rumanbsl haha fortunately that is not the case. I have not had nearly as much time as I would have liked to put into this project, but I'm definitely not abandoning it :). Thanks for asking for clarification.

[thedodd]

Wow! Stoked to see this too: https://github.com/mongodb/mongo-rust-driver/releases/tag/v0.10.0 Will have to put a plan together on how to best support both sync & async interfaces. This is a good problem to have though.

[rumanHuq]

It would be nice to have the latest mongo driver support, especially the async support. Looking forward to get a glimpse of your plan. I would like to help if I can. Can you outline some pre-req knowledge to contribute :)

[saghm]

Will have to put a plan together on how to best support both sync & async interfaces. This is a good problem to have though.

There is still a sync API in the driver; it's just under a feature flag (which I'm looking into getting to render on docs.rs right now, as it unfortunately currently isn't showing up). I think the most straightforward way to wrap it would be to have a feature flag for wither as well, although you could also just use only the async API and then use some method of blocking on the futures in the case of sync (which is what we do internally).

[thedodd]

@saghm nice. I'll take a look at how you and your team have implemented that internally (I'm assuming you mean internal to the mongodb crate) and probably just base the design off of that. One of my main goals with this crate has been to stay as close as possible to the way things are done in the underlying driver where possible.

[thedodd]

@rumanbsl so ... the async support is g2g now.

@Bajix & @Figments I'm planning on closing this issue. If you would like to open issues for any of the design items mentioned above, please feel free to do so. @Bajix especially related to the pipelines/aggregation which you mentioned previously.

I'll close this issue soon.

[surfingtomchen]

is the population feature supported by the official driver now?

[srikarm99]

Hi, Is this project not maintained?

[thedodd]

@srikarm99 that last release was quite some time ago. I do not have sufficient time to give it additional attention and maintenance. If you are interested in helping out, please let me know.