Bookshelf Next - New Project Organisation, call for developers and ideas

[Playrom]

Hello Developers!

Last month the project creator and main maintainer @tgriesser has granted administrative power and ownership of bookshelf to a new organisation, which will oversee the future of our beloved ORM.

Reference: #1511


Me and @dj-hedgehog are some of the members of this organisation.

We want to restart the development of bookshelf, and we have already started the work with the merge of new Pull Requests, and the release of version 0.10.4

We are also analysing the issues backlog

.

But we need your help to thrive the project, in form of collaborations, new ideas, PRs etc…

If you want to help please follow this guidelines

1. Use this thread to discuss about the future of the project or to propose yourself as developer
2. Open a new issue for every idea you might want to be implemented in the future
3. Submit a pull request

In the scope of reduce the huge size of the backlog we will close all the PRs and Issues older then December 2016, asking to reopen them if issues are still relevant, and PRs are still valid.

We will use the Github Projects to organise our work

We can't work alone, because this is why bookshelf started to fail this winter, we need your help and experties.

Thank You for your love for bookshelf and for every advice and help you will give us

[ricardograca]

I just have one (probably unpopular) suggestion and that is to drop babel. Since node versions 0.10 and 0.12 are no longer supported is it really still necessary?

[Playrom]

@ricardograca open an issue about that, I'm open to a discussion about this point ( I hate babel and no-es6 javascript )

[ebramanti]

@Playrom I became a member of the organization back when I was doing some documentation and light work on the API with @rhys-vdw, but I'm very open to helping implement a few features here and there in my off time from work.

[tkrotoff]

Please don't close old issues.

[joepie91]

Does this also affect Knex, or just Bookshelf?

Also, I feel like closing old issues is the wrong thing to do here. It's not going to make any actual issues go away, and may result in information about hard-to-reproduce issues (that occur semi-regularly but are rarely reported) getting lost in the shuffle for a long time. A coordinated triaging effort is probably more useful.

I just have one (probably unpopular) suggestion and that is to drop babel. Since node versions 0.10 and 0.12 are no longer supported is it really still necessary?

The first version of Node.js with full-ish ES6 support was Node.js 6.x, so Babel is not totally unnecessary yet, as Node 4.x is still supported.

EDIT: For the sake of completeness, the Node.js support schedule can be found here, and feature support can be found here.

[ricardograca]

...so Babel is not totally unnecessary yet...

... if you want to write code using ES6/7 features. If you don't then it is.

This change only affects Bookshelf.

Agreed on closing old issues. I'm afraid they will come back to bite us eventually and nobody will remember that there was an issue about it by then.

[crunchtime-ali]

We are willing to re-open any issues when devs speak up for it. There were 420 open issues. I clamped them down to 175. Due to work, studies etc. @Playrom, @ricardograca and I would likely never get around to work on most of the issues.

This way we see which of the old issues are still considered important so we can prioritise our efforts.

[Playrom]

@tkrotoff @joepie91 there are almost 500 issues, a lot of them back in years, and there were 7 months of total nightmare in this repo this year, without any kind of support...

we need to be realistic and focus the most recent or important issues and prs...

also if someone get some bad behaviour in bookshelf usage a simple search can prompt an old issue which can be reopened without any problem

[joepie91]

We are willing to re-open any issues when devs speak up for it. There were 420 open issues. I clamped them down to 175.

The developer reporting it may no longer be around, even if the rest of the users would still benefit from the fix.

Due to work, studies etc. @Playrom, @ricardograca and I would likely never get around to work on most of the issues.

Understandable, but that can be solved through a coordinated triaging effort. That does not have to be undertaken by just the maintainers; anybody can contribute to that. I suspect that even "close all the unanswered question-marked issues" alone would already significantly reduce the work required, without affecting issue resolution.

I'd personally be happy to help triage some issues (reviewing them, leaving comments on whether they should be closed or are already fixed, etc.) as I find the time to do so. At that point, maintainers just need to look at the comments left and follow up with a close/label/etc. I'd imagine there'd be more users willing to do so, if asked.

also if someone get some bad behaviour a simple search can prompt and old issue that can be reopened without any problem

This isn't realistic. GitHub's issue search is pretty crap, and if somebody doesn't already know exactly what they are looking for (and especially for hard-to-reproduce bugs that is very likely to be the case), they won't find anything, decide that they don't have enough data to file an issue, and just not bother.

[Playrom]

@ricardograca @joepie91 ok if we can find a nice group of people that can triage all the past issue for me it is not a problem for me to reopen them all! ( but @dj-hedgehog will be very angry for a few hours lost :D ). Just All the issues must be triaged, not just a few, it would be useless...

We are trying to do the best for the project, don't hate us please. We want to create a good community with a lot of developers to thrive the future of bookshelf

[joepie91]

We are trying to do the best for the project, don't hate us please. We want to create a good community with a lot of developers to thrive the future of bookshelf

No worries, absolutely no animosity intended. I'm just trying to help get the best possible result here for Bookshelf :)

ok if we can find a nice group of people that can triage all the past issue for me it is not a problem to reopen them all!

Hereby the request to anybody reading this thread: please leave a comment if you'd like to help out with an effort to triage old issues!

Triaging would mostly just involve reading the issues, classifying them (is it a question? a bug report?), and determining whether they are still relevant (is the bug fixed already? is the issue reproducible, or does it need more information?), then leaving a comment with the verdict. It can be done whenever people have time, doesn't require scheduling.

[Playrom]

ok nice idea @joepie91

by the way I think that all the questions can be set as close as dead without any thinking

[davidfurlong]

💪💪💪 appreciate the effort & will try to find time to contribute

[ricardograca]

I know it's a terrible thing to do, but I think we should close questions either immediately or after 3 days at most if there's no answer. This isn't stack overflow, and I think these take a lot of our time.

[jkantr]

I can lend a hand. Not super sure how a 'triage' works per se but if we can throw up some guidance to point people to I'm sure we can round up a few experienced devs...

Also as far as the 'questions' go.. it's not uncommon for projects to only field actual issues and bug reports. I'm sure we can find some reasonable middle ground.

[joepie91]

Also as far as the 'questions' go.. it's not uncommon for projects to only field actual issues and bug reports. I'm sure we can find some reasonable middle ground.

Perhaps a reasonable metric would be "is this already covered in the documentation?". If yes, close it and point to the documentation; if no, treat it as a documentation bug (and update the documentation with the answer, instead of just posting the answer in the issue thread). Things that shouldn't be in the documentation in the first place (eg. "how do I combine Bookshelf with X") should then just be directed to StackOverflow or such.

(Also, GitHub really needs better facilities for dealing with usage questions...)

[ricardograca]

@jkantr I think the best thing to do is to go over the issues that are not labeled and figure out what they are and apply a label.

If you are sure it's a bug report, then finding out if it's still an issue would help a lot. This won't be easy because most of the time there's no code example on how to reproduce the bug.

@joepie91 That's a good point. However sometimes the questions are about advanced usages of Bookshelf that may even require dropping down to knex. Is this something we want in the docs? Maybe we do, in an "Advanced Topics" section or something like that.

[joepie91]

This won't be easy because most of the time there's no code example on how to reproduce the bug.

These are probably a good candidate for "ask for a minimal testcase, close the issue after 2 weeks if none has been provided, unless there's a good reason for it" or a similar policy. The issue template should also probably be updated to ask for such a testcase more explicitly, as it's currently a bit vague on what's expected and what's optional.

However sometimes the questions are about advanced usages of Bookshelf that may even require dropping down to knex. Is this something we want in the docs?

I think it's reasonable to explain non-obvious and frequently-asked-about cases in the documentation, but not everything should be listed there. For example, "how do I represent a multi-level many-to-many relationship with Bookshelf and Knex" is a question that's worth covering in the documentation, but "how do I use Bookshelf with Express" probably is not.

I think the best thing to do is to go over the issues that are not labeled and figure out what they are and apply a label.

It's worth noting that only maintainers/collaborators can add labels, so to spread the workload amongst other users it would probably be ideal to:

1. Have a central issue thread tracking old issues
2. Ask 'triagers' to leave a comment on a thread suggesting a label / confirming reproducibility / etc.
3. Ask them to then leave a comment on the central issue thread to notify the maintainers when triage for an issue is "done".

This means that maintainers would only need to track a single thread to know where labels need to be applied, issues need to be closed, and so on.

[Playrom]

I totally agree with @joepie91 , also the lack of code example and explanations in the issues is the reason why I wrote the new issue/pr templates, their usage will make the triage of new entries a lot easier!

regarding the triage I think that yes the first priority is to label all the issues , assign yourself as assignee and maybe to set a milestone for the issue

regarding the questions I think that their presence in the issues is essential, we don't have any real community on stackoverflow so a lot of people will always come here to ask for advice... the best treatment I think is to leave the question open for a brief period and treat them as docs issue (if the question is not stupid) as suggested :)

It's important also to have a deeper use of the tools offered by GitHub, such as assignee, milestones and projects

[olalonde]

All I can say is I think this is great news. Bookshelf felt like an unmaintained project for a while.

[rhys-vdw]

I know it's a terrible thing to do, but I think we should close questions either immediately or after 3 days at most if there's no answer

@ricardograca I disagree with this. I've always seen questions as "documentation bugs". Bookshelf documentation is incomplete. The way I used to approach things was to answer the question if I could, and then open or request a PR adding the answer in the docs. If I couldn't answer it I'd leave a note saying I don't know - if the answer comes up then it can be added to docs and closed. These questions are key insights into possible omissions in Bookshelf's feature set or documentation.

[rhys-vdw]

Oh, but - I don't have time to contribute so feel free to disregard my opinion.

[Playrom]

@rhys-vdw every answer is a good answer!

and we are the one that have to thanks you for your past work on bookshelf! :)

[atrauzzi]

Redis, rethink, mongodb, couchbase, aerospike, neo4j.

[Playrom]

@atrauzzi ?

[joelmukuthu]

i've been working on an ORM for knex that supports some core features that i wished bookshelf had (validation, joins): https://github.com/joelmukuthu/knorm. i'm very willing to help bring the same features to bookshelf or even merge the two if we can find a way to do that.

[Playrom]

@joelmukuthu I love the fact that it's full ES6 <3

If you want feel free to open some issues or PRs , if you have coded something not currently present in bookshelf we will surely evaluate to add that code! :)

[joelmukuthu]

@Playrom awesome! i actually have working code for validation, joins and some field-name to column-name mapping. sorry there's no documentation but you can have a quick look through the tests (on travis perhaps) to see how it works. please take a look :)

[esatterwhite]

Before this project can be made great again, we must first define what was great about in the first place. And if those things are even important anymore. I think this thread does a pretty good job illustrating what is not great - largely that relations are largely broken and widely misunderstood. And to be fair, that is what and ORM is supported to do. As soon as the problem extends beyond a single join, the recommended solution is to drop down to knex and basically write the query by hand. Again the job of an ORM.

It should also be noted that there is a project built on knex that is doing what bookshelf has been trying to do for a number of years https://github.com/Vincit/objection.js

[Playrom]

bookshelf have issues, it's impossible to deny, but for my own experience I think that objection is sometimes an overkill, sometimes too complicated

Bookshelf is in a sweet spot between the easy of use and the power using, I think we need to improve our functionalities while maintain the simple way of usage that is our the core aspect

[esatterwhite]

I would disagree with that. Outside of the JSON schema validation, objection does what bookshelf suggests it can do.

[esatterwhite]

Can someone elaborate one the thumbs down? Objection docs suggest (minus things that json schema)

• An easy declarative way of defining models and relationships between them
• Simple and fun way to fetch, insert, update and delete objects using the full power of SQL
• Powerful mechanisms for eager loading and inserting object graphs
• Completely Promise based API
• Easy to use transactions

Transactions are actually handled by knex in both cases. so that is a wash. Both use promises, but that isn't really relevant to what an ORM does. Bookshelf gives a way to define models and relation ships between them. Both do that just as easily.

What bookshelf does not do very well is

• Simple and fun way to fetch, insert, update and delete objects using the full power of SQL
• Powerful mechanisms for eager loading and inserting object graphs

Which are basically the reasons behind using ORMs.

Once you've defined these relations with bookshelf models, doing anything outside of simple selects is rather difficult and in many cases just isn't possible with out writing one ore more queries by hand or with knex - which, does require you to know table names and column names. And in the case of joins and many to many relations, will also require you to inject aliases for tables and columns.

I'm not trying to take a cheap shot at bookshelf here, I have production code that runs on it, and open source projects that allow it as a backed orm option. I have a vested interest in this project. But for it to succeed, we need to be honest about what it's problems are.

[Playrom]

discussion is useful :) we can improve in a lot of different ways, feel free to open some issue, I will put in our project backlog for discussion and future release :)

[Unsigno]

Hi all , i am new here :) but i will try to help.

I still can not say much about functionality, but I can speak from a new user's point of view. And I do not have very good news, from Google I find little more than the documentation itself, and there are only few examples of some functions mixed with the API that are not enough to start using it. And attracting new users is an important point to keep a project alive.

That's why I propose a reorganization of the documentation, separate the examples of the API, creating a "Getting started" section to include them and add new ones. Maybe keeping a dropdown to continue showing them in the API (how now are, but hidden by default)

( Example ) I found Bookshelf by polymorphic models, and looks fine, the documentation shows how to define the models, but no more. Now i dont know if they manage the type internally, if i can create a ExtendedModel and automatically creates the BaseModel... I can define polymorphic models but i dont know how it works or how use it .

Maybe for people that know how Bookshelf works this sounds exaggerated, but forget your knowledge and try finding basic questions on Google .

If you agree with this point, I can help with it. If not I will try to collaborate with some PR.

I wish you 0 bugs and compilation errors :p

Best regards, Unsigno .

[guidsen]

If you need some extra hands. I'm willing to give up some weekend hours to contribute. Let me know!

[tkrotoff]

@Playrom

there are almost 500 issues, a lot of them back in years [...] we need to [...] focus [on] the most recent or important issues and prs...

Fixing important issues and PRs and closing all issues are just totally unrelated. You can tag important issues and PRs and sort them without disturbing users. There are GitHub projects with far more issues than Bookshelf and still thriving.

also if someone get some bad behaviour in bookshelf usage a simple search can prompt an old issue which can be reopened without any problem

This will almost never happen. GitHub search defaults to open issues + a user cannot reopen an issue: you need an admin for that and users won't ask 90% of the time. What will happen instead: users will open new issues instead of contributing to related existing issues since they are closed => losing unvaluable informations.

Also people subscribe to issues in order to be notified, now they will never know a particular feature has been released or a bug has been fixed.

=> what you have done here is cut the historical user base from the project.

Here my workflow to understand:

each time I have a problem with one of my dependency (feature needed or a bug):

• find the related issue(s)/PR(s) or create a new one
• comment on the issue if valuable
• subscribe to it
• add a TODO or FIXME in my code and the link to the GitHub issue

when getting a notification (email) about a feature implemented or a bug fix:

• search my projects for the related piece of code
• rework my code
• add comments on the related GitHub issue if valuable

[tkrotoff]

@dj-hedgehog

This way we see which of the old issues are still considered important so we can prioritise our efforts.

You can tag important issues instead

[rapzo]

I use bookshelf in a couple of projects and i know its source from head to toe and kinda stuck to it. Count me in on this.

[rapzo]

Couldn't we get a slack or gitter for this? Would be nice.

[theklr]

Have there been any updates on progress?

[thebergamo]

I've some projects running on Bookshelf and I really like the way the things are doing. I've faced with a lot of problems since to find information regarding how to use something to problems with transactions and other things.

What are the next steps? Count me in o/

[vinicius0026]

So, how are the efforts on bringing bookshelf back to life going? I would hate to see it go, but 2 weeks since the last commit, 13 days since the last comment on this thread are not good signals... Is there any other communication channel going on? I think a slack/gitter/[put your favorite chat platform here] group would facilitate communication between those willing to contribute and it would promote in depth discussions before controversial decisions are made (like closing all old issues). I very much agree with @esatterwhite. I've found myself writing many knex queries when things got just a little more complicated than fetching one item with related fields. I also think that docs is a huge spot for improvements. To wrap-up, I think a clear path forward is paramount for this project to thrive again. Count me in.

[Playrom]

hi guys, The answer is simple, I can't develop bookshelf alone, I currently don't have the time or the capabilities to have this responsibility

What I can do is to manage the project, such as merging PRs, organise the work of the contributors etc...

I've created a Google Form in order to know how many people wants to help and to start working on the future

https://goo.gl/forms/JSstGiMqJ2TGMa2N2

Please Sign Up If you want to help, without any help I can't make any promises

[vinicius0026]

I just did sign up. But to make things clear, I think no one expects you to maintain bookshelf alone, on the contrary, a lot of people had already shown intent to collaborate. We just need to put these people together and agree on what are the project priorities and its path forward, so that people can start collaborating.

[Playrom]

Exactly, but we are at the point that a lot of people are insteread but there are not A lot of useful PRs, so we need to talk each other

[davidfurlong]

This week I created a plugin which could also just be a PR: https://github.com/DeedMob/bookshelf-committed-plugin

It addresses some issues me and others were having with saved/updated/created events firing before the transaction has been completed

[mrhwick]

Jumping in a bit late on the discussion here.

I accepted the invitation to join the maintainers team for bookshelf. I currently have production projects making extensive use of bookshelf.js and knex.js, and a vested interest in seeing this project succeed.

As far as I can tell, we have a handful of different problems on our hands right now:

1. No clear project ownership / plan
   1. We have a new set of maintainers. This is a very good first step.
   2. Need to solidify the set of core contributors who can coordinate on management activities.
      • We have a mixture of collaborators/owners on the repo and bookshelf.js organization.
      • Some of the repo collaborators are not in the github organization.
      • Too complex, makes it hard to know who is involved and who is not.
   3. We do not have a coordinated effort to decide what comes next. I want to change this.
2. Confusion over communication channels
   1. Historically, there have been a variety of communication channels.
   2. Need to be realistic and not create idealistic expectations of our participation.
   3. Keep it simple, and don't overcommit our time as contributors.
3. Lack of system knowledge
   1. Do any of the maintainers know how the system is structured at a high level?
   2. Can't resolve bugs without familiarity of how the system actually functions.
   3. Can't make feature additions/improvements if no one knows where to start development.
4. There are a shit-ton of leftover issues and PRs backlogged on the github repo
   1. Need to identify what is important and what is not
   2. No way to know whether issues / PRs are still relevant
   3. No consensus on what to do with stale items (>1 year old)
5. Documentation is out of sync with the actual library
6. Missing core features expected of state-of-the-art ORMs in 2017
   1. I'll just give one example: composite primary keys
   2. I'm certain there are plenty of others that have been identified in the mess of leftover issues.
7. Release management / package publishing seems to be undiscussed completely at this point
8. Code quality issues
   1. Intermittent test failures on travis builds
   2. Refactoring is badly needed in some major system features, such as relation management

I'm sure this list is not exhaustive. There's a lot of stuff to catch up on.

I propose that we tackle the maintainer/support story first and foremost. We've got a mixture of repo collaborators and github organization members. We have open questions about communication mechanisms (see #973). If I were a user or potential user of this library, the lack of a cohesive support and maintenance story for this library would worry me greatly. We can do better.

I edited our first Github project on this repo to align it with establishing a baseline of expectations on maintenance and support. Once we have that in place, we can decide what is next priority.

[ricardograca]

Your analysis seems correct. You can do what you feel is the best in terms of "stale" issues, but I'm almost certain all of them are still valid.

My contributions are mostly in terms of labeling and triaging incoming issues and answering the most basic questions.

I've delved into the code base in the past, and it was a bit messy, very difficult to follow along, not enough or incorrect documentation, so I've mostly given up on that. I also haven't used Bookshelf in a while, so I'm not in the best position to fix issues or develop enhancements.

Good luck, you're going to need it :wink:

[mrhwick]

I'm concerned about the fact that we have a seemingly insurmountable wall of issues that would scare away prospective contributors. We need to start forward momentum somewhere, and the pile of leftovers makes it tough to know where to start. I would rather have a few key issues that explain the underlying problems we need to solve and reference the hundreds of other specific instances and examples.

If these old issues are still affecting our users, the history will be there for our reference.

[Playrom]

thank you very much @mrhwick , I think you are a better "head" then me in this moment, I just have currently so many personal organizational problem ...