keystone v5

[htor]

@JedWatson care to elaborate why the new keystone is being developed behind closed doors? as an oss project it's really essential that things are discussed out in the open, that the community can shape keystone into it's next form. if the community is left out (and invite-only members decide everything) the project will suffer. it does now – github issues+pr's are delayed infinitely because there's a secret v5 project nobody knows about. you even have a code name for it. why? when responding to bugs and pr's the devs say things like "please remind us later, we got lot's to do. will come back to you later". who are "we"? this is odd. open source is a collaborative effort and everyone contributing should be treated as equally important. at this point it's very obvious that keystone is being developed by and for the Thinkmill company, not the rest of the world.

[jesstelford]

Excellent question! I'm not sure why the decision was made to keep it private, but I do have a little more information that may help illuminate the situation a bit for you:

• The folks working on v5 (including me) are mostly not involved with v4 / the OSS packages that exist in the wild today
• This question has been raised internally at Thinkmill a number of times without resolution
  • However, there has been a specific intent to test v5 against real world apps to ensure we're not building into a corner, or repeating mistakes from <=v4
• v5 is being run pretty much as an OSS project within Thinkmill - multiple different projects/people using it, with different goals, contributing fixes and features as they come up.
• I want to open source it 😉

[Vultraz]

@jesstelford but will we (the general community) be able to use it when it's done? I really like Keystone, but there are a few really frustrating deficiencies with it (lack of nested object support in lists especially) that I was hoping would get resolved with further 4.x series patches. If 5.x is being developed in secret... will we ever see it? Or are we stuck with 4.x forever :(

[jesstelford]

will we (the general community) be able to use it when it's done?

100% yes! It's been built from the start with OSS in mind.

Ideally before it's "done".

Personally; I have a nascent business built on v5, which I am investing heavily into, so it's not going anywhere, and can only benefit from being OSS!

[jesstelford]

Re-reading your question, there's another interpretation:

will we be able to use it when it's done?

Like I said; it's being used in real world projects right from the start (from very small to very complex), so it's definitely usable.

Will it meet all expectations of an existing KS4 user? Probably not; it's a ground up rewrite, and we simply haven't gotten to all the usecases yet. But we (I) want it to support most usecases.

I'm not sure ~if you've poked around much~ how much you know, but a consistent theme in v5 is the amount of room for customising how it works, from different database adapters, to an extensive hooks API (not React hooks), to a detailed and well structured query schema. Hopefully this means that most usecases can be built on top of core without having to have things changed/added in v5.

[Vultraz]

a consistent theme in v5 is the amount of room for customising how it works

I actually haven't been able to find any info about v5 besides what's mentioned here and one small blurb on the Slack, so this is awesome to hear! I'm working on a project based on v4 right now and I'm banging my head against the wall due to the aforementioned lack of nested list objects, so hearing v5 will have more flexibility in general is great. 😃

Is there any general ETA for when it will become available? Or is that still very much up in the air? I'd kinda like to get my hands on it, since it might solve some of these problems I've been having with v4. 😛

[jesstelford]

I actually haven't been able to find any info about v5 besides what's mentioned here

D'oh, my bad! We haven't actually shared anything yet! 🤦‍♂️ (Because it really is run like an OSS project internally, I forget it's still not public)

Is there any general ETA for when it will become available?

This issue has prompted us (kicked us in the butt) to put together a proper timeline / plan, which we'll share shortly. But know that we've greatly accelerated the timeline to get it open sourced as soon as possible. Thank you! 🙏

it might solve some of these problems I've been having with v4. [..] lack of nested list objects

There is still no nested-list support 😅 The couple of times we've reached for that kind of functionality, we've realised that it's better expressed as a different data structure. Do you have a link to your specific usecase? While as yet we have no plans to build it into v5, it's good to have it in the back of our minds!

[Vultraz]

Do you have a link to your specific usecase?

@jesstelford I do, actually. I have a contract to revamp the entirety of this site. I'm rebuilding it from the bottom up since it's wayyy to old to simply be updated (it was built back in like 2001!). I decided to use Keystone as my CMS (I basically have total free reign over the design), and everything is mostly going fine, but I've run into a problem working on this page specifically.

My Keystone list looks like this:

... but my problem is Keystone only has a TextArray field. I really need a general object array field so I can create entries for the title subchapters without having to create hundreds of new documents in the database and link them with Types.Relationship.

In hindsight, perhaps I should have chosen a more mature, traditional Apache-based CMS (Drupal, maybe), but after exploring Node.js + Express I got hooked and went looking for a good Node-based CMS. I tried ApostropheCMS first, found its documentation lacking, and then settled on Keystone. It really is a good piece of software, but stuff like not having general-purpose Array support for database entries despite Mongoose and MongoDB supporting that out of the box is... frustrating. 😞

[jesstelford]

Oh yeah, we've hit that usecase ourselves; The need for a "write generic content" field.

Instead of going for a generic array-based approach, we've decided on a more specific type which handles that structure in a more formal way which will be part of v5. I don't want to over-promise, but it's way more powerful than nested lists / arrays ;)

[htor]

@jesstelford thanks for your replies

Excellent question! I'm not sure why the decision was made to keep it private, but I do have a little more information that may help illuminate the situation a bit for you:

* The folks working on v5 (including me) are mostly not involved with v4 / the OSS packages that exist in the wild today

i feel the structure and processes inside Thinkmill shouldn't be something the community has to deal with. even if i were to try knowing what's going on inside there, i can't because you don't/can't share this information with the community.

I actually haven't been able to find any info about v5 besides what's mentioned here

D'oh, my bad! We haven't actually shared anything yet! 🤦‍♂️ (Because it really is run like an OSS project internally, I forget it's still not public)

this is suprising, given the size of this project. even if it's run like an "oss" project internally, it's not visible to the rest of the community! this resembles the cathedral model of developing, only your version more closed off and most of the individuals in the (informally) elected core group represent the a company and its interests.

will we (the general community) be able to use it when it's done?

100% yes! It's been built from the start with OSS in mind.

Ideally before it's "done".

Personally; I have a nascent business built on v5, which I am investing heavily into, so it's not going anywhere, and can only benefit from being OSS!

a lot of important decisions are made right now that later can't be reverted and there's going to be frustration around architecture, features and non-revertible constraints, because it's not discussed openly.

"keystone" as a project is open source and is the result of many peoples work and contributions. if you make a new version of it you should honor that work – not open source it when you think it's "done" and to get your bugs fixed.

[JedWatson]

Thanks for jumping in @jesstelford

@htor I'll try and give some background and answers to your concerns.

First off, I've said in several places (here and on Medium) that we (Thinkmill) have been exploring what the future of Keystone could look like if we had a "do over" combining all the learnings since I started the project over five years ago.

Not to re-iterate what I've said elsewhere too much but pushing forward with Keystone 4's architecture just wasn't working out on an effort to results basis, and for that reason we went back to a blank slate several times with Keystone 5. Including one nontrivial implementation we scrapped mid last year.

Everyone has different experiences, but I believe that sometimes creativity needs time, space and patience. Room to experiment without fear or judgement, or accountability. And importantly, without burning other people's time and energy.

For better or worse, that's what we chose to do with Keystone 5. I hope that, when the result is opened up, it will be worth it for everyone who's been involved in its creation, and who's been waiting for it. I also hope it's of value to people who've never heard of or tried the project before.

Keystone is by no means the only significant open source project to operate this way; recently, the React team dropped a massive new feature (hooks) that was a well-thought-out solution to a problem they knew well from both the community and their own use-case, and that was worked on either entirely or mostly behind closed doors until the announcement at React Conf last year. Because they took the time to get it right, it was universally well received.

Now, I'm personally glad that when we decided to scrap our previous take on "new keystone" it was behind closed doors. It was the right call, and we made it without fear of burning anyone outside Thinkmill. Nobody had based a project on it, nobody had wasted time contributing to it, and our next iteration (what will be the public version) was much better. Any project going through this level of reinvention is complex, and I've seen people get frustrated with the apparent lack of progress on Keystone, but I've seen other projects burn contributors and early adopters worse. Neither approach is perfect.

So that's what's been going on. The good news is we're basically done with all that and are ready to open it up - partly because we're confident and excited about what we've done, and partly because I think keeping it closed is really diminishing returns at this point. It was always meant to be shared.

open source is a collaborative effort and everyone contributing should be treated as equally important. at this point it's very obvious that keystone is being developed by and for the Thinkmill company, not the rest of the world

Lessons have been learned about the sustainability of open source projects, and if Keystone works for Thinkmill that's absolutely a good thing for anyone else who's using it or contributing to it. We got into problems maintaining version 4 because it was a beast, but stopped being the core of our business, and that's unsustainable. That left it to a few maintainers (including myself) who tried to continue on personal energy alone, which is a recipe for burnout and frustration. Keystone 5 is, among other things, a massive investment to change that so the project has a bright long-term future.

That doesn't mean Keystone is developed only for Thinkmill. We can freely give our work away, and benefit from the contributions of others, in a healthy ecosystem. These two things are not in conflict. All the documentation, guides, and support has been (in the past) and will be (in the future) done because fundamentally we love sharing what we build with others and collaborating with incredible people all around the world.

[Vultraz]

Honestly, @JedWatson 's reasoning makes a lot of sense. I myself am the project admin and lead developer for the open source game Battle for Wesnoth, and we've recently begun attempting a total ground-up rewrite in a new engine as our current handmade engine has proven inadequate for what we want to do. Design decisions made long in the past continue to make life hard for us today and has resulted in burnout and very little enthusiasm to continue work with the current codebase.

The Wesnoth 2.0 effort is being undertaken in a semi-private context (private Discord channel, limited repo access) since I felt it would be better and easier to keep the initial team small, and to avoid unhelpful comments from the peanut gallery and naysayers. It's a huge undertaking and having too many cooks doesn't help.

Given that Keystone 5 was also a total rewrite, it makes a lot of sense that they would develop it in private. As Jed said, it gives a lot more flexibility to experiment and toss out what doesn't work.

Really looking forward to seeing what Keystone 5 brings once you finally make it public. v4 isn't bad at all, and if you've managed to clear out a lot of cruft I'm sure v5 will be even better!

[jesstelford]

Including one nontrivial implementation we scrapped mid last year [..] I'm personally glad that when we decided to scrap our previous take on "new keystone" it was behind closed doors.

I want to reinforce this point with my own experience; I built a non-trivial project on that version of keystone (which was also a total ground-up rewrite). It got a lot of things right, and was really nice to work with. But there were still some fundamentally rough edges which meant either a lot of work poured into fixing it, or start from scratch (again) and also incorporate other learnings we gained from using that first rewrite. It also meant rebuilding the project. Thankfully Thinkmill was in a position to be able to spend that time/effort, and it's better off for it.

If we had open sourced it, then ditched it, then open sourced another one (all while KS4 is still alive and well), we felt it was not great for anyone.

Now with the upcoming v5, I'm really happy with the results so far, and we're at the point where we want to get everyone else's feedback (and help) - there are still features missing, and still room to shape the future on top of the core we've built behind closed doors, but I'm sure you'll really like what's been done so far!

I love the passion of the community - it's great to have you participating and raising these concerns @htor @Vultraz, thank you! <3

[Vultraz]

@jesstelford I'd like your opinion. Do you think it's worth putting my v4-based project on hold for awhile and wait for v5? From what you say it sounds like v5 will have a lot of things that would make my work easier, and I don't know if it's worth trying to clobber solutions together using v4. Figuring out a scaleable and maintainable (what I produce can't be fragile) solution for the lack of a generic content field especially is giving me massive headaches. The best thing I've come up with is to use the Code field and require manual editing of JSON data, but as you can imagine that is extremely apt to break.

Do you think it worth waiting, in your professional opinion? I'm really trying to weigh continuing with v4 or waiting a few (hopefully) months for v5.

[jesstelford]

I honestly can't say one way or the other. Because v5 is a ground-up rewrite, there are some very fundamental differences which may mean you have to port a lot of code over (including your client/rendering code), ultimately outweighing the benefits of the new field.

But you shouldn't have to wait months to see v5 - we've got some immediate plans which should help you get an answer sooner.

[Vultraz]

Oh, that's awesome! I'll definitely be keeping a lookout.

[htor]

@jesstelford

I want to reinforce this point with my own experience; I built a non-trivial project on that version of keystone (which was also a total ground-up rewrite). It got a lot of things right, and was really nice to work with. But there were still some fundamentally rough edges which meant either a lot of work poured into fixing it, or start from scratch (again) and also incorporate other learnings we gained from using that first rewrite. It also meant rebuilding the project. Thankfully Thinkmill was in a position to be able to spend that time/effort, and it's better off for it.

If we had open sourced it, then ditched it, then open sourced another one (all while KS4 is still alive and well), we felt it was not great for anyone.

i agree that a small team doing a total re-write would be more efficient in the short term – but being a small team also makes things much harder:

@JedWatson

Lessons have been learned about the sustainability of open source projects, and if Keystone works for Thinkmill that's absolutely a good thing for anyone else who's using it or contributing to it. We got into problems maintaining version 4 because it was a beast, but stopped being the core of our business, and that's unsustainable. That left it to a few maintainers (including myself) who tried to continue on personal energy alone, which is a recipe for burnout and frustration.

to me it looks like the biggest problem here isn't really about maintaining a "beast", but rather falling into the classic trap of "let's rewrite it all to get everything right this time" for the nth time. how many rewrites are we going to endure before we can get some eyes on pull requests, issues and support? the community around github and slack isn't exactly thriving. many people have lost interest in the project because their contributions were not "aligned" with where thinkmill is headed. how can they know when everything is secret? discussing issues with a core dev is like talking to a salesperson.

Keystone 5 is, among other things, a massive investment to change that so the project has a bright long-term future.

how is the new keystone going to change the fact that software complexity grows over time? what is going to change?

[JedWatson]

@htor there's a lot to unpack here. by "a beast" I mean:

• monolithic (this wasn't the idea at the start, but it did end up this way)
• really difficult dev/test loop (including frequent invalid CI failures)
• hard to improve incrementally

I wrote all this up on Medium here when we launched keystone 4.

The tipping point for me was when I spent weeks of my time working on nested lists - by far and away the most requested feature from the community - and couldn't get it over the line. I was fighting a losing battle and realised that everything in Keystone felt like that: hard. Something had to change.

falling into the classic trap of "let's rewrite it all to get everything right this time" for the nth time

I don't know where you're coming from or your perspective. But it sounds like you're making massive assumptions. I think we've done the opposite - knowing when to go back and really fix some things we haven't given ourselves room to deal with for years, which have left the project falling behind.

discussing issues with a core dev is like talking to a salesperson

I don't actually understand what you mean by this. tbh it comes across a bit offensive, which I don't assume you mean to.

how is the new keystone going to change the fact that software complexity grows over time? what is going to change?

Really good question.

First, it's been designed to have a tight core with a strong extension system. This was probably the biggest failing in Keystone 4. This model has been proven successful by many of the largest, most useful open source projects and creates a much healthier community around them.

I've sadly blocked a number of contributions that people have wanted to make (and not been able to do my own) because Keystone's so complex and already handles such a wide variety of use-cases that solving one person's problem often breaks somebody else's solution. I think you're conflating that with "not being aligned with where Thinkmill is headed".

Other projects like babel, eslint, vscode, etc all have one thing in common: a well architected core system that empowers the community to build different tooling on top of them. Sometimes they get it right from day one, sometimes there's a painful transition when the main author(s) realise they need to re-architect their project into a platform.

To be clear: enabling and empowering a stronger community around Keystone was a major design goal for the new version. The right architecture is, in my opinion, one of the best ways of achieving that.

Second, we're clear on what Keystone is. Which I don't think I have been since day 1. Is it a website framework? a cms? a set of patterns for developing node.js apps? all of these?

I think that lack of clarity hurt the project in the long term, but it's taken years to figure out how to fix it.

Now, I'd describe keystone like this: (schema) => ({ api, adminUI })

You define your schema (along with extensions and rules) and you get an API and Admin UI based on that to work with. You can build a website, cms, or app with that. But the boundary is much clearer, and I think it'll be much easier for both maintainers and contributors in the future now we have that clarity.

[w01fgang]

Thank you @htor for the opening this topic! And thank you @JedWatson for finding time to respond! Once I tried Keystone.js, I fell in love with it. I was happy that Max Stoiber joined the Thinkmill team and worked to make it better. And I was really sad to see that Jed, Joss, and other contributors don't have time to continue the project. But reading these comments I'm happy to know that it wasn't abandoned, and it's alive! Thank you all guys!

The only thing I would add it's the possibility to run Keystone in the serverless environment like AWS Lambda or, my favorite, Zeit Now 2.0. It would be a great feature! Thanks again and looking for v5.

[Norbz]

The fact that keystone core is being rebooted by a small team seems reasonnable for me, any project needs a clear direction, as @JedWatson stated...

BUT, what's sad, once again, is the lack of communication. When this project was almost dead, and issues where addressed every two weeks about the status of the project, we had a new beginning, with the slack, V4 out of beta, and the promise to finally get some PR merged, more communication, shared roadmap etc...

Then what ? V4 came out of beta, sure, but then everything stopped, once again. We all understand that, as an open source project this may take more time, that personnal life may get into the way of deadlines, we all understand that, but even @autoboxer seemed clueless on slack about what was going on...

The very simple fact that we have news only when complaining in a github issue isn't healthy, and that's what makes people angry and suspicious about the future. Why not communicate about the work being done, why no roadmap ? Look at Strapi, it's develop by a company, they don't meet every deadline, they changed some core features, but at least we know why, and what's going on. It's exciting for the community.

I love Keystone, and I definitively look forward to testing V5, and start working with it, but this lack of communication hurts everyone : The users, the project, and in the end, even the core and maintainer team that gets way less credits than they deserve because of all the frustrations.

Cheers !

[j-holub]

Will v5 get rid of Cloudinary or offer an easy image gallery with locally stored images? I want to create my photography website and started doing it with KeystoneJS and then I found out, that the default Imagefield would store all my images on Cloudinary (which I do not want for obvious reasons). And I've seen workarounds using LocalField, but I'm not sure if that will give me AdminUI interface that the CloudinaryImageField gives me.

By the way I think your call on starting v5 behind close door was the right one, there's points where OpenSource is great and there's others where it's just not.

[jesstelford]

@w01fgang

the possibility to run Keystone in the serverless environment like AWS Lambda or, my favorite, Zeit Now 2.0. It would be a great feature!

That's a great idea! We'll have to add it to our backlog 👍

@Norbz Thanks for the feedback - we're going to work harder on clearly communicating the roadmap and current state of things. While Thinkmill is backing the development of Keystone, it's still a balancing act of time and priorities as with any other OSS project.

@00SteinsGate00

Will v5 get rid of Cloudinary

We will continue to support a Cloudinary image type.

the default Imagefield would store all my images on Cloudinary (which I do not want) [..] but I'm not sure if that will give me AdminUI interface that the CloudinaryImageField gives me.

You'll be glad to know that as well as a CloudinaryImage type, v5 also has a generic File type which can be used for images and much more!

[j-holub]

@jesstelford v4 has the generic File Type as well which I used to implement images, however, I don't see them as images in the admin interface, which isn't all that great but something I can work with for now. Will this change with v5? (or is there some way to add this functionality to the Admin Panel that I'm missing. Not trying to start a tech support discussion in this thread though).

[jesstelford]

I don't see them as images in the admin interface

Oh yes, in v5, if the File is detected as an image, it will show a preview in the Admin UI ;)

The Code❤️Design conf logo

I'm not sure about how to do the same in v4, sorry.

[Vultraz]

Man, v5 is gonna be awesome. ❤️

[j-holub]

I don't see them as images in the admin interface

Oh yes, in v5, if the File is detected as an image, it will show a preview in the Admin UI ;)

The Code❤️Design conf logo

I'm not sure about how to do the same in v4, sorry.

No worries, thanks for great news. I'll be looking forward to v5 and use the normal file field in v4 in the meantime.

[Vultraz]

@jesstelford Two more quick questions about v5, if I may:

1. Will any of the APIs underlying the Admin UI (such as search and filter functionality) be made available to use elsewhere in the app? For example, I notice this GET request sent when I use dropdown menu filtering in v4: GET /keystone/api/documents?basic&search=leg& 304 11.814 ms. I was thinking it might be useful if there were some public-facing API to allow the same sort of in-text search... or maybe that's possible already with Mongoose and I haven't found a way 😬 I'm not thinking clearly right this moment.

2. Will v5 have any sort of native React support, or a way to easily integrate with React? I noticed Jed mentioned native React support in his Medium post, but I was just wondering if that did indeed materialize.

Thanks!

[jesstelford]

1. Will any of the APIs underlying the Admin UI (such as search and filter functionality) be made available to use elsewhere in the app?

Yes! The Admin UI is 100% powered by the same API your application will consume.

1. available to use elsewhere in the app?
2. native React support

v5's focus is the API & AdminUI, being as non-prescriptive about your app layer as possible.

I have built a complicated React-based app on top of v5, and it's been a great match 👍

[jesstelford]

Just a quick update on this; We've been furiously working to get things in a state which we're happy to share. Turns out when we shook the blanket, a bunch of bugs crawled out 🐛

We're getting close to having something to release publicly. Bear with us ❤️

[Kiina]

Maybe I missed that part but if it's a complete rewrite will there still be compability to upgrade existing v3/4 projects? I still have a few running on the latest 0.3 beta due to a few regressions in 4.0 and it would be interesting to know if they could be updated to 5.0 in a reasonable effort.

[b1tn3r]

@jesstelford This is really good news, and I'm glad I checked out this thread.

I've been working on directly upgrading the Keystone-js project for awhile now that my own private project is build on top of. The first thing I noticed about it when I started editing the source code was that it could be structured and configured better for a superior developer experience. There have been numerous times when I seriously thought about doing this myself to switch it from browserify to webpack, give it hot module reloading and other configurations that I felt were sorely needed in the Keystone dev environment. Only reason I didn't was because it would've taken forever by myself, and now I'm definitely glad I didn't with this news.

I don't know when you're planning on releasing v5, but I have a number of features I have been working on (and some finished) directly on top of v4, could you tell me if any of these are going to be included in v5 so I know whether to stop working on them until v5 is available? Global search functionality, a new customizable UI, user-specific settings for the UI including appearance, security, and app/account bindings, user notifications and notification panel, user permissions and authority level, react-intl and Language provider, and built-in user registration. I've also been contemplating setting up a central file/image store as well so images and files can be easily accessed, reused and managed from a central location and UI.

Do you know if any of these features would be unnecessary with the new version? Or if v5 already includes or partially incorporates them?

Thanks in advance and I'm looking forward to v5. Here's to me hoping and fingers crossed that you configured the dev environment similar to Max Stoiber's react-boilerplate and used styled-components.

[chochihim]

Hey guys, I find that keystone 5's github repository can be accessed publicly now! I am sure that the team will share more information to us soon.

[b1tn3r]

@chochihim Thanks for the quick update. I'll check it out.

[JedWatson]

It's out! we quietly made it public last friday, but are still in a rush to get on top of the documentation, getting started experience and some papercuts before we say too much about it or draw attention to it.

That said, anyone who's made it this far down the thread is certainly welcome to check it out and let us know what you think 🙂

The repo is here: https://github.com/keystonejs/keystone-5 (give it a ⭐️ if you like our work!)

And the new docs site is available for preview here: https://v5.keystonejs.com/

We're still working on getting a demo up, but the v5 readme has instructions on how to clone the repo and run the demos locally.

There's a lot to do but we're really excited about how it's shaping up and and really glad to start being able to share the future of KeystoneJS with everyone - we hope you all like it! 🎉

@Kiina we're still working out what the upgrade story will be like for 3/4 => 5. Many of the concepts are familiar but version 5 is a complete rewrite in a way that hasn't happened before in Keystone's history and not all the concepts have come across. So it will depend a lot on how you're using Keystone 4 (and also, Keystone 5 is not really ready for prime time yet)

We'll make some noise both here and in the new repo when we're ready for people to seriously consider migrating and see how smooth we can make the process. In the meantime, feel free to have a look at the demo projects to get a feel for what's to come.

@b1tn3r I think you'll find a lot of your DX concerns are solved in the new version. It was great being able to start again with a completely modern setup for Keystone and we've found it much more productive to work on.

And definitely some, but not all, of the things you've mentioned are either in place or on the near-term roadmap including an extensible Admin UI, a rich permissions model for the Admin and API, and a react-friendly dev environment for the apps you build on it. We've also been talking about built-in user registration and internationalisation.

Hopefully the other things will also be easy (or become easier) to build on top of keystone itself, or contribute!

[Vultraz]

Whoot! I must say, even the new docs site looks so much sleeker than before. :D One thing I can't really find - is it still built on top of Express? The section on custom routing seems to imply so.

[jesstelford]

@vultraz Thanks for the kind words ♥️ The team has worked very hard to get it looking this great!

Yes, the API & admin UI are both built on top of express, and with a custom server you're able to extend that express server however you like: https://github.com/keystonejs/keystone-5/blob/master/README.md#custom-server 👍

[b1tn3r]

@JedWatson The new project looks really awesome guys. The monorepo structure is definitely a big step up from what you had before, and the user permissions and access control look very extensive. Also, I'm glad you removed the limitations and focus on Node template-driven architecture of Keystone 3 and 4. I have no idea how you're going to make that upgrade story from 4 to 5 though; it seems like an entirely different project.

My biggest question for you guys is when do you think it will be close to a stable release if you had to make a broad guess?

[JedWatson]

@b1tn3r thanks 🙂 we're really happy with how it's going, and our momentum is accelerating. Which is a really good sign that we got some things right.

Keystone 5 seems quite different from keystone 4 right now but a lot of the core ideas are the same. Lists, field types, etc. We're letting ourselves not worry about the upgrade story yet so we can focus on getting the new architecture right, but I've been thinking about it for a while and am hoping that when things settle we'll be able to make it an easier transition than it looks like at the moment.

In terms of timeline, I'm really hesitant to make any promises about it, because I can't say for sure we know what even needs to be done before it's stable. We're still working some things out.

I'd say that sometime this year it'll be something we're ready to move out of alpha and call "stable". Probably not in the next month or two. So broadly speaking, hopefully sometime in winter (or summer for our northern hemisphere friends)

At Thinkmill, we are building production projects on it. So from a functionality perspective what's there is reasonably reliable. We just want to continue iterating on the API, Admin UI and other fundamentals so we're cautious to put big warnings around that. It's also possible we'll remove / meaningfully change a few things. If you're happy to keep up with the churn, or want to actively contribute, it's in pretty good shape.

[laurenskling]

It's very nice to hear you've been working hard on keystone-alpha and sharing this with the world. A project I am looking forward to working with and contribute to. We cannot stop to thank you for the work all of you put into this.

But... Activity on keystone 4 has dried up for over 2 years. A couple of times, promises were made to get some hands on again. It never really lasted long. Creating a new feature, like React Hooks, behind closed doors isn't comparable to what you have built with keystone-alpha. React Hooks is a standalone optional feature that was added to the codebase, built while the codebase was still heavily under maintenance. By creating a fully new project called keystone-alpha, which has actually nothing to do with keystone except the history, you have completely dropped all support of keystone4, by which it is now dead.

I don't see a compatibility to the new codebase. Massive changes are brought in. I won't be upgrading my keystone projects to keystone-alpha. It's too much change.

There has been signs of do-overs and reboots. We are probably all excited to try this new codebase for our future projects. But all people asked for in this repo was some support. There are many people out there running keystone projects that weren't getting bugfixes and now probably never will. All we wanted was to get sortable working correctly, or a bunch of CloudinaryImage features, that were lost, working again. I have a PR open for just 1 line of code that is requested by many people, but it's just not getting attention. The community wasn't given the powers to take charge. All of our keystone projects turned unsupported legacy. Which saddens me.

[aleygues]

I totally agree with @laurenskling. I used KeystoneJS for production projects the last 2 years thinking the soft was ready to be used in prod, but by releasing a new V5 totally breaking with the old version and stopping support for the V4 as @laurenskling pointed out, you make me rethinking about the maturity of KeystoneJS for production projects.

I hope you'll keep this in mind for futur big releases (V6, ...)!

[jesstelford]

To be clear; v4 isn't going anywhere. It will continue to work in perpetuity. npm install keystone@^4 will always still work.

We've also got plans to make sure there's no ambiguity so folks using v4 won't accidentally install/use v5 packages.

If you have a successful app today which is built on v4; it will continue to work even after v5 stable is released 👍

OTOH; If you have the desire/time/budget to upgrade to v5, it will be more work than upgrading v3 -> v4 (we recognise that and are accepting the trade-off to enable all the great things v5 brings), but as @jedwatson said:

[We're] hoping that when things settle we'll be able to make it an easier transition than it looks like at the moment.

[autoboxer]

I have to agree with @laurenskling and @aleygues. The issue isn't that Keystone 4 will cease to exist, the issue is that Keystone 4 literally just exited beta, and there is zero support or communication from the team. I volunteered to help, and was graciously brought on, but with no communication from anyone who knows the codebase, there's no way I can be effective. There are glaring issues with Keystone 4 as it stands now, such as broken session management, broken drag and drop, issues with dates being handled differently in lists vs in models (both UTC as an option and the displayed format), and fundamental things like Boolean fields can't default to true, among a laundry list of other issues that can't reliably be fixed as the people who know the code aren't present to help. I have a lot of respect for anyone who builds a system like Keystone, and I have several projects built using it, but the lack of support and communication makes it very difficult to get excited about the next big thing. I have fixes I'd like to make to Keystone 4, will hopefully be able to help with Keystone 5 features, and genuinely hope a critical mass can be hit where the community drives this version to be amazing. I'd love for this to be the moment where Keystone support matches Keystone development, and communication becomes a front and center focus, but I think it's important to be honest about why there may be some unrest around how prior versions of Keystone were handled.

[jesstelford]

@autoboxer Very valid points. I'm sorry; I didn't mean to trivialise your and other folks experience & situations with Keystone v4.

To reiterate: the team working on v5 are not the same as the v4 team.

Whether this is better or not is yet to be seen, but as a v5 core contributor, I'm excited for the future of the project and community.

[JedWatson]

a new V5 totally breaking with the old version and stopping support for the V4

To be honest, this response is part of why we've been hesitant to open up Keystone 5 yet. I wonder, if we'd kept it behind closed doors for another few months, how much this could have been corrected (not that this changes anything, I'm just talking about the impression being created)

Regardless, the overriding request was for us to get back to working in the open, so we have - and I do think it's been the right call, overall.

If anybody thinks we're abandoning Keystone's history or community, my apologies for leaving that possibility open to interpretation. We're not, and I'm not. But for reasons that have already been discussed at length in issues here, we're taking a long path forward that I believe will be the best outcome in the years to come.

you have completely dropped all support of keystone4, by which it is now dead

Again, this isn't my view. I totally understand why the timeline is frustrating, but all this is just that - time.

Here's something I should share: there was a prototype worked on that ran purely against postgres, with a significantly different API. I pushed back hard on that, and forced two things to happen for the Keystone 5 project we've just opened up:

1. We got the mongoose adapter working first, so there's a viable upgrade path for projects that use the mongoose API and MongoDB.
2. The List syntax is, if not API compatible, very similar to Keystone 4

These decisions were made, and a lot of work done, specifically to benefit the people who've been on Keystone's journey with us.

Additionally, Keystone 5's architecture is very piecemeal - as many parts are independent as possible. So even if the first version post-alpha doesn't include a lot of Keystone 4's features (a good example would be the express server helpers for websites), it's much easier for the community to step in and add these features if there's value to bringing them forward for other users.

I see this modularity as one of the most important changes to keep Keystone 5 healthy in the long term. A lot of the features and innovation that people tried to contribute to v4, or that complicated the project over time, will be manageable and independent from the core in v5. Most other major projects that have been successful and built thriving communities - eslint and babel are two that come to mind - have implemented a similar principle.

I used KeystoneJS for production projects the last 2 years thinking the soft was ready to be used in prod

Maybe I'm missing the point, but if 4 has worked for the last 2 years, doesn't this mean it's ready to be used in prod? I know of a very large number of projects that have been built on it, despite the issues that @autoboxer and @laurenskling mention.

I keep mentioning that v4 is hard to work on. Part of that is because I spend as much time (when I'm working on it) trying to make sure that a change doesn't break for someone, as I do trying to fix things. The number of times I've jumped into trying to merge a PR or fix an issue, and it's escalated way beyond the time I've hard to spend on it, is ridiculous.

Rule #1 of building something big: everybody complains when you don't add or fix things, nobody notices when you don't totally break something for thousands of users, even though the second is way more work.

the issue is that Keystone 4 literally just exited beta

Let's not diminish what happened there. Several people took several totally unpaid days (and nights) to work through what we thought was a list of the critical issues so Keystone 4 got into a stable state. The issues today are a mix of questions, support, feature requests and actual bugs. So part of what's holding us back is the signal-vs-noise problem.

anyway enough ranting from me. I really appreciate the comments from everybody who recognises how much work this is. Let's talk about the future.

Now that Keystone 5 is out and the path forward is becoming clear, I think we can do something about the state of Keystone 4. Also it sounds like several people have an understanding of what's broken that needs attention, which is great, and has been hard for me to understand from the current state of issues and PRs.

I'll write up a new issue with a summary of how I think it can work, and we can go from there.

[laurenskling]

I hope my answer didn't come across too harsh, we from The Netherlands are known for being straight forward.

Let's think about where these topics come from. Not from your lack of trying to get Keystone forward, which seems pretty obvious with the new codebase (still have to take a closer look, am looking forward in doing so!).

Questions about what the plans for the future are seem to become a yearly thing now, https://github.com/keystonejs/keystone/issues/4281 in 2017, https://github.com/keystonejs/keystone/issues/4551 in 2018 and now again. Why are these questions asked? Because it is very unclear with the plans for the future are!

I don't think I even mind that you took the step to redo the entire codebase, if you read back you have actually pretty clearly stated you were going to do so: https://github.com/keystonejs/keystone/issues/4551#issuecomment-380143967 But in the same topic, @autoboxer was assigned to be the new Keystone 4 hero; but I (and himself if I read correctly) haven't seen him got a lot of help with his efforts. Commits since the 4.0.0 release have mostly been docs updates and the few things that are merged are never released in a 4.1.0 for example, so we are still working on the version last created juli 2018

It seems clear to me, that questions about where this project is going is not an attack on your efforts, but simple a cry for help from people to get some insight about what to expect.

If you had clearly stated: "you know guys, codebase is too hard, we are not going to get big features into keystone4 anymore, we will be working on a new codebase under water with a similar bit different API. @autoboxer will do some minor bugfixes while the new codebase is done and we'll help him do so." (which has actually been the story, right?) If this was the message april 2018, there wouldn't be this topic. There would only be: "OMG OMG you've shared the new codebase 🦄".

I understand you've redid the codebase, I think it's a good decision to step away from the 2013 codestyle, but I would have loved to see ongoing bugfixes for 4 in the meantime and for a little while after 5 would be released. I can't tell my clients to just wait a couple of years for some fixes and pay me to upgrade to the codebase to get it. I'm pretty sure this is where most people in these topic come from. Sure, it'll be nice to have nested lists, but if it can't be done in the keystone 4 codebase, it can't be done. It not here right now and still we are running all these projects..

Getting Keystone 4 to a stable version with as little bugs as possible, that's all we ask for. (while looking forward to running new projects on your awesome new codebase)

[JedWatson]

I added an issue to discuss v4 maintenance: #4913

[JedWatson]

Thanks for the clear and constructive messages @laurenskling. Challenging, and rightfully so, but not too harsh! You've made a lot of really good points that I agree with 🙂

FWIW I think part of the mixed messaging here was because we weren't that clear ourselves. At the start, Keystone 5 was an experiment (although as you pointed out, I did tell everyone what we were doing...) and now that we've found something that's working, it's a lot easier to say what we're doing.

Probably also, we really did try several times to reboot Keystone 4 within the codebase, but they all ended with frustration. Given the annual questions, we could have called it sooner.

All this is clearer in hindsight, and a lot of lessons have been learned ❤️

What you described is now what I want to do with v4. Let's clean the project up now that we have a clear scope for maintenance, and I'll make a higher priority for reviewing and merging bug fixes and minor updates so we don't leave people hanging on 4.0.0.

[htor]

thanks for answering!