Drupal 10 with aegir/drupal-project-composer

[macmladen]

As Drupal 10 is slowly approaching and beta is available, is there any chance we can expect it on Packagist?

[macmladen]

Now that Drupal 10.0 is available, when can we expect it in Packagist?

Or you suggest some our way?

Or the drush will be a serious obstacle?

[omega8cc]

Drupal 10 support in Aegir is still under development, because it breaks backward compatibility with Drush 8 which even if supports (mostly) PHP 8.1, is based on older version of Symfony. We will announce the progress once we have the major obstacles removed (which are pretty serious this time).

_{Sent with GitHawk}

[macmladen]

Any progress?

[omega8cc]

We will announce compatibility with Drupal 10 once it's ready to use. The work is pretty complex this time.

[macmladen]

Is there any way we can help? I suppose it is stuck on drush?

[omega8cc]

Here's the situation:

Aegir Provision -- Symfony 3.4.0 (PHP 7.4 max) (Drupal 6-9)

Drush 8.4.12.4  -- Symfony 2.8.3 (PHP 8.2 max) (Drupal 6-9)
Drush 10.6.2    -- Symfony 4.4.0 (PHP 7.4 max) (Drupal 8-9)
Drush 11.5.1    -- Symfony 5.4   (PHP 7.4+)    (Drupal 9-10)
Drush 12.0.0    -- Symfony 6.1   (PHP 8.1+)    (Drupal 10+)

Drupal core 7   -- no Symfony  (PHP 8.2 since 7.94)
Drupal core 8   -- Symfony 3.4 (PHP 7.4 since 8.8.3)
Drupal core 9   -- Symfony 4.4 (PHP 8.1 since 9.3.0)
Drupal core 10  -- Symfony 6.1 (PHP 8.2 since 10.0.0)

To support Drupal core 10 we need to make Aegir fully compatible with PHP 8.2, then both Drush 8 and Provision work on Symfony 6 without breaking support for hosting Drupal core 6, 7, 8 and 9. Quite a challenge if you count all relevant Drush dependencies.

[joseph-bluedrop]

it is obviously a complex work to support all drupal versions, but we need some visibility especially since Drupal 9 EOL is november 2023. is there any way we can help? do you think you will be able to support drupal 10 before drupal 9 EOL?

[omega8cc]

Yes, we aim to finalize support for Drupal 10 during the summer— meaning July or August the latest. We have a plan B for this if the Drush/Symfony workarounds will not be sufficient, so it will happen one way or another.

_{Sent with GitHawk}

[aDarkling]

As I don't know what Plan A or B are, may I offer my suggestion that there should just be a clean break between the Drush instances.

As with many others, I also have a few D7 sites that I haven't migrated yet. Some may never be migrated because the client won't pay for that. I work with small Charities.

To avoid the inevitable Drush collisions, I fully expect to host those sites on a separate BOA 4.x server. Forever, if needs be. Any security issues that occur will happen over there. The Clients have been warned that that version is EOL.

This would allow you to not spend time worrying about Drush incompatibility issues. Drush 8 for BOA 4, Drush 12 for BOA 5. All migrations between them are already strictly controlled by the higher version, so there would be no collisions.

I look forward to hearing your thoughts. Thank you so much for working on this!

[omega8cc]

That’s not a solution to the problem we have, unfortunately.

Here are some hard facts and requirements BOA must meet to stay relevant in this context:

1. Drush is no longer supporting standalone mode for years and system Drush 10 is the last version still working on command line, albeit with issues caused by conflicts with dependencies it shares with Drupal core due to constantly changing versions of these dependencies.

2. The only Drush version Aegir will ever be able to use is Drush 8. This will not change not just because Hostmaster is running Drupal 7, but because all Drush versions newer than 8 cannot be used reliably as standalone versions, with version 12 declaring officially that it’s not possible to use as standalone no matter how hard you try.

3. Any Drush version newer than 8 should be actually used as site-local exclusively, so Aegir must stop deleting local Drush.

4. We must support upgrade from Drupal 9 to Drupal 10 so any idea to make separate version is simply wrong and repeating the biggest Drupal mistake of ignoring the lack of upgrade path for Drupal 7 which resulted with huge loss of market share and is now doubling down on pressing hundreds of thousands D7 sites to drop Drupal (pun intended). We don’t want to commit this type of project suicide.

Basically, we either make Drush 8 compatible with Drupal 10 or we will drop Drush for Drupal 10 sites with some sort of magic in the backend to replace Drush.

We have already fixed the PHP versions compatibility for Aegir and Drush 8 so we can use PHP 8.1 without issues.

Now we either fix/clone Drush 8 for Drupal 10 or use other backend tools to make Drupal 10 supported within Aegir context.

_{Sent with GitHawk}

[joseph-bluedrop]

Any new updates on the subject? are you going to proceed with plan A or opt for plan B?

[TwoD]

I would personally disagree with trying to keep backwards compatibility with older versions. Not having a forced (and most likely very unreliable) D7 to D8+ upgrade path, and instead going with migrations, was a huge relief for us. This was also the direction I expected Aegir to go with.

A forced backwards compatibility with D9 instead of a clean cut for D10 would feel like dragging along old baggage nobody neeeds, especially since upgrading from D8 through to D10 is way easier than going from D7 to D8, so we don't need D9 compatibility (or rather drush 8) at all. D9 is even less relevant than D7 now so just ignore it and step over the Drush incompatibilitites.

Either way, time's pretty much up, what's plan B?

The only realistic choice we have - to be able to move all our sites to D10 in time - is to completely drop Aegir, or hack it ourselves to rip out anything not relevant for D10.

Aegir 5, while promising, sounds like it's a massive undertaking and far from being done in time. Not to mention it appears to to rely on Kubernetes and the kitchen sink, so we would have to replace most of our infrastructure anyway, negating any benefits of being able to easily pull in any existing multisites on the platforms we have.

If there is a plan B, we'd really like to know before essentially starting our own fork very soon.

[omega8cc]

Well, it’s either going to be the biggest announcement since BOA first commit in 2009 or it’s going to be as simple as possible set of dirty workarounds, because we have worked for months on two approaches — one very deep rewriting of entire system including Aegir backend and our Drush fork, and another less complex but leaving Drush legacy partly behind and opening new possibilities for Aegir and the BOA stack, although with shortcomings initially.

We didn’t know which approach is going to work better so we decided to work on both, even if it resulted with enormous effort normally taking years in the open source community.

The problem we face now is twofold: we don’t have any time left because it should be already deployed a month ago, and second: once we pick the first approach based on deep rewrite, there’s no return to the old and its workarounds, while it couldn’t be rock solid despite extensive testing and debugging, but if we pick less spectacular approach based on workarounds we can still move to complex one in the future.

You know, it turned into rabbit hole already but we have to remember that for BOA users it doesn’t matter how it was done, the only thing which matters is having complete Drupal 10 compatibility and overall reliability required in production environment, with as little changes to workflow and existing external scripting as possible.

We have a hard internal deadline for deployment next week so basically it’s the last weekend for testing and making decision on the approach to deploy on all our servers, although we are leaning towards the less ambitious but otherwise working reliably, and go with deep rewrite option later once it's published as separate BOA release and tested by the community.

Regarding the backward compatibility, we do appreciate everyone's input, even if disagree with some suggestions, not because of our preferences, though, but because of majority of BOA stack users requirements, which don't leave us space for some frivolous development moves the Drupal itself has done -- now scrambling to help hundreds of thousands of D7 sites owners to migrate to D9 -- too late probably for many of them because too many simply dropped Drupal already over the years since D8 introduction. We will not repeat this upstream project mistake.

Perhaps the confusion about this approach is coming from the fact that Drupal developers usually have limited responsibility in managing codebase lifecycle, while our responsibility as web stack developers is very different and we can't afford any radical changes leaving many users in the dust.

Let's hope we are able to release new BOA with D10 support in the next few days.

Cheers!

[macmladen]

Dear Omega8cc, first and foremost, I think no one can thank you enough for the work and effort you made so far. I have been a speaker at Drupal events and on several occasions, I presented BOA and how one can create their own server and Drupal hosting. I've helped some make and fix it and tried to be sensible and active as much as possible.

I totally understood the problem from the beginning and I know it is a tough one.

However, I'd vote for the other, 'less complex but leaving Drush legacy partly behind and opening new possibilities for Aegir and the BOA stack'

The rationale is that we really need to move forward, Drupal 10 is almost a year old, it is 10.1 at the moment and it seems to steer away from some solutions which may make it even more complex to cope with in the future.

Whatever the radical solution is, I vote for that direction and for the future Drupals. Not sure which Drupal versions could be supported by it (8, 9, 10, 11? ...) but the future is more important in my opinion.

Those who need Drupal 7 (myself included) may keep the instance of the current BOA and create a new one for the 8+ projects. So no one is left behind, just frozen in time and the solution will continue to work for them but as with any legacy technology, it will be obsoleted by PHP which will be obsoleted by OS.

As you already pointed out, many gave up on their Drupal 7 and have gone to other platforms or solutions or migrated to Drupal 8 -> 9 -> 10. Or, at least, should be forced to by agencies and developers as they will struggle to find support and maintenance.

However complicated it is to move sites to the new instance, it is something I see I have to do to be able to move forward.

That is why I do not expect to update the server (as I do have two Drupal 7 sites) but to create a new one and move others to the new one.

Those that have updated BOA and are only in the Drupal 8+ domain will probably have a smoother path.

[omega8cc]

I wish it was that simple. It's not, because we have many servers (not just our customers) with hundreds of D9 sites, some D8 sites and many D7 sites where moving to different server is not an option for just two reasons -- changing DNS for hundreds of sites is a nightmare with that number of clients (especially when the person who managed DNS left four years ago) plus some of these sites must be upgraded from D9 to D10 without changing their DNS while some D8 and D7 sites may eventually upgrade later, but to keep these customers you can't just tell them to coordinate IP changes or leave. It's easy only when you control all aspects, which is rare in real world.

[macmladen]

I did not intend to present it as simple, I just wanted to say that those servers that need to stay at the old version may continue to stay that way while new ones may move forward.

Probably, some tradeoff must be made, as there is not a single solution that can fit all.

Therefore, my vote goes for the approach that favours the future and those needing the old can stay with the current, creating new sites on the new BOA.

I had to move from one server to another so I used a script and API to take care of DNS.

@omega8cc I wish you luck whatever choice you make and hope we will soon be able to somehow host Drupal 10 on Barracuda on Aegir.

[omega8cc]

Almost there!

[macmladen]

@omega8cc Everybody is certainly excited about this, sooo... have you any, even the wildest estimation for the release?

[acc]

Great job. Are these changes going to be able to be merged in with non-BOA Aegir?

[omega8cc]

@omega8cc Everybody is certainly excited about this, sooo... have you any, even the wildest estimation for the release? @macmladen

How about.. now? :-)

[omega8cc]

Great job. Are these changes going to be able to be merged in with non-BOA Aegir? @acc

Yes, that's the plan, although you will need our Drush 8 fork plus D10 patch for core: https://github.com/omega8cc/drush

[omega8cc]

Hello everyone!

We are delighted to announce that we have completed all tests and published new BOA with Drupal 10 support.

You can test how it works for you after upgrading both Barracuda and Octopus by either creating new Drupal 10 codebase with Composer on your server and then adding it as platform in Aegir as usual, or you can add new keywords to your Octopus configuration file: ~/static/control/platforms.info and launch Aegir upgrade by creating empty ~/static/control/run-upgrade.pid file.

For more details please read the built-in docs in the ~/static/control/README.txt file or read it online at https://github.com/omega8cc/boa/blob/5.x-dev/aegir/tools/system/conf/control-readme.txt

We are working on documentation explaining important details and any edge case scenarios once we gather enough feedback, but generally everything should work like for previous core versions, with only few important points and caveats:

1. Aegir no longer removes Drush from any Drupal 8 or later codebase. Instead it locks it temporarily using just permissions on the Drush directory — typically located in vendor/drush to avoid conflicts. When you need to use the site-local drush, simply run either Disable or Enable task on any site on that platform — it will be handy to have a dummy site installed for this purpose. Then when you need to run other tasks which require locked local Drush, run the platform Verify task first.

2. Any Drupal 10 codebase needs simple patch and it will be applied automatically for you on the first Verify task for the platform. If you will overwrite it later, please apply it again — you will find it in your codebase web root: https://github.com/omega8cc/boa/blob/5.x-dev/aegir/patches/drupal-ten-aegir-01.patch

3. It’s still possible that your codebase will need similar patch for another module you are using, like it was required for Varbase distribution where we had to patch also the syslog module enabled by its installation profile — see the attached screenshot.

4. In our testing we could easily upgrade Drupal 9.5 to either Drupal 10.0 or directly to Drupal 10.1, but we have tested only basic installation profiles, so there can be more issues to cover later, like those explained in the blog posts linked on drupal.org, for example: https://blogs.jamesrome.net/drupal10_upgrade

There are other new features to discuss, like instant PHP-CLI version configuration for Drush and Composer on command line — independent of the Aegir backend version defined in the ~/static/control/cli.info file, but we will explain them in the docs and email updates shortly.

Thank you for your patience while we worked on this important milestone.

Enjoy your new Aegir with Drupal 10 support!

Your Omega8.cc Dev Team

[socialnicheguru]

Great job. Are these changes going to be able to be merged in with non-BOA Aegir? @acc

Yes, that's the plan, although you will need our Drush 8 fork plus D10 patch for core: https://github.com/omega8cc/drush

The D10 patch are you referring to is: https://github.com/omega8cc/boa/blob/5.x-dev/aegir/patches/drupal-ten-aegir-01.patch

And to be clear, you are saying that with our current AEgir install from Drupal.org, we can add your version of Drush and the D10 patch you refer to and it should upgrade AEgir3.x at /var/aegir to handle D10 sites?

[omega8cc]

@socialnicheguru It’s not enough, you also need our version of Provision and Hosting. We will commit upstream what we can, but we need to be careful to not introduce unrelated BOA changes. It may still break features we don’t use, like remote servers in Aegir, so anyone interested will have to test and submit patches once we backport our work.

_{Sent with GitHawk}

[socialnicheguru]

I just installed BOA "in-head" but I think I need to install "in-dev".

If I want to take advantage of D10 changes in 5.x-dev which contain Drupal 10, do I need to install "in-dev"?

If so how do I install "in-dev" once I installed using in-head?

[omega8cc]

@socialnicheguru You don’t need dev, head is fully D10 compatible, please check the changelog.

_{Sent with GitHawk}

[darthsteven]

This is a very interesting approach! I'm wondering how you've overcome the challenges faced by Drupal 10 shipping with Symfony 6, as described in #1752 where some quite low-level libraries now have type hints that make the Drush 8 versions incompatible etc?

[omega8cc]

Hi @darthsteven nice to see you here!

We still need to document this properly but it’s actually pretty dirty hack we have made while debugging step by step the errors we have encountered after making Aegir compatible with PHP 8.1 and 8.2

Basically, we are using Provision to copy only one tiny library from Aegir Drush 8 to Drupal 10 codebase — it doesn’t conflict and doesn’t break anything important and doesn’t affect the D10 site but allows Drush 8 to work.

It’s done automatically on every platform verify (if needed) plus we also lock local Drush and one more non-essential Symfony library (with stupid permissions lock), so it doesn’t interfere with Aegir Drush.

We have also added the Drush unlock task which restores original D10 codebase libs and unlocks everything so the user can use local Drush on command line while not running any Aegir tasks.

Honestly, we have been surprised how easy it was to achieve after months of overthinking.

_{Sent with GitHawk}

[omega8cc]

Ah, one more thing, simple D10 core patch is also needed and applied automatically on platform verify.

_{Sent with GitHawk}

[darthsteven]

Yeah, I guess the bit that I don't get is how Drush 8 can be compatible with Symfony 6, since the typehints got added to the interfaces etc. and those aren't there in Drush 8.

[omega8cc]

That’s the point. In theory it shouldn’t work. We haven’t made any meaningful progress until we stopped believing it’s not possible and just tried to make it work nonetheless instead of rewriting Aegir etc.

_{Sent with GitHawk}

[omega8cc]

To be precise, we didn’t make Aegir compatible with Symfony 6. Instead we managed to make Drush 8 with its own Symfony version operate on Drupal 10 without conflicts.

_{Sent with GitHawk}

[darthsteven]

With the version of Drush 8 that you're maintaining here: https://github.com/omega8cc/drush/tree/8-boa-micro ?

I'll certainly give it go if that's the case. I'm still not sure how it can work if half the code is referencing one version of Symfony, and the other half a different version! But I suppose that if you were to carefully avoid calling/using the same libraries at different versions, it would work.

[omega8cc]

Yes, that’s the Drush 8 fork we use with our Aegir fork.

_{Sent with GitHawk}

[omega8cc]

We think it was possible because Drush 8 is using only pretty basic Symfony stuff. We certainly couldn’t make that happen with any newer Drush version due to its heavy reliance on Symfony, which makes standalone Drush 10 and 11 impossible to use due to constant conflicts.

_{Sent with GitHawk}

[darthsteven]

I see! I've followed along and made a relatively standard version of Aegir work with Drupal 10!

It's almost magic! But it was a lot of hard work and code from @omega8cc and friends, so thanks.

I might try and push some PHP8 and Drupal 10 stuff into Aegir 3's branches in the next few weeks, just to leave it all a bit closer for anyone who follows along at home :)

[omega8cc]

Fantastic! Thank you @darthsteven ! It would be great if you could do that, much appreciated!

_{Sent with GitHawk}

[acc]

I see! I've followed along and made a relatively standard version of Aegir work with Drupal 10!

It's almost magic! But it was a lot of hard work and code from @omega8cc and friends, so thanks.

I might try and push some PHP8 and Drupal 10 stuff into Aegir 3's branches in the next few weeks, just to leave it all a bit closer for anyone who follows along at home :)

I would appreciate that @darthsteven and would be happy to test anything you commit.

[darthsteven]

@acc have a read of my latest thinking: https://www.computerminds.co.uk/articles/aegir-3-and-drupal-10-just-about-working

[omega8cc]

Great write up @darthsteven !

Since we know which parts of our forks are absolutely required to support Drupal 10, and which are just BOA features, we could probably still commit back all the right stuff without going too far, plus we know that Aegir 3 is essentially dead without these changes anyway, so any attempt to bring it back to life outside of BOA context may still be a good thing.

Since we plan to go in different direction with BOA itself licensing wise, it could be the best we can do to say thanks to all other Aegir developers like you and the project founder.

_{Sent with GitHawk}

[acc]

@acc have a read of my latest thinking: https://www.computerminds.co.uk/articles/aegir-3-and-drupal-10-just-about-working

Thanks @darthsteven , very helpful write up

Since we know which parts of our forks are absolutely required to support Drupal 10, and which are just BOA features, we could probably still commit back all the right stuff without going too far

@omega8cc that would be great if you did that I think. I am sure there are others not participating in this discussion that would appreciate it immensely.

[acc]

@darthsteven

From your article:

I suppose we could have some code maintained in an issue fork or a patch.

I really think this is worthwhile if you are prepared to kick it off.