search

ddev / ddev

Docker-based local PHP+Node.js web development environments
https://ddev.com
Apache License 2.0
2.49k stars 588 forks source link

Preserve the sanctity of #ddev-generated files via new customization options #6357

Open nickchomey opened 1 week ago

nickchomey commented 1 week ago

Is there an existing issue for this?

Is your feature request related to a problem?

It was discovered during testing of the Traefik v3 migration PR that an error related to additional_hostnames and additional_fqdns was due to my having overriden the .ddev/traefik/static_config.yaml file by removing the #ddev-generated line from the top. This prevented ddev from setting some new settings in static_config.yaml, notably 

core:
  defaultRuleSyntax: v2

This static_config.yaml issue is just one more example of frustrations related to #ddev-generated files.

For example, even without this migration to v3, in order to use the Traefik middleware plugins, I had to override both the global static_config.yaml and each project's traefik dynamic yaml. But, then if I wanted to change the project config.yaml - say to add additional_hostnames - those changes could only be applied if I added #ddev-generated to the yaml, which then wipes out my middleware config. So, you have to copy the middleware config, ddev restart to regenerate, re-add the middleware config, and ddev restart again. And then get confused when it inevitably gets out of sync again in the future.

Likewise, when I excitedly shared DDEV with people in the Wordpress world recently, the immediate feedback by some people was "why does it keep breaking my wp-config.php file? This is ridiculous" I told them what was happening, but I suspect they just stopped using it altogether and will never return. Thats a great shame and shouldn't happen again.

Root Problem Diagnosis

The problem, fundamentally, is that DDEV dynamically generates various config files at runtime, but doesn't expose sufficient ways for users to add customizations to them.

It does so with config.*.yaml and docker-compose.*.yaml, but doesn't do so with respect to the Traefik router, and some of the project types (at least Wordpress) generate config files that conflict with user workflows.

This forces users to manually override (at DDEV's instruction!) files that really should be completely, 100%, off-limits.

Describe your solution

#ddev-generated files should be sacred and off-limits - all customizations should be done in other config files which then get dynamically merged at runtime.

We should:

This would ensure that no one ever has to futz with and get blindsided by #ddev-generated again.

#ddev-generated Files that should be sacred but are currently missing extension mechanisms:

(This list will be updated as more files are identified below)

I have been working on a PR that I will submit soon that automatically generates a .middleware-template.yaml for each project, which then gets dynamically merged by /pkg/ddevapp/traefik.go into the project's .ddev/traefik/config/<project>.yaml file each time you ddev start. This allows me to use the middleware while also letting ddev dynamically generate the traefik yaml based on the project's config.yaml. I can probably extend the scope of this to include an extension for ~/.ddev/traefik/static_config.yaml

Likewise, I have been working on an overhaul to pkg/ddevapp/wordpress.go which gets rid of #ddev-generated in wp-config.php but maintains it in wp-config-ddev.php. It will work such that no one will ever have to touch wp-config-ddev.php - they can configure the core wp-config.php however they need for their production site, and then ddev-specific variables will be loaded from wp-config-ddev.php via the value of the WP_ENVIRONMENT_TYPE environment variable.

You'll get to see and review the mechanisms when I submit the PRs soon.

rfay commented 1 week ago

Thanks for the good thinking and for taking the time to write up an issue.

We'll be looking forward to solutions that work globally. This approach is widely known and widely documented. And it's simple, and there's code to support it.

We'd be interested in creative long-term approaches.

In the config.yaml, we support config.*.yaml, which is a better technique. In docker-compose.*.yaml each item is merged, making that a better situation. For add-ons a docker-compose.varnish_extras.yaml is preferred over editing and "taking over" docker-compose.varnish.yaml for example.

nickchomey commented 1 week ago

This approach is widely known and widely documented. And it's simple, and there's code to support it.

I'm assuming you mean #ddev-generated? As I've pointed out, its not all that simple and causes immense friction and frustration even with very standard use of ddev - to the point that newcomers immediately rejected ddev because of it.

The solution that I've proposed is very much alike to what you've just described:

In the config.yaml, we support config..yaml, which is a better technique. In docker-compose..yaml each item is merged, making that a better situation. For add-ons a docker-compose.varnish_extras.yaml is preferred over editing and "taking over" docker-compose.varnish.yaml for example.

Similar extension mechanisms should be created everywhere such that no one will ever need to touch the files that were previously #ddev-generated. In fact, we could/should go so far as to remove any documentation that suggests the possibility of overriding #ddev-generated mechanisms. Instead, modifications should always happen in optional user-managed config files that then get dynamically merged, just as config.*.yamls get merged.

I've detailed two such mechanisms for middleware and wp-config.php, and others could be added through a more thorough review.

nickchomey commented 1 week ago

One useful clarification: we probably shouldn't remove #ddev-generated. It is a useful indication that it is a file that is dynamically generated/managed by DDEV.

What needs to change is that it should be made VERY clear that #ddev-generated means DONT TOUCH. In order to facilitate this, we need to make it possible to do any customizations elsewhere.

My upcoming traefik.go and wordpress.go changes allow for this by exposing middleware-template.yaml and wp-config.php as the places where users can seamlessly make customizations. Similar changes could be made elsewhere in DDEV to preserve the sanctity of the #ddev-generated files

Ive changed the title of this Issue and contents of the original post to better reflect this (and also appear less disruptive and threatening to the existing codebase).

rfay commented 1 week ago

Are you missing that things like the settings.ddev.php for Drupal use #ddev-generated, and that it's used for .ddev/.gitignore, and on and on? You seem like you're focused on traefik-related things, when this pattern is used throughout DDEV.

nickchomey commented 1 week ago

Im sure I'm missing quite a lot! I started a tracker in the OP earlier for which files are ddev-generated and don't have extension mechanisms. I've only noticed for WordPress and traefik so far, but assume there are ddev-generated files everywhere. But probably only some of those are lacking extension mechanisms.

So, if you and others can identify more, I'll add them to the list. The goal being that no one will ever have to directly modify any ddev-generated file again

rfay commented 1 week ago

Please don't go down that path until it's adequately discussed. Just because you see an opportunity doesn't mean that it will be implemented or that your implementation will be accepted.

nickchomey commented 1 week ago

This issue is simply meant to serve as a place to bring the matter to light, and identify and discuss specific possible fixes. I have no intention of doing anything beyond my own immediate concerns.

As such, I'll submit draft PRs for what Ive already done for wordpress, traefik and cloudflare tunnels soon. It'll be up to others to address this for other aspects of DDEV.