Using Xdebug with the CiviCRM buildkit on Docker

[johanv]

I noticed that you want to use Xdebug with the buildkit on docker (#19). I got it working with Ubuntu 18.04 and PhpStorm. I added a line to docker-compose.yml, and a file README-xdebug.md with a howto.

This could need some improvements. Suggestions welcome.

[michaelmcandrew]

Hey @johanv - this looks very interesting :)

Thanks for the contribution - would love to get it in and documented.

A couple of initial thoughts and questions.

Are you sure you need to do the firewall config? Did you look into exposing the port on the host? That seems more docker like but maybe I am missing something - I have never fully grokked how xdebug communicaton between php and the ide works.

You said that MetalArend had done work on this - do you have a link so I could check it out in more detail?

I'm using Atom - I'll have a play around and see if I can get it working,

[johanv]

I could not get it working without the firewall config. The php server should have a way to connect to port 9000 (or whatever port you configure) of your docker host, because your IDE is listening there. Exposing a port exposes a port of the guest as a port on the host, but we need it the other way round: the guest needs to connect back to the host.

@metalarend created similar documentation for a project we've been working on together (of which the code is not publicly available). It is more or less the same, I just changed some small details, like port numbers and path mappings.

I've never used XDebug with Atom. But I had a quick look at the documentation of the php-debug package for Atom, and I guess it should work, if you:

• change idekey=PHPSTORM to idekey=xdebug-atom in docker-compose.yml, because for the php-debug package the IDE key does not seem to be configurable. If you use xdebug-atom as ide key in docker compose, users of PhpStorm (and I think NetBeans as well), can configure their IDE to use 'xdebug-atom' as key. Restart your docker containers after changing docker-compose.yml.
• set up the path mapping in the xdebug-atom package settings, as described in README-xdebug.md.

[MetalArend]

Awesome to see this being of use for the community!

About configuring the firewall: Xdebug connects back to the host by resolving host.docker.internal to an ip address. This connection seems not to be a connection within Docker itself, but from the Docker container connecting to your host from outside the Docker network. The Xdebug documentation has some more documentation on this scenario: [https://xdebug.org/docs/remote]("Communication Set-up" at https://xdebug.org/docs/remote).

[artfulrobot]

Hey thanks so much for this repo (Michael) and contributers to this issue!

I was able to get Vdebug working in my local vim (neovim) with this.

[michaelmcandrew]

I'm not sure why but host.docker.internal does not exist in the linux flavour of docker. One way around this would be to hardcode the subnet of the buildkit network in the docker-compose.yml.

It seems like docker automatically assigns subnets from 172.16.0.0/16 – 172.31.0.0/16 (the host gets the first available IP). So if we were specifiy the subnet, we know how to reach the host.

And in the unlikely event that it ends up conflicting with a pre-existing subnet on a host, they can change it.

[artfulrobot]

I can't help thinking this could be simpler if there was a way to route a port from the host to the docker vm. e.g. in the dockerfile you can do it the other way around, e.g. this ports line says expose the docker images' 8080 port to the hosts's 8080 port, but that assumes that docker is the server on that port. So it's like ssh -L8080:localhost:8080 remotevm.

But what we need is ssh -R 9000:localhost:9000 remotevm -because our IDE will be the server that XDEBUG is the client of.

If we could do that in a dockerfile, I think we could avoid the other stuff. Is that a possibility?

[michaelmcandrew]

If we could do that in a dockerfile, I think we could avoid the other stuff. Is that a possibility?

Are you suggesting installing ssh on the civicrm container and creating a tunnel to the host? If so, then I think you would need to know the host IP (or maybe I am misunderstanding).

Or if you mean can we map ports the other way in docker compose, then the answer is no.

BUT you can access ports on the host if you know the IP address to connect to.

So all you have to do is define the subnet and we should be fine.

Unless you have a strong objection / reason to hard-coding the subnet?

[artfulrobot]

Are you suggesting installing ssh on the civicrm container and creating a tunnel to the host? If so, then I think you would need to know the host IP (or maybe I am misunderstanding).

No, was just using SSH as an analogy... But if you did install ssh on the container and expose the container's ssh port to the host at port 21000 or something then you wouldn't have to worry about its IP. Anyway, no, that's not what I was suggesting.

Or if you mean can we map ports the other way in docker compose, then the answer is no.

Ah, ok. I'm new to docker, so there you go. Thanks.

Unless you have a strong objection / reason to hard-coding the subnet?

not me.

[michaelmcandrew]

I suspect that there is neat solution that involves forwarding ssh ports and having them exposed back again on the host, but I would be worried that we might rip a hole in the fabric of space time if something goes wrong during our experimentation.

Joking aside, I will probably try the 'easy way' for now and if people are up for exploring alternative approaches that don't require a hardcoded subnet, then that would be cool

[wmortada]

I hadn't seen this post but I managed to get xdebug working on my machine (Ubuntu 18.04 with PhpStorm) by just adding a php.ini file with three additional lines to the civicrm container. I didn't have to change any other settings.

The additional lines are:

xdebug.remote_enable = On
xdebug.remote_autostart = 1
xdebug.remote_host = 172.20.0.1

And here are the actual commands that I typed into the terminal to do this:

bkb
sudo su 
cp /usr/local/etc/php/php.ini-development /usr/local/etc/php/php.ini
echo 'xdebug.remote_enable = On' >> /usr/local/etc/php/php.ini
echo 'xdebug.remote_autostart = 1' >> /usr/local/etc/php/php.ini
echo 'xdebug.remote_host = 172.20.0.1' >> /usr/local/etc/php/php.ini
exit
exit

This worked for me, but is obviously only a temporary fix as these changes are lost every time the container is rebuilt. However, I think this could be somehow baked into the container build. Not sure what to do about hardcoding the IP address though. Hope that helps!

[michaelmcandrew]

FYI, I've released a new version of this repo with xdeug enabled. See https://github.com/michaelmcandrew/civicrm-buildkit-docker/releases/tag/0.8.0

Also, I discovered this fairly interesting approach to the problem: https://github.com/thecodingmachine/docker-images-php/blob/7.2-v1/utils/docker-entrypoint-as-root.sh#L68

[artfulrobot]

@michaelmcandrew I'm using 0.8 and still fighting to get XDEBUG to work, am I doing something wrong?

I have got it working by following these steps. (I'm posting with the dual purpose of helping others and checking if I'm doing anything stupid!)

1. clone this repo and cd into it

2. Edit docker-compose.yml and add this line to the environment section:

   XDEBUG_CONFIG: "default_enable=0 remote_enable=1 remote_handler=dbgp remote_port=9000 remote_autostart=0 remote_connect_back=0 idekey=NVIM_FTW remote_host=172.26.0.1"

   Noting that this is taken from the top bit of the suggested README-xdebug.md (thanks for that!) except for:

   1. you may prefer a commercial closed source IDE instead of a FOSS one, and it may prefer something other than NVIM_FTW, e.g. PHPSTORM ;-)

   2. I've put 172.26.0.1 as the remote host, which I think is what @michaelmcandrew has pinned the docker-host's IP to? (works for me).

3. Bring it up e.g. with bku

4. In your local docker host system run your IDE and configure it to look out for the appropriate IDE key (e.g. PHPSTORM or whatever you chose), and also set up the path maps. In vdebug for (neo)vim, I did this in my config file with:

   let g:vdebug_options = { 'path_maps': {'/buildkit/build/dmaster':'/home/rich/civicrm-buildkit-docker/build/dmaster'} }  

[michaelmcandrew]

hey @artfulrobot

I'm not using this a huge amount at the moment but just checked again now and it does appear to be 'working for me'.

From what you wrote, I am a little confused about whether you have it working or not, and what the problem is :) In any case, a couple of thoughts on what you wrote above, that might be obvious but...

• I didn't write README-xdebug.md - that was johanv - and you shouldn't need to follow the steps in it. Not saying that it isn't a useful or valid approach (it was useful to me in setting up this stuff) - just saying that it is not necessarily relevant to the implementation I came up with.
• I didn't need to do step 2 that you mentioned above. I added some config to https://github.com/michaelmcandrew/civicrm-buildkit-docker/blob/master/civicrm/buildkit.ini instead. You might want to check that it is appearing here: /usr/local/etc/php/conf.d/buildkit.ini within the container.
• FYI I didn't need the idekey to get things working in visual studio code (which is the editor that I have been taking for a spin recently). I do see idekey mentioned a lot in online tutorials. not sure if this is cargo-culting or IDE specific. Just saying I didn't need it :)

I did do some config similar to you to map the directories. Mine looked like this:

    "version": "0.2.0",
    "configurations": [

        {
            "name": "Listen for XDebug",
            "type": "php",
            "request": "launch",
            "port": 9000,
            "pathMappings": {
                "/buildkit/build/dmaster": "${workspaceRoot}"
            }
        }
    ]
}

I then set a break point, turned on listening, loaded a page, and it worked, i.e. it switched to my ide on at the breakpoint with the variables outputted to the screen.

What happened for you?

One thing I found useful when trying to get this working for the first time, was using curl in the container to test to check if/when port 9000 was open on the host.

I did

buildkit@civicrm:~/build/dmaster$ curl 172.26.0.1:9000
curl: (7) Failed to connect to 172.26.0.1 port 9000: Connection refused
buildkit@civicrm:~/build/dmaster$ 

Then started listening in visual studio and repeated the command:

buildkit@civicrm:~/build/dmaster$ curl 172.26.0.1:9000
[waiting for the prompt to return]

Waiting for the prompt to return signalled to me that the port was open, i.e. that I could communicate with the host.

Let me know if I can help some more...

[michaelmcandrew]

FWIW looks like xdebug has a mode where it gets the remote ip from the header: https://xdebug.org/docs/all_settings#remote_connect_back

Might be useful

[artfulrobot]

Thanks Michael. Hmmm. I can see that in the buildkit ini file but not in my container's /usr/local/etc/php/conf.d/buildkit.ini. I'll try deleting everything and starting again, perhaps there was something that didn't get updated.

[michaelmcandrew]

perhaps there was something that didn't get updated

Hey @artfulrobot, yes - I think that was probably my fault. I forgot to update the image on docker hub, which is what the docker-compose file is pulling by default. Sorry about that.

I've published a new version on docker hub that automatically calculates the host IP when you start the container so we no longer need to fix the IP in the docker-compose.yml (which is also updated).

FWIW, the image name has been renamed to michaelmcandrew/civicrm-buildkit (since I am also working on michaelmcandrew/civicrm-{drupal,wordpress,base}).

Hopefully you should be able to pull the changes to the repo do a docker-compose up -d --build (no need to delete everything) and things should start working.

Feel free to ping me on mattermost or via some other real time method if you can't get it up and running, and I will help.

[artfulrobot]

hmmm. I've done that and tried starting from scratch again but still no joy. will look out for you on mattermost, thanks!

[lolaslade]

Using PhpStorm 2019.1 I didn't have any luck until I set xdebug.remote_connect_back = 1 using XDEBUG_CONFIG in the docker-compose.yml environment section

[artfulrobot]

@lolaslade here's what I've ended up doing:

Edit docker-compose.yml, adding this line under the environment: key

    XDEBUG_CONFIG: foo=bar

Then, to start the containers I use a script I called bkstart:

#!/bin/bash
my_ip=$(ip addr show | egrep 'inet.*scope global' | sed  '1s/^ \+inet \([0-9.]\+\).*$/\1/p' -n)
echo "My IP: $my_ip"
sed -i 's/XDEBUG_CONFIG:.*/XDEBUG_CONFIG: remote_host='$my_ip' ide_key=foo/' /home/rich/civicrm-buildkit-docker/docker-compose.yml
docker-compose --file $HOME/civicrm-buildkit-docker/docker-compose.yml up -d

This will replace the XDEBUG_CONFIG line with one that includes the right remote_host, then start them. I have my ide_key set to foo because it doesn't matter for Vdebug (in vim/neovim) but you might need something for phpstorm.

This script + path translation config seems to work for me.

[michaelmcandrew]

@artfulrobot - https://github.com/michaelmcandrew/civicrm-buildkit-docker/blob/master/civicrm/docker-civicrm-entrypoint sets the environment variable XDEBUG_REMOTE_HOST based on the output of /sbin/ip route so you shouldn't need that line anymore.

adding an XDEBUG_CONFIG environment variable for extra variables that you want to set that are specific to your set up sounds like a good idea. I'll document that.

aside: I suspect this is not useful knowledge for you, but vscode does not need an ide_key - just sayin'.

@lolaslade - looking at the docs for xdebug.remote_connect_back = 1 here: https://xdebug.org/docs/all_settings, it says "If enabled, the xdebug.remote_host setting is ignored and Xdebug will try to connect to the client that made the HTTP request". It might be helpful to look at phpinfo(); and share the values you see there for xdebug.remote_host, and the $_SERVER['HTTP_X_FORWARDED_FOR'] and $_SERVER['REMOTE_ADDR'] variables?

@jaapjansma - did you get things working reasonably well with phpstorm in the end - any chance you can help @lolaslade.

If it is OK with you lot, I'm going to close this PR now, since the codebase has moved on and we now have support for xdebug, for one editor at least (vscode :slightly_smiling_face:) and suggest that we open new issues for help getting xdebug working with specific editors.

@lolaslade and @jaapjansma - here's an issue for phpstorm: https://github.com/michaelmcandrew/civicrm-buildkit-docker/issues/39

[artfulrobot]

@michaelmcandrew fair enough closing the PR. I was still needing that workaround on a recent (within last month) build, so thought I'd share it. I'll retest when I get a mo with a fresh copy of master and see what's what. And, jussayin' neovim + coc brings vs language servers and completion to vim :-D

[michaelmcandrew]

@artfulrobot

fair enough closing the PR. I was still needing that workaround on a recent (within last month) build

That's surprising - I added in that ip calculation in November 18: https://github.com/michaelmcandrew/civicrm-buildkit-docker/commit/eb6e09e252d10bc4e79090a820516a76ea39e0aa. Happy to investigate what is happening if you wanted to open an issue.

neovim + coc brings vs language servers and completion to vim :-D

:-D