YCM and Containers

[fvdnabee]

Issue Prelude

Please complete these steps and check these boxes (by putting an x inside the brackets) before filing your issue:

• [x] I have read and understood YCM's [CONTRIBUTING][cont] document.
• [x] I have read and understood YCM's [CODE_OF_CONDUCT][code] document.
• [x] I have read and understood YCM's [README][readme], especially the [Frequently Asked Questions][faq] section.
• [x] I have searched YCM's issue tracker to find issues similar to the one I'm about to report and couldn't find an answer to my problem. (Example Google search.)
• [ ] If filing a bug report, I have included the output of vim --version.
• [ ] If filing a bug report, I have included the output of :YcmDebugInfo.
• [ ] If filing a bug report, I have attached the contents of the logfiles using the :YcmToggleLogs command.
• [ ] If filing a bug report, I have included which OS (including specific OS version) I am using.
• [ ] If filing a bug report, I have included a minimal test case that reproduces my issue, using vim -Nu /path/to/YCM/vimrc_ycm_minimal, including what I expected to happen and what actually happened.
• [ ] If filing a installation failure report, I have included the entire output of install.py (or cmake/make/ninja) including its invocation
• [x] I understand this is an open-source project staffed by volunteers and that any help I receive is a selfless, heartfelt gift of their free time. I know I am not entitled to anything and will be polite and courteous.
• [x] I understand my issue may be closed if it becomes obvious I didn't actually perform all of these steps.

Thank you for adhering to this process! It ensures your issue is resolved quickly and that neither your nor our time is needlessly wasted.

Feature Details

Use case

This issue contains a feature request for using YCM with containers. The use case are python projects where the development environment assumes containerization of the project for e.g. dependency management. IDEs like vscode and pycharm support remote development inside containers.

Setup

After some discussions on gibber, a setup where ycmd runs in the development container and ycm on the host machine was identified as a feasible approach. The project folder would be bind mounted in the container, so that changes made in VIM on the host system are visible to ycmd in the container.

Changes to ycm(d)

The following changes were identified:

• Run ymcd in the container. This will likely require ycmd to be built within the container, in order for ycm_core.so to be compatible with the container's environment. There would be a dedicated ycmd install within the container, apart from the host ycmd install.
• Adapt ycm to start the container/ycmd. Starting the container in this case, would start ycmd. I'm thinking .ycm_extra_conf.py would be a good candidate to signal to ycm that it should run ycmd inside the dev docker for that project.
• Expose ycmd's http server port from the container to the host system and thus to ycm.
• Point ycm to the exposed ycmd port (might not require changes to ycm at all, as the port would be accessible via lo; via e.g. SSH port forwarding the container might even run on a different machine)
• Translate all file paths that are returned from ycmd in ycm. As the file systems of the container (running ycmd) and the host system (running vim/ycm) differ, some form of translation between the two is necessary. One option is to prefix the location of a (sshfs) mount of the container's root file system on the host. In this case sshfs does the actual file transfers from container to host and changes to ycm would be minimal.

Feedback welcomed

I've opened this issue to solicit feedback on the feature request. I'm hoping to find some time this month to work on a PR. Any blatant problems or anticipated problems with the approach outlined above are welcomed.

[bstaletic]

Hi. We've had this discussion with another user in gitter.

Mostly, you have identified the big problems. There's one more thing.

bstaletic@Gallifrey YouCompleteMe  (git)-[remotes/origin/HEAD]-% rg '(from|import) ycmd' python/ycm --vimgrep
python/ycm/omni_completer.py:20:1:from ycmd import utils
python/ycm/omni_completer.py:21:1:from ycmd.completers.completer import Completer
python/ycm/base.py:19:1:from ycmd import identifier_utils
python/ycm/paths.py:38:3:  from ycmd import utils
python/ycm/paths.py:73:3:  from ycmd import utils
python/ycm/youcompleteme.py:28:1:from ycmd import utils
python/ycm/youcompleteme.py:29:1:from ycmd.request_wrap import RequestWrap
python/ycm/signature_help.py:21:1:from ycmd import utils
python/ycm/vimsupport.py:23:1:from ycmd.utils import ( ByteOffsetToCodepointOffset,
python/ycm/client/completion_request.py:20:1:from ycmd.utils import ToUnicode
python/ycm/client/command_request.py:20:1:from ycmd.utils import ToUnicode
python/ycm/client/base_request.py:25:1:from ycmd.utils import ToBytes, GetCurrentDirectory
python/ycm/client/base_request.py:26:1:from ycmd.hmac_utils import CreateRequestHmac, CreateHmac
python/ycm/client/base_request.py:27:1:from ycmd.responses import ServerError, UnknownExtraConf
python/ycm/tests/conftest.py:30:1:from ycmd.utils import CloseStandardStreams, WaitUntilProcessIsTerminated
python/ycm/tests/youcompleteme_test.py:45:1:from ycmd.responses import ServerError
python/ycm/tests/postcomplete_test.py:29:1:from ycmd.utils import ToBytes
python/ycm/tests/completion_test.py:27:1:from ycmd.responses import ServerError
python/ycm/tests/vimsupport_test.py:27:1:from ycmd.utils import ToBytes
python/ycm/tests/syntax_parse_test.py:25:1:from ycmd.utils import ReadFile
python/ycm/tests/event_notification_test.py:29:1:from ycmd.responses import ( BuildDiagnosticData, Diagnostic, Location, Range,
python/ycm/tests/test_utils.py:31:1:from ycmd.utils import GetCurrentDirectory, OnMac, OnWindows, ToUnicode

bstaletic@Gallifrey YouCompleteMe  (git)-[remotes/origin/HEAD]-% rg default_settings plugin --vimgrep
plugin/youcompleteme.vim:89:38:        \ '/../third_party/ycmd/ycmd/default_settings.json'
plugin/youcompleteme.vim:301:3:" default_settings.json. This ensures that any part of ycm that uses ycmd code

YCM directly imports utilities from ycmd. This is done by having ycmd in PYTHONPATH. I see two ways of solving this:

1. Duplicate the code, so that YCM gets its own copy of the utilities it needs.
2. Make something like protoycmd that both projects depend on.
3. Make ycmd's path configurable, but keep it in PYTHONPATH.

Solving this problem would also help packaging ycmd. It would be nice to be able to do python -m pip install ycmd, but it's a pain in the ass and only slightly related to making a containerized ycmd.

[fvdnabee]

Thanks for your input @bstaletic, here and on gitter.

After some hacking, I got ycmd to respond to completion requests from ycm in vim on the host machine.

• About your comment, currently I followed approach 1 where I duplicated the code. I have a ycm install on the host, which uses ycmd on the host system for utils/other requirements from ycmd. A second is running in the docker, where ycmd starts jedi using a python interpreter from the container.

Some findings:

• Due to docker networking, it seems that ycmd does not receive requests over the loopback address, which causes HMAC verification to fail. From my understanding this is because ycm(d) assumes they will always communicate over the loopback address (as I gather from the HMAC logic, and e.g. BaseRequest.server_location). With container port publishing via docker-run -p, traffic stems from the private network shared between the container and the host.
• As a workaround I've disabled HMAC in both ycm and ycmd. As of yet I don't know a solution that would not open ycm(d) to potentially malicious request from external addresses. Perhaps explicit whitelistening of an adapter/ip address that is used only by the host/container. At least docker uses a private address by default (172.17.0.2/16 here). Suggestions are welcomed.
• I don't (yet) start the ycmd in the container in ycm in YouCompleteMe:_SetUpServer. The SafePopen should be able to call docker-run directly. Also monitoring the docker run process should provide similar semantics as is currently the case for monitoring the ycmd process directly (via .poll()). A nice side effect is that when vim exits, the container stops as ycmd receives a shutdown request (ycmd is the entrypoint of the container).
• Initially, I changed PathToPythonInterpreter and DIR_OF_YCMD in ycm/paths.py, but specifying the interpreter in the projects .ycm_extra_conf.py seems like a more satisfactory solution.
• I didn't bother with file translation yet. Instead, I made sure all files were available at the same paths in the host and in the container via a bind mount in the container and a second bind mount on the host,. Additionally I made the the container's python virtualenv available on the host at the same location as in the container (via sshfs). This is necessary for e.g. GoToDefinition for project dependencies. I also made sure to start vim from the bind mounted location on the host, so that the file paths for ycm and ycmd were identical. I'm unaware the downsides of sshfs, but I'm guessing it isn't a filesystem known for its performant IO; even over a private network on the same machine.
• Not having a proper path translation is crippling the usability of this approach , as host and container need to have the same file system layout (for files you want to access from vim at least). YCM could be extended to implement this translation, where paths send to ycmd are translated to container paths and paths from ycmd are translated back to host paths. Would this be a feasible task for ycm?
• Unfortunately, there will still be files that are only in the container (e.g. the python virtualenv folder) and that aren't accessible on the host while they should be (e.g. a module of a dependency in the virtualenv folder). This will require some sort of file transfer (e.g. via sshfs) as I don't think it's possible in docker to share such files with the host machine.

[wffurr]

I'd love to run ycmd on a remote server perhaps over an ssh tunnel to make it look like localhost traffic. Any progress or advice on this?

[puremourning]

I just found this, which is very pretty interesting https://github.com/AlexandreCarlton/ycmd-docker

It's not quite the same use case (just running ycmd in a container to minimise dependency hell), but it's an interesting take.

[puremourning]

Running ycmd In a container or on a remote host is not considered in scope at the moment.