better diagnostics when Clang completer returns no completions?

[renewagner]

Hi,

I recently installed emacs-ycmd. Overall, I'm quite impressed! With some files, however, I don't get any semantic completions at all. *ycmd-server* then contains:

2018-04-25 13:27:26,005 - INFO - Received completion request
2018-04-25 13:27:26,006 - DEBUG - Using filetype completion: True
2018-04-25 13:27:26,098 - INFO - Received completion request
2018-04-25 13:27:26,099 - DEBUG - Using filetype completion: True
2018-04-25 13:27:26,100 - ERROR - Exception from semantic completer (using general): Traceback (most recent call last):
  File "...../ycmd/ycmd/../ycmd/handlers.py", line 103, in GetCompletions
    .ComputeCandidates( request_data ) )
  File "...../ycmd/ycmd/../ycmd/completers/completer.py", line 229, in ComputeCandidates
    candidates = self._GetCandidatesFromSubclass( request_data )
  File "...../ycmd/ycmd/../ycmd/completers/completer.py", line 240, in _GetCandidatesFromSubclass
    raw_completions = self.ComputeCandidatesInner( request_data )
  File "...../ycmd/ycmd/../ycmd/completers/cpp/clang_completer.py", line 150, in ComputeCandidatesInner
    raise RuntimeError( PARSING_FILE_MESSAGE )
RuntimeError: Still parsing file, no completions yet.

2018-04-25 13:27:26,220 - ERROR - Exception from semantic completer (using general): Traceback (most recent call last):
  File "...../ycmd/ycmd/../ycmd/handlers.py", line 103, in GetCompletions
    .ComputeCandidates( request_data ) )
  File "...../ycmd/ycmd/../ycmd/completers/completer.py", line 229, in ComputeCandidates
    candidates = self._GetCandidatesFromSubclass( request_data )
  File "...../ycmd/ycmd/../ycmd/completers/completer.py", line 240, in _GetCandidatesFromSubclass
    raw_completions = self.ComputeCandidatesInner( request_data )
  File "...../ycmd/ycmd/../ycmd/completers/cpp/clang_completer.py", line 165, in ComputeCandidatesInner
    raise RuntimeError( NO_COMPLETIONS_MESSAGE )
RuntimeError: No completions found; errors in the file?

Is there any way to diagnose what exactly the clang completer is choking on? Also, why would an error be a problem in the first place? Shouldn't it just ignore errors given that clang was designed to be able to parse code despite (certain) errors (which evidently works in Xcode)?

I should note that running clang manually I've been able to fix some errors clang was originally complaining about. The remaining errors are caused by clang failing to understand some (valid!) code involving certain Eigen templates. I can, however, get clang to output its AST just fine using clang -Xclang -ast-dump -fsyntax-only -fno-diagnostics-color $(python ~/path/to/.ycm_extra_conf.py) file.cc (where I've modified my .ycm_extra_conf.py to output all compiler flags to stdout).

I'm using Emacs 25.3 and HEAD from the master branches of all additional Emacs packages and ycmd. As for clang, I'm currently stuck at 3.8.0. Let me know if you need any additional information! I'm afraid I cannot share the file that's causing the issue as is.

Cheers,

Rene

[renewagner]

The errors in the file? part appears to be a red herring. I managed to sort out all compiler warnings and errors clang was reporting when invoked manually on the command line and I'm still seeing the same exceptions in *ycmd-server. I've tried both clang 3.8 and 6.0.0.

Thus, I'm leaning towards thinking that this is in fact a problem with ycmd. Before I report an issue there I have a few more questions:

Vim users are instructed to use :YcmDiags. Is there an equivalent in emacs-ycmd?

My impression is that parsing in ycmd is too quick compared with clang on the command line. Are there any timeouts that I might be running into? Is there any way to configure those?

Cheers,

Rene

[abingham]

It looks like there's progress being made over on the ycmd issue, so that's good.

Vim users are instructed to use :YcmDiags. Is there an equivalent in emacs-ycmd?

We have ycmd-show-debug-info which might show something.

My impression is that parsing in ycmd is too quick compared with clang on the command line. Are there any timeouts that I might be running into? Is there any way to configure those?

There are a few controls you have. One is ycmd-idle-change-delay, though this seems unlikely to be related to what you're seeing. There are also the various company settings that you can play with. Some people (like me) even disable automatic semantic completion, binding it instead to a specific chord to get it when we want it.

I hope this helps. I suspect you'll find more answers in the ycmd issue, but let me know if you need more info.

[renewagner]

Thanks for the input! The ycmd issue is resolved now. It was a small but pretty fatal mistake in my .ycmd_extra_conf.py code that parses the include search path from clang output.

As far as emacs-ycmd is concerned, I think there are two potential improvements so other users won't end up in the same pitfalls:

1. Add a "Troubleshooting" section to the documentation.

   • It should mention that incorrect flags passed to ycmd/libclang are the most common source of completion failures or ycmd issures in general (judging by the responses in Valloric/ycmd#1016)
   • libclang lacks a default include search path although all Clang command line tools do have one. This can be obtained by parsing clang or gcc output. Perhaps point people to your implementation of that functionality in the examples/tests so they can re-use it. I asked about this in the ycmd issue but ycmd is unlike to include that functionality.
   • Users should verify that the flags listed in ycmd-show-debug-info exactly match their expectations.
   • Instead of the vim command :YcmDiags users should install flycheck and flycheck-ycmd. I took these as optional but in fact they are not. This would have made it a lot easier to track down my issue since, once installed, flycheck did in fact complain about uint8_t being undefined which explains why completions of std::vector<uint8_t> cannot work. It did not complain about the root cause of the problem, the failing #include <stdint.h> because Clang ships a version of that which relies on the system stdint.h being present elsewhere in the search path (which it wasn't).

2. With the original, rather complex translation unit the time spent parsing it actually is a problem. I get exceptions in the *ycmd-server* buffer but no other indication in the regular UI. Perhaps this could be handled with a simple "Completion not possible until parsing is done. Please try again later." message in the same way that ycmd-parse-buffer prints the "Parsing buffer done"?

Cheers,

Rene

[abingham]

I'm glad you got up and running!

Add a "Troubleshooting" section to the documentation.

That's such a good idea that we've had an open issue since 2015! I'd love to find time for it, but unfortunately I'm pulled in too many other directions right now.

users should install flycheck and flycheck-ycmd. I took these as optional but in fact they are not.

I'm not sure I follow. I take it that flycheck-ycmd would have made your initial problems more apparent, but I don't see how that makes flycheck or flycheck-ycmd mandatory for using ycmd. I'm probably missing your point here, though.

Perhaps this could be handled with a simple "Completion not possible until parsing is done. Please try again later."

This is a very good idea. Would you mind making an issue for it?

[renewagner]

we've had an open issue since 2015!

How about if I prepare a PR for it? I'd include the above and some of the stuff mentioned in #233. The only thing I'm unsure about is the logging part so I may require some input from you on that one.

I take it that flycheck-ycmd would have made your initial problems more apparent, but I don't see how that makes flycheck or flycheck-ycmd mandatory for using ycmd.

Perhaps my wording wasn't perfect. What I meant to say was that ycmd developers simply expect an equivalent of :YcmDiags to be present for troubleshooting. flycheck-ycmd provides that. And in some cases installing flycheck-ycmd straight away will make certain sources of errors apparent immediately.

Would you mind making an issue for it?

Will do.

[renewagner]

I've tidied up my troubleshooting suggestions and added them as a comment in #233. Let me know if you'd like me to turn that into a PR there. The UI enhancement request is now #467.

I consider this issue resolved and will close it now. Thanks again!

[abingham]

Awesome, thanks! This level of follow-through is a rare thing in open source issue reports :)

Let me know if you'd like me to turn that into a PR there.

I'd be delighted to get a PR for this.