Overal improvements

[yaroslavyaroslav]

Hey, thank you for this awesome project, it's really the breath for making it possible to work with Xcode projects without it, and the most reliable this kind solution so far I've tried.

And therefore I'd be happy to participate with improving this project even further. Here's the places where I see a room for improvements:

• overall reliability (currently the server could working in unreliable state depending on an completeness of a build logs that user is provided to it, another one issue here is with the incremental updates relying on the Xcode post build command)
• background updates (currently I've faced such behavior that if I create a new type within my coding session it's keeping unreachable to such server until the next rebuild, where (obviously) it gets available pretty instantly within Xcode itself), so as I saw in the command output this server relies in a some way on Xcode NoIndex db, as the source of the completion, so I wonder if this could be fixed with a relatively low effort?
• raw xcodebuild output support. I'm seeing this enhancement as a logical step further for this solution, to make it possible something like the follow xcodebuild -project YourWorkspace.xcodeproj -scheme YourScheme -destination "generic/platform=iOS Simulator" build | xcode-build-server --whatever, this one will provide users with the option to completely avoid back switching to Xcode to perform incremental build to increase their completion experience.

Though so far I have a lack of knowledge within both lsp, bsp servers and your implementation, I'd really appreciate any hints or other kind of support that could lowering the learning curve within all of that scope for me.

[SolaWing]

@yaroslavyaroslav I'm glad that you can participate in improving this project.

For the basic knowledge, the only thing sourcekit-lsp needs is to provide the compilation parameters corresponding to the file. this is what xcode-build-server does: parsing xcode build log to extract compilation parameters (xclog_parser.py), read cache to get the compilation parameters of file(compile_database.py) and provide to lsp(server.py).

The compilation parameters rarely change, and when change, build and restart lsp is a easy way to update it. That's whyI haven't implemented incremental updates yet, but I'm happy to see this improvement.

Here's what I think about your suggestion：

Firstly, raw xcodebuild output already support: just xcodebuild -project YourWorkspace.xcodeproj -scheme YourScheme -destination "generic/platform=iOS Simulator" build | xcode-build-server parse. this will write .compile and buildServer.json to current dir.

And for the overall reliability: we may provide more easy way(in one command) to setup xcode project bridge, watch and sync automatically. the pipeline would be:

1. setup buildServer.json and record build dir(user may provide scheme target or custom build dir)
2. xcode-build-server is start and check and watch newest xcactivitylog(by compare modification time)
3. user trigger build(either by xcode or xcodebuild command), update xcactivitylog
4. xcode-build-server watch newest xcactivitylog, parse and update cache, notify sourcekit-lsp

Among them, the default build_dir can be obtained through xcodebuild -showBuildSettings -workspace *.xcworkspace/ -scheme Target | grep "\bBUILD_DIR =" | head -1 | awk '{print $3}' | tr -d '"'

The above solutions still require the user to trigger the build to update the parameters, but the setup will be more friendly and there is no need to restart lsp.

For the background indexing, sourcekit-lsp has not supported yet. see https://github.com/apple/sourcekit-lsp#indexing-while-building. cross module symbol won't visible to each other until build(Xcode may have added additional completion sources). if in the same module, sourcekit-lsp did support index on the fly. but if add new file, parameter changes and we still need to update it for including the new file and notify sourcekit-lsp, this missing file case we may be able to modify the parameters directly without trigger build.

[yaroslavyaroslav]

Hi @SolaWing thank you so much for such detailed answer.

I'll take some time to consider your answer with the amount of the attention that it deserves and to think a bit about my suggesting/their connections, and then will come back with my comments/Abington questions if any, but just for now, after a glance like at it let me point it out that this

Firstly, raw xcodebuild output already support: just xcodebuild -project YourWorkspace.xcodeproj -scheme YourScheme -destination "generic/platform=iOS Simulator" build | xcode-build-server parse.

This is the absolutely outstanding news for me, like this was the most pain I've had with this solution. Thank you for such a great job I've again.

[yaroslavyaroslav]

Hi, sorry for such a long delay, it took time to think what've you said and to play a bit more with the tool in its given state. So my suggestion is drifted to the following:

1. Pass through the script the compiler errors at the end of the file. I mean the xcodebuild is not so user friendly with that, while keeping hidden the exact error reason somewhere within its ordinary std output rather forwarding to err output, but I believe we can work around it with a relatively small effort.
2. Improve UX of this tool out of the scope of Xcode, because I believe that it's most useful as a foundation for 3rd party editors not the Xcode itself. So I believe that it's a better idea to straighten user path of those who decided to use it with VSC, Sublime, vim you name it in completely autonomous way. How's that?
3. Force to restart/rebuild the index as a new file arises.

Let me elaborate a bit all of the above.

During last weeks I've used this tool with Sublime Text editor to work with my main product, quite complex and tangle iOS app. And it works really well in the most cases, except those mentioned above. And by "it works really well" I mean that those are the most reasons why I had to switch back to Xcode because this tool failed to handle that.

The main reason why I had to repeatedly switch back and forth to Xcode was to seek the reason why the app wasn't building. This is because the exact reason (e.g., detailed error message) appears only within main xcodebuild log and does not duplicate them within error output. This overlaps with suppressive behaviour of the xcode-build-server, leading to complete loss of errors. That leaves user with inability to fix compilation issues.

The second most annoying issue was the need to rebuild the project after either adding a new file to the project or performing cross-file refactoring. Both of these actions led to a situation where the lack of background indexing broke the completion feature. And because my project is quite large, this need drains quite a bit of both time and battery.

Speaking about UX improvement, other than those mentioned above I wonder why one would even take Xcode UX on the first place? I mean, Xcode already has an awesome completion feature right out of the box. The workflow where a user expected to switch from their 3rd party editor to Xcode just to update the index, brings even more annoyance to their experience. So what do you think about improving xcode-build-server in another direction, for instance, by providing a straightforward wrapper around xcodebuild tool to handle the project needs? This would provide all the basic features to get project built and index updated with appropriate project scheme and target?

Just to say, I'd be happy to implement most of it myself at a not-so-rapid pace, if you're ok with that direction.

I would be happy to hear from you.

[SolaWing]

@yaroslavyaroslav Thank you for your suggestion, It sounds great to stay away from Xcode. It is indeed annoying to constantly switch between Xcode and Editor. I even have a shortcut key specifically for this.

Generally speaking, the development process is edit-compile-run-debug cycle. I developed this project to improve the editing experience. Compile and run's can also improved by encapsulating commands, but so far, I have not encountered a better debugging experience than Xcode. Anyway, at least we can make the edit-compile-run cycle completely out of Xcode.

Regarding your question:

1. For error msg, there are serveral way to handle it. you can use raw output, or parse error and put it to end, xcpretty is also good at this. However, if you want to take the experience a step further, most editors have corresponding plug-in integrations that can directly display errors and jump to the corresponding files, but that is beyond the scope of this project.

2. For the scenario of adding new files, as I said before, we can indeed monitor file changes and then update the compilation parameters of the new files to lsp. This requires some effort, but users should have a perfect experience. In addition, if the newly added files need to be compiled without the use of xcode, we need to have some way to add the new files to the corresponding location of xcodeproj. It may be to automatically add the corresponding target according to the same folder, or provide a command. Both need to edit the xcodeproj, I know ruby and swift can do this.

3. In the cross-file refactoring scenario, if there is no cross-module, I tested that it can take effect after saving the file, so the corresponding editor should ensure batch saving after refactor. But cross-modules must be recompiled. We may be able to automate compilation in the background, but compilation is also expensive, and I'm not sure if automatic triggering of compilation is a good option. If the compilation is triggered, it is also best to have relevant progress prompts to the editor, which depends on whether lsp supports this prompt. However, it is possible to provide command line interface, to simplify xcodebuild usage. Running the program also requires a easy command line interface. This way you should no longer need xcode except for debugging.

Finally, I'm very glad that you can participate in improving this project together.

[yaroslavyaroslav]

Thanks for your generous response. I'd like to start it by adding error parsing to the script and maybe adding warning parsing as well, not sure about the latter though. I'm aware of xcpretty itself as well as how powerful it is. I've used it just a few times though. Actually there was a time when I tried to even use it for the exact same purpose — as a support provider for the LSP server, unfortunately with no success.

Anyway, I think that using it here for the mentioned task would be some kind of overhead. While enhancing an internal parser a bit to be able to capture error and warning messages by a strict rule would be just right and wouldn't take a significant amount of effort.

Also I'd like to mention a nice point about how it would be useful to provide an option to update users' Xcode project on our side. I hadn't thought about that, due to my current project being organised mostly within local packages to avoid exactly that, but indeed my case is quite rare and this feature is a must overall. I'm considering it as a second stage to work on.

[SolaWing]

@yaroslavyaroslav FYI: this weekend I have developed a feature, like the first discussed overall reliability, to simply xcodeproj bind by use xcode-build-server config -workspace *.xcworkspace -scheme XXX, and watch log file generated, to automatically parse log, update .compile and notify lsp.

In my current test, it works for build in xcode app. so no need to setup post action script. but for xcodebuild commandline, I found it only generate xcactivitylog when specify -resultBundlePath. though manual parse by xcodebuild ... | xcode-builid-server parse -ao .compile should also work.

This feature current is on watch branch. If you developed new feature, should best base on this branch. when I try it enough time and think it stable, I will merge it.

[SolaWing]

watch branch is merged. it works well in my local environment. if anyone has any problem, please report issue.