Indexing a new file

[wojciech-kulik]

I'm thinking about: https://github.com/wojciech-kulik/xcodebuild.nvim/issues/56

@SolaWing are you aware if there is any CLI mechanism to trigger indexing without the whole build? What does Xcode do under the hood that it works?

[fireplusteam]

xcode sends specific request to XCBBuildServer to retrieve the index data for a specific file, it's very similar like to build a single file. I wrote a proxy for XCBBuildSever in python and logged everything. It's just binary messages behind the scenes So a clients (xcode) sends:

INDEXING_INFO_REQUESTED\xc5\x17\x83{"filePath":"/Users/Ievgenii_Mykhalevskyi/Desktop/source7/Experiences/Shop/Sources/UI/ShopViewController.swift

XCCBuildServer communicate with swiftc or whats ever and then sends back some sort of response

INDEXING_INFO_RECEIVED\x92\xd9@7b584ba584266c03a8382b4234005f322ce020d2de46fd50b2ccb49369b7ccfd\xc5\x1dAbplist00\xa1\x01\xd8\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x13\\_\x10\x0fLanguageDialect

[wojciech-kulik]

Great discovery! Is it possible to replicate this behavior in this tool? Or would it be too complex?

[fireplusteam]

it's simple to recreate

You need to install pyinstaller on a mac then unarchive the Archive.zip, you need to build python file like a binary

pyinstaller --onefile XCBBuildService.py

Then rename XCBBuildService file in xCode folder to XCBBuildService-origin which is located in

/Applications/Xcode.app/Contents/SharedFrameworks/XCBuild.framework/Versions/A/PlugIns/XCBBuildService.bundle/Contents/MacOS/XCBBuildService

create a simlink to a newly generated binary by pyinstaller

ln -s path/to/Your_XCBBuildServiceProxy/dist/XCBBuildService /Applications/Xcode.app/Contents/SharedFrameworks/XCBuild.framework/Versions/A/PlugIns/XCBBuildService.bundle/Contents/MacOS/XCBBuildService

you may want to modify the script to log to a file (you can edit it and forward to any file). there're multiple options and see the results. Currently it supports to run a modify the request from xcodebuild tool to compile a single file without rebuilding the whole project. (Just runtime injection into json) or you can run the build without stopping the building process if you get some compile errors (useful if you made a modification but have some compile errors, so it will update the indexes even there 1000 errors :) )

Archive.zip

[wojciech-kulik]

Hmm, does it require to disable SIP? Modifying files inside the Xcode.app won't work without that, I guess?

Thank you for the detailed description! I will definitely check it out!

@SolaWing maybe it would be worth embedding it in xcode-build-server if possible?

[fireplusteam]

I did it without disabling SIP

[SolaWing]

for full indexed, whole build is needed. this is true even in xcode: when there have multiple targets deps, dep need to build first to make user’s import works. But for simple update the outdated data, rerun this file’s build command normally enough.

xcode-build-server can optimize to index in background after a full build with log generated and reuse file flags last build. but for new file, still has to trigger incremental build to update flags. currently all flag info is come from build logs, so xcode-build-server can’t recognize new file without build. (maybe add a command to notice new file without build and integrate with other plugin? new file is easy to inject to module compile flags) actually I have a debug command to trigger single file compile command for test update index, but for background index, it need more works like observe file change, etc.

I'm less inclined to use XCBBuildService because it's an internal private service of xcode and has potential stability compatibility risks. Unless background compilation of a single file can not workable and this private service is the only way.

[wojciech-kulik]

In my plugin, I know whenever the file is added, so I could forward some command to xcode-build-server if needed.

[fireplusteam]

@wojciech-kulik , anyway Xcode starts a build to get indexes with the parameter

continueBuildingAfterErrors = True

to mimic this you can use xcodebuild -jobs 4 or -jobs 2 and export the following environmental variable if you use xcbuildservice proxy.

export continueBuildingAfterErrors=True

That works for me if you have a lot of modules in the project, as in my case. You can trigger it only if you switch between files belonging to different modules and you make some modification in any of the modules or add a new file. If you have a lot of modules, refresh the compile flags seems is not working as you need to compile a changed module anyway

[fireplusteam]

@wojciech-kulik , you can write the script which would setup proxy for xcbbuildserver automatically and uninstall it and use in a plugin

[fireplusteam]

Also there's a drawback which I don't know how to resolve (seems like limitation of sourcekit), if you've made a change in module A, and there're errors in module A, even a recompile would not refresh indexes and in module B new added classes in module A would not be indexed (only old ones). Only successful build of module A refreshes indexes in module B.

[wojciech-kulik]

@fireplusteam The approach with xcodebuild service would probably work. However, as it's supposed to be for general use, I would like to avoid messing around with someone's Xcode and modifying Xcode.app directly.

I think it should be addressed either on the LSP or BSP side if possible.

actually I have a debug command to trigger single file compile command for test update index

@SolaWing is it possible to integrate it with my plugin? I know which file is added/deleted/modified, I could notify xcode-build-server about those changes.

[SolaWing]

@wojciech-kulik I plan to detected the new file when no flags and infer it from other file in same directory. it need some time to implement. this way no need to others except tell me there is a new file.

[wojciech-kulik]

Great to hear, no rush! I'm glad to hear that potentially it could resolve the problem!

[fireplusteam]

Not sure if we want to have this as files can move to another folder/target/rename/delete etc and it's not something that xcode build server should handle, if you need to update flags, you can trigger low level task with with xcodebuild and it works great for me even for really big big project

[wojciech-kulik]

I'm not familiar with the internals of BSP and Xcode indexing. Could you please provide some information on what should be done after CRUD operation to fix the LSP? Do you mean to just run the build?

[SolaWing]

Not sure if we want to have this as files can move to another folder/target/rename/delete etc and it's not something that xcode build server should handle, if you need to update flags, you can trigger low level task with with xcodebuild and it works great for me even for really big big project

@fireplusteam so this feature I plan to only enable when explicit set. and when flags updated from build, will use the build flag and drop the temporary flags, so the new file flags only affected gap between new file and build. if something wrong, rebuild should be ok.

I'm not familiar with the internals of BSP and Xcode indexing. Could you please provide some information on what should be done after CRUD operation to fix the LSP? Do you mean to just run the build?

build did fix the new file issue. but xcode-build-server only get new flags after building finish. Therefore based on compilation time, this update time may be longer.

[fireplusteam]

@wojciech-kulik here's my extension to VS code https://github.com/fireplusteam/ios_vs_code I stole one file from you related to project management written in ruby :) You can see how it works in files AutocompleteWatcher.ts and compile_module.sh, you can set number of jobs to xcodebuild and rerun it only on certain types of events like (adding a new file, something changes in a module and you switch to another one, delete the file, or move a file from one place to another). Basically, I run incremental rebuild if project file is change or some file is changed in some module and a user switched over to another module in the editor (works great). If a user runs build manually autowatcher is cancelled as it's not needed at the moment. Also if you run xCode with my proxy you will see that xCode indexing system does the same. It starts the continuous build with continueBuildingAfterErrors=True of a project with low priority. So my proxy is just to set overide continueBuildingAfterErrors variable when building with xcodebuild and go. You can use -jobs of xcodebuild tool to simulate the same behavior.

extension supports tests,snapshots, debug, builds, built-in xcode-build-server, modify project structure/etc

@SolaWing I would rather request the async parser just in time while it's building

[SolaWing]

for repo which build fast, trigger incremental build to update flags and index, indeed is a much reliable way. It should be default and recommend to most users.

async parser can further reduce the responsive time as piping build log to parser..

background build and index may also implement in xcode-build-server as long as I know how to trigger a build. file changes event can also be detected by check xcode project file. this way may improve experience to all editor users.

I previous thought to treat no-flags file as new file, and hack flag from neighbors file, will still usable, but it should only used by which repo is very large and build is extremely slow and can't wait for building.

[wojciech-kulik]

I added a new file and tried running xcodebuild on the side. However, it's very slow. In my case after adding a file, the incremental build takes 22 seconds and after that I need to restart LSP, otherwise I still see errors.

Isn't there any more efficient way to update the index, even if only within the same module?

I noticed that Xcode starts swift-frontend tool for some operations. Not sure if it's relevant for this process. You can get a path to it by xcrun -f swift-frontend.

[SolaWing]

@wojciech-kulik you can check your log to see how time spending. if your new-file module is at first compile, then real-time log parser may be help to update the flags quickly. else if the expected compile still slow, we have to hack the flags temporarily to make it work, as I previously says.

[wojciech-kulik]

As I understand, to introduce this "hack" it would require changes on your side, right?

I think most commercial projects will run incremental build 20s+, even for small changes. So I think it's too long to wait for something in the background. In this case it's better to run a build, at least a user can see the progress.

So, I think the only way to ease the UX in this case is some "hack" as you said, based on other flags from this modules, to do it quickly even if it's not 100% correct.

[SolaWing]

As I understand, to introduce this "hack" it would require changes on your side, right?

Yes, flags is editor independent, and only I can pass the final flags to lsp.

I think most commercial projects will run incremental build 20s+, even for small changes. So I think it's too long to wait for something in the background. In this case it's better to run a build, at least a user can see the progress.

though background build is possible, BSP not interact with editor directly and also currently no api to notify progress to soucekit-lsp. I think I should first implement the new file hack, then real-time log parser, and then background build.

[wojciech-kulik]

Sounds good 👍

[fireplusteam]

Thinking about that! Do we really want the background build implemented on xcode-build-server as it's up to the client to decide when to start the build and how to start it? The only responsibility of xcode-build-server is to async parse the log file while it's building if possible? @wojciech-kulik , Also I don't have to restart the lsp server after a new build is started. My project is really really big (more then 200+ modules and lot of libraries), and 20s is not so much. I tested in Xcode and when you add some new files to some module, it usually takes some time to update indexes as well in Xcode because it starts the same kind of rebuild but the only difference is that Xcode is quicker to report the issue. I suspect that the same kind or almost the same speed can be achieved by async parser. Of course Xcode indexing system can prioritise what to rebuild first but that's something that hard to achieve at this point

[wojciech-kulik]

Yes, I agree, the build triggered by BSP is not necessary, the client can decide when and how to trigger that. It also gives some flexibility like progress tracking. Also, I think sourcekit-lsp is considering background indexing.

[fireplusteam]

@wojciech-kulik , looks like I reproduced your case, I used to watch to xcode build with xcode build server, which automatically refreshes the source-lsp, but by manually parsing -a of build logs, I need to restart LSP to make a new flag working. @SolaWing could you please take a look if it properly notifies LSP if we manually parse logs with parse -a <some_file> and add/rename a file?

[SolaWing]

@fireplusteam how reproduce the case? xcode-build-server will watch buildServer.json and it’s compile files, when the file updated, xcode-build-server will reload to notify lap new flags. normally this should works, including add new file and rebuild. except a known limited case: index store path is pass to sourcekit-lsp as init response and can’t changes. index store path normally won’t change unless build changes build root

if you enable debug logger, you can see the flags xcode-build-server sent to lsp and check if it is correct and updated after updating compile file

[fireplusteam]

server.py.zip I did that change and it started to work fine with parse -a

[SolaWing]

server.py.zip I did that change and it started to work fine with parse -a

@fireplusteam seems you comment out branch and always treat as xcode kind.. it seems not affect the autoreload, as long as you updated the compile file, or auto updated by xcactivitylog...

also notice to avoid modify .compile file same, there is a compile_path.lock to ensure to no parallel parse. when already has a parse, further parse will be ignored. if you run your manual parse, the background log parse after build will be blocked and ignored, until the lock file removed(may not remove when parser is killed..) or outdated(180s). But no matter which way, LSP flags should updated when .compile file changes, except there is exception and BSP crashs..

you can export SOURCEKIT_LOGGING=3 to see details logs, to see how BSP send flags and what flags to LSP, and then reproduce the problem, then we can confirm which part is wrong.

[fireplusteam]

@SolaWing , I restarted a Mac and looks like it fixes something with lsp server, now it works, thanks I'm not using manual parse, just tried it out but something went wrong, now it also works :)

[lelonco]

Yesterday I also had the same issue. When I added a new file to the project, rebuilt it but lsp still couldn't recognize the new class. I also tried to make a clean build, but it didn't help. After restarting lsp it starts to work properly. I'm using manual build log parse command from the tutorial

[SolaWing]

@lelonco did your index-store-path changed? if not, please export SOURCEKIT_LOGGING=3 to see details logs and reproduce it, to check if the flags from xcode-build-server to LSP is correct

[SolaWing]

In newest master, I hack unknown swift file as new file, and If other files in the same directory have compile flags, this file will be inserted into the module compile flags and notify LSP for module compile flags changes. the hack flags now is only available in memory, restart will lost it until reopen the new file. a newest compile is needed to make flags persistent.

this feature now is only enabled by environment variable XBS_FEAT_NEWFILE=1. I'm not quite sure yet if this feature is stable enough as a default behavior. You can try it first and let me know what you think. @wojciech-kulik @fireplusteam

[wojciech-kulik]

I tried it and it worked like a charm on the first try 🔥 🚀! This will significantly improve the workflow!

I will test it more next week. Thank you for all the effort! 💪 🍻

[Woit]

awesome! i will waiting this version in homebrew

[fireplusteam]

@SolaWing , I tested without XBS_FEAT_NEWFILE=1 set. I'm adding a new file and recompile but indexes are not updated until I restart lsp server.

[fireplusteam]

@SolaWing it's working with 9425d7b081bbe209fe0639f3679abee4d1f2cba0 commit I use "kind": "xcode"

[SolaWing]

@fireplusteam can you give me the detail logs by set env var SOURCEKIT_LOGGING=3?

[fireplusteam]

@SolaWing , here is the log with XBS_FEAT_NEWFILE=1 enabled. For me it's working in some cases. Here is the log when it's not working after I added a new file and triggered recompile NewFile6.swift was added Without XBS_FEAT_NEWFILE it's not working as well, however with the mentioned above commit everything working fine example.txt

[fireplusteam]

example.zip Test project in which I reproduced it

[SolaWing]

@fireplusteam I fixed this bug by newest master. cause a condition that python treat empty dict as false and fallback to global cache, and won't clear..

[fireplusteam]

@SolaWing , working great, thanks I will test it more in the next couple of days

[SolaWing]

The New File Infer Flag Feature currently default to true.

[hyzyla]

Hello! I'm here to provide feedback on the XBS_FEAT_NEWFILE feature.

1. I've installed the latest version of xcode-build-server:

   brew upgrade --fetch-HEAD xcode-build-server

2. I added environment variables to my .vscode/settings.json to enable the Swift VSCode extension to start the build server with the required settings:

   {
   "swift.swiftEnvironmentVariables": {
   "XBS_FEAT_NEWFILE": "1",
   "SOURCEKIT_LOGGING": "1"
   }
   }

3. I restarted the LSP service using the Command Palette (⌘+P):

   > Swift: Restart LSP Server

Everything works smoothly; after adding a file, it becomes visible to the LSP server. Thank you for your work!

[wojciech-kulik]

@SolaWing will you release the new version to Homebrew?

[SolaWing]

@SolaWing will you release the new version to Homebrew?

Brew New Version Wait Merge: https://github.com/Homebrew/homebrew-core/pull/169037

[wojciech-kulik]

I would love to confirm how it works, but recently my LSP is getting super crazy, constantly stops working and I have to restart Neovim. I'm not sure what's causing that. Most likely not xcode-build-server because I even tried downgrading it to 1.0.0 with the same result

[akaralar]

@wojciech-kulik that was happening to me last week before this release too but I haven't been able to nail it down.

[wojciech-kulik]

Maybe some update to some plugin is crashing LSP? nvim-lspconfig, telescope, or something using LSP directly?

[SolaWing]

I would love to confirm how it works, but recently my LSP is getting super crazy, constantly stops working and I have to restart Neovim. I'm not sure what's causing that. Most likely not xcode-build-server because I even tried downgrading it to 1.0.0 with the same result

maybe lsp crash, or you has so many compile and cpu&memory is not enough?