Not seeing output building using VC menu with Make and clang

[alecazam]

Can I simply add -ftime-trace to our compiler settings in Make? I'm only seeing an empty UI from building the VC solution, but it's mostly a call over to Make and then the files for reference and setting breakpoints.

[Viladoman]

Yes! That's the idea with clang. Just add the -ftime-trace flag and it should output a .json file just next to each object file with all the profiling data. The clang invocation needs to have the flag, not sure how your build system is expanding its flags to the single compiler invocations.

That being said, I have seen some regressions on the contents of the reports in the last clang versions. What version of Clang are you using?

I would make sure first that when you build those .json files are generated. Then if so, make sure the score generator is pointing to the correct folder where those are.

[alecazam]

Thanks, I'll give it a try. I'm on latest Xcode, VS 2019 Pro, and Android Studio. I think they each have different clang revisions, but it's nice that we have one compiler everywhere now. I saw several Twitter gamedev users posting how they'd optimized their builds. I have a personal ktx/ktx2 builder on github that I should probably optimize too. Thanks for making such a nice little tool. I wonder if Xcode/Android Studio could use too?

[alecazam]

So I added the line to our clang builds, and it outputs the json files. This is on VS2019, but after a full rebuild with the extension, VS locks up to the point I have to kill it. Task manager reports almost no activity. VS 2019 is 32-bit, and maybe the sheer size of a project and all that json is too much. It's too much when I look through it, since I've written to this profile format below. Also opening these json files individual in Perfetto isn't all that useful since it just shows "Source".

It would be useful to be able to run the extension on an existing output directory without having to rebuild. Right now after the VS crash, I'd have to do another rebuild and hope it doesn't lock up again. Each json file is 400KB or more which is crazy, since only really need the filename, and duration of the "Source" elements.

Seems like it would be easy to reprocess and collect the json by file into a much smaller binary format, and then write out the Perfetto/Catapult json. Not sure if that's what the extension does, but how I'd approach it.

[Viladoman]

The .json file contains everything that clang is profiling, this can be a lot of symbols not only sources. The cleanup will be done by the ScoreExtractor if it is setup to ignore some of the data. The ScoreExtractor is the one that generates a .scor binarized format containing all the information already digested. This format is then common from Clang and MSVC as they both provide the build time data differently. Then this binaries is what the extension and viewers understand and display.

If you have the Extension configured for Clang, having the Clang Traces Path field in the Extension settings pointing to the correct folder location, you can run a full folder extraction by running Extensions/Compile Score/Clang Full Score Generation. This fill just walk the whole Clang Traces Path directory recursively opening all .json files and parsing the ones that contain clang trace events.

Alternatively, you can just run the Score Generator directly from the command line.

I hope this helps.

[alecazam]

Yeah, thanks so much. I tried that "Full Score Generation". The process starts, but 15 minutes later, VS is at 0% cpu, and the processing hasn't finished, and there is not progress indicator. It's just that the extension is all dimmed out, and never seems to return. I guess I could leave it overnight, but I don't have a good feeling about this ever finishing. Unfortunately, we can't move to VS 2022.

I did take a look at you JsonParser, and it seemed nicely coded for perf. Just seems like it's a bit overwhelmed with only 4 cores in my Parallels VM. Thanks for making this available. I know many Twitter folks are using it and it works for them. But they may be using Cmake or VS projects directly. Will keep trying.

[23:36:16] Calling ScoreDataExtractor with -clang -extract -tp 100 -td 1 -d 1 -i ....

[Viladoman]

What version of Clang are you using? I would suggest going one step at a time and trying parsing a folder with only 1 .json file and see what happens. Also if possible do it from VS in the ScoreExtraction solution so we can see where is the executable getting stuck.

Thanks for your help.

[alecazam]

I'm just using clang that ships on consoles. I think we just have a lot of files generated. I think our Win builds are clang 10 from Win 2019 Pro. I'll also try SeeProfiler too, and see if that can process the json folder.