Add indicator if parser/indexer is still working

[bluehazzard]

I would like to add a indicator for active parsing at file or project level.
For example a icon in the toolbar:

• Black -> No information available
• Red -> Parsing started
• Yellow -> Information partially available (for ex for this file)
• Green -> parser finished

Is there any api that gives me the status for the parser?

[yvesdm3000]

An indicator to show that the system is indexing could be added by sending a message to the GUI thread that indexing has started/finished. For active-file parsing the system is essentially always parsing when doing code-completion, except for the initial parse or reparse when the file is focussed. It might be helpful to add these indicators only for the no-code-completion-case. Also beware the file could be parsing at the same time when indexing is running.

[bluehazzard]

I tried to get a overview of the code, but i don't get behind it that fast... Do you think you find the time to write a quick overlook of the classes and events?

What are the ClangProxy, ClangIndexer and ClIndexToken classes? Just two sentences would be great...

Where is the indexer started and where is detected if a file is indexed/parsed?

[yvesdm3000]

Hi, ClangProxy serves as the mediator between the GUI thread and the world where Clang can play. The original code from Alpha0010 didn't use any threads and therefore blocked the GUI for a very long time while parsing a file or calculating code-completion which was very-very annoying. He already had the class with this name as a façade to Clang, I simply extended it with 'Jobs' that are sent to the Clang thread, that way the functions that are directly on ClangProxy can be called from this new thread. To avoid Mutex-madness, the basic idea is simple: The GUI thread posts a job to the Clang thread to execute it and posts the result back to wx GUI thread as a wx Event. Now there are different types of Jobs, most notable an EventJob is a job that is posted to the Clang thread and at the end will be posted back to wxWidgets GUI thread as an event to process the result (either succeeded with its data or failed). Sadly not all operations in Code::Blocks can be implemented in an asynchronous way so another type of job was created SyncJob where the job can be posted and Code::Blocks can wait for its result, with a certain timeout.

[yvesdm3000]

Hi, ClangIndexer is actually a ClangPluginComponent. At a certain point, the plugin began to grow with a lot of spaghetti code where code of different use-cases were mixed. I proposed to the CodeBlocks development team to divide the plugin into multiple plugins, each one using the same Clang engine since we don't want to parse the same file multiple times do we? But they were strongly against that so instead I divided the plugin itself into multiple parts. That's why we now have "ClangCC" doing the code completion GUI stuff, "ClangDiagnostics" that is doing the inline diagnostics part, "ClangIndexer" that is traversing all files of the project and sending the next one on the list when the previous file is finished, "ClangRefactoring" that is doing the "goto definition", "goto declaration" and "call hierarchy" things (and many future things!!) and finally "ClangToolbar" that is in charge of the toolbar. Each of these plugins can send a Job to the ClangProxy and can subscribe to events when such a job is finished (and even other jobs, for example when a file is parsed). These events generally are named clEVT_XXXXXXX_FINISHED.

[yvesdm3000]

ClIndexToken is the token that is persistent and essentially no longer linked to Clang. Each piece of parsed content, e.g. varable type, function name, function parameter is a 'token" in Clang. Some operations like code-completion don't need an indexer, but if you want to search the whole project (or solution?) for a certain function call, you don't want to parse the whole project just to offer you a list (on the projects I work on, that would take about 40 minutes!!!), so we do a light parse of every file in the project and store the tokens we've encountered, indexing on token so under a token there is a list of ClIndexTokenLocation that says which file has this token and where in the file content. This Indexing-database is strongly linked to the files that are on disk. To cover the case of files that are being edited (you know, people type new text that are new tokens), there is a database that is NOT persistent which stores 'AbstractToken' objects. Once someone presses 'Save', these AbstractToken objects are therefore converted into IndexTokens and the indexer-database is updated without the need of the indexer itself. This way this indexer-database is always synchronized with the state on disk.

[yvesdm3000]

For the indexer itself, it is started in ClangIndexer. Look at https://github.com/yvesdm3000/ClangLib/blob/01b27cef472e772ba4db80eda6e0a413c70aa931/src/clangindexer.cpp#L70 where it does a GetFilesList to get all the files from the project and adds it to m_StagingFiles. If this list was empty, it starts the indexing by sending the first file to the Clang-indexer-thread at pClangPlugin->BeginReindexFile and when the reindexing of that file is finished, this ClangIndexer class will receive a clEVT_REINDEXFILE_FINISHED and ClangIndexer::OnReindexFileFinished will make sure the next file in the list gets parsed. There are other event handlers to tune this indexing, e.g. stop indexing when the project is closed etc. I know there is still an issue that files don't get reindexed when they are modified outside CodeBlocks, e.g. there should be a timer that does what ClangIndexer::OnProjectOpen does on all opened projects but then when the indexer is idle e.g. when m_StagingFiles is empty.

[bluehazzard]

Thank you very much! this helps a lot to understand all this.

I missuse this thread now to ask question about the code ;)

For example, why is OnClangUpdateTokenDatabaseFinished never called ? It is not registered anywhere...

https://github.com/yvesdm3000/ClangLib/blob/01b27cef472e772ba4db80eda6e0a413c70aa931/src/clangplugin.cpp#L1483

[yvesdm3000]

You are correct, it is clearly no longer used. It can be removed I guess...