[1.1.5] Cannot finish indexing when using this plugin for EU4 modding

[szczesniak-piotr]

Hello, I have been corresponding with you already regarding this issue, but I wanted to post it here, so it does not go missing.

The indexing in pycharm/IntelliJ never finishes after adding the directory of the EU4 game. Probably there are too many files in that game compared to Stellaris. Indexing for Stellaris works fine, but not for EU4.

I even left it overnight, and it did not finish. CPU and RAM pegged at 100% used. I've got a Ryzen 5950X and 32 GB 3800mhz RAM, so I guess it should be enough?

I've checked it on a different machine and it's the same. Files are getting indexed, but very slowly.

[DragonKnightOfBreeze]

Hi, I have made some optimization, please install the following plugin package manually, check and profile indexing for eu4 game directory again, and then send me the .jfr file for about 5~10 minutes.

This update should optimize the indexing logic for definitions, but there are many other things to be indexed.

Paradox-Language-Support-1.1.6.zip

[szczesniak-piotr]

Will try it out in a second, thank you very much!!

[szczesniak-piotr]

It did not help unfortunately. Looks about the same. One difference I noticed is that RAM usage is lower. It capped out around 4.5 GB even tho I have 8 GB limit in the IDE settings.

Here is the jfr file: idea64_exe_2023_08_11_084202.zip

[DragonKnightOfBreeze]

Under my expect

[DragonKnightOfBreeze]

During this jfr, it takes 60% time (4000s+) to parse script files, 30% time (2000s+) to indexing. at max 20% time (1400s+) can be optimized by optimizing indexing logic.

[szczesniak-piotr]

So this is just because there are a lot of files in the EU4 game, or at least mostly because of that. (and the files are long)

[szczesniak-piotr]

Do you think it is possible for the plugin to work for this game?

[DragonKnightOfBreeze]

Not sure, optimize indexing logic helps nearly nothing, but perhaps scripts files to parse during indexing can be optmized and reduced. Should install eu4 and check manually.

[DragonKnightOfBreeze]

This finally should be a out of memory problem. Over 60,000,000+ config objects are generated by deep copy, deep merge and inline. Approximately 4G of memory is occupied. There should be no other problems. Indexing process should be finished in 20 minutes If this problem are resolved.

[szczesniak-piotr]

So you are going to be able to fix this issue?

[DragonKnightOfBreeze]

Not sure for those generated config objects are hard to cache.

[DragonKnightOfBreeze]

After lastest commit IDE will freeze during indexing event files.

[szczesniak-piotr]

Does it go through folders alphabetically?

[DragonKnightOfBreeze]

Yeah IDEA index files (that should be indexed) in project one by one alphabetically.

[szczesniak-piotr]

So there is quite a lot more to index.

[szczesniak-piotr]

I thought it could be fixed by giving more ram to the IDE but it just made it consume all of it regardless of how much you gave it.

[DragonKnightOfBreeze]

Cwtools shall not index expressions (e.g. a event target) but only almost definitions and localisations. So it's enough fast. But for my plugin, it shall index expressions. It take too much time to get configs from expressions, to infer it's a event target or some other.

[DragonKnightOfBreeze]

It should only use the memory you give to IDEA. For Stellaris, 4G shall be ok.

[DragonKnightOfBreeze]

Until this problem is resolved, maybe thoese additional indices can be disabled, to use PLS together with eu4.

[szczesniak-piotr]

It would be spectacular if you managed to fix this plugin for EU4.

[DragonKnightOfBreeze]

Should be very hard after many updates.

[DragonKnightOfBreeze]

Indexing for eu4 game directory is almost finished after 15 minutes (CPU: 8 cores, RAM: 8G)

[szczesniak-piotr]

Does it work normally after the process is finished? Last time I waited for it to finish, the process would start over every time I had added a new file to the project structure. Does it reindex quickly now (15 minutes is quick)?

[szczesniak-piotr]

If you'd like to, I could test it on my end, but you would need to send the new plugin version.

[szczesniak-piotr]

1.1.7 still does not work fine for me.

[DragonKnightOfBreeze]

Need some more optimization

[DragonKnightOfBreeze]

The config objects generated during indexing process are too many (3000,000+, 3G+), also the time to consume. The eu4 game directory is just too crazy. Plugin should parse all script files and localisation files to index definitions, localisations, flags, event targets, and more. Other files should not be indexed by plugin ,they are just shown by IDEA during scanning.

[DragonKnightOfBreeze]

I'm sorry but this time I cannot resolve this performance problem. Since there are few remaining features to be implemented, Next several versions should try to do some optimization.

[szczesniak-piotr]

Sure, I hope someday it will be fixed. Thank you!

[DragonKnightOfBreeze]

Indexing process for eu4 game directory is once finished over 1 hour under 8 cores CPU & 8 G RAM from my last test. Note that cwt configs exclude Stellaris is not very perfect, so it may show a lot of errors when open a script file. You may need to send PR to cwtools-eu4-config by yourself as I don't have so much time and vigor.

[DragonKnightOfBreeze]

although it was once done, since it's too slow, if you open game directory by IDE twice or make lots of changes, it may still cause a soft IDE freeze (you can operate anything but some IDEA background process is just cannot be finished for a long time)

[DragonKnightOfBreeze]

Indexing process for eu4 game directory is finished over about 4 minutes at 1.1.12.

This version I merge several indices (that require resolving cwt configs) into one index. So the speed to indexing should be much faster when there are too many files to indexing. but the speed to access index should be slower since there are more index data to visit.

Please still note that cwt configs exclude Stellaris is not very perfect, so it may show a lot of errors when open a script file.

Since loading local cwt configs is not supported by PLS currently, you may still need to send PR to cwtools-eu4-config by yourself once 1.1.12 is released.