search

DragonKnightOfBreeze / Paradox-Language-Support

IDEA Plugin: Paradox Language Support.
https://windea.icu/Paradox-Language-Support
MIT License
36 stars 4 forks source link

[1.1.5] Cannot finish indexing when using this plugin for EU4 modding #49

Closed szczesniak-piotr closed 10 months ago

szczesniak-piotr commented 1 year ago

Hello, I have been corresponding with you already regarding this issue, but I wanted to post it here, so it does not go missing.

The indexing in pycharm/IntelliJ never finishes after adding the directory of the EU4 game. Probably there are too many files in that game compared to Stellaris. Indexing for Stellaris works fine, but not for EU4.

I even left it overnight, and it did not finish. CPU and RAM pegged at 100% used. I've got a Ryzen 5950X and 32 GB 3800mhz RAM, so I guess it should be enough?

I've checked it on a different machine and it's the same. Files are getting indexed, but very slowly.

DragonKnightOfBreeze commented 1 year ago

Hi, I have made some optimization, please install the following plugin package manually, check and profile indexing for eu4 game directory again, and then send me the .jfr file for about 5~10 minutes.

This update should optimize the indexing logic for definitions, but there are many other things to be indexed.

Paradox-Language-Support-1.1.6.zip

szczesniak-piotr commented 1 year ago

Will try it out in a second, thank you very much!!

szczesniak-piotr commented 1 year ago

It did not help unfortunately. Looks about the same. One difference I noticed is that RAM usage is lower. It capped out around 4.5 GB even tho I have 8 GB limit in the IDE settings.

Here is the jfr file: idea64_exe_2023_08_11_084202.zip

DragonKnightOfBreeze commented 1 year ago

Under my expect

DragonKnightOfBreeze commented 1 year ago

During this jfr, it takes 60% time (4000s+) to parse script files, 30% time (2000s+) to indexing. at max 20% time (1400s+) can be optimized by optimizing indexing logic.

szczesniak-piotr commented 1 year ago

So this is just because there are a lot of files in the EU4 game, or at least mostly because of that. (and the files are long)

szczesniak-piotr commented 1 year ago

Do you think it is possible for the plugin to work for this game?

DragonKnightOfBreeze commented 1 year ago

Not sure, optimize indexing logic helps nearly nothing, but perhaps scripts files to parse during indexing can be optmized and reduced. Should install eu4 and check manually.

DragonKnightOfBreeze commented 1 year ago

This finally should be a out of memory problem. Over 60,000,000+ config objects are generated by deep copy, deep merge and inline. Approximately 4G of memory is occupied. There should be no other problems. Indexing process should be finished in 20 minutes If this problem are resolved.

szczesniak-piotr commented 1 year ago

So you are going to be able to fix this issue?

DragonKnightOfBreeze commented 1 year ago

Not sure for those generated config objects are hard to cache.

DragonKnightOfBreeze commented 1 year ago

After lastest commit IDE will freeze during indexing event files.

szczesniak-piotr commented 1 year ago

Does it go through folders alphabetically?

DragonKnightOfBreeze commented 1 year ago

Yeah IDEA index files (that should be indexed) in project one by one alphabetically.

szczesniak-piotr commented 1 year ago

So there is quite a lot more to index.

szczesniak-piotr commented 1 year ago

I thought it could be fixed by giving more ram to the IDE but it just made it consume all of it regardless of how much you gave it.

DragonKnightOfBreeze commented 1 year ago

Cwtools shall not index expressions (e.g. a event target) but only almost definitions and localisations. So it's enough fast. But for my plugin, it shall index expressions. It take too much time to get configs from expressions, to infer it's a event target or some other.

DragonKnightOfBreeze commented 1 year ago

It should only use the memory you give to IDEA. For Stellaris, 4G shall be ok.

DragonKnightOfBreeze commented 1 year ago

Until this problem is resolved, maybe thoese additional indices can be disabled, to use PLS together with eu4.

szczesniak-piotr commented 1 year ago

It would be spectacular if you managed to fix this plugin for EU4.

DragonKnightOfBreeze commented 1 year ago

Should be very hard after many updates.

DragonKnightOfBreeze commented 12 months ago

Indexing for eu4 game directory is almost finished after 15 minutes (CPU: 8 cores, RAM: 8G)

szczesniak-piotr commented 12 months ago

Does it work normally after the process is finished? Last time I waited for it to finish, the process would start over every time I had added a new file to the project structure. Does it reindex quickly now (15 minutes is quick)?

szczesniak-piotr commented 12 months ago

If you'd like to, I could test it on my end, but you would need to send the new plugin version.

szczesniak-piotr commented 12 months ago

1.1.7 still does not work fine for me.

DragonKnightOfBreeze commented 12 months ago

Need some more optimization

DragonKnightOfBreeze commented 12 months ago

The config objects generated during indexing process are too many (3000,000+, 3G+), also the time to consume. The eu4 game directory is just too crazy. Plugin should parse all script files and localisation files to index definitions, localisations, flags, event targets, and more. Other files should not be indexed by plugin ,they are just shown by IDEA during scanning.

DragonKnightOfBreeze commented 12 months ago

I'm sorry but this time I cannot resolve this performance problem. Since there are few remaining features to be implemented, Next several versions should try to do some optimization.

szczesniak-piotr commented 12 months ago

Sure, I hope someday it will be fixed. Thank you!

DragonKnightOfBreeze commented 12 months ago

Indexing process for eu4 game directory is once finished over 1 hour under 8 cores CPU & 8 G RAM from my last test. Note that cwt configs exclude Stellaris is not very perfect, so it may show a lot of errors when open a script file. You may need to send PR to cwtools-eu4-config by yourself as I don't have so much time and vigor.

DragonKnightOfBreeze commented 12 months ago

although it was once done, since it's too slow, if you open game directory by IDE twice or make lots of changes, it may still cause a soft IDE freeze (you can operate anything but some IDEA background process is just cannot be finished for a long time)

DragonKnightOfBreeze commented 10 months ago

Indexing process for eu4 game directory is finished over about 4 minutes at 1.1.12.

This version I merge several indices (that require resolving cwt configs) into one index. So the speed to indexing should be much faster when there are too many files to indexing. but the speed to access index should be slower since there are more index data to visit.

Please still note that cwt configs exclude Stellaris is not very perfect, so it may show a lot of errors when open a script file.

Since loading local cwt configs is not supported by PLS currently, you may still need to send PR to cwtools-eu4-config by yourself once 1.1.12 is released.