CryShana / CryBarEditor

Age of Mythology Retold resource manager, BAR editor, modding tool
13 stars 0 forks source link

Trying to update the game to a newer patch while this program is vieweing the Data.bar fails the update #6

Open CodesbyRobot opened 1 month ago

CodesbyRobot commented 1 month ago

Understandable but workarounds?

CryShana commented 1 month ago

This happens because the program has the file opened for reading - it holds it opened so it can read directly from it as needed without having to load it into memory entirely. That would be bad for big files like texture BAR files that are over 2.5GB big.

The possible workarounds are:

But my main question is Why? Just reopen the program 😅

CodesbyRobot commented 1 month ago

What if the program makes a temporary copy in the temp folder and deletes it after closing the program?

CryShana commented 1 month ago

For some BAR files this would mean copying GBs of data to temporary folder every time you click it. This would slow down switching between BAR files significantly.

If you wish to separate it - make a copy of whole folder yourself and use that folder as Root directory.

CodesbyRobot commented 1 month ago

How does notepad ++ do it?

CryShana commented 1 month ago

Notepad++ loads the entire file into memory. If you try opening one of the bigger files into it, they even show a warning that it will be slow. And it is, even on an NVMe SSD it takes a few seconds to open - on anything slower it will be significantly slower. notepad++_7LXb4kWamQ

And the memory usage reflects that: Taskmgr_0cAUpbo9UA

For comparison, this is CryBarEditor opening the same file instantly: Taskmgr_aYvH3Rkujh

It would be very slow when switching files AND would waste memory.

The way my tool handles archive files is the same way any other archives are handled. If you try opening a ZIP/RAR/7z file with 7zManager, it will also keep a stream opened to the archive to keep browsing it / extracting it fast. But I digress.


A better solution to achieve what you want, would be the lazy loading approach as described in my 2nd point - which means immediately closing the file stream once not needed and reopening it every time you click on any entry inside it. (Or an even fancier approach: closing the file only after some idle time, to maximize performance).

This way we would still avoid loading entire file into memory but wouldn't need to keep it opened. It would still be slightly slower because of the overhead of opening the file every time, but not as slow as opening the entire file into memory. I think Resource Manager uses the same approach.

This would still be worse performing than current solution and would need me to make exceptions for stuff like searching and exporting the content. And handling file changes. I could make it an optional "opt-in" option for users to enable this mode of reading BAR archives... HOWEVER, this may still not solve your main issue... the fact is that the file stream has to be opened any time you click on a different entry inside an opened BAR file -- if you do this while a patch is ongoing and modifying these files - you could still have issues. You could cause the patch to fail mid-way because you accidentally clicked on an entry in that time - searched for something or exported it...

Either way, it's a lot better to just close the app and reopen it whenever thre is a patch, which is not often anyway. The app also remembers where you left off, so it's not like you lose much time either. And if you are that impatient, just make a copy of the \games folder and open the copy in the program.


Another possibility is a mixed approach, where smaller BAR files are loaded entirely into memory (<300MB or similar) while others use a file stream to be read. This could allow you to work with common files such as Data.bar without any issues. The issue would only occur if you were to open any of the bigger art resources BAR files.

But again... this kind of functionality introduces not just extra work, also extra complexity. Which if avoidable, I would like to avoid. And I don't really see this issue as a real issue. It would happen at most once a month and only for a few seconds/minutes - AND can be avoided by making a copy yourself.

How much does this issue affect you? I would need to gauge the severity of this issue to know how to handle it.

CodesbyRobot commented 1 month ago

It doesn't affect me that much. The other functionality to remove the alpha channel on png images so instead of the transparent layer, we get the full original picture, would be better to have.