[ISSUE] Bug, Maximum data size limit reached for DAT: 040000

[MaterialMiko]

The Problem:

Installing many mods leaves you with an error message during the Data Writing stage of installing new mods, that locks the whole thing. Trying to enable/disable mods after encountering this issue results in not being able to enable mods anymore because the error comes up again and again no matter what you do(including refreshing) besides disabling everything or using 'reset' to delete everything and start over. Getting past the max data size even once means you have to delete everything and re-install everything if you want to be able to use the program properly and mod the game normally. Very tiresome.

It seems like ffmt doesn't handle memory as efficiently as Textools yet, and saves duplicates of the same files over one another, making file size increase more than it's meant to. And that the memory cap is around 8gb or lower, and not at 32gb like Textools managed to do.

To Reproduce using ffmt 0.9.10.0 with Manjaro Linux, ext4 drives. Install a shitload of mods, the heavier they are, the faster the issue will come. It seems like installing the same mod over itself may increase the file size? That a single same mod being installed over and over again will bring this error message faster? To test.

My hope: Some solution to the problem, of any kind, that doesn't force me to spend 2~3 hours re-installing all of my mods and filtering out the ones that are great but too heavy for me to afford installing because of this bug. Thank you for the software, happy to be able to mod things on Linux.

[fosspill]

Thanks for the nicely formatted issue!

It took me a bit of code reading and searching, but this is not an FFMT issue.

This is an issue due to the xivModdingFramework, specifically this line.

Wrote a short test and it seems like we'll have to contribute to the moddingframework with linux file systems which supports large enough files.

Drive /mnt/data
  Drive type: Fixed
  Volume label: /disk1
  File system: ext3
Drive /mnt/data2
  Drive type: Fixed
  Volume label: /disk2
  File system: ext3

I will be keeping this issue open until the issue is resolved "upstream"

[fosspill]

pull requested: https://github.com/TexTools/xivModdingFramework/pull/32

[fosspill]

The PR was accepted into xivModdingTool which means this bug will be fixed during once FFMT is released based on a newer framework version.

@MaterialMiko could you confirm that the PR covers your filesystem?

EDIT: I see I missed that detail in your issue. Next FFMT release should indeed fix it for you.

[taylor85345]

I wanted to add to this that there may be a cleanup issue contributing to this problem. I've had this error crop up a few times when applying a large mod that overwrites an existing large mod, such as an update to Hair Defined. Multiple attempts at installing the same mod sees the list of "failed" imports get longer with each attempt. Resetting and installing all mods fresh usually avoids the error, so I suspect that there may be lingering data from overwritten mods inflating the size of the indexes.

[fosspill]

@taylor85345 The XIV Modding Framework does this by design. It always adds mods onto the end. Which means that if you remove older mods you'd just leave empty gaps that it wont fill.

TexTools does have a sort of reclaim space feature to work around this. May it be possible for us to include this in here as well @shinnova ?

[shinnova]

TexTools does have a sort of reclaim space feature to work around this. May it be possible for us to include this in here as well @shinnova ?

Looks like it will be very straightforward to add, the framework has a dedicated function for defragmenting the game's dat files.

[fosspill]

:partying_face: A new Framework release is out which includes the PR! @shinnova is working on a release which includes that framework update and a way of reclaiming space

[shinnova]

We've released a pre-release now that should stop this error from appearing again any time soon. @taylor85345 could you see if the defragmenting resolves the cleanup issue? And @MaterialMiko could you see if you still get the size limit error now?

[taylor85345]

Tested the size limit issue using the current AUR release and then the pre-release. The AUR version failed, and the new pre-release succeeded without issue. I would say the size limit is resolved.

Regarding the defrag function, it looks like the function is pointing to the wrong directory. Running `ffmt defragment" returned the following error:

Validating cache...
Defragmenting...
ERROR: An error occurred during the defragmentation process: One or more errors occurred. (Could not find file '/home/taylor/.steam/steam/steamapps/common/XivMods.json'.)

The file is located in /home/taylor/.steam/steam/steamapps/common/FINAL FANTASY XIV Online/game/XivMods.json on my system.

[fosspill]

Tested the size limit issue using the current AUR release and then the pre-release. The AUR version failed, and the new pre-release succeeded without issue. I would say the size limit is resolved.

Awesome. This very likely means that if people run textools through wine then the fix counts for them too.

[taylor85345]

Saw the pull request fixing the incorrect file path, so I went ahead and compiled ffmt myself to try it out. ffmt now points to the correct directory, but the defragmentation process errors out halfway through, leaving FFXIV in an incomplete and unbootable state:

❯ ./ffmt defragment
Validating cache...
Defragmenting...
Saving updated Index Files...ERROR: An error occurred during the defragmentation process: One or more errors occurred. (Unable to determine Dat Number from one of the following files

/home/taylor/final-fantasy-xiv-online/dosdevices/c:/Program Files (x86)/SquareEnix/FINAL FANTASY XIV - A Realm Reborn/game/sqpack/ffxiv/040000.win32.dat4
/home/taylor/final-fantasy-xiv-online/dosdevices/c:/Program Files (x86)/SquareEnix/FINAL FANTASY XIV - A Realm Reborn/game/sqpack/ffxiv/040000.win32.dat7.
/home/taylor/final-fantasy-xiv-online/dosdevices/c:/Program Files (x86)/SquareEnix/FINAL FANTASY XIV - A Realm Reborn/game/sqpack/ffxiv/040000.win32.dat2
/home/taylor/final-fantasy-xiv-online/dosdevices/c:/Program Files (x86)/SquareEnix/FINAL FANTASY XIV - A Realm Reborn/game/sqpack/ffxiv/040000.win32.dat3
/home/taylor/final-fantasy-xiv-online/dosdevices/c:/Program Files (x86)/SquareEnix/FINAL FANTASY XIV - A Realm Reborn/game/sqpack/ffxiv/040000.win32.dat6.
/home/taylor/final-fantasy-xiv-online/dosdevices/c:/Program Files (x86)/SquareEnix/FINAL FANTASY XIV - A Realm Reborn/game/sqpack/ffxiv/040000.win32.dat0
/home/taylor/final-fantasy-xiv-online/dosdevices/c:/Program Files (x86)/SquareEnix/FINAL FANTASY XIV - A Realm Reborn/game/sqpack/ffxiv/040000.win32.dat5
/home/taylor/final-fantasy-xiv-online/dosdevices/c:/Program Files (x86)/SquareEnix/FINAL FANTASY XIV - A Realm Reborn/game/sqpack/ffxiv/040000.win32.dat1
)

The function seems to be replacing 040000.win32.dat6 with an incomplete backup file called 040000.win32.dat6., and also creating a mystery 040000.win32.dat7. file.

[shinnova]

Could you try with this version and see if that works? I've tested extensively and haven't been able to reproduce the issue.

[taylor85345]

same issue. Didn't create the extra dat7 file, but otherwise same result.

FFXIV_Modding_Tool-linux-0.10.1/FFXIV_Modding_Tool-linux/ffmt 
❯ ./ffmt defragment
Validating cache...
Defragmenting...
Saving updated Index Files...ERROR: An error occurred during the defragmentation process: One or more errors occurred. (Unable to determine Dat Number from one of the following files

/home/taylor/ffxiv/SquareEnix/FINAL FANTASY XIV - A Realm Reborn/game/sqpack/ffxiv/040000.win32.dat4
/home/taylor/ffxiv/SquareEnix/FINAL FANTASY XIV - A Realm Reborn/game/sqpack/ffxiv/040000.win32.dat2
/home/taylor/ffxiv/SquareEnix/FINAL FANTASY XIV - A Realm Reborn/game/sqpack/ffxiv/040000.win32.dat3
/home/taylor/ffxiv/SquareEnix/FINAL FANTASY XIV - A Realm Reborn/game/sqpack/ffxiv/040000.win32.dat6.
/home/taylor/ffxiv/SquareEnix/FINAL FANTASY XIV - A Realm Reborn/game/sqpack/ffxiv/040000.win32.dat0
/home/taylor/ffxiv/SquareEnix/FINAL FANTASY XIV - A Realm Reborn/game/sqpack/ffxiv/040000.win32.dat5
/home/taylor/ffxiv/SquareEnix/FINAL FANTASY XIV - A Realm Reborn/game/sqpack/ffxiv/040000.win32.dat1
)

Side note, I just moved my install to a new location and symlinked into my wine prefix, which is why the file paths don't match my last post.

[taylor85345]

To expand on this, I've encountered the same error on two separate installs of Manjaro Linux on different machines. I also just tried starting over with a clean install of FFXIV and importing only the "Hair Defined" modpack, and I still encountered the same error. It seems to consistently impact the file ending in .dat6 and occassionally .dat7

[fosspill]

We are still looking into this, rather casually.

Should likely make a little debug build of the framework to figure this one out

[DianaNites]

Would it be possible to get a new release with just the data size fixes? I use the AUR version and I just hit this issue, only to find out "oh its been fixed for months"

Thank you

[fosspill]

Would it be possible to get a new release with just the data size fixes? I use the AUR version and I just hit this issue, only to find out "oh its been fixed for months"

Aiming to have a new version out tonight with endwalker fixes and the data size fixes. No defrag, however.

[DianaNites]

Rad!

That said.. I just tried to reinstall my mods on the pre-release and still got this error?

Im on Arch, ext4 file system, it should work?

[fosspill]

Yes, ext4 is supported by the framework since https://github.com/TexTools/xivModdingFramework/pull/32

There is still a limit though.

Are you importing modpack after modpack or are you importing one large backup?

[DianaNites]

Im just installing a few mods for the first time, and I wasnt able to install any more than I was before the pre-release, which is odd because the limit should be higher right? It still failed at the same place

[fosspill]

That's indeed odd, but there may be issues due to endwalker as well.

I'm wondering though if you could double-check which file system your game is actually installed on, and later tonight test with a new ffmt version that may fix it all

[DianaNites]

I did double check, it's ext4. Looking forward to the new version! For now I guess i'll remain on vanilla

[DianaNites]

Theres no link to the executable, and i'm not set up to build C# code

[fosspill]

Yeah sorry, had some technical issues so I deleted the post.

Here we go again: I have a feeling something else is going on here.

@DianaNites could you please run this on your machine on your machine with the drive path as an argument?

The code is rather simple, in case you'd prefer to build it yourself.

using System;
using System.IO;
namespace DriveInfo
{
 class Program
 {
   static void Main(string[] args)
   {
               var driveInfo = new System.IO.DriveInfo(args[0]);
               Console.WriteLine(driveInfo.DriveFormat);
   }
 }
}

You'll then run driveinfo /path/to/game/drive after unzipping the file

Please report back what output you get.

EDIT: I'm rusty and don't remember how to start the file. 2 sec. EDIT3: Third time....

[fosspill]

I've ran it and @shinnova did too. We both are on encrypted systems and got "lofs" as the output. For this release we'll add that as a large file system, but preferable driveinfo would show the real file system, even for encrypted drives.

Please let us know what your output is as well so that we can ensure the issue won't happen in future releases

[DianaNites]

Third times the charm

It reported lofs, which is odd and just plain wrong. The only thing I can find describing that is this random manual page https://www.unix.com/man-page/linux/7FS/lofs/

Meanwhile, where I have the game installed

Screenshot


edit: Yeah I get lofs too. BUT: I don't have an encrypted drive?

[fosspill]

I honestly have no clue. Tested this back in august with no issues.

Likely something changed with how linux reports file system to dotnet, or maybe you have something else special set up? bcache or something like it?

Regardless though, we'll just add "lofs" to the list and hope for the best :) I can't find much documentation on this topic, nor can I find any other c# project that has ran into the same issue.

[DianaNites]

Please let us know what your output is as well so that we can ensure the issue won't happen in future releases

we'll just add "lofs" to the list and hope for the best

I have to ask: Why even have this check at all? Would it be better to just remove it?

Can xix even be installed on a filesystem that cant have files bigger than 2GB?

Presumably the filesystem will error if you do something wrong and you can just clean up work?

Plus, none of those limits are accurate, or the function name is wrong. Ext4 can have single files up to 16TiB, and NTFS can have single files up to 256 TiB. Even FAT16 can have up to 4GB single files, and it has a default smaller than even ancient filesystems like FAT16 had?

Or if its some limit the game has, well, why does it vary by drive then?

[fosspill]

That's indeed a valid question, and something we'd have to take upstream. @shinnova could we override this functionality on our side and just ignore what upstream says?

[fosspill]

I have to ask: Why even have this check at all? Would it be better to just remove it?

^

Can xix even be installed on a filesystem that cant have files bigger than 2GB?

That's indeed a good question too

Presumably the filesystem will error if you do something wrong and you can just clean up work?

Sure, but we've been leaning on the framework to deal with those sort of functions

Plus, none of those limits are accurate, or the function name is wrong. Ext4 can have single files up to 16TiB, and NTFS can have single files up to 256 TiB. Even FAT16 can have up to 4GB single files, and it has a default smaller than even ancient filesystems like FAT16 had?

That's entirely correct. I threw together a list of common linux file systems and kept the frameworks highest current limit. It would be fantastic if you'd send them a pull request with proper values.

Or if its some limit the game has, well, why does it vary by drive then?

There is, or at least used to be, an upper limit in the game. Why those exact limits are set I'm unsure about, however. Likely a case of "too many cooks" and quick fixes

[fosspill]

Yet another reply for you @DianaNites: This is a quote from the NTFS part of the code: "2 ^35 is the maximum addressable size in the Index files. (28 precision bits, left-shifted 7 bits (increments of 128)"

So that would be the max the game can read, I presume

[fosspill]

Sent them a pull request to default to max limit and only reducing it where needed. https://github.com/TexTools/xivModdingFramework/pull/40

[fosspill]

@DianaNites (and everyone else who follows this issue) sorry about the spam, but I may be the bearer of good news. Please test the following prerelease and confirm if your issue still persists.

https://github.com/fosspill/FFXIV_Modding_Tool/releases/download/v0.10/FFMT-linux-v0.10.1-prerelease.zip

Download the file, unzip it, go into the folder and run ./ffmt mpi xxxxx.ttmp2

[fosspill]

After some testing we've decided to release it: https://github.com/fosspill/FFXIV_Modding_Tool/releases/tag/v0.10.1

Please let me know if there are any issues so we can reopen this issue

[DianaNites]

Thanks a lot for your work on this and the new release!