Generate Bethesda.net file lists from project data

[Arthmoor]

This is a feature request for some future time, but I hope it's simple enough to be something that can be done quickly by someone who knows how these things work.

Uploading a mod to Bethesda.net requires an ESP and BSA pair. Due to a bug in the CK, which Bethesda seems unwilling or unable to fix, that BSA file needs to be manually packed within the CK interface. This can lead to forgetting files and getting wrong files included.

Fortunately the CK also has an option to import a file list that exists from a previous packaging session. These lists are just text files. All they are are path names to files already in the folders, which means that adding a feature to the project menu to make these lists can save a considerable amount of time for SE mod authors who upload to Bethesda.net. I have attached one such file as an example, the one I use to package Cutting Room Floor. It should be more than sufficient to show what needs to be generated and in what format.

The files should be named the same as their plugin file for the mods they go to, and are stored in the Mod Info Exports folder. The filename would be as such for CRF:

"C:\Steam\steamapps\common\Skyrim Special Edition\Mod Info Exports\Cutting Room Floor.achlist"

C:\Steam\steamapps\common\ would obviously be the path to where the user has SSE installed.

Any file found in a folder in the project root should be included. Cutting Room Floor.zip

[Utumno]

If I understand the attachment right those are all the destination paths in the Data dir. Then this boils down to extracting a list of the destination files, should one install the project via BAIN - right?

[Utumno]

I added a commit on fallout3/nv branch adding an Export Achlist menu item give that a whirl https://github.com/wrye-bash/wrye-bash/tree/150-fo3-fnv-support

[Arthmoor]

Didn't expect this to even be looked at right now but seems to be working.

One problem though is that it's including the ESP file. The plugin files should not be included in the list. I don't know how that being there might affect things.

It would also be helpful for checking against the available resources if it sorted the output in the file alphabetically.

[Utumno]

Ok, I pushed a fixed version - I exclude esls esps and esms. The resources added to the achlist are not the entire project folder, just the resources selected for installation. So if you have a complex package it will only add to the achlist the files from the selected subpackages

[Arthmoor]

That's fine about the subpackages since Bethesda.net can't do variable install setups anyway. I'll give the new update a go.

[Arthmoor]

Looks like the update is good. It pulled all of the needed files from the mods I tested and the alphabetical listing is very much appreciated for easier troubleshooting. I'd say this one is good to go unless there's weird edges cases we haven't thought of.

[Utumno]

Could you write a help item I can add to the readme for this new menu - linking also to some definition of those files?

Just for mind-numbing clarity - the list I output is the destinations of the installers files given the current configuration of the installer. It's not the source paths inside the installer - this would take a bit more time to calculate.

[Utumno]

Also note that if I understand above correctly I see no reason to limit this in projects.

Should this menu be active only for SSE?

[Arthmoor]

Not sure what the distinction is between the destination and source is in this case, the list appears to be properly generated based on the contents of the project folder.

The main reason I suggest using it on projects is that in theory the person using this should also have the mod installed into the Data folder already as it's something being worked on. It's not really suitable for use on regular archives downloaded from Nexus or wherever.

I'd have to double check on it for FO4, but the menu option should only be available for FO4 and SSE.

I'll see about writing up a help entry for it.

[Utumno]

Thanks @Arthmoor, doesn't need to be long - just mention the Mod Info dir, some info (link) on the aiclist files and be clear about what this does (see below also)

Not sure what the distinction is between the destination and source is in this case

If you have a complex package and that wants to install data\meshes\foo\bar.nif this may come from installer(.7z)\001 one subpackage\meshes\foo\bar.nif or installer(.7z)\002 another subpackage\meshes\foo\bar.nif. Those last two are the source files - the first one is the destination file, in the data dir. I don't look at the project/archive files OR the data dir - I just use the cached info on what BAIN should install from the installer(.7z) in its current configuration.

It would not make sense to list all files for a complex package (or looking at the data dir for that matter).

[Arthmoor]

Fallout 4 uses the same achlist format, so that's good there.

As I said, since this option is for uploading to Bethesda.net, it only makes sense to go by what's currently installed in the project folder. Which in theory should match what BAIN thinks is there or there'd be an orange box or something to tell you it's out of sync.

This should do for the help entry:

Export ACHList

This exports a file list of the contents of a project for use when uploading to Bethesda.net. In the CK, this list is used when the CK asks for a list of files. Import the ACHList Bash generated and it will then package your files into the proper archives for upload. Available only for Skyrim Special Edition and Fallout 4.

[Utumno]

Thanks!

As "project folder " I understand the actual folder inside Bash Installers dir - this may be a complex archive that has the same data data\meshes\foo\bar.nif in different folders. So what I use is the files configured to be installed by this installer (for simple packages it's the contents of the folder). This has the benefit that changing install options on a complex package will export a different achlist file

[Arthmoor]

I see, that makes sense then because you can then make multiple packages of the same thing if you want to and upload them separately to Bethesda.net. So I'd say the feature is working as planned now.

[Utumno]

Great I pushed a finalish version with the docs (edited) and also changed the sorting to group directories together - this will make it easier to read I think. Same branch

[Arthmoor]

Don't know if it's caused by this last push or something else, but it isn't starting up now. Just getting the following error:

Traceback (most recent call last): File "bash\bash.py", line 227, in main _main(opts) File "bash\bash.py", line 310, in _main bush.detect_and_set_game(opts.oblivionPath, bashIni, retCode) File "bash\bush.py", line 176, in detect_and_set_game setGame(name, u' Using %(gamename)s game:') File "bash\bush.py", line 167, in setGame game.init() File "bash\game\skyrimse__init.py", line 71, in init from .records import MreCell, MreWrld, MreFact, MreAchr, MreDial, \ File "C:\Wrye Bash 308\Mopy\Wrye Bash Launcher.pyw", line 63, in load_module mod = imp.load_source(fullname,filename+ext,fp) File "bash\game\skyrimse\records.py", line 26, in from ..skyrim.records import * File "C:\Wrye Bash 308\Mopy\Wrye Bash Launcher.pyw", line 63, in load_module mod = imp.load_source(fullname,filename+ext,fp) File "bash\game\skyrim\records.py", line 1117, in class MreHeader(MreHeaderBase): File "bash\game\skyrim\records.py", line 1133, in MreHeader slots = MreHeaderBase.slots__ + melSet.getSlotsUsed() ImportError: caused by ImportError(u'caused by TypeError(\'can only concatenate tuple (not "list") to tuple\',)',)

[Utumno]

No, I have refactored a bit - thanks for testing :P

Should be fixed now

[Arthmoor]

Heh, thanks. All good now. I think this achlist ticket can be closed as completed if there's nothing left to check.

[Utumno]

Thanks for prompt assistance :P - let me close this from the commit on dev as usual

[Arthmoor]

Woops, before you do that, another file cropped up in one of my checks that shouldn't be in the list.

An ini file at the same level as the esp file.

Alternate Start - Live Another Life.esp was correctly excluded. Alternate Start - Live Another Life.ini was not excluded. It should not be in the list.

Basically it's unlikely you'll need any files in the top level of the project. Only the folders storing other stuff beneath that.

[Sharlikran]

Did you ever have any issues with xEdit exporting that list? If so maybe mention it to @zilav and maybe he can refine it a bit more.

[Arthmoor]

I didn't even know xEdit had such a function. The one Utumno wrote up is pretty much there except for picking up ini fragments it's not supposed to and it's pretty straightforward to use.

It's been rather helpful - found several dumb mistakes in some of my mods where files either got left out or added when they shouldn't have been. Which is exactly what I was hoping for.

I'll take a look at the one in xEdit and compare what it returns vs what Bash is returning now.

[Arthmoor]

Right, so the xEdit function dumps far too much information about assets the mod merely uses but does not supply itself. That's not very useful. The one for Bash is dumping only what the project actually needs.

[Sharlikran]

Ahhhhh...I think that's like when I use the asset manager to copy files, it copies vanilla assets. Zilav told me how to avoid that. I forget though. Thanks for the feedback.

[zilav]

Assets manager can skip vanilla files, but it is supported for copying assets mode only. Check vanilla archives only (or any other archives you don't want to copy files from) in Processed containers list and click Build checksums, this could take several minutes because it internally unpacks all files and calculates their checksums. This checksums index is saved into a file in xEdit folder, so you need to do that only once.

Afterwards select all containers in Processed containers, and in Copy referenced assets mode check Skip copying files with matching checksums. Keep in mind that script doesn't pick up implicitly used assets like facegen and voices files for example, read the description at the top of the script.

[Utumno]

Thanks for feedback everybody.

@Arthmoor - so do I have to exclude all top level files? Including bsas for instance?

On Tue, Jul 10, 2018, 08:36 zilav notifications@github.com wrote:

Assets manager can skip vanilla files, but it is supported for copying assets mode only. Check vanilla archives only (or any other archives you don't want to copy files from) in Processed containers list and click Build checksums, this could take several minutes because it internally unpacks all files and calculates their checksums. This checksums index is saved into a file in xEdit folder, so you need to do that only once.

Afterwards select all containers in Processed containers, and in Copy referenced assets mode check Skip copying files with matching checksums. Keep in mind that script doesn't pick up implicitly used assets like facegen and voices files for example, read the description at the top of the script.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/wrye-bash/wrye-bash/issues/411#issuecomment-403705829, or mute the thread https://github.com/notifications/unsubscribe-auth/ABTUV0Snk0op3WiNsyNxTKdUAKrLTohwks5uFD1BgaJpZM4T5iI3 .

[Arthmoor]

Yes, all top level files. The point of making an achlist is that you're going to ask the CK to make new BSA files for uploading. So if for some reason those are in the project folder they too need to be ignored.

[Utumno]

Thanks pushed (same branch for now, soon back to utumno-wip).

So although you were right that

it's simple enough to be something that can be done quickly by someone who knows how these things work

see how many details need be specified (order, top level files, help messages, games it applies etc). Start with the help message next time :P

[Utumno]

if for some reason those are in the project folder they too need to be ignored

You certainly mean those are meant to end up to the top level of the Data folder :P

[Arthmoor]

Well wherever they end up if they're at the top level they should be left out :P

Also may want to exclude the Docs folder as well since nobody will be able to access those files via Bethesda.net either.

[Utumno]

Well wherever they end up if they're at the top level they should be left out :P

You certainly mean wherever they are, if they end up at the top level they should be left out :P

Docs excluded in latest iteration and readme updated (duh)