Is this tool ready for sga packing?

[CannibalToast]

I cannot seem to figure out what the commands are for packing sga files. Is there any documentation on the commands?

(SGAV2)

[ModernMAK]

TLDR at the bottom

Currently, no commands are present, its intended to be used as a library, a CLI for sga packing would likely require an .ini file to specify what files to pack as flat files and what files to compress. (Which is how the archive.exe that is distributed with the modding tools packs files).

Packing can still be done via pyfilesystem (assuming I haven't broken it, I mostly test against opening filesystems), from memory, the big issue is manually specifying the storage type for files.

I'm assuming you want to pack Dawn of War assets? SGA V2 is not currently compatible with Impossible Creatures Steam Edition, which is also V2 despite not being compatible with DoW's format.

You'll have to forgive the lack of documentation, I've certainly been overambitious in trying to learn about the various file formats. If you do want to use this as a library I can provide a snippet in this thread later which can pack files.

TLDR This is a library, no Command-Line Interface exists. SGA-V2 only supports Dawn of War, it unfortunately does not support Impossible Creatures: Steam Edition. I can provide a snippet to pack files, if you are still interested.

[CannibalToast]

Hello! Thank you so much for the reply and explanation,

Please provide the snippet!

I am currently in the process of creating an automation script to pack folders automatically.

I have the unpacking down. I just need the packing.

I would be able to code in an implementation, injecting directories into an INI file. I just need to know the correct formatting.

My script currently creates a .arciv file every time it is ran and auto opens mod packager from the Dawn of war tools.

Any insight into this matter would be greatly appreciated :)

Cannibal Toast.

Cannibal Toast

---

From: Marcus Kertesz @.> Sent: Monday, October 9, 2023 4:26:41 AM To: MAK-Relic-Tool/Issue-Tracker @.> Cc: CannibalToast @.>; Author @.> Subject: Re: [MAK-Relic-Tool/Issue-Tracker] Is this tool ready for sga packing? (Issue #35)

TLDR at the bottom

Currently, no commands are present, its intended to be used as a library, a CLI for sga packing would likely require an .ini file to specify what files to pack as flat files and what files to compress. (Which is how the archive.exe that is distributed with the modding tools packs files).

Packing can still be done via pyfilesystem (assuming I haven't broken it, I mostly test against opening filesystems), from memory, the big issue is manually specifying the storage type for files.

I'm assuming you want to pack Dawn of War assets? SGA V2 is not currently compatible with Impossible Creatures Steam Edition, which is also V2 despite not being compatible with DoW's format.

You'll have to forgive the lack of documentation, I've certainly been overambitious in trying to learn about the various file formats. If you do want to use this as a library I can provide a snippet in this thread later which can pack files.

TLDR This is a library, no Command-Line Interface exists. SGA-V2 only supports Dawn of War, it unfortunately does not support Impossible Creatures: Steam Edition. I can provide a snippet to pack files, if you are still interested.

— Reply to this email directly, view it on GitHubhttps://github.com/MAK-Relic-Tool/Issue-Tracker/issues/35#issuecomment-1752550293, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANARIZYRIECQC327Q2LSMV3X6OYMDAVCNFSM6AAAAAA5TMUXBSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJSGU2TAMRZGM. You are receiving this because you authored the thread.Message ID: @.***>

[ModernMAK]

A crude CLI that should meet your needs.

CLI Script

```python import argparse import json import os import re from dataclasses import dataclass from pathlib import Path from typing import List, Optional, Dict from relic.sga.core import StorageType from relic.sga.core.filesystem import EssenceFS from relic.sga.v2.serialization import essence_fs_serializer as v2_serializer # Schema ~ ~ ~ Not actually used @dataclass class SolverConfig: match: str query: Optional[str] = None # 'size >= X' ~ could handle any python string, but size storage: Optional[StorageType] = None # [STORE, BUFFER, STREAM] @dataclass class DriveConfig: solvers: List[SolverConfig] working_dir: Optional[str] = None # Relative or absolute @dataclass class Config: ... # 'alias':DriveConfig # Dict[str, DriveConfig] # End of Schema ~ ~ ~ _CHUNK_SIZE = 1024 * 1024 * 4 # 4 MiB _PRINT = True def _join(l: Optional[str], r: Optional[str]) -> str: if l is None: return r elif r is None: return l elif l is None and r is None: raise NotImplementedError("Cannot join two null strings") else: return os.path.join(l, r) def _print(*args, **kwargs): if _PRINT: print(*args, **kwargs) def _resolve_storage_type(s: Optional[str]): _HELPER = { "STORE": StorageType.STORE, "BUFFER": StorageType.BUFFER_COMPRESS, "STREAM": StorageType.STREAM_COMPRESS } if s is None: return StorageType.STORE s = s.upper() if s in _HELPER: return _HELPER[s] else: return StorageType[s] def pack(working_dir: str, outfile: str, config: Dict): _print(f"Packing `{outfile}`") # Create 'SGA' sga = EssenceFS() name = os.path.basename(outfile) sga.setmeta({ "name": name, "header_md5": "0" * 16, # Must be present due to a bug, recalculated when packed "file_md5": "0" * 16, # Must be present due to a bug, recalculated when packed }, "essence") # Specify name of archive # Walk Drives for alias, drive in config.items(): _print(f"\tPacking Drive `{alias}`") sga_drive = None # sga.create_drive(alias) # CWD for drive operations drive_cwd = _join(working_dir, drive.get("path")) # Try to pack files _print(f"\tScanning files in `{drive_cwd}`") frontier = set() _R = Path(drive_cwd) # Run matchers for solver in drive["solvers"]: # Determine storage type storage = _resolve_storage_type(solver.get("storage")) # Find matching files for path in _R.rglob(solver["match"]): if not path.is_file(): # Edge case handling continue # File Info ~ Name & Size full_path = str(path) if full_path in frontier: continue path_in_sga = os.path.relpath(full_path, drive_cwd) size = os.stat(full_path).st_size # Dumb way of supporting query query = solver.get("query") if query is None or len(query) == 0: ... # do nothing else: result = eval(query, {"size": size}) if not result: continue # Query Failure # match found, copy file to FS # EssenceFS is unfortunately, _print(f"\t\tPacking File `{os.path.relpath(full_path,drive_cwd)}` w/ `{storage.name}`") frontier.add(full_path) if sga_drive is None: sga_drive = sga.create_drive(alias) with open(full_path, "rb") as unpacked_file: with sga_drive.openbin(path_in_sga, "w") as packed_file: while True: buffer = unpacked_file.read(_CHUNK_SIZE) if len(buffer) == 0: break packed_file.write(buffer) sga_drive.setinfo(path_in_sga, {"essence": {"storage_type": storage}}) _print(f"Writing `{outfile}` to disk") # Write to binary file: with open(outfile, "wb") as sga_file: v2_serializer.write(sga_file, sga) _print(f"\tDone!") if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("src", type=str, help="Source Directory") parser.add_argument("out_sga", type=str, help="Output SGA File") parser.add_argument("config_file", type=str, help="Config .json file") args = parser.parse_args() with open(args.config_file) as json_h: args_config = json.load(json_h) pack(args.src, args.out_sga, args_config) ```

Sample config.json

```json { "data": { "solvers": [ { "match": "*.fda", "storage": "stream" }, { "match": "*.sample", "storage": "buffer", "query": "size > 0" }, { "match": "*.*" } ] }, "attrib": { "solvers": [] } } ```

First install SGA-V2 with pip install relic-tool-sga-v2~=1.0.0

The package version is important, as I'm working on a 2.0.0 release which will almost certainly not work with this script.

Copy the sample config file, or look at the schema at the top of the CLI to write your own. Solvers are resolved IN ORDER. By default, if 'storage' is not specified, it defaults to 'STORE'.

After installing, copy the script into a dump_sga.py file, and run it with python dump_sga.py "dir-to-pack" "path-to-save.sga" "path-to-config.json"

I've never actually tested the importance of a drive's contents, if i recall correctly, assets typically go in data drives and game data typically goes in attrib drives.

If you encounter an issue with this tool, or with Dawn Of War while using archives packed using this tool, please let me know, I'd be quite interested in the results.

[CannibalToast]

Awesome!!! I was able to get it to work. Quick question. Is it possible to compress the .sga file aswell?

[ModernMAK]

The contents can be compressed, (using a storage type of STREAM or BUFFER in the config.json file), this will zlib compress the individual files that get packed in the archive.

Certain assets like .fda audio files are typically stream compressed, and I expect DoW can handle any asset that's compressed, but I have not tested this myself. The sample config has an example of this.

 {
  "match": "*.fda",
  "storage": "stream"
}

The SGA archive container format itself does not support compression, (at least not in the V2 spec), if you want Dawn of War to be able to read them, you'll need to leave the SGA uncompressed and only compress the packed files.

[CannibalToast]

I see, I am going to put in different file types to compress using this format.

[CannibalToast]

Hello! So i am currently trying to make an unpack script by reverse engineering the script that you made for me. Unfortunately i am not fluent in python and am not having any luck. I hate to ask again but when you have the time would it be possible to make an extract script? Here is what i have.

import argparse
import json
import os
import re
from dataclasses import dataclass
from pathlib import Path
from typing import List, Optional, Dict

from relic.sga.core import StorageType
from relic.sga.core.filesystem import EssenceFS
from relic.sga.v2.serialization import essence_fs_serializer as v2_serializer

# Schema ~ ~ ~ Not actually used
@dataclass
class SolverConfig:
    match: str
    query: Optional[str] = None  # 'size >= X' ~ could handle any python string, but size
    storage: Optional[StorageType] = None  # [STORE, BUFFER, STREAM]

@dataclass
class DriveConfig:
    solvers: List[SolverConfig]
    working_dir: Optional[str] = None  # Relative or absolute

@dataclass
class Config:
    ... # 'alias':DriveConfig # Dict[str, DriveConfig]

# End of Schema ~ ~ ~

_CHUNK_SIZE = 1024 * 1024 * 4  # 4 MiB
_PRINT = True

def _join(l: Optional[str], r: Optional[str]) -> str:
    if l is None:
        return r
    elif r is None:
        return l
    elif l is None and r is None:
        raise NotImplementedError("Cannot join two null strings")
    else:
        return os.path.join(l, r)

def _print(*args, **kwargs):
    if _PRINT:
        print(*args, **kwargs)

def _resolve_storage_type(s: Optional[str]):
    _HELPER = {
        "STORE": StorageType.STORE,
        "BUFFER": StorageType.BUFFER_COMPRESS,
        "STREAM": StorageType.STREAM_COMPRESS
    }
    if s is None:
        return StorageType.STORE

    s = s.upper()
    if s in _HELPER:
        return _HELPER[s]
    else:
        return StorageType[s]

def pack(working_dir: str, outfile: str, config: Dict):
    ...

def unpack(infile: str, outdir: str):
    _print(f"Unpacking `{infile}`")

    # Read binary file
    with open(infile, "rb") as sga_file:
        sga = v2_serializer.read(sga_file)

    # Iterate over drives in the SGA file
    for drive in sga.getdrives():
        alias = drive.alias
        _print(f"\tUnpacking Drive `{alias}`")

        # Create the output directory for the drive
        drive_dir = os.path.join(outdir, alias)
        os.makedirs(drive_dir, exist_ok=True)

        # Iterate over files in the drive
        for path in drive.walk():
            if not path.isdir():  # Skip directories
                # Get the relative path of the file in the drive
                rel_path = os.path.relpath(path.fullpath, "/")

                # Create the output directory for the file
                file_dir = os.path.join(drive_dir, os.path.dirname(rel_path))
                os.makedirs(file_dir, exist_ok=True)

                # Create the output file
                out_file = os.path.join(file_dir, os.path.basename(rel_path))

                # Open the file in the drive and copy its contents to the output file
                with path.openbin("r") as packed_file:
                    with open(out_file, "wb") as unpacked_file:
                        while True:
                            buffer = packed_file.read(_CHUNK_SIZE)
                            if len(buffer) == 0:
                                break
                            unpacked_file.write(buffer)

    _print(f"Unpacking complete!")

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("src", type=str, help="Source Directory")
    parser.add_argument("out_sga", type=str, help="Output SGA File")
    parser.add_argument("config_file", type=str, help="Config .json file")
    parser.add_argument("--unpack", action="store_true", help="Unpack the SGA file")
    args = parser.parse_args()

    with open(args.config_file) as json_h:
        args_config = json.load(json_h)

    if args.unpack:
        unpack(args.src, args.out_sga)
    else:
        pack(args.src, args.out_sga, args_config)

[ModernMAK]

Unpacking is actually much easier, the archive doesn't need to 'setup' the filesystem, so we can just use PyFilesystem's copy_fs function.

Something like:

def unpack(infile: str, outdir: str):
    import fs # import here or at the top
    _print(f"Unpacking `{infile}`")
   fs.copy.copy_fs(f"sga://{infile}',f'osfs://{outdir}')

If you wanted print messages you could use:

def _copy_callback(src_fs, src_path, dst_fs, dst_path):
   _print(f"\tUnpacking `{src_path}`")

def unpack(infile: str, outdir: str):
    import fs # import here or at the top
    _print(f"Unpacking `{infile}`")
   fs.copy.copy_fs(f"sga://{infile}',f'osfs://{outdir}', on_copy=_copy_callback)

I haven't tested the snippets, but it should be real close to what needs to be done.

[ModernMAK]

Packer & Unpacker

import argparse
import json
import os
import re
from dataclasses import dataclass
from pathlib import Path
from typing import List, Optional, Dict

import fs.copy
from fs.base import FS
from relic.sga.core import StorageType
from relic.sga.core.filesystem import EssenceFS
from relic.sga.v2.serialization import essence_fs_serializer as v2_serializer

# Schema ~ ~ ~ Not actually used
@dataclass
class SolverConfig:
    match: str
    query: Optional[str] = None  # 'size >= X' ~ could handle any python string, but size
    storage: Optional[StorageType] = None  # [STORE, BUFFER, STREAM]

@dataclass
class DriveConfig:
    solvers: List[SolverConfig]
    working_dir: Optional[str] = None  # Relative or absolute

@dataclass
class Config:
    ...  # 'alias':DriveConfig # Dict[str, DriveConfig]

# End of Schema ~ ~ ~

_CHUNK_SIZE = 1024 * 1024 * 4  # 4 MiB
_PRINT = True

def _join(l: Optional[str], r: Optional[str]) -> str:
    if l is None:
        return r
    elif r is None:
        return l
    elif l is None and r is None:
        raise NotImplementedError("Cannot join two null strings")
    else:
        return os.path.join(l, r)

def _print(*args, **kwargs):
    if _PRINT:
        print(*args, **kwargs)

def _resolve_storage_type(s: Optional[str]):
    _HELPER = {
        "STORE": StorageType.STORE,
        "BUFFER": StorageType.BUFFER_COMPRESS,
        "STREAM": StorageType.STREAM_COMPRESS
    }
    if s is None:
        return StorageType.STORE

    s = s.upper()
    if s in _HELPER:
        return _HELPER[s]
    else:
        return StorageType[s]

def pack(args):
    # Extract Args
    working_dir: str = args.src_dir
    outfile: str = args.out_sga
    config_file: str = args.config_file
    with open(config_file) as json_h:
        config: Dict = json.load(json_h)

    # Execute Command
    _print(f"Packing `{outfile}`")

    # Create 'SGA'
    sga = EssenceFS()
    name = os.path.basename(outfile)
    sga.setmeta({
        "name": name,  # Specify name of archive
        "header_md5": "0" * 16,  # Must be present due to a bug, recalculated when packed
        "file_md5": "0" * 16,  # Must be present due to a bug, recalculated when packed
    }, "essence")

    # Walk Drives
    for alias, drive in config.items():
        _print(f"\tPacking Drive `{alias}`")
        sga_drive = None  # sga.create_drive(alias)

        # CWD for drive operations
        drive_cwd = _join(working_dir, drive.get("path"))

        # Try to pack files
        _print(f"\tScanning files in `{drive_cwd}`")
        frontier = set()
        _R = Path(drive_cwd)

        # Run matchers
        for solver in drive["solvers"]:
            # Determine storage type
            storage = _resolve_storage_type(solver.get("storage"))
            # Find matching files
            for path in _R.rglob(solver["match"]):
                if not path.is_file():  # Edge case handling
                    continue
                # File Info ~ Name & Size
                full_path = str(path)
                if full_path in frontier:
                    continue
                path_in_sga = os.path.relpath(full_path, drive_cwd)
                size = os.stat(full_path).st_size

                # Dumb way of supporting query
                query = solver.get("query")
                if query is None or len(query) == 0:
                    ...  # do nothing
                else:
                    result = eval(query, {"size": size})
                    if not result:
                        continue  # Query Failure

                # match found, copy file to FS
                # EssenceFS is unfortunately,
                _print(f"\t\tPacking File `{os.path.relpath(full_path, drive_cwd)}` w/ `{storage.name}`")
                frontier.add(full_path)
                if sga_drive is None:  # Lazily create drive, to avoid empty drives from being created
                    sga_drive = sga.create_drive(alias)

                with open(full_path, "rb") as unpacked_file:
                    with sga_drive.openbin(path_in_sga, "w") as packed_file:
                        while True:
                            buffer = unpacked_file.read(_CHUNK_SIZE)
                            if len(buffer) == 0:
                                break
                            packed_file.write(buffer)
                    sga_drive.setinfo(path_in_sga, {"essence": {"storage_type": storage}})

    _print(f"Writing `{outfile}` to disk")
    # Write to binary file:
    with open(outfile, "wb") as sga_file:
        v2_serializer.write(sga_file, sga)
    _print(f"\tDone!")

def unpack(args):
    infile: str = args.src_sga
    outdir: str = args.out_dir

    _print(f"Unpacking `{infile}`")
    def _callback(_1: FS, srcfile: str, _2: FS, _3: str):
        _print(f"\t\tUnpacking File `{srcfile}`")

    fs.copy.copy_fs(f"sga://{infile}", f"osfs://{outdir}", on_copy=_callback)
    _print(f"\tDone!")

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    subparsers = parser.add_subparsers(title="commands")

    pack_cmd = subparsers.add_parser("pack")
    pack_cmd.add_argument("src_dir", type=str, help="Source Directory")
    pack_cmd.add_argument("out_sga", type=str, help="Output SGA File")
    pack_cmd.add_argument("config_file", type=str, help="Config .json file")
    pack_cmd.set_defaults(func=pack)

    unpack_cmd = subparsers.add_parser("unpack")
    unpack_cmd.add_argument("src_sga", type=str, help="Source SGA File")
    unpack_cmd.add_argument("out_dir", type=str, help="Output Directory")
    unpack_cmd.set_defaults(func=unpack)

    args = parser.parse_args()
    args.func(args)  # call function for command

Pack: python sgav2cli.py pack src_dir out_sga config_file Unpack python sgav2cli.py unpack src_sga out_dir

---

Edit: added print statements.

[CannibalToast]

It seems to work. One little issue though. It seems to rename the files with the directory they are supposed to be in instead of putting the files in the folders that they belong.

[ModernMAK]

It seems to rename the files with the directory they are supposed to be in instead of putting the files in the folders that they belong.

I don't think I understand what you mean. I assume this is a problem when packing since you posted a screencap of the archive viewer. Can you give me an example of whats happening and what you expect to happen?

E.G Actual: Packing a file sound\custom sounds\mysound.fda becomes sound\custom sounds.fda in the archive.

Expected: Packing a file sound\custom sounds\mysound.fda becomes sound\custom sounds\mysound.fda in the archive.

[CannibalToast]

I see,

The issue is that the files are not being put into folders and sub folders like they are in the regular folder. The directory of the files is not being matched inside of the SGA. Instead of there being separate folders of different things all of the files are inside the sga"s base directory with no folder.

[ModernMAK]

I think i understand now, looking into it.

[CannibalToast]

This is what the sga file above should look like after packing. With sub folders and stuff. Thanks for looking into this.

[ModernMAK]

Think I've found the problem; made an issue #39

[ModernMAK]

The patch breaks the CLI, since that bug hid a separate bug in the CLI,

Since this CLI more-or-less works, working on rolling it into the proper repos. Will probably close this issue when that's done.

[ModernMAK]

The CLI has been rolled into relic-tool-sga-v2, simply update it via pip install relic-tool-sga-v2 -U and all dependencies should update as well.

Commands are now executed directly e.g. relic sga pack v2 ... instead of python script.py pack ...

A new guide is located here, which goes into more depth of how the packing.json files works. Alternatively check out the main page for an alternate link to the guide.

Any further problems and bugs probably deserve their own issue. But feel free to continue fielding questions here.

[CannibalToast]

Hello! The unpack works perfectly. Unfortunately i am getting an error with packing.

PS C:\Users\Admin\Desktop> relic sga pack v2 "D:\uasound" "D:\uasound.sga" "D:\config.json" PackingD:\uasound.sga Packing Drivetest Scanning files inD:\uasound` Packing File sound\ultimate_apocalypse_grand_release.fda w/ STORE Packing File sound\_default.fda w/ STORE Packing File sound\_default.rat w/ STORE Packing File sound\assorted\_default.rat w/ STORE Packing File sound\assorted\nature\earth\earthquake.fda w/ STORE Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "C:\Python311\Scripts\relic.exe__main__.py", line 7, in File "C:\Python311\Lib\site-packages\relic\core\cli.py", line 53, in run exit_code = self._run(ns) ^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\relic\core\cli.py", line 39, in _run result: Optional[int] = cmd(ns) ^^^^^^^ File "C:\Python311\Lib\site-packages\relic\sga\v2\cli.py", line 122, in command with sga_drive.makedirs(parent, recreate=True) as folder: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\fs\base.py", line 1132, in makedirs self.makedir(path, permissions=permissions) File "C:\Python311\Lib\site-packages\fs\memoryfs.py", line 436, in makedir raise errors.ResourceNotFound(path) fs.errors.ResourceNotFound: resource 'sound\assorted\nature\earth' not found`

Here is the config file i am using

{ "test": { "solvers": [ { "match": "store.*", "storage": "store" }, { "match": "*.buffer", "storage": "buffer" }, { "match": "stream.txt", "storage": "stream" }, { "match": "*.*", "storage": "store", "query": "size > 0" }, { "match": "*.fda", "storage": "stream" } ] }, "data": { "path": "mymodata", "solvers": [ { "match": "*.fda", "storage": "store" }, { "match": "*.*", "storage": "stream", "query": "size >= 1000000000" }, { "match": "*.*", "storage": "store" } ] }, "attrib": { "path": "mymodattributes", "solvers": [ { "match": "*.*", "storage": "store" } ] } }

I double checked and the files inside that directory are all where they should be for packing.

Edit: I am also appearing to get a weird permissions error when i try to pack on the OS drive.

PS C:\Users\Admin\Desktop> relic sga pack v2 "C:\Users\Admin\Desktop\uasound.sga" "C:\Users\Admin\Desktop\uasound" "C:\Users\Admin\Desktop\config.json" PackingC:\Users\Admin\Desktop\uasound Packing Drivedata Scanning files inC:\Users\Admin\Desktop\uasound.sga\mymodata Packing Driveattrib Scanning files inC:\Users\Admin\Desktop\uasound.sga\mymodattributes WritingC:\Users\Admin\Desktop\uasoundto disk Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "C:\Python311\Scripts\relic.exe\__main__.py", line 7, in <module> File "C:\Python311\Lib\site-packages\relic\core\cli.py", line 53, in run exit_code = self._run(ns) ^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\relic\core\cli.py", line 39, in _run result: Optional[int] = cmd(ns) ^^^^^^^ File "C:\Python311\Lib\site-packages\relic\sga\v2\cli.py", line 135, in command with open(outfile, "wb") as sga_file: ^^^^^^^^^^^^^^^^^^^ PermissionError: [Errno 13] Permission denied: 'C:\\Users\\Admin\\Desktop\\uasound'

[ModernMAK]

First problem needs more testing; makedirs is complaining that a directory doesn't exist, but it's job is to create those directories. 🙄 Think the issue might be packing from the root of a drive, I don't have a test case for that anywhere, so that's my first assumption.

This isn't your problem, but should be said; The example config is mostly to explain how to use the various features; explained here, it shouldn't be used without some modification. I suppose I should mention that in the guide, and provide an alternative 'minimal' config.

Looking at the log, your packing stuff into test which isn't a normal drive. Since you're packing sounds, they typically go in the data drive, which means data's path should be removed or point to the subfolder where all your data files are. attrib can also be removed; i think the only think in attrib is *.rgd files, and I'm working on the assumption you don't have any.

That being said, the config format hasn't changed, the config file you were using before should still work with the new CLI.

Second problem;

PS C:\Users\Admin\Desktop> relic sga pack v2 "C:\Users\Admin\Desktop\uasound.sga" "C:\Users\Admin\Desktop\uasound"  "C:\Users\Admin\Desktop\config.json"

Your second run swapped the src_dir and out_sga arguments, that's why you're getting a permission error, you are trying to open a directory as if it was a file. Which admittedly, deserves a check to print a better error.

Try

PS C:\Users\Admin\Desktop> relic sga pack v2 "C:\Users\Admin\Desktop\uasound" "C:\Users\Admin\Desktop\uasound.sga" "C:\Users\Admin\Desktop\config.json"

Alternatively, the tool should respect current working directories so...

PS C:\Users\Admin\Desktop> relic sga pack v2 "uasound" "uasound.sga" "config.json"

should get you the same result.

[ModernMAK]

Hot-fix out, update with pip install relic-tool-sga-v2 -U

covers problem 1, problem 2 I spaced on, so I'll make an issue and tackle that later.

[CannibalToast]

The hotfix worked! I am able to get it working, and from the desktop now.

Thank you so much for taking all this time and effort to make this possible. :)

[CannibalToast]

Hello, i was playing around with the script a bit. I made a simple powershell wrapper for it. When i put the sga into dawn of war it appeared to not be seen. Looks like the TOC header information is not being transferred or made. The archive name. ie. 2.arciv.txt This for example is what the .arciv file looks like when packing with modpackager (excluding the thousands of files/data inbetween these two strings) and then the archive header information that is used to create the sga and a manifest of file extensions stored

    {
        ArchiveName = "uasound2",
    },

    TOCHeader =  
            {
                Alias = "data",
                Name = "uasound2",
                RootPath = [[C:\Users\Admin\Desktop\New folder (2)\UltimateApocalypse_THB\data\]],
                Storage =  
                {

                    {
                        MaxSize = 100,
                        MinSize = -1,
                        Storage = 0,
                        Wildcard = "*.*",
                    },

                    {
                        MaxSize = -1,
                        MinSize = -1,
                        Storage = 0,
                        Wildcard = "*.mp3",
                    },

                    {
                        MaxSize = -1,
                        MinSize = -1,
                        Storage = 0,
                        Wildcard = "*.wav",
                    },

                    {
                        MaxSize = -1,
                        MinSize = -1,
                        Storage = 0,
                        Wildcard = "*.jpg",
                    },

                    {
                        MaxSize = -1,

                        MinSize = -1,
                        Storage = 2,
                        Wildcard = "*.lua",
                    },
                },
            },
[I attached the file above for reference. it is normally a .arciv file but needed to rename it .txt for github]

i'm wondering if I'm doing something wrong with my configuration or if it's something that has yet to be implemented 

[ModernMAK]

Are you able to open the SGA in mod packager, or the archive viewer? If it opens, then yay. The TOC shouldn't be the main problem. It may warn you that it can't open because the SGA is being used elsewhere, which just means you need to close the other tool before opening it again.

It does look like I dropped a todo for specifying the drive's name, but I don't believe that's the issue. My initial tests let me open and extract files when repacking official archives. BUT, it may be why DoW can't see it, if it expects a unique name to load archives, but that's all speculation.

I'm assuming that most of the TOCHeader metadata in the .arciv file is only used for packing; only alias and name are included (as part of the drive's definition in the ToC list). ModPackager and ArchiveViewer don't complain (mostly) when I test against my samples, so I think the TOC block is fine, but I'll have to look into that too. Fixing the name specification should be the easiest thing to fix, unlike the other things, I'm about to mention.

According to a few tests, it seems that the data block is more complicated than I had originally thought. There appears to be checksums associated with every file, which I thought wasn't included until SGA V4 or V5. (They did, after all, have two checksums in the header itself.) Unfortunately, the Xentax forums are closing/closed so I can't check if they mentioned this, and it's not in my original notes when I studied the format.

Reopening issue because it looks like 'no' this tool is not ready for SGA packing. It'll take a while to delve deeper and see exactly what I'm missing.

[CannibalToast]

Looks like the toc name is blank. I think the archive name is blank aswell

[ModernMAK]

The archive name is written; DXP2DataKeys, the funky dots are due to the archive using utf-16 encoding.

Edit: I should clarify, the highlighted section is the 'root folder name'. Near the top is where the archive name is written.

That missing value should display the name of the root folder (I keep calling it a drive), which as I said;

It does look like I dropped a todo for specifying the drive's name, but I don't believe that's the issue. ... Fixing the name specification should be the easiest thing to fix, unlike the other things, I'm about to mention.

I can get a hotfix out quick to fix that, but I'm pretty sure this won't solve the problem of DoW failing to read the SGA.

[ModernMAK]

New patch, like before, run the pip command to update all dependencies.

pip install relic-tool-sga-v2 -U

You MUST add a new name field in the config.json file. Check the guide if you want a visual example.

I've tested swapping images when repacking assets, so I'm fairly confident that this should let DoW read files.

Do let me know if there are further issues.

[CannibalToast]

There appears to be one more variable.

The archive name. I dont think that it goes by name of the file but by name of the archive.

While this latest patch helps significantly i think that this should be the last thing to fix before dawn of war can see the file.

In some previous screenshots that you have shown, i did see that you were running dawn of war dark crusade, while i am running soulstorm. I am not sure, is there a difference in packing procedure between those two games?

I am unable to verify what the archive names in this case are for the default files because i do not see any way of actually getting that information. Mod packager and sga viewer do not have that capability.

[ModernMAK]

I've tested opening all official archives from steam, DoW Gold, DoW Winter Assault, DoW DC, and DoW SS. While SS is a bit weird (it wasn't made by relic, they had a 3rd party make it), there doesn't seem to be any weirdness with how they packed the SGA files, so that's a relief.

I can assure you, when packed, the 'archive name' is the name of the file (without the extension). At least for all official SGAs.

The root folder name (the name that shows up in mod packager / archive viewer) is typically the name of the file, without the extension and all lowercase, exceptions do apply. E.G. Dawn of War Dark Crusade\Engine\Engine.sga has a root folder name of enginedata instead of engine.

If you had a hex editor handy, you could determine the root folder of an SGA by looking for a "data" string followed by a lot of zeros (60 to be exact). The next 64 bytes are the name of the root folder.

But, I found a new bug while testing this, so I need to roll out a new patch anyways, I can add a tool to the CLI to print metadata information or something. I'll post an update when that's released

[ModernMAK]

New patch, this one is a dependency, so it's not the same command as before

pip install relic-tool-sga-core -U

The new Info CLI:

usage: relic sga info [-h] [-q] sga [log_file]

Dumps metadata packed into an SGA file.

positional arguments:
  sga          SGA File to inspect
  log_file     Optional file to write messages to, required if `-q/--quiet` is used

optional arguments:
  -h, --help   show this help message and exit
  -q, --quiet  When specified, SGA info is not printed to the console

Personally I'd use

relic sga info -q sga [log_file]

because it will print ALL metadata, and the log file allows you to quickly go back to it.

The CLI guide doesn't have this command yet, I'll need to get around to updating it.

[CannibalToast]

That tool is very useful. It did find an error with one of the sga's i packed using relic-tool. The log only shows the name of the sga. There appears to be a checksum error of some sort

PS C:\Users\Admin\Desktop> relic sga info -q uasound4.sga toast.log Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "C:\Python311\Scripts\relic.exe\__main__.py", line 7, in <module> File "C:\Python311\Lib\site-packages\relic\core\cli.py", line 53, in run exit_code = self._run(ns) ^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\relic\core\cli.py", line 39, in _run result: Optional[int] = cmd(ns) ^^^^^^^ File "C:\Python311\Lib\site-packages\relic\sga\core\cli.py", line 222, in command with fs.open_fs(f"sga://{sga}") as sgafs: # type: ignore ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\fs\opener\registry.py", line 220, in open_fs _fs, _path = self.open( ^^^^^^^^^^ File "C:\Python311\Lib\site-packages\fs\opener\registry.py", line 177, in open open_fs = opener.open_fs(fs_url, parse_result, writeable, create, cwd) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\relic\sga\core\filesystem.py", line 208, in open_fs return self.factory.read(sga_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\relic\sga\core\filesystem.py", line 161, in read return handler.read(sga_stream) ^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\relic\sga\core\serialization.py", line 824, in read assembler.assemble(essence_fs) File "C:\Python311\Lib\site-packages\relic\sga\core\serialization.py", line 420, in assemble self.assemble_drive(fs, drive_def, folder_defs, file_defs) File "C:\Python311\Lib\site-packages\relic\sga\core\serialization.py", line 405, in assemble_drive self._assemble_container( File "C:\Python311\Lib\site-packages\relic\sga\core\serialization.py", line 352, in _assemble_container self.assemble_folder( File "C:\Python311\Lib\site-packages\relic\sga\core\serialization.py", line 374, in assemble_folder self._assemble_container( File "C:\Python311\Lib\site-packages\relic\sga\core\serialization.py", line 352, in _assemble_container self.assemble_folder( File "C:\Python311\Lib\site-packages\relic\sga\core\serialization.py", line 374, in assemble_folder self._assemble_container( File "C:\Python311\Lib\site-packages\relic\sga\core\serialization.py", line 352, in _assemble_container self.assemble_folder( File "C:\Python311\Lib\site-packages\relic\sga\core\serialization.py", line 374, in assemble_folder self._assemble_container( File "C:\Python311\Lib\site-packages\relic\sga\core\serialization.py", line 352, in _assemble_container self.assemble_folder( File "C:\Python311\Lib\site-packages\relic\sga\core\serialization.py", line 374, in assemble_folder self._assemble_container( File "C:\Python311\Lib\site-packages\relic\sga\core\serialization.py", line 349, in _assemble_container self.assemble_file(container, file_def) File "C:\Python311\Lib\site-packages\relic\sga\v2\serialization.py", line 270, in assemble_file raise MismatchError("CRC Checksum", crc32_generated, crc32) relic.core.errors.MismatchError: Unexpected CRC Checksum; gotb'#\xee\xb3', expected b'+\x90\x9d\x91'!`

[ModernMAK]

More of an oversight on my part. CRC checks are enabled by default.

Was uasound built using modpackager, this tool, or corsix?

If you open the sga in modpackager and verify checksums, does it say all checksums pass?

I should add a flag to toggle checksum verification though, so I'll make an issue and get back to that later.

[CannibalToast]

I see, yeah modpackager appears to say the checksum is correct.

Unfortunately I still cannot get the game to recognize the file.

While I was able to get the game to recognize more sounds. More beeps in game (beep = sound not found)

I cannot seem to figure out why the sounds are not just playing. I think it's a settings issue with the archiving method. I am going to try other archiving methods.

I'm gonna also install a hex editor and see what the differences are when it comes to one that is from mod packager and one that is packaged using relic tool.

[ModernMAK]

I'm sorry this tool wasn't as prepared as I thought, but I do thank you, your help as given me more insight to the file format than I'd originally had.

Good luck on your modding.

[CannibalToast]

It's okay! This tool has significantly reduced the time my mod team takes to unpack these archives. The old tools we used would crash on archives more than 2gb so we can pack more in and not need to split up the archives.

I can attest to how infuriating it is to reverse engineer 20 year old code. You've done an excellent job here.

[CannibalToast]

Hello. It has been a while since we last spoke. I was wondering what the plans for this program would be in the future? If you plan to keep working on it or archive it?

[ModernMAK]

SGA-V2 still gets a few updates, I've been archiving a lot of the other repos that I just won't have time to get updated.

SGA V2 got a massive rework to better support reading/writing files, and supports .arciv directly instead of my half-baked JSON.

As of right now, not sure how stable it is. I updated my test-cases which is always scary. Not sure when I'll get the next release out.

[ModernMAK]

You're welcome to try my latest dev code, but it's not tested or well documented.

pip install git+https://github.com/MAK-Relic-Tool/Relic-Tool-Core@indev
pip install git+https://github.com/MAK-Relic-Tool/Relic-Tool-SGA-Core@code-review
pip install git+https://github.com/MAK-Relic-Tool/SGA-V2@migration-to-2.0

Assuming that installs properly.

The following should tell you how the new pack function works; it now uses the DOW .arciv files.

relic sga pack -h 

I don't think unpacking works, since V2 also has tidbits for Impossible Creatures, which really mucks up everything.

[CannibalToast]

Awesome!

So it seemed to install correctly.

a package called ply was missing so i just installed that and it seemed to work and give help for commands.

I just tried to unpack out of curiosity and it seems to have found a mismatch.

Traceback (most recent call last): File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\Scripts\relic.exe__main__.py", line 7, in File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\relic\core\cli.py", line 168, in run return super().run() File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\relic\core\cli.py", line 122, in run exit_code = self._run(ns, sys.argv) File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\relic\core\cli.py", line 91, in _run result: Optional[int] = func(ns) File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\relic\sga\core\cli.py", line 170, in command copy_fs(sga, f"osfs://{outdir}", on_copy=_callback, preserve_time=True) File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\fs\copy.py", line 50, in copy_fs return copy_fs_if( File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\fs\copy.py", line 110, in copy_fs_if return copy_dir_if( File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\fs\copy.py", line 450, in copy_dir_if copier.copy(_src_fs, dir_path, _dst_fs, copy_path) File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\fs_bulk.py", line 144, in copy copy_file_internal( File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\fs\copy.py", line 281, in copy_file_internal _copy_locked() File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\fs\copy.py", line 275, in _copy_locked copy_modified_time(src_fs, src_path, dst_fs, dst_path) File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\fs\copy.py", line 521, in copy_modified_time src_meta = _src_fs.getinfo(src_path, namespaces) File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\relic\sga\v2\essencefs\definitions.py", line 2346, in getinfo return node.getinfo(namespaces) # type: ignore File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\relic\sga\v2\essencefs\definitions.py", line 533, in getinfo return self._backing.getinfo(namespaces) File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\relic\sga\v2\essencefs\definitions.py", line 313, in getinfo modified=self.modified_unix, File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\relic\sga\v2\essencefs\definitions.py", line 361, in modified_unix return self._data_info.header.modified File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\relic\sga\v2\serialization.py", line 266, in modified buffer = self._serializer.read_bytes(*self.Meta.modified_ptr) File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\relic\core\lazyio.py", line 636, in read_bytes value = self._cache[key] = _read() File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\relic\core\lazyio.py", line 629, in _read raise MismatchError("Read Mismatch", len(b), size) relic.core.errors.MismatchError: Unexpected Read Mismatch; got 0, expected 4!

I will try packing now

[CannibalToast]

I just tested the packing and it seems to have encountered some sort of limit error.

PS C:\Users\Admin\Desktop> relic sga pack v2 test.arciv test.sga Traceback (most recent call last): File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\Scripts\relic.exe__main__.py", line 7, in File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\relic\core\cli.py", line 168, in run return super().run() File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\relic\core\cli.py", line 122, in run exit_code = self._run(ns, sys.argv) File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\relic\core\cli.py", line 91, in _run result: Optional[int] = func(ns) File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\relic\sga\v2\cli.py", line 69, in command elif not _check_parts(out_path): File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\relic\sga\v2\cli.py", line 59, in _check_parts return _check_parts(d) File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\relic\sga\v2\cli.py", line 59, in _check_parts return _check_parts(d) File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\relic\sga\v2\cli.py", line 59, in _check_parts return _check_parts(d) [Previous line repeated 986 more times] File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\relic\sga\v2\cli.py", line 55, in _check_parts d, f = os.path.split(_path) File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\ntpath.py", line 212, in split seps = _get_bothseps(p) File "C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\ntpath.py", line 36, in _get_bothseps if isinstance(path, bytes): RecursionError: maximum recursion depth exceeded while calling a Python object

Here is the arciv file for reference. i renamed it json for github test.json

[ModernMAK]

Cool, i'll try to take a peak later this week