search

Scighost / Starward

Game Launcher for miHoYo - 米家游戏启动器
https://starward.scighost.com
MIT License
3.56k stars 164 forks source link

[Document] 原神启动器的 Chunk 下载模式分析 #725

Open Scighost opened 8 months ago

Scighost commented 8 months ago

一直以来，原神官服启动器的下载模式饱受玩家诟病，其采用的压缩包-解压的模式会占用两倍于压缩包的存储空间。从原神 4.5 版本的预下载开始，启动器加入了全新的 Chunk 下载模式，解决了这个缺点。在此模式下，单个游戏资源文件被切分为多个块分别被下载，最后在本地组合成一个整体。

灰度测试

Chunk 下载模式正处于灰度测试中，但是可以通过修改 API 的返回值强制开启。启动器会请求下列 API 若干次：

https://abtest-api-data.mihoyo.com/data_abtest_api/config/experiment/list

在某一次的请求中会有如下的返回值，只需把 "downloadMode": "file" 修改为 "downloadMode": "chunk" 即可启用 Chunk 下载模式。

{
  "retcode": 0,
  "success": true,
  "message": "",
  "data": [
    {
      "code": 1010,
      "type": 2,
      "config_id": "347",
      "period_id": "",
      "version": "",
      "configs": {
        "downloadMode": "file"
      },
      "sceneWhiteList": false,
      "experimentWhiteList": false
    }
  ]
}

版本信息

启动器通过下列 API 获得游戏的版本数据，此外还可以通过添加查询参数 tag=4.5.0 获取特定版本的信息。非预下载期间，预下载 API 的返回值为空。

官服正式版
https://api-takumi.mihoyo.com/downloader/sophon_chunk/api/getBuild?branch=main&package_id=s8m3Yf3j6G&password=QlYzM79uF6va&plat_app=cxgf44wie1a8

官服预下载
https://api-takumi.mihoyo.com/downloader/sophon_chunk/api/getBuild?branch=predownload&package_id=s8m3Yf3j6G&password=izObT6iTHAqq&plat_app=cxgf44wie1a8

B服正式版
https://api-takumi.mihoyo.com/downloader/sophon_chunk/api/getBuild?branch=main&package_id=yxzhR2pRqE&password=mPNvWfvLBO8b&plat_app=cxgf44wie1a8

B服预下载
https://api-takumi.mihoyo.com/downloader/sophon_chunk/api/getBuild?branch=predownload&package_id=yxzhR2pRqE&password=16dFsdfbRBPO&plat_app=cxgf44wie1a8

国际服正式版
https://sg-public-api.hoyoverse.com/downloader/sophon_chunk/api/getBuild?branch=main&package_id=Ul0pnnZo8h&password=SSaKNEvZjrK0&plat_app=cxhpq4g4rgg0

国际服预下载
https://sg-public-api.hoyoverse.com/downloader/sophon_chunk/api/getBuild?branch=predownload&package_id=Ul0pnnZo8h&password=yran8QsvU88T&plat_app=cxhpq4g4rgg0

这里展示官服 4.5.0 版本的信息，JSON 的结构和命名非常清晰，通过节点 manifest 和 manifest_download 可以组合出资源清单文件的下载链接，通过节点 chunk_download 和清单文件中的内容可以组合出游戏资源文件的下载链接。

API 中存在 encryption 和 password 参数，后续可能会对资源文件加密。

版本信息示例 ``` json { "retcode": 0, "message": "OK", "data": { "build_id": "D7g0SFeiFg9o", "tag": "4.5.0", "manifests": [ { "category_id": "10017", "category_name": "游戏资源-外网", "manifest": { "id": "manifest_233f3acd5276c84e_890ab337d4ec8edf6c98c4dcf702b8bf", "checksum": "890ab337d4ec8edf6c98c4dcf702b8bf", "compressed_size": "4736513", "uncompressed_size": "9818860" }, "chunk_download": { "encryption": 0, "password": "", "compression": 1, "url_prefix": "https://autopatchcn.yuanshen.com/client_app/sophon/chunks/cxgf44wie1a8/62nfHL6ocpNF", "url_suffix": "" }, "manifest_download": { "encryption": 0, "password": "", "compression": 1, "url_prefix": "https://autopatchcn.yuanshen.com/client_app/sophon/manifests/cxgf44wie1a8/62nfHL6ocpNF", "url_suffix": "" }, "matching_field": "game", "stats": { "compressed_size": "75506162967", "uncompressed_size": "76752839912", "file_count": "16115", "chunk_count": "74156" }, "deduplicated_stats": { "compressed_size": "75443433225", "uncompressed_size": "76650339773", "file_count": "16115", "chunk_count": "74077" } }, { "category_id": "10018", "category_name": "语音包-中文-外网", "manifest": { "id": "manifest_fe557261015fe099_f13178635a13eb92613660a2989a8d48", "checksum": "f13178635a13eb92613660a2989a8d48", "compressed_size": "697187", "uncompressed_size": "1333393" }, "chunk_download": { "encryption": 0, "password": "", "compression": 1, "url_prefix": "https://autopatchcn.yuanshen.com/client_app/sophon/chunks/cxgf44wie1a8/62nfHL6ocpNF", "url_suffix": "" }, "manifest_download": { "encryption": 0, "password": "", "compression": 1, "url_prefix": "https://autopatchcn.yuanshen.com/client_app/sophon/manifests/cxgf44wie1a8/62nfHL6ocpNF", "url_suffix": "" }, "matching_field": "zh-cn", "stats": { "compressed_size": "11858894718", "uncompressed_size": "13671657756", "file_count": "129", "chunk_count": "11952" }, "deduplicated_stats": { "compressed_size": "11858620147", "uncompressed_size": "13671337332", "file_count": "129", "chunk_count": "11951" } }, { "category_id": "10019", "category_name": "语音包-英文-外网", "manifest": { "id": "manifest_3de8d7c6c1a4bef4_1daceaa6e906c0aa3d6621e6039f8b84", "checksum": "1daceaa6e906c0aa3d6621e6039f8b84", "compressed_size": "789952", "uncompressed_size": "1505333" }, "chunk_download": { "encryption": 0, "password": "", "compression": 1, "url_prefix": "https://autopatchcn.yuanshen.com/client_app/sophon/chunks/cxgf44wie1a8/62nfHL6ocpNF", "url_suffix": "" }, "manifest_download": { "encryption": 0, "password": "", "compression": 1, "url_prefix": "https://autopatchcn.yuanshen.com/client_app/sophon/manifests/cxgf44wie1a8/62nfHL6ocpNF", "url_suffix": "" }, "matching_field": "en-us", "stats": { "compressed_size": "14728100079", "uncompressed_size": "15959256895", "file_count": "129", "chunk_count": "13500" }, "deduplicated_stats": { "compressed_size": "14727266762", "uncompressed_size": "15958274971", "file_count": "129", "chunk_count": "13499" } }, { "category_id": "10021", "category_name": "语音包-日文-外网", "manifest": { "id": "manifest_06dfea23fc844733_2470cf8019c95daa74fe918640414b07", "checksum": "2470cf8019c95daa74fe918640414b07", "compressed_size": "870736", "uncompressed_size": "1666416" }, "chunk_download": { "encryption": 0, "password": "", "compression": 1, "url_prefix": "https://autopatchcn.yuanshen.com/client_app/sophon/chunks/cxgf44wie1a8/62nfHL6ocpNF", "url_suffix": "" }, "manifest_download": { "encryption": 0, "password": "", "compression": 1, "url_prefix": "https://autopatchcn.yuanshen.com/client_app/sophon/manifests/cxgf44wie1a8/62nfHL6ocpNF", "url_suffix": "" }, "matching_field": "ja-jp", "stats": { "compressed_size": "14072069356", "uncompressed_size": "17912647294", "file_count": "129", "chunk_count": "14959" }, "deduplicated_stats": { "compressed_size": "14072069356", "uncompressed_size": "17912647294", "file_count": "129", "chunk_count": "14959" } }, { "category_id": "10020", "category_name": "语音包-韩文-外网", "manifest": { "id": "manifest_3da522d853512954_f3b98736693bf599efb3d14cdf8717cc", "checksum": "f3b98736693bf599efb3d14cdf8717cc", "compressed_size": "657108", "uncompressed_size": "1255144" }, "chunk_download": { "encryption": 0, "password": "", "compression": 1, "url_prefix": "https://autopatchcn.yuanshen.com/client_app/sophon/chunks/cxgf44wie1a8/62nfHL6ocpNF", "url_suffix": "" }, "manifest_download": { "encryption": 0, "password": "", "compression": 1, "url_prefix": "https://autopatchcn.yuanshen.com/client_app/sophon/manifests/cxgf44wie1a8/62nfHL6ocpNF", "url_suffix": "" }, "matching_field": "ko-kr", "stats": { "compressed_size": "11594504922", "uncompressed_size": "13434855981", "file_count": "129", "chunk_count": "11244" }, "deduplicated_stats": { "compressed_size": "11594504922", "uncompressed_size": "13434855981", "file_count": "129", "chunk_count": "11244" } } ] } } ```

在此 API 中，官服和B服的内容完全一致，B服的 PCGameSDK.dll 仍需通过以下 API 获取：

https://hk4e-launcher-static.mihoyo.com/hk4e_cn/mdk/launcher/api/resource?channel_id=14&key=KAtdSsoQ&launcher_id=17&sub_channel_id=0

资源清单

游戏资源清单和游戏资源文件均使用 Zstandard (zstd) 压缩算法，清单文件在解压后需要使用 Protocol Buffers (protobuf) 解析其内容。

这里使用 .NET protobuf-net 的形式展示其数据模型

List<GameFile> manifest = Serializer.Deserialize<List<GameFile>>(manifest_bytes);

/// <summary>
/// 游戏资源文件
/// </summary>
[ProtoContract]
public class GameFile
{
    /// <summary>
    /// 文件或文件夹相对于游戏根目录的路径
    /// </summary>
    [ProtoMember(1)]
    public string File { get; set; }

    [ProtoMember(2)]
    public List<Chunk> Chunks { get; set; }

    [ProtoMember(3)]
    public bool IsFolder { get; set; }

    [ProtoMember(4)]
    public long Size { get; set; }

    [ProtoMember(5)]
    public string MD5 { get; set; }
}

/// <summary>
/// 资源文件块
/// </summary>
[ProtoContract]
public class Chunk
{
    /// <summary>
    /// 文件块的下载链接后缀
    /// </summary>
    [ProtoMember(1)]
    public string UrlSuffix { get; set; }

    /// <summary>
    /// 文件块解压后的 MD5 值
    /// </summary>
    [ProtoMember(2)]
    public string MD5 { get; set; }

    /// <summary>
    /// 文件块解压后，相对于文件头的偏移量
    /// </summary>
    [ProtoMember(3)]
    public long Offset { get; set; }

    /// <summary>
    /// 解压前的大小
    /// </summary>
    [ProtoMember(4)]
    public long CompressedSize { get; set; }

    /// <summary>
    /// 解压后的块大小
    /// </summary>
    [ProtoMember(5)]
    public long Size { get; set; }

    /// <summary>
    /// 未知哈希算法
    /// </summary>
    [ProtoMember(6)]
    public ulong Unknown { get; set; }
}

资源文件示例

{
    "File": "YuanShen_Data/StreamingAssets/AssetBundles/blocks/00/05130550.blk",
    "Chunks": [
        {
            "Suffix": "fc9327bfb71c7a8a_5a8d03f4c687b6808304166e9babde95",
            "MD5": "5a8d03f4c687b6808304166e9babde95",
            "Offset": 0,
            "CompressedSize": 1754811,
            "Size": 1754756,
            "Unknown": 15817881018744700928
        },
        {
            "Suffix": "88ea771167162221_ff7e21a86b098f3ee055c08e1d2b44a4",
            "MD5": "ff7e21a86b098f3ee055c08e1d2b44a4",
            "Offset": 1754756,
            "CompressedSize": 1208916,
            "Size": 1208873,
            "Unknown": 1333711227053146112
        },
        {
            "Suffix": "b9b9aaf28252291b_8c01b876372f43bd50694236b352bd3c",
            "MD5": "8c01b876372f43bd50694236b352bd3c",
            "Offset": 2963629,
            "CompressedSize": 83939,
            "Size": 83923,
            "Unknown": 0
        }
    ],
    "IsFolder": false,
    "Size": 3047552,
    "MD5": "e89988407ded973318bf8298976bfcf8"
}

下载步骤

首先下载游戏资源文件块到根目录的 chunk 文件夹中，此时文件块还未被解压，文件夹中也不存在子文件夹。但是，下载后的文件块名称并不和 MD5 值相同，命名方式我没有分析出来。

接下来解压对应的文件块，在 staging 文件夹中组合成一个完整的游戏资源文件，此文件夹中存在着和游戏根目录相同的子文件夹结构。组合完成后再移动资源文件到正确的路径。

而在预下载过程中，对比新旧两个版本的资源清单，仅下载有差异的部分。

总结

相比于老启动器的 File 下载模式，新的 Chunk 下载模式解决了需要预留两倍存储空间的问题，下载过程中也能更好地利用多线程提高下载速度。但是版本更新过程中音频资源文件不再使用 HDiffPatch，下载的数据量相比之前有所增加。

bangbang23333 commented 7 months ago

encryption 和 password 参数应该是给测试服包体下载用的，测试服启动器是要登录账号的

prpjzz commented 7 months ago

I have two questions like this:

  1. Can we apply this to Starward?
  2. What's the old game compression algorithm for blk files?
Lightczx commented 7 months ago

我来构成交叉引用 https://blog.amarea.cn/archives/genshin-launcher-chunk-download-mode.html

Lightczx commented 7 months ago

chunk.db

-- auto-generated definition
create table depot_file_data
(
    file_main_key LONGVARCHAR
        primary key,
    package_id    LONGVARCHAR,
    build_id      LONGVARCHAR,
    depot_id      LONGVARCHAR,
    file_id       LONGVARCHAR,
    file_ver      LONGVARCHAR,
    file_status   INTEGER default 0,
    install_dir   LONGVARCHAR
);

chunk_config.db

-- auto-generated definition
create table config
(
    package_id_branch_depot_id LONGVARCHAR
        primary key,
    local_version              LONGVARCHAR,
    server_version             LONGVARCHAR,
    local_build_id             LONGVARCHAR,
    server_build_id            LONGVARCHAR,
    package_id                 LONGVARCHAR,
    branch                     LONGVARCHAR,
    depot_id                   LONGVARCHAR,
    matching_field             LONGVARCHAR,
    install_dir                LONGVARCHAR
);

chunk_manifest.db

-- auto-generated definition
create table depot_manifest
(
    package_id_build_id_depot_id     LONGVARCHAR
        primary key,
    package_id                       LONGVARCHAR,
    build_id                         LONGVARCHAR,
    depot_id                         LONGVARCHAR,
    version                          LONGVARCHAR,
    depot_manifest_id                LONGVARCHAR,
    depot_manifest_checksum          LONGVARCHAR,
    depot_manifest_compressed_size   INTEGER default 0,
    depot_manifest_uncompressed_size INTEGER default 0,
    depot_manifest_url_prefix        LONGVARCHAR,
    depot_manifest_encryption        INTEGER default 0,
    depot_manifest_password          LONGVARCHAR,
    depot_manifest_compression       INTEGER default 0,
    depot_manifest_md5               LONGVARCHAR
);