brave / brave-browser

Brave browser for Android, iOS, Linux, macOS, Windows.
https://brave.com
Mozilla Public License 2.0
17.67k stars 2.31k forks source link

Support delta installer for updates on Windows #9565

Closed simonhong closed 2 years ago

simonhong commented 4 years ago

Let's start investigation about delta update with below archive type. Maybe INCREMENTAL_ARCHIVE_TYPE is used for delta installer.

enum ArchiveType {
  UNKNOWN_ARCHIVE_TYPE,     // Unknown or uninitialized.
  FULL_ARCHIVE_TYPE,        // Full chrome.7z archive.
  INCREMENTAL_ARCHIVE_TYPE  // Incremental or differential archive.
};

Currently, mini_installer target uses courgette algorithm for generating delta installer. Courgette: https://www.chromium.org/developers/design-documents/software-updates-courgette

According to the below code in chrome/installer/setup/setup_main.cc, maybe setup.exe generates output file(newer version file) by using patch file(new delta file?) and input file(current version). and output file would be a complete file for update to next version.

 } else if (cmd_line.HasSwitch(installer::switches::kPatch)) {
    const std::string patch_type_str(
        cmd_line.GetSwitchValueASCII(installer::switches::kPatch));
    const base::FilePath input_file(
        cmd_line.GetSwitchValuePath(installer::switches::kInputFile));
    const base::FilePath patch_file(
        cmd_line.GetSwitchValuePath(installer::switches::kPatchFile));
    const base::FilePath output_file(
        cmd_line.GetSwitchValuePath(installer::switches::kOutputFile));

    if (patch_type_str == installer::kCourgette) {
      *exit_code = installer::CourgettePatchFiles(input_file,
                                                  patch_file,
                                                  output_file);
    } else if (patch_type_str == installer::kBsdiff) {
      *exit_code = installer::BsdiffPatchFiles(input_file,
                                               patch_file,
                                               output_file);
#if BUILDFLAG(ZUCCHINI)
    } else if (patch_type_str == installer::kZucchini) {
      *exit_code =
          installer::ZucchiniPatchFiles(input_file, patch_file, output_file);
#endif  // BUILDFLAG(ZUCCHINI)
    } else {
      *exit_code = installer::PATCH_INVALID_ARGUMENTS;
    }
  }

create_installer_archive.py generates patch file by mini_installer target. Uncompressed chrome.7z file is used to get diff.

template("generate_mini_installer")
simonhong commented 4 years ago

Got strange error while generating delta installer. Previous version's chrome.7z extracting is failed. but extracted properly when I do it in console. Curious why it think as an overwrite??? - Before running GetPrevVersion, this script delete previous temp dir. So, it should not be an overwrite when extracting.

=> Found the reason when below temp_dir is passed with -o'"temp_dir"', temp_dir is generated under C:/. However, temp_dir is generated in cwd when it's passed with -o'temp_dir'. Because of this, it warns about overwriting. Normally, that temp_dir becomes empty because MakeStagingDirectory deletes it before extracting it into temp_diir.

  cmd = [lzma_exec,
         'x',
         '-o%s' % temp_dir,
         prev_archive_file,
         'Chrome-bin/*/chrome.dll',]
[1/279] ACTION //components/resources:about_credits(//build/toolchain/win:win_clang_x64)
[2/9] ACTION //chrome/installer/mini_installer:mini_installer_archive(//build/toolchain/win:win_clang_x64)
FAILED: chrome.7z setup.ex_ gen/chrome/installer/mini_installer/mini_installer/packed_files.rc chrome.packed.7z
C:/Projects/brave/brave-browser/vendor/depot_tools/bootstrap-3_8_0_chromium_8_bin/python/bin/python.exe ../../chrome/tools/build/win/create_installer_archive.py --build_dir . --staging_dir gen/chrome/installer/mini_installer/mini_installer --input_file ../../chrome/installer/mini_installer/chrome.release --resource_file_path gen/chrome/installer/mini_installer/mini_installer/packed_files.rc --target_arch=x64 --distribution=_brave --output_dir . --chrome_runtime_deps gen/chrome_component.runtime_deps --setup_runtime_deps gen/setup.runtime_deps --last_chrome_installer=C:/build/installer --setup_exe_format=DIFF --diff_algorithm=COURGETTE --verbose --skip_signing --enable_hidpi=1 --depfile gen/chrome/installer/mini_installer/archive.d
['../../chrome/tools/build/win/create_installer_archive.py', '--build_dir', '.', '--staging_dir', 'gen/chrome/installer/mini_installer/mini_installer', '--input_file', '../../chrome/installer/mini_installer/chrome.release', '--resource_file_path', 'gen/chrome/installer/mini_installer/mini_installer/packed_files.rc', '--target_arch=x64', '--distribution=_brave', '--output_dir', '.', '--chrome_runtime_deps', 'gen/chrome_component.runtime_deps', '--setup_runtime_deps', 'gen/setup.runtime_deps', '--last_chrome_installer=C:/build/installer', '--setup_exe_format=DIFF', '--diff_algorithm=COURGETTE', '--verbose', '--skip_signing', '--enable_hidpi=1', '--depfile', 'gen/chrome/installer/mini_installer/archive.d']
C:/build/installer\chrome.7z
gen/chrome/installer/mini_installer/mini_installer\temp_installer_archive
Running .\..\..\third_party\lzma_sdk\Executable\7za.exe x -o"gen/chrome/installer/mini_installer/mini_installer\temp_installer_archive" C:/build/installer\chrome.7z Chrome-bin\*\chrome.dll
Traceback (most recent call last):
  File "../../chrome/tools/build/win/create_installer_archive.py", line 665, in <module>
    sys.exit(main(options))
  File "../../chrome/tools/build/win/create_installer_archive.py", line 538, in main
    options.output_name)
  File "../../chrome/tools/build/win/create_installer_archive.py", line 188, in GetPrevVersion
    RunSystemCommand(cmd, options.verbose)
  File "../../chrome/tools/build/win/create_installer_archive.py", line 232, in RunSystemCommand
    (e.cmd, e.returncode, e.output))
Exception: Error while running cmd: ['.\\..\\..\\third_party\\lzma_sdk\\Executable\\7za.exe', 'x', '-o"gen/chrome/installer/mini_installer/mini_installer\\temp_installer_archive"', 'C:/build/installer\\chrome.7z', 'Chrome-bin\\*\\chrome.dll']
Exit code: 2
Command output:

7-Zip (a) [64] 15.14 : Copyright (c) 1999-2015 Igor Pavlov : 2015-12-31

Scanning the drive for archives:
1 file, 220406223 bytes (211 MiB)

Extracting archive: C:\build\installer\chrome.7z
--
Path = C:\build\installer\chrome.7z
Type = 7z
Physical Size = 220406223
Headers Size = 2965
Method = Copy
Solid = -
Blocks = 187

Would you like to replace the existing file:
  Path:     \gen/chrome/installer/mini_installer/mini_installer\temp_installer_archive\Chrome-bin\81.1.8.90\chrome.dll
  Size:     134749328 bytes (129 MiB)
  Modified: 2020-05-05 11:27:19
with the file from archive:
  Path:     Chrome-bin\81.1.8.90\chrome.dll
  Size:     134749328 bytes (129 MiB)
  Modified: 2020-05-05 11:27:19
? (Y)es / (N)o / (A)lways / (S)kip all / A(u)to rename all / (Q)uit?
ERROR: Can't allocate required memory!

Archives with Errors: 1

ninja: build stopped: subcommand failed.
null
null
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! brave@1.10.23 create_dist: `node ./scripts/commands.js create_dist "Release" "--skip_signing"`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the brave@1.10.23 create_dist script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\simon\AppData\Roaming\npm-cache\_logs\2020-05-07T02_20_15_743Z-debug.log
mihaiplesa commented 4 years ago

Getting the following:

13:33:37  The variable "build_delta_installer" was set as a build argument
13:33:37  but never appeared in a declare_args() block in any buildfile.
13:33:37  
13:33:37  To view all possible args, run "gn args --list <out_dir>"
13:33:37  
13:33:37  The build continued as if that argument was unspecified.
simonhong commented 4 years ago

@mihaiplesa Did you build with both PRs (b-b and b-c) together? build_delta_installer is declared in https://github.com/brave/brave-core/pull/5484/files#diff-a1d6e4460dc2ec6f82e6a22b56c1bbd8R37

simonhong commented 4 years ago

Set release-notes/exclude label because server-side stuff also should be ready for delta-installer. This issue is only for client side support for delta installer.

mihaiplesa commented 4 years ago

Reopening since this should include Windows x86 which still fails as above.

mihaiplesa commented 2 years ago

Fresh log from a Windows x86 build with delta error below.

0.020614s\r\n[0117/161749.066:VERBOSE1:ensemble_create.cc(208)] Matched old(1,256182034,784896) to new(1,422865086,784896) --> 32\r\n[0117/161749.066:VERBOSE1:ensemble_create.cc(218)] done FindGenerators found 16 in 49.3365s\r\n[0117/161751.924:VERBOSE1:simple_delta.cc(35)] GenerateSimpleDelta 31 31\r\n[0117/161751.924:VERBOSE1:bsdiff_create.cc(96)] Start bsdiff\r\n[0117/161751.924:VERBOSE1:bsdiff_create.cc(123)] done divsufsort 0.000336\r\n[0117/161751.924:VERBOSE1:bsdiff_create.cc(338)] Control tuples: 1 copy bytes: 31 mistakes: 0 (skips: 1) extra bytes: 0\r\nUncompressed bsdiff patch size 34\r\nEnd bsdiff 0.000409\r\n[0117/161806.146:VERBOSE1:adjustment_method_2.cc(1238)] Adjuster::Adjust\r\n[0117/162037.342:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 148.357\r\n[0117/162751.365:VERBOSE1:adjustment_method_2.cc(1279)] Adjuster::Solve 432.799\r\n[0117/162754.396:VERBOSE1:label_manager.cc(34)] 667504 of 686005 labels pre-assigned.\r\n[0117/162754.397:VERBOSE1:label_manager.cc(55)] fill forward 18501\r\n[0117/162754.397:VERBOSE1:label_manager.cc(77)] fill backward 0\r\n[0117/162754.398:VERBOSE1:label_manager.cc(93)] infill 0\r\n[0117/162754.403:VERBOSE1:label_manager.cc(34)] 1517676 of 1543242 labels pre-assigned.\r\n[0117/162754.405:VERBOSE1:label_manager.cc(55)] fill forward 25566\r\n[0117/162754.406:VERBOSE1:label_manager.cc(77)] fill backward 0\r\n[0117/162754.408:VERBOSE1:label_manager.cc(93)] infill 0\r\n[0117/162759.792:ERROR:patch_generator_x86_32.h(92)] Cannot serialize encoded 'new' input. (old(1,18118014,166675456) => new(1,32,166675456))\r\n-gen failed.\r\n"

mihaiplesa commented 2 years ago

Apparently Windows x86 deltas got fixed by migrating to zucchini in https://github.com/brave/brave-core/pull/12098

GeetaSarvadnya commented 2 years ago

Verification PASSED on

Brave | 1.35.102 Chromium: 98.0.4758.102 (Official Build) (64-bit)
-- | --
Revision | 273bf7ac8c909cde36982d27f66f3c70846a3718-refs/branch-heads/4758@{#1151}
OS | Windows 10 Version 21H2 (Build 19044.1526)

x64 bit stub installers

Case 1: Release channel_x64bit_ Admin Privileges Yes_PASSED - Install stub installer `BraveBrowserSetup.exe` - Confirm latest release version `1.35.101` is pulled from the Omaha server - Upgrade the profile to test channel stub installer `BraveBrowserSetup-64-r-test.exe` - Confirm able to upgrade the profile to `1.35.102` Stub installer | Test channel -------|------- ![image](https://user-images.githubusercontent.com/38657976/154296093-dd627455-ed8d-4bd9-8c4c-558626ead4ad.png) | ![image](https://user-images.githubusercontent.com/38657976/154296165-1b22e7f8-ee37-4bed-b03a-d0123cfc0e9d.png)
Case 2: Release channel_x64bit_ Admin Privileges No * ensured that `BraveBrowserSetup-64-r-test.exe` downloads/installs `1.35.102 Chromium: 98.0.4758.102` without any issues * ensured that upgrading from `1.35.101 Chromium: 98.0.4758.87` -> `1.35.102 Chromium: 98.0.4758.102` using `BraveBrowserSetup-64-r-test.exe` worked without any issues Stub installer | Test channel -------|------- ![1](https://user-images.githubusercontent.com/2602313/154353039-a898ab35-5383-434e-8ab0-4d0bb1e3ee19.png) | ![2](https://user-images.githubusercontent.com/2602313/154353051-f5dcf2d5-9e3f-4063-9890-d08a1755692e.png)
Case 3: Beta channel_x64bit_ Admin Privileges Yes_PASSED - Install brave beta standalone installer `1.36.90` `BraveBrowserStandaloneBetaSetup.exe` - Upgrade the profile to test channel `BraveBrowserBetaSetup-64-be-test.exe` - Confirm that the beta profile is upgraded to test channel `1.36.96` Stub installer | Test channel -----|---- ![image](https://user-images.githubusercontent.com/38657976/154345507-227884b4-9a34-4ab4-a816-2a5cc5895736.png) | ![image](https://user-images.githubusercontent.com/38657976/154345570-ccd3d764-9a09-4e5d-8d2a-8b130f4b8a93.png)
Case 4: Beta channel_x64bit_ Admin Privileges NO * ensured that `BraveBrowserBetaSetup-64-be-test.exe` downloads/installs `1.36.96 Chromium: 98.0.4758.87` without any issues * ensured that upgrading from `1.36.94 Chromium: 98.0.4758.87` -> `1.36.96 Chromium: 98.0.4758.87` using `BraveBrowserBetaSetup-64-be-test.exe` worked without any issues Stub installer | Test channel -----|---- ![1](https://user-images.githubusercontent.com/2602313/154354891-86d4abcf-8591-447d-8514-6f518efa50e3.png) | ![2](https://user-images.githubusercontent.com/2602313/154354910-ba2a77ff-3566-4989-91f9-b26b980c8fe8.png)
Case 5: Nightly channel_x64bit_ Admin Privileges Yes_PASSED - Instal the nightly version `1.37.34` - `BraveBrowserStandaloneNightlySetup.exe` - Upgrade the profile to test channel `BraveBrowserBetaSetup-64-ni-test.exe` - Confirm that the nightly profile is upgraded to test channel `1.37.53` Stub installer | Test channel -----|----- ![image](https://user-images.githubusercontent.com/38657976/154346327-bf6497f3-6b0f-46fc-97f3-3e590174d1ee.png) | ![image](https://user-images.githubusercontent.com/38657976/154346389-2717260b-4650-4f1f-a17b-68cfa51ef39e.png)
Case 6: Nightly channel_x64bit_ Admin Privileges NO * ensured that `BraveBrowserNightlySetup-64-ni-test.exe` downloads/installs `1.37.53 Chromium: 98.0.4758.102` without any issues * ensured that upgrading from `1.37.52 Chromium: 98.0.4758.102` -> `1.37.53 Chromium: 98.0.4758.102` using `BraveBrowserNightlySetup-64-ni-test.exe` worked without any issues Stub installer | Test channel -----|----- ![1](https://user-images.githubusercontent.com/2602313/154361720-92316d84-f357-479f-b41f-2a696dbe3c29.png) | ![2](https://user-images.githubusercontent.com/2602313/154361735-b4137130-aea2-436c-afec-531ee8f8648f.png)

x32 bit stub installers

Case 1: Release channel_x32bit_ Admin Privileges Yes_PASSED - Install stub installer `BraveBrowserSetup32.exe` - Confirm latest release version `1.35.101` is pulled from the Omaha server - Upgrade the profile to test channel stub installer BraveBrowserSetup32-86-r-test` - Confirm able to upgrade the profile to test channel `1.35.102` Stub installer | Test channel -------|------- ![image](https://user-images.githubusercontent.com/38657976/154346894-f3c05877-b7c8-402a-868b-2411ae5eb2a6.png) | ![image](https://user-images.githubusercontent.com/38657976/154346943-9a924393-25a5-4147-89e1-1ec658ff9780.png)
Case 2: Release channel_x32bit_ Admin Privileges No * ensured that `BraveBrowserSetup32-86-r-test.exe` downloads/installs `1.35.102 Chromium: 98.0.4758.102` without any issues * ensured that upgrading from `1.35.101 Chromium: 98.0.4758.87` -> `1.35.102 Chromium: 98.0.4758.102` using `BraveBrowserSetup-64-r-test.exe` worked without any issues Stub installer | Test channel -------|------- ![image](https://user-images.githubusercontent.com/2602313/154363220-d42648d6-055c-4a59-8801-d5eeb93ae833.png) | ![image](https://user-images.githubusercontent.com/2602313/154363373-16456091-0b13-425f-bf7b-9a94f0cfb69e.png)
Case 3: Beta channel_x32bit_ Admin Privileges Yes_PASSED - Install brave beta `1.36.90` `BraveBrowserStandaloneBetaSetup32.exe` - Upgrade the profile to test channel `BraveBrowserBetaSetup32-86-be-test` - Confirm profile is upgraded to test channel `1.36.94` Stub installer | Test channel -----|------ ![image](https://user-images.githubusercontent.com/38657976/154348300-34189e7c-265f-462d-bca2-d66d4da1b98e.png) | ![image](https://user-images.githubusercontent.com/38657976/154348667-0e5ecd64-9de4-45ec-95cc-76e3e81e65a1.png)
Case 4: Beta channel_x32bit_ Admin Privileges NO * ensured that `BraveBrowserBetaSetup32-86-be-test.exe` downloads/installs `1.36.94 Chromium: 98.0.4758.87` without any issues * ensured that upgrading from `1.36.90 Chromium: 98.0.4758.87` -> `1.36.94 Chromium: 98.0.4758.87` using `BraveBrowserBetaSetup32-86-be-test.exe` worked without any issues Stub installer | Test channel -----|------ ![image](https://user-images.githubusercontent.com/2602313/154367217-f12367d6-def1-4cbd-b93d-3b3537da1f94.png) | ![image](https://user-images.githubusercontent.com/2602313/154367293-094dbbe7-e72b-4115-9d62-1659d04232f4.png)
Case 5: Nightly channel_x32bit_ Admin Privileges Yes_PASSED - Install nightly build `1.37.33` `BraveBrowserStandaloneNightlySetup32.exe` - Upgrade the profile to test channel `BraveBrowserNightlySetup32-86-ni-test` - Confirmed that profile is upgraded to test channel `1.37.53` Stub installer | Test channel -----|----- ![image](https://user-images.githubusercontent.com/38657976/154349913-79e0229e-47a0-4c48-ab86-fbe4d69b27b9.png) | ![image](https://user-images.githubusercontent.com/38657976/154349970-baf2f159-4ea8-4ed3-968c-0816f05aa8e9.png)
Case 6: Nightly channel_32bit_ Admin Privileges NO * ensured that `BraveBrowserNightlySetup32-86-ni-test.exe` downloads/installs `1.37.53 Chromium: 98.0.4758.102` without any issues * ensured that upgrading from `1.37.50 Chromium: 98.0.4758.87` -> `1.37.53 Chromium: 98.0.4758.102` using `BraveBrowserNightlySetup32-86-ni-test.exe` worked without any issues Stub installer | Test channel -----|------ ![image](https://user-images.githubusercontent.com/2602313/154368001-d2d0e62e-7eb9-4eb7-b23a-863c787c5875.png) | ![image](https://user-images.githubusercontent.com/2602313/154368012-fe91fb38-fa45-4706-b6ff-2c656d6fb58a.png)
GeetaSarvadnya commented 2 years ago

Covered the DELTA installer verification via https://bravesoftware.slack.com/archives/C0YL5KMA8/p1645118243641749?thread_ts=1645054477.458409&cid=C0YL5KMA8

Nightly image (11)

Beta image (13)

Release image (12)

GeetaSarvadnya commented 2 years ago

Enabled logs IsEnabledLogToFile via Regedit key and ensured logs file is showing up the delta installer file brave_installer-delta-x64.exe and upgrade is happening via delta installer, verification is done in the slack thread below: https://bravesoftware.slack.com/archives/GAA4017R7/p1645173242350309?thread_ts=1645031809.437009&cid=GAA4017R7 image