ESMification

[Timvde]

I want to inform you about the ESMification that was recently completed in Firefox: the code was migrated away from Mozilla's JSM modules to ES6 modules.

In about a year (Firefox 136, planned for March 2025), Firefox will completely remove support for ChromeUtils.import and similar APIs.

All information about migrating to ES6 modules can be found in this migration document: https://docs.google.com/document/d/14FqYX749nJkCSL_GknCDZyQnuqkXNc9KoTuXSV3HtMg/edit

Some more information (the documentation about the migration in the Firefox codebase itself) is here: https://docs.google.com/document/d/1cpzIK-BdP7u6RJSar-Z955GV--2Rj8V4x2vl34m36Go/edit

[onemen]

I am using ChromeUtils.importESModule and ChromeUtils.defineESModuleGetters for Fireofx files that ends with .sys.mjs

see https://github.com/onemen/TabMixPlus/blob/main/addon/modules/ChromeUtils.jsm

Do you see any problem with Tab Mix Plus?

[117649]

I am using ChromeUtils.importESModule and ChromeUtils.defineESModuleGetters for Fireofx files that ends with .sys.mjs

see https://github.com/onemen/TabMixPlus/blob/main/addon/modules/ChromeUtils.jsm

Do you see any problem with Tab Mix Plus?

@onemen Well I know something:https://github.com/onemen/TabMixPlus/blob/c4c1c2afd48bf0e88cce51ceb6b583d4deddf888/addon/modules/TabmixSvc.jsm#L425-L428

And it is very bad.

[Timvde]

Do you see any problem with Tab Mix Plus?

No, sorry for being unclear about that :) I just did a search on the code and noticed that some of the affected APIs were still used. I filed this issue only to inform you ahead of time in case you weren't aware. If this is not applicable to TMP because you are already using ES6 modules everywhere and the JSM modules are only there for compatibility with older versions, that's great :D

[onemen]

Thank you

[117649]

@onemen Just realize this will cause bootstraploader script stop working. I've reviewed loader script again I think we can have a solution by set legacy addon's sign state to 'not required' and freeze appDisable to false during install or we can just set it to 'privileged'.

Plus do we have a way to changeCode() an ES6 module obj?

[onemen]

We will have to change all Tab Mix Plus and Firefox scripts jsm files to be es6 modules

[117649]

@onemen

We will have to change all Tab Mix Plus and Firefox scripts jsm files to be es6 modules

First attempt of BootstrapLoader.jsm .

Left some that are not yet es6 un touched.

Have freeze 'appDisabled' and 'signedState' on addon internal.

/* This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, You can obtain one at http://mozilla.org/MPL/2.0/. */

'use strict';

let EXPORTED_SYMBOLS = [];

const { XPCOMUtils } = ChromeUtils.importESModule('resource://gre/modules/XPCOMUtils.sys.mjs');
const Services = globalThis.Services;

XPCOMUtils.defineLazyModuleGetters(this, {
  Blocklist: 'resource://gre/modules/Blocklist.jsm',
  ConsoleAPI: 'resource://gre/modules/Console.jsm',
  InstallRDF: 'chrome://userchromejs/content/RDFManifestConverter.jsm',
});

Services.obs.addObserver(doc => {
  if (doc.location.protocol + doc.location.pathname === 'about:addons' ||
      doc.location.protocol + doc.location.pathname === 'chrome:/content/extensions/aboutaddons.html') {
    const win = doc.defaultView;
    let handleEvent_orig = win.customElements.get('addon-card').prototype.handleEvent;
    win.customElements.get('addon-card').prototype.handleEvent = function (e) {
      if (e.type === 'click' &&
          e.target.getAttribute('action') === 'preferences' &&
          this.addon.__AddonInternal__.optionsType == 1/*AddonManager.OPTIONS_TYPE_DIALOG*/ && !!this.addon.optionsURL) {
        var windows = Services.wm.getEnumerator(null);
        while (windows.hasMoreElements()) {
          var win2 = windows.getNext();
          if (win2.closed) {
            continue;
          }
          if (win2.document.documentURI == this.addon.optionsURL) {
            win2.focus();
            return;
          }
        }
        var features = 'chrome,titlebar,toolbar,centerscreen';
        win.docShell.rootTreeItem.domWindow.openDialog(this.addon.optionsURL, this.addon.id, features);
      } else {
        handleEvent_orig.apply(this, arguments);
      }
    }
    let update_orig = win.customElements.get('addon-options').prototype.update;
    win.customElements.get('addon-options').prototype.update = function (card, addon) {
      update_orig.apply(this, arguments);
      if (addon.__AddonInternal__.optionsType == 1/*AddonManager.OPTIONS_TYPE_DIALOG*/ && !!addon.optionsURL)
        this.querySelector('panel-item[data-l10n-id="preferences-addon-button"]').hidden = false;
    }
  }
}, 'chrome-document-loaded');

const {AddonManager} = ChromeUtils.importESModule('resource://gre/modules/AddonManager.sys.mjs');
const {XPIDatabase, AddonInternal} = ChromeUtils.importESModule('resource://gre/modules/addons/XPIDatabase.sys.mjs');

// const { defineAddonWrapperProperty } = Cu.import('resource://gre/modules/addons/XPIDatabase.jsm');
// defineAddonWrapperProperty('optionsType', function optionsType() {
//   if (!this.isActive) {
//     return null;
//   }

//   let addon = this.__AddonInternal__;
//   let hasOptionsURL = !!this.optionsURL;

//   if (addon.optionsType) {
//     switch (parseInt(addon.optionsType, 10)) {
//       case 1/*AddonManager.OPTIONS_TYPE_DIALOG*/:
//       case AddonManager.OPTIONS_TYPE_TAB:
//       case AddonManager.OPTIONS_TYPE_INLINE_BROWSER:
//         return hasOptionsURL ? addon.optionsType : null;
//     }
//     return null;
//   }

//   return null;
// });

XPIDatabase.isDisabledLegacy = () => false;

ChromeUtils.defineLazyGetter(this, 'BOOTSTRAP_REASONS', () => {
  const {XPIProvider} = ChromeUtils.importESModule('resource://gre/modules/addons/XPIProvider.sys.mjs');
  return XPIProvider.BOOTSTRAP_REASONS;
});

const {Log} = ChromeUtils.importESModule('resource://gre/modules/Log.sys.mjs');
var logger = Log.repository.getLogger('addons.bootstrap');

/**
 * Valid IDs fit this pattern.
 */
var gIDTest = /^(\{[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\}|[a-z0-9-\._]*\@[a-z0-9-\._]+)$/i;

// Properties that exist in the install manifest
const PROP_METADATA      = ['id', 'version', 'type', 'internalName', 'updateURL',
                            'optionsURL', 'optionsType', 'aboutURL', 'iconURL'];
const PROP_LOCALE_SINGLE = ['name', 'description', 'creator', 'homepageURL'];
const PROP_LOCALE_MULTI  = ['developers', 'translators', 'contributors'];

// Map new string type identifiers to old style nsIUpdateItem types.
// Retired values:
// 32 = multipackage xpi file
// 8 = locale
// 256 = apiextension
// 128 = experiment
// theme = 4
const TYPES = {
  extension: 2,
  dictionary: 64,
};

const COMPATIBLE_BY_DEFAULT_TYPES = {
  extension: true,
  dictionary: true,
};

const hasOwnProperty = Function.call.bind(Object.prototype.hasOwnProperty);

function isXPI(filename) {
  let ext = filename.slice(-4).toLowerCase();
  return ext === '.xpi' || ext === '.zip';
}

/**
 * Gets an nsIURI for a file within another file, either a directory or an XPI
 * file. If aFile is a directory then this will return a file: URI, if it is an
 * XPI file then it will return a jar: URI.
 *
 * @param {nsIFile} aFile
 *        The file containing the resources, must be either a directory or an
 *        XPI file
 * @param {string} aPath
 *        The path to find the resource at, '/' separated. If aPath is empty
 *        then the uri to the root of the contained files will be returned
 * @returns {nsIURI}
 *        An nsIURI pointing at the resource
 */
function getURIForResourceInFile(aFile, aPath) {
  if (!isXPI(aFile.leafName)) {
    let resource = aFile.clone();
    if (aPath)
      aPath.split('/').forEach(part => resource.append(part));

    return Services.io.newFileURI(resource);
  }

  return buildJarURI(aFile, aPath);
}

/**
 * Creates a jar: URI for a file inside a ZIP file.
 *
 * @param {nsIFile} aJarfile
 *        The ZIP file as an nsIFile
 * @param {string} aPath
 *        The path inside the ZIP file
 * @returns {nsIURI}
 *        An nsIURI for the file
 */
function buildJarURI(aJarfile, aPath) {
  let uri = Services.io.newFileURI(aJarfile);
  uri = 'jar:' + uri.spec + '!/' + aPath;
  return Services.io.newURI(uri);
}

var BootstrapLoader = {
  name: 'bootstrap',
  manifestFile: 'install.rdf',
  async loadManifest(pkg) {
    /**
     * Reads locale properties from either the main install manifest root or
     * an em:localized section in the install manifest.
     *
     * @param {Object} aSource
     *        The resource to read the properties from.
     * @param {boolean} isDefault
     *        True if the locale is to be read from the main install manifest
     *        root
     * @param {string[]} aSeenLocales
     *        An array of locale names already seen for this install manifest.
     *        Any locale names seen as a part of this function will be added to
     *        this array
     * @returns {Object}
     *        an object containing the locale properties
     */
    function readLocale(aSource, isDefault, aSeenLocales) {
      let locale = {};
      if (!isDefault) {
        locale.locales = [];
        for (let localeName of aSource.locales || []) {
          if (!localeName) {
            logger.warn('Ignoring empty locale in localized properties');
            continue;
          }
          if (aSeenLocales.includes(localeName)) {
            logger.warn('Ignoring duplicate locale in localized properties');
            continue;
          }
          aSeenLocales.push(localeName);
          locale.locales.push(localeName);
        }

        if (locale.locales.length == 0) {
          logger.warn('Ignoring localized properties with no listed locales');
          return null;
        }
      }

      for (let prop of [...PROP_LOCALE_SINGLE, ...PROP_LOCALE_MULTI]) {
        if (hasOwnProperty(aSource, prop)) {
          locale[prop] = aSource[prop];
        }
      }

      return locale;
    }

    let manifestData = await pkg.readString('install.rdf');
    let manifest = InstallRDF.loadFromString(manifestData).decode();

    let addon = new AddonInternal();
    for (let prop of PROP_METADATA) {
      if (hasOwnProperty(manifest, prop)) {
        addon[prop] = manifest[prop];
      }
    }

    if (!addon.type) {
      addon.type = 'extension';
    } else {
      let type = addon.type;
      addon.type = null;
      for (let name in TYPES) {
        if (TYPES[name] == type) {
          addon.type = name;
          break;
        }
      }
    }

    if (!(addon.type in TYPES))
      throw new Error('Install manifest specifies unknown type: ' + addon.type);

    if (!addon.id)
      throw new Error('No ID in install manifest');
    if (!gIDTest.test(addon.id))
      throw new Error('Illegal add-on ID ' + addon.id);
    if (!addon.version)
      throw new Error('No version in install manifest');

    addon.strictCompatibility = (!(addon.type in COMPATIBLE_BY_DEFAULT_TYPES) ||
                                 manifest.strictCompatibility == 'true');

    // Only read these properties for extensions.
    if (addon.type == 'extension') {
      if (manifest.bootstrap != 'true') {
        throw new Error('Non-restartless extensions no longer supported');
      }

      if (addon.optionsType &&
          addon.optionsType != 1/*AddonManager.OPTIONS_TYPE_DIALOG*/ &&
          addon.optionsType != AddonManager.OPTIONS_TYPE_INLINE_BROWSER &&
          addon.optionsType != AddonManager.OPTIONS_TYPE_TAB) {
            throw new Error('Install manifest specifies unknown optionsType: ' + addon.optionsType);
      }

      if (addon.optionsType)
        addon.optionsType = parseInt(addon.optionsType);
    }

    addon.defaultLocale = readLocale(manifest, true);

    let seenLocales = [];
    addon.locales = [];
    for (let localeData of manifest.localized || []) {
      let locale = readLocale(localeData, false, seenLocales);
      if (locale)
        addon.locales.push(locale);
    }

    let dependencies = new Set(manifest.dependencies);
    addon.dependencies = Object.freeze(Array.from(dependencies));

    let seenApplications = [];
    addon.targetApplications = [];
    for (let targetApp of manifest.targetApplications || []) {
      if (!targetApp.id || !targetApp.minVersion ||
          !targetApp.maxVersion) {
            logger.warn('Ignoring invalid targetApplication entry in install manifest');
            continue;
      }
      if (seenApplications.includes(targetApp.id)) {
        logger.warn('Ignoring duplicate targetApplication entry for ' + targetApp.id +
                    ' in install manifest');
        continue;
      }
      seenApplications.push(targetApp.id);
      addon.targetApplications.push(targetApp);
    }

    // Note that we don't need to check for duplicate targetPlatform entries since
    // the RDF service coalesces them for us.
    addon.targetPlatforms = [];
    for (let targetPlatform of manifest.targetPlatforms || []) {
      let platform = {
        os: null,
        abi: null,
      };

      let pos = targetPlatform.indexOf('_');
      if (pos != -1) {
        platform.os = targetPlatform.substring(0, pos);
        platform.abi = targetPlatform.substring(pos + 1);
      } else {
        platform.os = targetPlatform;
      }

      addon.targetPlatforms.push(platform);
    }

    addon.userDisabled = false;
    addon.softDisabled = addon.blocklistState == Blocklist.STATE_SOFTBLOCKED;
    addon.applyBackgroundUpdates = AddonManager.AUTOUPDATE_DEFAULT;

    addon.userPermissions = null;

    addon.icons = {};
    if (await pkg.hasResource('icon.png')) {
      addon.icons[32] = 'icon.png';
      addon.icons[48] = 'icon.png';
    }

    if (await pkg.hasResource('icon64.png')) {
      addon.icons[64] = 'icon64.png';
    }

    Object.defineProperty(addon, 'appDisabled', {
      value: false,
      writable: false
    });

    Object.defineProperty(addon, 'signedState', {
      value: AddonManager.SIGNEDSTATE_NOT_REQUIRED,
      writable: false
    });

    return addon;
  },

  loadScope(addon) {
    let file = addon.file || addon._sourceBundle;
    let uri = getURIForResourceInFile(file, 'bootstrap.js').spec;
    let principal = Services.scriptSecurityManager.getSystemPrincipal();

    let sandbox = new Cu.Sandbox(principal, {
      sandboxName: uri,
      addonId: addon.id,
      wantGlobalProperties: ['ChromeUtils'],
      metadata: { addonID: addon.id, URI: uri },
    });

    try {
      Object.assign(sandbox, BOOTSTRAP_REASONS);

      XPCOMUtils.defineLazyGetter(sandbox, 'console', () =>
        new ConsoleAPI({ consoleID: `addon/${addon.id}` }));

      Services.scriptloader.loadSubScript(uri, sandbox);
    } catch (e) {
      logger.warn(`Error loading bootstrap.js for ${addon.id}`, e);
    }

    function findMethod(name) {
      if (sandbox[name]) {
        return sandbox[name];
      }

      try {
        let method = Cu.evalInSandbox(name, sandbox);
        return method;
      } catch (err) { }

      return () => {
        logger.warn(`Add-on ${addon.id} is missing bootstrap method ${name}`);
      };
    }

    let install = findMethod('install');
    let uninstall = findMethod('uninstall');
    let startup = findMethod('startup');
    let shutdown = findMethod('shutdown');

    return {
      install(...args) {
        install(...args);
        // Forget any cached files we might've had from this extension.
        Services.obs.notifyObservers(null, 'startupcache-invalidate');
      },

      uninstall(...args) {
        uninstall(...args);
        // Forget any cached files we might've had from this extension.
        Services.obs.notifyObservers(null, 'startupcache-invalidate');
      },

      startup(...args) {
        if (addon.type == 'extension') {
          logger.debug(`Registering manifest for ${file.path}\n`);
          Components.manager.addBootstrappedManifestLocation(file);
        }
        return startup(...args);
      },

      shutdown(data, reason) {
        try {
          return shutdown(data, reason);
        } catch (err) {
          throw err;
        } finally {
          if (reason != BOOTSTRAP_REASONS.APP_SHUTDOWN) {
            logger.debug(`Removing manifest for ${file.path}\n`);
            Components.manager.removeBootstrappedManifestLocation(file);
          }
        }
      },
    };
  },
};

AddonManager.addExternalExtensionLoader(BootstrapLoader);

if (AddonManager.isReady) {
  AddonManager.getAllAddons().then(addons => {
    addons.forEach(addon => {
      if (addon.type == 'extension' && !addon.isWebExtension && !addon.userDisabled) {
        addon.reload();
      };
    });
  });
}

[onemen]

@117649,

try to turn BootstrapLoader.jsm to BootstrapLoader.sys.mjs and use ChromeUtils.importESModule in config.js

[117649]

@117649,

try to turn BootstrapLoader.jsm to BootstrapLoader.sys.mjs and use ChromeUtils.importESModule in config.js

NO.

BootstrapLoader.jsm does not export any object so I'll try Services.scriptloader.loadSubScript first. But before any of those there are RDFManifestConverter.jsm & RDFDataSource.jsm to go.

[117649]

@onemen utils.zip

[onemen]

since xiaoxiaoflood/firefox-scripts is not active lets do all firefox-scripts discussions and code in https://github.com/onemen/firefox-scripts

can you create a PR?