autoUpdater.quitAndInstall fails to relaunch on Mac OS 11 Big Sur

[koenoe]

When restarting an upgraded Electron application with autoUpdater.quitAndInstall, the application quits and upgrades successfully, but is NOT relaunched. There's also an issue about this in the Electron repository.

It works as expected for me with Electron 11.1.1, but it fails with 11.1.1-wvvmp. It only fails on Big Sur and succeeds on Catalina and Mojave. I've created a minimal code example to reproduce:

import {
  BrowserWindow,
  Notification,
  app,
  autoUpdater,
  dialog,
  globalShortcut,
} from 'electron';
import * as path from 'path';

if (!process.env.NODE_ENV) {
  process.env.NODE_ENV = 'production';
}

interface VersionInfo {
  appVersion: string;
  packageVersion: string;
}

function getVersion(): VersionInfo {
  return {
    appVersion: process.env.APP_VERSION || 'DEVELOP',
    packageVersion: process.env.PACKAGE_VERSION || '',
  };
}

function onAppReady(cb: Function) {
  if (app.isReady()) {
    cb();
  } else {
    app.on('ready', () => cb());
  }
}

function appReady(): Promise<void> {
  return new Promise((resolve) => {
    onAppReady(() => resolve());
  });
}

const UPDATE_URL =
  process.platform === 'darwin'
    ? `https://update-server-mac.com/updates/latest?v=${getVersion().packageVersion}`
    : 'https://update-server-windows.com/desktop/windows/';

function createWindow() {
  const webPreferences = {
    contextIsolation: true,
    enableRemoteModule: true,
    nodeIntegration: false,
    plugins: true,
    webSecurity: true,
    worldSafeExecuteJavaScript: true,
  };

  const mainWindow = new BrowserWindow({
    backgroundColor: '#ff0000',
    enableLargerThanScreen: true,
    frame: false,
    icon: path.resolve(`${__dirname}/../app/assets/icons/icon.png`),
    show: false,
    title: 'My app',
    titleBarStyle: 'hiddenInset',
    webPreferences,
    width: 640,
    height: 480,
  });

  // Prevents title from being set to <title> of web
  mainWindow.on('page-title-updated', (e) => e.preventDefault());

  const webContents = mainWindow.webContents;

  webContents.on(
    'certificate-error',
    (
      _: Electron.Event,
      url: string,
      error: string,
      certificate: Electron.Certificate,
      callback: (isTrusted: boolean) => void,
    ) => {
      console.error(
        new Error(
          `Certificate error (${error}) on "${url}" with certificate "${certificate.issuerName}"`,
        ),
      );
      callback(false); // reject everything
    },
  );

  webContents.on('render-process-gone', (_, { reason }) => {
    console.error('Renderer process crashed', reason);
  });

  webContents.on('responsive', () => {
    console.log('Renderer process is responsive again');
  });

  webContents.on('unresponsive', () => {
    console.warn('Renderer process is unresponsive');
  });

  webContents.once('destroyed', () => {
    console.log('Web content was destroyed');
  });

  webContents.on('new-window', (event: Electron.Event) => {
    event.preventDefault();

    return false;
  });

  webContents.on(
    'did-fail-load',
    (
      _: Electron.Event,
      errorCode: number,
      errorDescription: string,
      validateURL: string,
    ) => {
      if (validateURL === webContents.getURL()) {
        console.error(
          new Error(
            `Web Content failed to load (${errorCode}: ${errorDescription}) on "${validateURL}"`,
          ),
        );
      }
    },
  );

  mainWindow.loadURL(`file://${__dirname}/../app/assets/index.html`);

  mainWindow.on('close', () => {
    console.log('mainWindow on close');
  });

  // Must be called here, else it doesn't show up
  mainWindow.once('ready-to-show', () => {
    console.log('ready-to-show');
    mainWindow.show();
  });
}

function startAutoUpdater() {
  autoUpdater.on('checking-for-update', () =>
    console.log('Checking for updates'),
  );
  autoUpdater.on('error', (error: Error) => {
    console.error(error);
    const notification = {
      title: 'Auto update error',
      body: error.message,
    };
    new Notification(notification).show();
  });
  autoUpdater.on('update-available', () => {
    console.log('Update available');
    const notification = {
      title: 'Auto update',
      body: 'Update available',
    };
    new Notification(notification).show();
  });
  autoUpdater.on('update-downloaded', (_, releaseNotes, releaseName) => {
    console.log('Update downloaded');
    const dialogOpts = {
      type: 'info',
      buttons: ['Restart', 'Later'],
      title: 'Application Update',
      message: process.platform === 'win32' ? releaseNotes : releaseName,
      detail:
        'A new version has been downloaded. Restart the application to apply the updates.',
    };

    dialog.showMessageBox(dialogOpts).then((returnValue: any) => {
      if (returnValue.response === 0) autoUpdater.quitAndInstall();
    });
  });
  autoUpdater.on('update-not-available', () => {
    console.log('No update found');
    const notification = {
      title: 'Auto update',
      body: 'No updates found',
    };
    new Notification(notification).show();
  });

  autoUpdater.setFeedURL({
    url: UPDATE_URL,
  });
}

const main = async () => {
  // Prevent multiple app instances
  const gotTheLock = app.requestSingleInstanceLock();
  if (!gotTheLock) {
    app.exit();
    return;
  }

  app.allowRendererProcessReuse = true;
  // Allow playback to be started from electron (without DOM interaction)
  app.commandLine.appendSwitch('autoplay-policy', 'no-user-gesture-required');

  await appReady();

  const notification = {
    title: 'App version',
    body: getVersion().appVersion,
  };
  new Notification(notification).show();

  startAutoUpdater();

  globalShortcut.register('CommandOrControl+U', () => {
    console.log('Shortcut triggered to check for updates');
    autoUpdater.checkForUpdates();
  });

  createWindow();
};

try {
  main();
} catch (error) {
  console.error('Failed to execute main:', error);
}

[khwaaj]

I can confirm that the updated version of Squirrel/ShipIt is in the v11.1.1-wvvmp release, which should mean this is actually fixed in ECS as well. I can see no apparent reason for ECS to fail on Big Sur only based on the changes that it provides compared to stock Electron. I guess it could be possible that some kind of conflict arises between the Widevine CDM updater and the auto updater, but if that is the case I believe we should see the problem on Catalina and Mojave as well.

Unfortunately I don't have the ability to test this myself at this time, but it would be interesting to see the logs produced by your example code, especially if logs are added for the Widevine related events and the single instance lock check. This could give some valuable hints as to where the issue arises and if there is some relation with the Widevine events.

[koenoe]

I can confirm that the updated version of Squirrel/ShipIt is in the v11.1.1-wvvmp release, which should mean this is actually fixed in ECS as well. I can see no apparent reason for ECS to fail on Big Sur only based on the changes that it provides compared to stock Electron. I guess it could be possible that some kind of conflict arises between the Widevine CDM updater and the auto updater, but if that is the case I believe we should see the problem on Catalina and Mojave as well.

Unfortunately I don't have the ability to test this myself at this time, but it would be interesting to see the logs produced by your example code, especially if logs are added for the Widevine related events and the single instance lock check. This could give some valuable hints as to where the issue arises and if there is some relation with the Widevine events.

Thanks for the quick reply. In my current example I don't sign it with the Widevine certificates as I basically started stripping our build process until I got a working version with the 'pure' Electron build (as multiple people confirmed it was working for them in Electron 11).

I'll add the Widevine signing back + some more logging and I'll report my findings back to you.

[koenoe]

I've done some more debugging/logging and found some useful information.

Current code example (for my test with ECS, I return success() in onWidevineReady for my test with Electron):

import {
  BrowserWindow,
  Notification,
  app,
  autoUpdater,
  dialog,
  globalShortcut,
} from 'electron';
import * as path from 'path';
import * as logger from './logger';

if (!process.env.NODE_ENV) {
  process.env.NODE_ENV = 'production';
}

logger.configureLogger();
logger.interceptConsole(console);

interface VersionInfo {
  appVersion: string;
  packageVersion: string;
}

function getVersion(): VersionInfo {
  return {
    appVersion: process.env.APP_VERSION || 'DEVELOP',
    packageVersion: process.env.PACKAGE_VERSION || '',
  };
}

function onAppReady(cb: Function) {
  if (app.isReady()) {
    cb();
  } else {
    app.on('ready', () => cb());
  }
}

function appReady(): Promise<void> {
  return new Promise((resolve) => {
    onAppReady(() => resolve());
  });
}

let widevineIsReady: boolean;
let widevineHasError: any;

function onWidevineReady(success: Function, error: Function) {
  if (widevineIsReady) {
    console.log('Widevine already installed');
    return success();
  }

  if (widevineHasError) {
    console.log(`Widevine already errored: ${widevineHasError.message}`);
    return error(widevineHasError);
  }

  app.on('widevine-error', (err: Error) => {
    console.error(`Widevine error: ${err.message}`);
    widevineIsReady = true;
    success();
  });

  // @ts-ignore
  app.on('widevine-ready', (version, lastVersion) => {
    if (lastVersion !== null) {
      console.log('Widevine upgraded', { version, lastVersion });
    } else {
      console.log('Widevine installed', { version });
    }

    widevineIsReady = true;
    success();
  });

  // @ts-ignore
  app.on('widevine-update-pending', (curVersion, pendingVersion) => {
    console.log('New Widevine update pending', {
      currentVersion: curVersion,
      pendingVersion,
    });
  });
}

function widevineReady(): Promise<void> {
  return new Promise((resolve, reject) => {
    onWidevineReady(resolve, reject);
  });
}

const UPDATE_URL =
  process.platform === 'darwin'
    ? `https://update-server-mac.com/updates/latest?v=${getVersion().packageVersion}`
    : 'https://update-server-windows.com/desktop/windows/';

function createWindow() {
  const webPreferences = {
    contextIsolation: true,
    enableRemoteModule: true,
    nodeIntegration: false,
    plugins: true,
    webSecurity: true,
    worldSafeExecuteJavaScript: true,
  };

  const mainWindow = new BrowserWindow({
    backgroundColor: '#ff0000',
    enableLargerThanScreen: true,
    frame: false,
    icon: path.resolve(`${__dirname}/../app/assets/icons/icon.png`),
    show: false,
    title: 'MyApp',
    titleBarStyle: 'hiddenInset',
    webPreferences,
    width: 640,
    height: 480,
  });

  // Prevents title from being set to <title> of web
  mainWindow.on('page-title-updated', (e) => e.preventDefault());

  const webContents = mainWindow.webContents;

  webContents.on(
    'certificate-error',
    (
      _: Electron.Event,
      url: string,
      error: string,
      certificate: Electron.Certificate,
      callback: (isTrusted: boolean) => void,
    ) => {
      console.error(
        new Error(
          `Certificate error (${error}) on "${url}" with certificate "${certificate.issuerName}"`,
        ),
      );
      callback(false); // reject everything
    },
  );

  webContents.on('render-process-gone', (_, { reason }) => {
    console.error('Renderer process crashed', reason);
  });

  webContents.on('responsive', () => {
    console.log('Renderer process is responsive again');
  });

  webContents.on('unresponsive', () => {
    console.warn('Renderer process is unresponsive');
  });

  webContents.once('destroyed', () => {
    console.log('Web content was destroyed');
  });

  webContents.on('new-window', (event: Electron.Event) => {
    event.preventDefault();

    return false;
  });

  webContents.on(
    'did-fail-load',
    (
      _: Electron.Event,
      errorCode: number,
      errorDescription: string,
      validateURL: string,
    ) => {
      // If the initial request fails, we assume that the user doesn't have
      // a network connection. Show them the offline page.
      if (validateURL === webContents.getURL()) {
        console.error(
          new Error(
            `Web Content failed to load (${errorCode}: ${errorDescription}) on "${validateURL}"`,
          ),
        );
      }
    },
  );

  mainWindow.loadURL(`file://${__dirname}/../app/assets/index.html`);

  mainWindow.webContents.openDevTools();

  mainWindow.on('close', () => {
    console.log('mainWindow on close');
  });

  // Must be called here, else it doesn't show up
  mainWindow.once('ready-to-show', () => {
    console.log('ready-to-show');
    mainWindow.show();
  });
}

function startAutoUpdater() {
  autoUpdater.on('checking-for-update', () =>
    console.log('Checking for updates'),
  );
  autoUpdater.on('error', (error: Error) => {
    console.error(error);
    const notification = {
      title: 'Auto update error',
      body: error.message,
    };
    new Notification(notification).show();
  });
  autoUpdater.on('update-available', () => {
    console.log('Update available');
    const notification = {
      title: 'Auto update',
      body: 'Update available',
    };
    new Notification(notification).show();
  });
  autoUpdater.on('update-downloaded', (_, releaseNotes, releaseName) => {
    console.log('Update downloaded');
    const dialogOpts = {
      type: 'info',
      buttons: ['Restart', 'Later'],
      title: 'Application Update',
      message: process.platform === 'win32' ? releaseNotes : releaseName,
      detail:
        'A new version has been downloaded. Restart the application to apply the updates.',
    };

    dialog.showMessageBox(dialogOpts).then((returnValue: any) => {
      if (returnValue.response === 0) autoUpdater.quitAndInstall();
    });
  });
  autoUpdater.on('update-not-available', () => {
    console.log('No update found');
    const notification = {
      title: 'Auto update',
      body: 'No updates found',
    };
    new Notification(notification).show();
  });

  autoUpdater.setFeedURL({
    url: UPDATE_URL,
  });
}

const main = async () => {
  // Prevent multiple app instances
  const gotTheLock = app.requestSingleInstanceLock();
  if (!gotTheLock) {
    console.warn('got the multiple instance lock.');
    app.exit();
    return;
  }

  app.allowRendererProcessReuse = true;

  // Allow playback to be started from electron (without DOM interaction)
  app.commandLine.appendSwitch('autoplay-policy', 'no-user-gesture-required');

  await appReady();
  await widevineReady();

  const notification = {
    title: 'App version',
    body: getVersion().appVersion,
  };
  new Notification(notification).show();

  startAutoUpdater();

  globalShortcut.register('CommandOrControl+U', () => {
    console.log('Shortcut triggered to check for updates');
    autoUpdater.checkForUpdates();
  });

  createWindow();
};

try {
  main();
} catch (error) {
  console.error('Failed to execute main:', error);
}

This results in the following logs. ECS:

(12/01/21 14:10:20.583) [info] - Widevine installed
{
  "version": "4.10.1582.2"
}
(12/01/21 14:10:20.911) [info] - ready-to-show [object Object]
(12/01/21 14:10:26.905) [info] - Shortcut triggered to check for updates [object Object]
(12/01/21 14:10:26.909) [info] - Checking for updates [object Object]
(12/01/21 14:10:27.128) [info] - No update found [object Object]
(12/01/21 14:10:46.386) [info] - Shortcut triggered to check for updates [object Object]
(12/01/21 14:10:46.389) [info] - Checking for updates [object Object]
(12/01/21 14:10:47.355) [info] - Update available [object Object]
(12/01/21 14:10:50.903) [info] - Update downloaded [object Object]
(12/01/21 14:10:52.752) [info] - mainWindow on close [object Object]
(12/01/21 14:10:52.769) [info] - Web content was destroyed [object Object]

Electron:

(13/01/21 10:42:04.025) [info] - Shortcut triggered to check for updates [object Object]
(13/01/21 10:42:04.028) [info] - Checking for updates [object Object]
(13/01/21 10:42:05.155) [info] - Update available [object Object]
(13/01/21 10:42:08.716) [info] - Update downloaded [object Object]
(13/01/21 10:42:10.111) [info] - mainWindow on close [object Object]
(13/01/21 10:42:10.125) [info] - Web content was destroyed [object Object]
(13/01/21 10:42:14.298) [info] - ready-to-show [object Object]

Other than that you can see that we never get a ready-to-show after updating in ECS, there's not much other useful information here. However, the ShipIt_stderr.log gives more useful information.

ECS:

2021-01-12 14:10:50.935 ShipIt[44621:499958] Detected this as an install request
2021-01-12 14:10:52.865 ShipIt[44621:499963] Beginning installation
2021-01-12 14:10:53.136 ShipIt[44621:499964] Moved bundle from file:///Applications/MyApp.app/ to file:///var/folders/v9/vhfc6_rx5798pcthq2s0myrw0000gn/T/com.MyApp.desktop.ShipIt.6NSyyEGo/MyApp.app
2021-01-12 14:10:53.137 ShipIt[44621:499964] Moved bundle from file:///var/folders/v9/vhfc6_rx5798pcthq2s0myrw0000gn/T/com.MyApp.desktop.ShipIt.8vCXjoJQ/MyApp.app to file:///Applications/MyApp.app/
2021-01-12 14:10:53.184 ShipIt[44621:499964] Couldn't remove owned bundle at location file:///var/folders/v9/vhfc6_rx5798pcthq2s0myrw0000gn/T/com.MyApp.desktop.ShipIt.8vCXjoJQ/MyApp.app, error NSError { domain: NSCocoaErrorDomain, code: 4, description: "“MyApp.app” couldn’t be removed.", failureReason: "The file doesn’t exist.", underlying error: NSError { domain: NSPOSIXErrorDomain, code: 2, description: "The operation couldn’t be completed. No such file or directory" }, userInfo: {
    NSFilePath = "/var/folders/v9/vhfc6_rx5798pcthq2s0myrw0000gn/T/com.MyApp.desktop.ShipIt.8vCXjoJQ/MyApp.app";
    NSUserStringVariant =     (
        Remove
    );
} }
2021-01-12 14:10:53.233 ShipIt[44621:500019] On main thread and launching: file:///Applications/MyApp.app/
2021-01-12 14:10:53.233 ShipIt[44621:500019] Bundle URL is valid
2021-01-12 14:10:53.233 ShipIt[44621:500019] Attempting to launch app on lower than 11.0
2021-01-12 14:10:53.234 ShipIt[44621:499964] Installation completed successfully

Electron:

2021-01-13 10:42:10.872 ShipIt[49171:537067] On main thread and launching: file:///Applications/MyApp.app/
2021-01-13 10:42:10.872 ShipIt[49171:537067] Bundle URL is valid
2021-01-13 10:42:10.873 ShipIt[49171:537067] Attempting to launch app on 11.0 or higher
2021-01-13 10:42:10.873 ShipIt[49171:537067] Launching new ShipIt at /Applications/MyApp.app/Contents/Frameworks/Squirrel.framework/Resources/ShipIt with instructions to launch file:///Applications/MyApp.app/
2021-01-13 10:42:10.873 ShipIt[49171:536969] Installation completed successfully
2021-01-13 10:42:13.203 ShipIt[49175:537098] Detected this as a launch request
2021-01-13 10:42:13.289 ShipIt[49175:537098] Successfully launched application at file:///Applications/MyApp.app/
2021-01-13 10:42:13.289 ShipIt[49175:537098] ShipIt quitting
2021-01-13 10:42:13.291 ShipIt[49171:537067] New ShipIt exited
2021-01-13 10:42:13.291 ShipIt[49171:537067] ShipIt quitting

The reason restarting fails in ECS, is because of this I think:

2021-01-12 14:10:53.233 ShipIt[44621:500019] Attempting to launch app on lower than 11.0

I also noticed someone commenting on the original Electron issue about this. But not sure whether that person is using Electron or a fork.

[khwaaj]

Thanks for the detailed response! This is actually exactly what I was suspecting was happening and it suggests there is something fishy in the build environment rather than a problem with the ShipIt workaround itself, maybe something related to the macOS SDK.

Xcode on the build machine was recently updated but I can't be sure if it was done before or after v11.1.1-wvvmp was released. I'm preparing release builds for v11.2.0-wvvmp right now, would you be willing to try out a preliminary build of that and see if it fixes the issue?

[koenoe]

Xcode on the build machine was recently updated but I can't be sure if it was done before or after v11.1.1-wvvmp was released. I'm preparing release builds for v11.2.0-wvvmp right now, would you be willing to try out a preliminary build of that and see if it fixes the issue?

Absolutely! I'm using 11.1.1 now as I wanted to make sure it happened with the latest version too. But the problem is happening with every ECS 11 build I've tried.

[khwaaj]

Thanks, you can download the preliminary build here.

You won't be able to use EVS to sign it as it is not yet official, but for this test it won't matter.

[koenoe]

Could you clarify how I can get this to work with a local ZIP file? Right now we have this in our package.json when we use ECS:

"devDependencies": {
  "electron": "https://github.com/castlabs/electron-releases#v11.1.1-wvvmp"
  ...
},
"build": {
    "electronVersion": "11.1.1",
    "electronDownload": {
      "version": "11.1.1-wvvmp",
      "mirror": "https://github.com/castlabs/electron-releases/releases/download/v"
    }
  }

And I think we need to sign it as Electron Auto Updater otherwise won't work?

Error: Could not get code signature for running application

[khwaaj]

To get electron-builder to use the build when packaging you can unpack the zip, say in /tmp/v11.2.0-wwwmp or replacing the contents of node_modules/electron/dist, and then use the electronDist option to point to it (instead of using electronDownload):

"build": {
  "electronDist": "/tmp/v11.2.0-wwwmp"
}

The signing error you see has to be related to code-signing, not VMP-signing, so it is unrelated to EVS. I'm not familiar with the Electron Auto Updater, but I suppose it requires the build to be code signed?

[koenoe]

To get electron-builder to use the build when packaging you can unpack the zip, say in /tmp/v11.2.0-wwwmp or replacing the contents of node_modules/electron/dist, and then use the electronDist option to point to it (instead of using electronDownload):

"build": {
  "electronDist": "/tmp/v11.2.0-wwwmp"
}

The signing error you see has to be related to code-signing, not VMP-signing, so it is unrelated to EVS. I'm not familiar with the Electron Auto Updater, but I suppose it requires the build to be code signed?

Awesome, will try that out. Thank you very much! You're right, I didn't realise you meant VMP-signing. Should be fine without, correct.

[koenoe]

This is becoming a bit more troublesome as I expected. At this moment I'm only able to build our app on the CI (don't have the signing certificates locally), so I'd need to add the preliminary build in my orphan branch to the repository. Unfortunately some files in preliminary build exceeds GitHub's file size limit of 100MB 😅.

Would it be possible to pre-release this build? It would make it a lot easier for me to test at this moment.

[khwaaj]

Ok, that does sound like trouble. I'll see if I can publish a "prelelease" somehow.

[khwaaj]

Try this, you should just need to update the tag in your URL to #v11.2.0-wvvmp-prerelease.1.

[khwaaj]

Since I suspect the issue actually lies in the macOS SDK that is used by the build I preemptively created another prerelease build that forces the use of the latest SDK. The build machine has multiple SDKs installed and electron defaults to the oldest supported SDK when building, in this case 10.15, so that is what is used in prerelease.1. For prerelease.2 I made sure the 11.1 SDK is used instead. In case the first prerelease displays the same auto-updater issue, please try with the second one to see if that fixes the problem.

It is available here, or by using the tag #v11.2.0-wvvmp-prerelease.2.

[koenoe]

I can confirm it works with #v11.2.0-wvvmp-prerelease.2! 💃🕺

[khwaaj]

Nice, thank you for your collaboration @koenoe! I'll be publishing the official v11.2.0-wvvmp release sometime during the day.

[koenoe]

Thank YOU. Awesome, will report back as soon as I'm able to test with the v11.2.0-wvvmp release

[khwaaj]

v11.2.0-wvvmp is now published.

[koenoe]

I can confirm it also works in v11.2.0-wvvmp. Thanks again!

[koenoe]

@khwaaj Unfortunately this fails again with 11.2.3-wvvmp. Same line in ShipIt_stderr.log that's incorrect:

Attempting to launch app on lower than 11.0

[khwaaj]

That is very strange since the only SDK available on the build machine is the latest one. The releases in between works as expected?

[koenoe]

That is very strange since the only SDK available on the build machine is the latest one. The releases in between works as expected?

That is strange indeed. Haven't tried it since v11.2.0-wvvmp, so will revert back to that one for now. And then I will test with a version in between and let you know.

[koenoe]

@khwaaj I double checked this morning and it broke in v11.2.1-wvvmp

[khwaaj]

Ok, so it broke immediately in the next release. I have a nagging suspicion about what could be causing it, but if that is the case I would be a bit disturbed by the implications. Anyway, I'll try to schedule a new build so that we can verify if it is the culprit here.

[koenoe]

Ok, so it broke immediately in the next release. I have a nagging suspicion about what could be causing it, but if that is the case I would be a bit disturbed by the implications. Anyway, I'll try to schedule a new build so that we can verify if it is the culprit here.

Let me know when you've got a new build you'd like me to try 👍

[khwaaj]

@koenoe, I apologize for the delay, but I just published v11.3.0-wvvmp which contains the small fix in the build environment that could possibly be the culprit here.

[khwaaj]

@koenoe, any updates here? Are things back to working as expected?

[koenoe]

@koenoe, any updates here? Are things back to working as expected?

I think so, but I'm not working for the company anymore where I encountered this problem. @osmestad could you verify?

[khwaaj]

Ok, thanks for letting me know @koenoe. Since I've heard nothing new I'll consider this resolved and close the issue. Please reopen if it turns out this is still an issue.

[osmestad]

Sorry for the slow reply! as far as I know this has been working fine for us lately :-)