Wrap follow-redirects around Electron's net module

[tomgrice]

Provides better flexibility and improved handling of redirects. Addresses an issue where redirects were on occasion failing due to server's reverse proxy settings.

[tomgrice]

Just noticed this allows the TSO download but breaks the Mesh bundle download. I’ll fix shortly

[tomgrice]

All works for me now.

[ItsSim]

Thanks. I wanted to change these requests to use the custom download.js function (the first one you changed) but never got around to it.

Will try to do a full installation on PC and Mac tonight with these changes and merge this PR if everything works correctly.

[tomgrice]

Yeah I did think reusing the function would be best, one to do in spare time :)

I do think it would be more efficient to port over to purely use the net module instead of using it via follow-redirects, and the net module has redirection handling built in. It is missing some functions though so would require a bit of a rewrite, for negligible performance gains. Javascript and Node/Electron aren't my expertise by any means so maybe one for someone else to try haha.

Thanks anyway, fixing this will solve a minor inconvenience not being able to install on my cloud gaming instance haha.

[ItsSim]

Tested on:

• Windows 11 Pro 22000.376
• macOS Monterey 12.0.1 (M1) (and on a different network)

There is an issue only with the following download: https://beta.freeso.org/LauncherResourceCentral/MacExtras. The rest work as expected.

It isn’t handling the redirect from https://beta.freeso.org/LauncherResourceCentral/MacExtras to https://beta.freeso.org/TempMacExtras.zip, and the response headers are incorrect:

{
  date: 'Fri, 24 Dec 2021 02:44:45 GMT',
  'content-type': 'text/html; charset=UTF-8',
  location: 'https://beta.freeso.org/TempMacExtras.zip',
  'cf-cache-status': 'DYNAMIC',
  'expect-ct': 'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"',
  'report-to': '{"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=AKrRo05g%2Bc%2FDNj4%2F5Dv06s2mymLZXrZaBdUxL7IJhKwT%2F2UpnIaXyLL%2FEg6IenaVhp8v4yOFnF265MAyu1UJIHxWZkaaK9GXz24EUE4eK%2F%2FrXmD8sCB79mEhMJfAf6GbEUIGy4bki3HrQaTCOaw%3D"}],"group":"cf-nel","max_age":604800}',
  nel: '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}',
  server: 'cloudflare',
  'cf-ray': '6c267d59dd1e3ff5-CDG',
  'alt-svc': 'h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400'
}

Tried reverting the changes of this PR and it handles this redirect again, returning the expected headers:

{
  date: 'Fri, 24 Dec 2021 03:05:54 GMT',
  'content-type': 'application/zip',
  'content-length': '11272146',
  connection: 'close',
  'last-modified': 'Mon, 27 Apr 2020 21:24:04 GMT',
  etag: '"5ea74d74-abffd2"',
  expires: 'Sun, 26 Dec 2021 02:34:55 GMT',
  'cache-control': 'public, max-age=172800',
  'x-download-optimized': '1',
  pragma: 'public',
  'cf-cache-status': 'HIT',
  age: '1859',
  'accept-ranges': 'bytes',
  'expect-ct': 'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"',
  'report-to': '{"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=aHHWCcahUnprYWdZ5LVv2Xzeuv7%2BSmSYTZN%2FV31j21p%2FjglYVnvtmFCZA9E0A5nTrR664i00EoQcMu3lVAz7024B60F480KcSp0onfP6koHJ3%2FJPc8pVP%2FnHCzFOf9Yrpw8%3D"}],"group":"cf-nel","max_age":604800}',
  nel: '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}',
  server: 'cloudflare',
  'cf-ray': '6c269c51de0383dd-BRU',
  'alt-svc': 'h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400'
}

This is very strange because the RemeshPackage.docx (https://beta.freeso.org/LauncherResourceCentral/3DModels) download does work as expected... 😩

Note that the MacExtras, as the name implies, is only downloaded when using the launcher on macOS, but this can be reproduced on Windows easily by changing the remesh package download URL here: https://github.com/ItsSim/fsolauncher/blob/30d875d82cb139f80110552799b28f5b63ac30e5/src/fsolauncher/library/installers/remeshes-installer.js#L13

to https://beta.freeso.org/LauncherResourceCentral/MacExtras.

I don’t get why this is happening only for that zip file redirect. Will investigate some more later. Not ready to merge this one until I figure this out.

[tomgrice]

Very strange. I was having trouble at first but it seems to be redirecting and downloading fine now?

https://beta.freeso.org/LauncherResourceCentral/MacExtras {
  date: 'Fri, 24 Dec 2021 04:54:27 GMT',
  'content-type': 'text/html; charset=UTF-8',
  location: 'https://beta.freeso.org/TempMacExtras.zip',
  'cf-cache-status': 'DYNAMIC',
  'expect-ct': 'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"',
  'report-to': '{"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=hsqsfsk3WJxyAjpmQ8Et0X2%2FEHqCpelsgMmYfUHZqyRPzS3GjQOEx3VK%2F%2BhJYpKosME909zKRvDYy0LzUAKJIeBCYPk0KDJoTlmK7FoVT86L8zY%2Bm21UN4xImebDkDLOXGI%3D"}],"group":"cf-nel","max_age":604800}',
  nel: '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}',
  server: 'cloudflare',
  'cf-ray': '6c273b57bcfc76a3-LHR',
  'alt-svc': 'h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400'
}

Results in mono runtimes being downloaded and extracted.

[ItsSim]

It does download, but since it’s not returning the final headers (after redirection), there’s no Content-Length, which is used to display the progress of the download. You end up with a broken progress item in the download tab:

[ItsSim]

Perhaps a solution would be to just show a 100% progress bar and the text: "Downloading client files... {megabytes downloaded} MB" without showing a percentage if the Content-Length data isn’t available. But it still irks me that the RemeshPackage download returns the correct headers every time...

[tomgrice]

I think I've sussed this one out and you should be able to keep the progress bar as-is. I think Cloudflare is suggesting a cache time to the built in Chromium browser, so it is just returning some weird metadata most of the time. Can override it with a http header:

_request = httpModule.get( from, { headers: { 'Pragma': 'no-cache' } }, _onDownload ); In const run in the download.js file.

Hopefully that fixes it on your end too. I was thinking of all kinds of hacky solutions but this should work as a workaround.

[ItsSim]

That seems to have fixed it! Merging and will push a new version ASAP, hopefully this also fixes issue https://github.com/ItsSim/fsolauncher/issues/19 for those who have been having download issues.