search

rfvgyhn / min-ed-launcher

Minimal Elite Dangerous Launcher
MIT License
252 stars 9 forks source link

Odyssey Alpha #14

Closed mcgillij closed 3 years ago

mcgillij commented 3 years ago

Plans to add a param to launch / download the alpha?

nonchip commented 3 years ago

it'll launch it alright, but downloading is the issue: since it's not there already, it doesn't try to "update" it (it even says it's found it in the "authorised products" but because it cant find its versioninfo file it removes it from the list of "updateable products"). which would be great to have because the fdev launcher keeps crashing for me during the download process...

sirkkalap commented 3 years ago

What will be the option to lauch Odyssey alpha with MinEdLauncher?

rfvgyhn commented 3 years ago

I have no plans to implement content downloads in the near future. I don't have a frontier-only account which makes it difficult to figure out the download protocol and test it. Perhaps after the non-steam/lutris feature is properly implemented and tested. Just like that feature, I'll have to blindly implement it and then rely on someone else to test it.

I also don't have access to the alpha so I'm unsure of what the launch param is. I would guess /edo but I'd need the latest version of EDLaunch.exe to verify. From what I can tell, fdev hasn't released updated files on steam or epic, so, I think, you'll have to wait for that. I'd guess a temporary workaround would be to launch the fdev launcher, sign in with a non-steam/frontier account, let it download the game files and then then launch it via steam/epic.

rfvgyhn commented 3 years ago

Another way to get the launch param is to sniff your own web traffic. The launcher makes a request to https://api.zaonce.net/3.0/user/purchases which responds with json containing info about the products you own. The launch param would match what's returned for the filter property of the odyssey entry.

{
    "purchases": [
        {
            "filter": "ed",
            "product_name": "Elite Dangerous",
        },
        {
            "filter": "eda",
            "product_name": "Elite Dangerous: Arena",
        },
        {
            "filter": "edh",
            "product_name": "Elite Dangerous: Horizons",
        }
    ]
}
xolophreny commented 3 years ago

Posted in a Linux thread, slightly relevant to this launcher. Currently launching the alpha by mangling (or, well, moving) Horizons' VersionInfo.txt so the launcher doesn't recognise it as a valid product and launches the next product in line being the alpha

rfvgyhn commented 3 years ago

I released a new version that allows you to select which product to run. After you've selected a product to run, you'll also find the autorun filter in the log file. It will look something like [DBG] User selected Elite Dangerous: Horizons - FORC-FDEV-D-1013 - edh.

This also means that you'll now have to specify the /autorun flag in your launch options to get the same behavior as in previous releases.

rfvgyhn commented 3 years ago

A thought on implementing content downloads, if someone were to give me a log of the requests and responses (e.g. mitmproxy) made when the vanilla launcher is downloading an update, it might be enough to point me in the right direction for getting an initial implementation done. From there, if people are willing to test it, we might be able to get a working implementation.

xolophreny commented 3 years ago

Tried that, think it wasn't as successful as it should've been... So, steps taken:

  1. Change version number in Products/PUBLIC_TEST_SERVER_64/VersionInfo.txt to something lower
  2. Delete EliteDangerous64.exe there just for good measure
  3. Run sudo tcpdump -l 'tcp port 80' | tee tcpdump.log without any other applications using internet
  4. Start the launcher with steam, successfully upgrade the alpha, stop the launcher
  5. Repeat steps 1, 2
  6. Run ./mitmproxy -p 9200 -w mitm.log
  7. Start the launcher with steam using http_proxy=http://127.0.0.1:9200/ %command% launch option, successfully upgrade the alpha, close the launcher
  8. Inspect /launcher-status/status.json: 
    {
"status": 2,
"text": "OK"
}
  9. Try to look at mitm.log as a text file and see that it's not a text file
  10. Whoops, maybe ./mitmdump -p 9200 -w mitmdump.log?
  11. Repeat steps 1, 2, 7
  12. mitmdump.log is also not a text file... Bleh, just in case copy mitmproxy's TUI contents into mitm1.log.

So here are the resulting files. The tcpdump output shows the launcher getting what seems like this alpha version's file manifest, but I don't exactly know why mitmproxy doesn't catch that or why the actual file downloads aren't showing up even on tcpdump. Seeing how I didn't have a clear idea of what I am doing through all of this, maybe with some help I could produce more meaningful results

mitmdump.log mitm.log mitm1.log tcpdump.log

nonchip commented 3 years ago

btw to mitigate the crashiness of fdev's launcher (at least somewhat) it helps to disable threaded downloads (i think the key is something like disable_fastdownload) in the launcher's config file. also apparently disabling e/fsync helps. just for people looking for a quick workaround while you're doing network magic :D

also @xolophreny pretty sure mitmproxy uses tcpdump-esque binary dumps instead of textfiles

rfvgyhn commented 3 years ago

@xolophreny It looks like it didn't capture any of the HTTPS traffic. Did you install the mitmproxy cert? If you're on an arch based system, you can run sudo trust anchor ~/.mitmproxy/mitmproxy-ca-cert.cer and sudo trust anchor --remove ~/.mitmproxy/mitmproxy-ca-cert.cer to remove it later.

FYI, if you run mitmweb --listen-port 9200, you get an interface for inspecting all the traffic that's similar to browser debug tools. It will show you the traffic in realtime and also allows to load the binary log files you created. I found it to be a bit more intuitive for a few quick sessions instead of having to learn the TUI keys.

For those that want to try and disable fast download mode in the vanilla launcher, the config setting is in AppData\Local\Frontier_Developments\EDLaunch.exe_URL_[randomchars]\[version]\user.config under the setting Option_DisableFastDownload.

xolophreny commented 3 years ago

Right, I see, many thanks. Made sure to correctly install the certificate systemwide this time, was able to successfully look at https traffic from the browser with a configured proxying, or anything launched with http_proxy=127.0.0.1:9200 https_proxy=127.0.0.1:9200.

However, launching Elite from steam with http_proxy=127.0.0.1:9200 https_proxy=127.0.0.1:9200 %command% still yields the same results as before with only one or two weird image requests through https from the launcher and nothing at all during the upgrade, but this time the launcher also complained about certificate issues, no change after selecting "Yes" when asked to proceed anyway. I don't know if wine/proton has its own certificate storage, would explain this if it actually does...

Currently trying to plug all http/s traffic through this, but simply redirecting ports 80 and 443 to 9200 yields 502 Bad Gateway pretty much everywhere, and setting a proper systemwide proxy is unfortunately not trivial with just bare NetworkManager, from what I can tell

rfvgyhn commented 3 years ago

I think this getting beyond my knowledge. Which version of proton are you using? Perhaps pressure-vessel is interfering?

I normally use a windows vm to do my launcher/network inspection, but my steps to get it to work on linux where as follows:

  1. Delete/rename current proton prefix (steamapps/compatdata/359320)
  2. Force use of proton 5.0-10 to avoid potential pressure-vessel issues
  3. Launch game so new proton prefix is created
  4. Use protontricks to remove default wine-mono and install wine-mono-6.1.0_ED (couldn't get launcher to run without this step) protontricks -c 'wine64 uninstaller' 359320
  5. Trust mitmproxy cert
  6. Start mitmweb
  7. Start game via http_proxy=127.0.0.1:9200 %command% and selecting ed: horizons in steam prompt
  8. See requests to FDev's api at https://api.zaonce.net in web ui
xolophreny commented 3 years ago

All of the above was indeed performed with proton 5.0-10 because for some reason I couldn't get the stock launcher to work at all now on 5.9-GE or any pressure-vessel-using versions, with neither wine-mono_ED nor dotnet472.

So steps taken with mitmproxy certificates trusted systemwide (moved to /usr/local/share/ca-certificates/mitmproxy.crt and imported with sudo update-ca-certificates) and forced proton 5.0-10:

Removed the existing prefix, launched it to create a new one, protontricks 359320 -q dotnet472 win7 because for some reason wine-mono makes my launcher crash on pressing play/upgrade. (think that's because actually requires Proton 5.13 and above?)

With the prefix set up, set launch options to http_proxy=127.0.0.1:9200 %command% and selected default launcher to update the alpha; not seeing any https again, nor requests to the API. mitmweb.log

Then tried again after changing VersionInfo.txt for Horizons and starting the horizons launcher. It doesn't even try to update horizons because it relies on steam for that. mitmweb-horizons.log

Welp, still seems that I'm failing at getting the launcher (or proton?) to respect the proxy in some way. Will probably try to screw around with Wireshark over the next few days, that should allow to inspect all traffic going on. Hopefully someone else figures it out by then >.>

kerberizer commented 3 years ago

All of the above was indeed performed with proton 5.0-10 because for some reason I couldn't get the stock launcher to work at all now on 5.9-GE or any pressure-vessel-using versions, with neither wine-mono_ED nor dotnet472. (...) Removed the existing prefix, launched it to create a new one, protontricks 359320 -q dotnet472 win7 because for some reason wine-mono makes my launcher crash on pressing play/upgrade. (think that's because actually requires Proton 5.13 and above?)

FWIW, I had the same problem with crashes on play/upgrade. In the end, 6.5-GE-1 with wine-mono_ED seemed to work for me.

A-UNDERSCORE-D commented 3 years ago

Correctly detects and launches the alpha for me after symlinking directories (ln -s ./PUBLIC_TEST_SERVER_OD PUBLIC_TEST_SERVER_64)

kerberizer commented 3 years ago

@xolophreny @Rfvgyhn Re capturing traffic with mitmproxy: 1) One way to achieve it is by transparently redirecting (almost) all traffic to mitmproxy. 2) The above is still tricky because of certificate trust. Long story (~2h testing) short: append the mitmproxy certificate (in PEM format) to the existing <steamdir>/steamapps/common/SteamLinuxRuntime_soldier/var/deploy-0.20210317.0/files/etc/ssl/certs/ca-certificates.crt. Naturally, the version after deploy may need to be adjusted. Likewise for the scout version of the pressure vessel: <steamdir>/steamapps/common/SteamLinuxRuntime/var/scout/files/etc/ssl/certs/ca-certificates.crt.

@Rfvgyhn Re downloading an update: 1) Long story short: have you seen this? It may work (haven't tested it, but the code seems reasonable in the light of what traffic and behaviour from the installer I've seen) and the latest manifest for the OD Alpha seems to be http://cdn.zaonce.net/elitedangerous/win/manifests/Win64_4_0_0_10_Alpha+%282021.04.09.263090%29.xml.gz 2) Here's what comms I managed to gather. Apologies for not posting the raw traffic, but security paranoia kicks in :D

A few notes in advance:

GET https://api.zaonce.net/1.1/server/time
{
    "unixTimestamp": 1618096603
}
GET https://api.zaonce.net/1.1/client/version
version: 0.4.6600.0
{
    "versionLatest": "0.4.6568.0",
    "versionStatus": "future"
}
GET https://api.zaonce.net/3.0/user/steam/auth
steamTicket: <redacted>
machineId:   <redacted>
fTime:       1618096603.34384
{
    "authToken": "<redacted>",
    "machineToken": "<redacted>",
    "registeredName": "<redacted>"
}
POST https://api.zaonce.net/1.1/eventLog
eventTime:    1618096604.05182
event:        SteamAuthenticated
machineToken: <redacted>
authToken:    <redacted>
fTime:        1618096603.90326
machineId:    <redacted>

action=SteamAuthenticated&date=20210410&time=231644
{
    "message": "Ok",
    "status": 200
}
POST https://api.zaonce.net/1.1/eventLog
event:        Authenticated

action=Authenticated&user=<redacted>&date=20210410&time=231644
POST https://api.zaonce.net/1.1/eventLog
event:        CollectHardwareSurvey

action=CollectHardwareSurvey&date=20210410&time=231644
GET https://api.zaonce.net/3.0/user/purchases
lang:  en
fTime: 1618096605.10003
-H "Authorization: bearer <redacted>"
...
        {
            "colour": "#b4956d",
            "directory": "PUBLIC_TEST_SERVER_64",
            "filter": "edo",
            "gameargs": "",
            "product_name": "Elite Dangerous: Odyssey Alpha - Public Test",
            "product_sku": "PUBLIC_TEST_SERVER_OD",
            "serverargs": "/Test",
            "sortkey": "01",
            "template": "http://hosting.zaonce.net/launcher-steam/odyssey/en.html"
        }
...
POST https://api.zaonce.net/1.1/eventLog
event:        AvailableProjects

action=AvailableProjects&&projects=[FORC-FDEV-D-1010, FORC-FDEV-D-1012, COMBAT_TUTORIAL_DEMO, FORC-FDEV-D-1013, PUBLIC_TEST_SERVER_OD]&date=20210410&time=231645
GET https://api.zaonce.net/3.0/user/installer
machineToken: <redacted>
authToken:    <redacted>
machineId:    <redacted>
sku:          PUBLIC_TEST_SERVER_OD
os:           win
fTime:        1618096612.84418
{
    "localFile": "2021.04.09.263090.xml.gz",
    "md5": "0e310a6b3c842b62bd02c0056b8c972e",
    "remotePath": "<redacted>",
    "size": "4158792",
    "version": "2021.04.09.263090"
}
POST https://api.zaonce.net/1.1/eventLog
event:        UpdateRequired

action=UpdateRequired&Project=PUBLIC_TEST_SERVER_OD&Directory=PUBLIC_TEST_SERVER_OD&Current=2021.03.30.261992&Remote=2021.04.09.263090&date=20210410&time=231654
POST https://api.zaonce.net/1.1/eventLog
event:        ClientAuthorised

action=ClientAuthorised&&clientversion=0.4.6600.0&date=20210410&time=231655
POST https://api.zaonce.net/1.1/eventLog
event:        SupportedAddress

action=SupportedAddress&IPv4=true&date=20210410&time=231655
POST https://api.zaonce.net/1.1/eventLog
event:        DownloadAndInstallProject

action=DownloadAndInstallProject&project=PUBLIC_TEST_SERVER_OD&date=20210410&time=233615
POST https://api.zaonce.net/1.1/eventLog
event:        DetermineInstaller

action=DetermineInstaller&Project=PUBLIC_TEST_SERVER_OD&Installer=2021.04.09.263090.xml.gz&Remote=/Win64_4_0_0_10_Alpha+%282021.04.09.263090%29.xml.gz&date=20210410&time=233733
POST https://api.zaonce.net/1.1/eventLog
event:        DownloadManifest

action=DownloadManifest&date=20210410&time=233734

This, of course, is getting the manifest. Apparently, the name of each manifest is hardcoded (or fetched by other means during initial installation, not upgrade) and the product version is appended to the name.

GET http://cdn.zaonce.net/elitedangerous/win/manifests/Win64_4_0_0_10_Alpha+%282021.04.09.263090%29.xml.gz
POST https://api.zaonce.net/1.1/eventLog
event:        DownloadManifestCompleted

action=DownloadManifestCompleted&date=20210410&time=233735
POST https://api.zaonce.net/1.1/eventLog
event:        FastSynchroniseStarted

action=FastSynchroniseStarted&date=20210410&time=233736
POST https://api.zaonce.net/1.1/eventLog
event:        FastSynchroniseStartProcess

action=FastSynchroniseStartProcess&Threads=8&Bundles=65493&date=20210410&time=233835

This is the download of the files itself. Seems likely that downloaded are the files that fail the hash check from latest manifest. Only one posted for example.

GET http://cdn.zaonce.net/elitedangerous/win/files/62/2451401c93cb555d2961d81248b2b6cdae261b

Some downloads have failed (empty response body despite the HTTP 200 code). As expected, was due to the proxy. These failures apparently get also pushed to the eventLog endpoint (the POST body does include the embedded newlines).

POST https://api.zaonce.net/1.1/eventLog
event:        DownloadException

action=DownloadException&exception=Exception starting download:

The operation has timed out.&date=20210410&time=234040

Finally, once all is over. This is with the errors, though, which perhaps explains the zeroes everywhere. The update did finish correctly without mitmproxy, but, of course, I don't have the log :) I suspect that some files were simply too large and maybe it was possible to avoid the errors with some configuration.

POST https://api.zaonce.net/1.1/eventLog
event:        FastSynchroniseCompleted

action=FastSynchroniseCompleted&Added=0&Updated=0&Skipped=0&Removed=0&Copied=0&Downloaded=0&date=20210411&time=003259
POST https://api.zaonce.net/1.1/eventLog
event:        CacheResults

action=CacheResults&Cache:virtual=0&date=20210411&time=003259
POST https://api.zaonce.net/1.1/eventLog
event:        FORCDownloadManagerReport

action=FORCDownloadManagerReport&filecount=1&date=20210411&time=003300

On exit.

POST https://api.zaonce.net/1.1/eventLog
event:        Closing

action=Closing&date=20210411&time=003556
rfvgyhn commented 3 years ago

@kerberizer Perfect. That's exactly what I needed. Thanks for digging into this. I should be able to start working on content downloads sometime this week.

Regarding the event log api, you're correct that it won't be needed. The vanilla launcher logs all sorts of stuff both locally and remotely and not sending remote logs is one of the things I consider to be a feature of the minimal launcher.

rfvgyhn commented 3 years ago

Some progress has been made. I have file verification + caching and downloading working. I believe I've worked out most of the edge cases. The branch is currently hard-coded to check updates for the main game so the CI build can't be used to test it yet. Shouldn't be long before there's a build for folks to test though.

rfvgyhn commented 3 years ago

A new build is available and ready for testing. If using a steam or epic account, you'll need to specify the new forceUpdate setting in settings.json. "forceUpdate": "PUBLIC_TEST_SERVER_OD",. This tells the launcher to get updates from FDev even though they should be managed by steam/epic.

Regarding the product directory (PUBLIC_TEST_SERVER_OD vs PUBLIC_TEST_SERVER_64), it looks like the vanilla launcher uses the SKU instead of the directory specified in the purchases response to determine where the product files exist if it's Steam or Epic.

@kerberizer @xolophreny @A-UNDERSCORE-D Before any folder renaming or symlinking, in which folder did the odyssey product files live and did you launch the vanilla launcher through steam or directly (i.e. without the /steam flag)?

A-UNDERSCORE-D commented 3 years ago

PUBLIC_TEST_SERVER_OD is what my install used (its what I symlinked PUBLIC_TEST_SERVER to), and I launched it though steam always. I need proton support et al.

A-UNDERSCORE-D commented 3 years ago 
╓ad@battlestar-galactica [05:05:38]:/games/steam/steamapps/common/Elite Dangerous/Products
╙─╴% la
total 59M
drwxr-xr-x 1 ad users  224 Mar 30 15:21 COMBAT_TUTORIAL_DEMO/
drwxr-xr-x 1 ad users  672 Apr 10 14:06 elite-dangerous-64/
lrwxrwxrwx 1 ad users   23 Apr 10 22:48 PUBLIC_TEST_SERVER_64 -> ./PUBLIC_TEST_SERVER_OD/
drwxrwxr-x 1 ad users 1.1K Apr 22 10:32 PUBLIC_TEST_SERVER_OD/
-rw-r--r-- 1 ad users  59M Apr 20 20:32 VirtualCache.xml

For reference.

kerberizer commented 3 years ago

Yes, it's PUBLIC_TEST_SERVER_OD here as well. I haven't done any symlinking and the launcher was run from Steam with the default parameters (double-checked it did use /Steam) .

A-UNDERSCORE-D commented 3 years ago

Just tested with the latest alpha update, did not work. launched, bottom left said phase 3, server said too new. it DID do an update though. Currently running the normal launcher to see if its an fdev bug or not. Possibly you didn't update some version file somewhere?

A-UNDERSCORE-D commented 3 years ago

update: vanilla launcher says there is an update available for me.

A-UNDERSCORE-D commented 3 years ago

oh I will also note that Horizons and Odyssey flipped in the selection list when the update ran. Possibly related?

rfvgyhn commented 3 years ago

New build addressing VersionInfo.txt not being updated properly and the unsorted selection list after an update. You'll need to delete the hash map from $XDG_CACHE_HOME/min-ed-launcher/ to account for the fixed bug.

A-UNDERSCORE-D commented 3 years ago

Just to verify behavior; I had both a hashmap.PUBLIC_TEST_SERVER_64.txt and a Win64_4_0_0_30_Alpha2021.04.28.265434/hashmap.txt in the cache dir. Former had hashes, latter was empty. I have removed both.

rfvgyhn commented 3 years ago

That's correct. Everything in the cache dir can safely be deleted at any time. It will just take longer to hash Elite's 35k files the next time there's an update. The hashmap.*.txt file contains hashes of the files in the product dir. The files in the folder are where updates are downloaded to. If all files are downloaded successfully, they are then moved to the product dir. The hashmap.txt file inside that folder is for hashes of the temp files incase an update gets interrupted before it finishes.

Your Win64_4_0_0_30_Alpha2021.04.28.265434 folder should have been deleted though had things worked properly. Let me know if that folder remains the next time you try an update.

If you'd like to force an update to happen, you can edit products/[product]/VersionInfo.txt and set the version number to be less than whatever it currently is and then delete/rename a file like products/[product]/AppConfig.xml.

A-UNDERSCORE-D commented 3 years ago

Sounds good! and I'd rather leave it until the next hotfix (unless you'd like a reprod now, anyway).

A-UNDERSCORE-D commented 3 years ago

Okay that worked perfectly (so far, anyway, just did a quick login after updating)

A-UNDERSCORE-D commented 3 years ago

Played a chunk more after that, no visible issues with the update.

rfvgyhn commented 3 years ago

Cool. I'll leave this open for a bit longer before I merge it in-case an issue crops up later. I'm still not positive the hash cache doesn't have a problem. If you could let me know how it works with an existing hash map file the next time you get an update, that would be great.

rfvgyhn commented 3 years ago

I guess it doesn't make much sense to wait for another update since the alpha is over. I've merged this into master and should have a new release soon. Feel free to open a new issue if problems are found.

kerberizer commented 3 years ago

Cool. I'll leave this open for a bit longer before I merge it in-case an issue crops up later. I'm still not positive the hash cache doesn't have a problem. If you could let me know how it works with an existing hash map file the next time you get an update, that would be great.

Unfortunately, due to an unrelated issue and the alpha ending I couldn't get a clear confirmation that hashing was working completely as intended, but it did look like it was, including in terms of idempotence. Overall, my impression was very good. The only minor thing was that I did need to create that PUBLIC_TEST_SERVER_64 symlink. I guess it remains to be seen if the release would still use the product_sku for the directory name instead of the more logical directory key.

rfvgyhn commented 3 years ago

I looked into the directory issue further and it turns out there's specific code to swap them depending on the platform. I've added that same logic (minus .rdr file support) which is included in the 0.4.0 release.

I'm unsure of the reason for it, but the logic looks like

If Steam or Epic, try the following paths

  1. Contents of first valid path in Products/[directory].rdr if it exists
  2. Products/[directory] if it exists
  3. Contents of first valid path in Products/[sku].rdr if it exists
  4. Products/[sku] else
  5. Contents of first valid path in Products/[sku].rdr if it exists
  6. Products/[sku] if it exists
  7. Contents of first valid path in Products/[directory].rdr if it exists
  8. Products/[directory]