search

goatcorp / FFXIVQuickLauncher

Custom launcher for FFXIV
https://goatcorp.github.io/
GNU General Public License v3.0
2.76k stars 327 forks source link

Mirroring patch files #363

Open Poliwrath opened 3 years ago

Poliwrath commented 3 years ago

There are no goats in the launcher.

Hello! I'm making a Github issue rather than talking about this on the Discord in case it gets buried in ImGui chatter.

Would it be a good idea to have patch hashes + file URLs stored in the launcher, and the ability to download them from a list of mirrors? (discard / tell user to notify discord / download from patch-dl if file hashes don't match or whatever). The URLs would purely be for people interested in hosting their own mirrors without needing to login, and hashes would be for verification (iirc official patch lists just use crc32? I could be misremembering entirely, can't check right now)

I can probably implement this myself (terribly) and offer a mirror with SOME patches (at 100Mbps if there's too much traffic, which is probably fine for most people because yay slow internet, also my VPS doesn't have a lot of storage space), just curious if this is something I should bother attempting to do.

baaaahs

reiichi001 commented 3 years ago

I could see hosting patch files / URLs to SE's servers as a "we probably shouldn't do this" given that patch files for the game are locked behind auth. While it's no worse than the people offering torrents of the game install/patches, it could put the project in a "SE mad" spotlight for hosting their copyrighted data.

AmenneHolelane commented 3 years ago

I believe that it was discussed and vetoed for precisely that reason, Discord search sucks so i can't find the message

Poliwrath commented 3 years ago

The files themselves don't require authentication pretty sure (unless that was changed), anyone can download the patches if they know the URLs, just the file lists themselves (i.e "here are the patches you need to go from version to version") require authentication. I was debating on just hosting them on a MEGA and link it to whoever complained about speeds / ISP fuckery on the discord but there's a download limit (5GB per day or something lame) as well as storage limits...

goaaats commented 3 years ago

This would be really cool to have, with some decent mirrors, and you have a point for sure - I'm just not 100% on the legal perspective.

Maybe I'll read up on it a bit later, in the end we just have to ask ourselves if SE would care. If we require authentication to download still, we might be okay.

pigsflew commented 3 years ago

Possibly an overcomplicated suggestion, but could FFXIVQL use a torrent library like MonoTorrent to download from other FFXIVQL users, with SE's server treated as an HTTP Seed source?

The barebones idea would be to get the patch list from SE, and then either somehow generate a magnet url based on those details or have a hosted tracker, then use local service discovery to prioritize peers on the local network if, for example, there are multiple players in the same household.

otoh, there's the sheer complexity vs benefit to consider...

goaaats commented 3 years ago

Hey, I decided to attempt implementing this in XL on the bt_patches branch.

At the moment, the process goes as follows:

Ideal would be finding a way around the "downloading .torrent" requirement.

@pigsflew Do you have any comments/ideas on how to improve this? @Poliwrath Would you still be open to seed/host these?

Poliwrath commented 3 years ago

What's the full size of the torrent files?

AmenneHolelane commented 3 years ago

I can help seed as well

From: Poliwrath @.> Sent: Monday, June 14, 2021 10:02:10 AM To: goatcorp/FFXIVQuickLauncher @.> Cc: Amenne @.>; Comment @.> Subject: Re: [goatcorp/FFXIVQuickLauncher] Mirroring patch files (#363)

What's the full size of the torrent files?

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/goatcorp/FFXIVQuickLauncher/issues/363#issuecomment-860802189, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALCHDM4BEXTF3PYZGAUKCODTSYRYFANCNFSM4WSNY22A.

goaaats commented 3 years ago

It's about 70GB altogether - I doubt there would be much traffic since it's intended to be a "fallback".

AmenneHolelane commented 3 years ago

True but more people means redundancy:P

From: goaaats @.***> Sent: Monday, June 14, 2021 14:36 To: goatcorp/FFXIVQuickLauncher Cc: Amenne; Comment Subject: Re: [goatcorp/FFXIVQuickLauncher] Mirroring patch files (#363)

It's about 70GB altogether - I doubt there would be much traffic since it's intended to be a "fallback".

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/goatcorp/FFXIVQuickLauncher/issues/363#issuecomment-860975072, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALCHDM4QTYFQN3YTIZMLCGDTSZR6VANCNFSM4WSNY22A.

pigsflew commented 3 years ago

@Poliwrath the actual .torrent file is 56kb for the test file (1.5GB total download file size)

A patchset might include multiple files, but in general, you're looking at the .torrent files including some static header/footer info (164 bytes in the test file), plus about 20 bytes per 512k of actual download size. For Patch 5.3, the total download size was about 3GB, which would mean 123 kb of .torrent chunk hashes.

@goaaats YOU ROCK. This looks like an awesome initial implementation! Some thoughts:

Note: I'm rusty at C# so please tell me if I'm wrong about my understanding of your code!

Clients Seeding after download

Currently, your implementation seems to disconnect as soon as you've hit 100% download--which is fine, but it might also be useful to distributing load if clients stayed connected after finishing the download, either for a set amount of time, or just while the app is running, or something like that.

This might need the "close on launch" setting disabled...

Trackers

I think you're using trackerless on purpose, but I'm actually not sure if it's better to use DHT & Local only or actually use public trackers.

Bittorrent Extensions to take advantage of HTTP Seeding?

I'm not 100% on how to work this, but Monotorrent should support BEP19 for http seeding via a standard file host, and I think this should be able to take advantage of that? That would mean nobody here would need to seed all the time. The clients would distribute load, but the initial downloads would come from Sqex themself.

Additionally it could use a single tracker per patchlist, by concatenating all the pieces together and indicating the length in each fileset.

The full torrent file would look more like this:

{
  "created by": "FFXIV QuickLauncher v5.5.3",
  "comment": "Patch file for Final Fantasy XIV, with HTTP seed from Square-Enix's servers",
  "creation date": 1623705725,
  "info": {
    "files": [
      {
        "length": 1499625959,
        "path": [
          "game",
          "4e9a232b",
          "H2017.06.06.0000.0001a.patch"
         ]
      }
    ],
    "name": "FFXIV Patch Data",
    "piece length": 1048576,
    "pieces": "<hex> ... </hex>"
  },
  "url-list": [ "http://patch-dl.ffxiv.com/" ]
}

Then you'd bencode it and theoretically it should fall back to downloading from ffxiv.com in the torrent lib.

Again, i'm not 100% on this, so some amount of experimentation might be needed...

goaaats commented 3 years ago

Regarding seeding after download:

This would be nice to do, but technically, if you don't disable it in the settings, the patcher deletes patches immediately after they are installed. This is needed for people with small C drives to be able to patch & install patches, since otherwise you'd have 2 games at 70GB on your hard drive, basically. Maybe it could be an option, I wouldn't want to enable it by default either way. Would need some work to be sane.

Trackers

I'm not educated enough on this. DHT worked fine in our tests. I don't want to run a tracker, so we'd have to use a public tracker somewhere if we don't go trackerless.

WebSeed

This works fine with the FFXIV patch server(I tried), but in the end I got speeds about 30% slower than the managed HTTP downloader we have built-in anyhow with just WebSeed without any peers, so I sort of dismissed it. Is there another benefit to webseed that I'm not seeing?

reiichi001 commented 3 years ago

RE: Seeding after download, what I would recommend on this regard is if the user checks the "keep patch downloads" option, we could then allow them to enable/disable "seed patch downloads to other users" as an option.

Since users can technically change where patches are downloaded/saved from the default location of %appdata%\xivlauncher\patches, hard disk space shouldn't be that much of an issue. ...but we'd need to have some sort of sanity check like "only seed patches this user has downloaded" unless we want to enforce an all or nothing approach of "if you want to share patches to other users, you also need them all"

Poliwrath commented 3 years ago

It's about 70GB altogether - I doubt there would be much traffic since it's intended to be a "fallback".

Storage space is an issue for me right now, so I might not be able to seed everything. Maybe just the most recent ones, and MAYBE the H patches if I have enough room (I think they are / were problematic with certain ISPS? I can't remember).

pigsflew commented 3 years ago

The advantage would be that we wouldn't need to run seed hosts specifically, just host the tracker files and let XQL host what it has--if you fall back to pure http, you're no longer in the torrent cloud providing more redundancy of sources/bandwidth.

re: speed, I think the reason the built in downloader is faster might be because of some config difference between the httpclient used by the torrent and the xql default client? I wonder if the lib is only allocating one thread per seed, and treating the server like one seed... I'll poke around at that tonight to see if I can figure out how to improve.

You're right about DHT, trackers shouldn't be necessary :)

Re: disk space, could also use a patch cache max size, cleaning up the oldest patchsets until the folder is under the max. I'd leave mine with the most recent 15GB of patch data all the time.

pigsflew commented 3 years ago

oh my gosh.

It waits a whole minute before it tries the WebSeed. Would that account for the 30% slowdown?

TorrentSettings.cs#L85

We could possibly configure that down to something way lower, like maybe 10s?