search

07th-mod / resources

Resources used to play the patches
0 stars 0 forks source link

Moving our files to our own server #6

Open ItaloKnox opened 6 years ago

ItaloKnox commented 6 years ago

Calling @drojf and @enumag for discussion. Everyone else on the team is also welcome to comment (I won't ping since I don't know how many of you are interested in this).

Okay, so this issue was raised (once again) on Discord, about how Github hasn't been reliable for us for some time already. This is particularly true for our Umineko department since their binaries go over the 5gb mark. We also get issues with Github every now and then with Higurashi, so this seems to be a problem we all face and it directly affects the end user more than us, developers.

The issue:

  1. We're basically abusing Github with how we handle the binaries. I don't think this is correct and even though we are technically inside their ruleset, it is not a done deal and we might have issues in the future.
  2. Github is not a reliable place for our binaries since they are pretty big and will often timeout, making it difficult for people with slower connections (myself included) to finish download the patch in any method available (browser download or installer).

The solution:

Before I carry on with the options we have and how we would handle the access, let's go through the pros and cons of doing this.

Pros:

Cons:

How to handle dev access

Of course, the server access will not be available to the public or everyone in the group. The plan is to give access only to a handful of trusted members who can manage a Linux server through the command line. I can handle the initial setup and file migration, and then everyone with access can use the server to set up tools or anything that can be beneficial to us. Of course, the Big Brother will be watching to make sure no one is abusing it.

About the donations

My only condition to run a server would be to have it running on a yearly payment basis. We could set a donation drive every year a couple months before expiration and have everyone contribute to keeping the server alive. Supposing we don't set for a high-performance server (something we certainly don't need), the donation pot shouldn't go over $200 yearly. We can create a Paypal.me for that. It is important to be as transparent as possible with this because the last thing I want is someone using the funds incorrectly, something I think we can all agree that is unacceptable. If we all agree on doing this, we can discuss further on how to handle this in a way we can keep track of the money and how it is spent.

A side note: I don't think a Patreon would be a nice idea before anyone even thinks about suggesting it.

Important server specs

Now, the juicy part. Here are the most important things we should have on a server:

Stuff such as CPU, cores, and threads are less important since we will mostly run downloads.

Service suggestions

We need is something that is affordable. We can get an amazing server for $35, but I think we don't need that much, we can actually go with less than that. The first thing that comes to my mind is Kimsufi. I have some experience with them and they are pretty good, except for the connection speed that is only 10mb/s. That can be problematic when you compare it to Github's gigabit connection.

My low-end suggestion:

We can also go a bit higher:

Online.net also has a few plans with better connection speeds and good specs:

It is worth noting that those are all dedicated servers, and they all charge a fee from 10 to 20 USD/EUR for setup. This happens only once.

About Online.net, I never used it but after some research, they seem to be pretty good.

So, that's all I have to say for now. I think we should discuss if this is viable or not in our current condition. Let me hear your thoughts.

ItaloKnox commented 6 years ago

Oh, just an addendum: I think the Online.net option is the most attractive one.

enumag commented 6 years ago

Hmm that's actually quite affordable. It's a good idea in my opinion, I'm just slightly worried about speed from different places around the world and availability when the demand is high (like when we release a patch for new chapter). Would it be possible / good idea to keep using GH as a mirror? Online.net indeed looks better.

I also agree with patreon not being a good idea here. There is no need for monthly donations like on patreon. One time donations when we need to pay the bill sounds better.

ItaloKnox commented 6 years ago

I was also worried about Online.net datacenter being in France but I went ahead and pinged it from Brazil and USA, these are my results:

Brazil/Rio de Janeiro: ~300ms USA/Seattle: ~230ms

Not really good but for any web things we can have CloudFlare handle that with a free CDN.

I also tested the downloads and could easily get to my top speed at home and on my server. The server actually could download from there at whopping 60mb/s with 16 connections, and 20mb/s with a single connection. I guess we don't have anything to worry about?

Demand also shouldn't be a problem, we have enough RAM for that and the gigabit connection is as good as it gets.

Here, try it yourself if you are interested: http://ping.online.net/

ItaloKnox commented 6 years ago

Forgot about one thing: I think we should try to distance ourselves from GH as much as possible with these files. @Petsnew suggested a seedbox so I believe having torrents should be a good alternative in case someone doesn't want to make a direct download or the server somehow fails.

enumag commented 6 years ago

Yeah latency and speed looks good on my side too. And since it works fine on your side of the atlantic too, we shouldn't need GH then. 👍

drojf commented 6 years ago

I was thinking about the installer, and I realized that there are no steps which you couldn't do just by extracting an archive (for umineko, that is), if we have our own server.

Eg if we did have our own server, you could set up the following (theoretically):

  1. Automatically prepare an archive with the game files on the server each time a new version of our patch is released
  2. Distribute the archive by Bitorrent or otherwise
  3. User installs the archive by placing the archive in their game directory and extracting it.

That was the simplified version to get the gist across of what I want to do. In reality, it would be more like this:

  1. The user downloads 3 archives, depending on what version of the game they want to install:
    1. There would be a large 'common' archive used to install the game (one variant for voices only, and one for full patch). These wouldn't be regenerated very often. Effectively they would be created by having the game files on the server, and then running a script to compile them into an archive.
    2. There would be a small archive which is updated each time the script changes - there would be variants for full patch, voice only, and any other variants. A script on the server would download the latest 0.utf and whatever other files needed (like the patched .exe), and compile them into this archive.
    3. There would be a cross platform "extractor and verifier" zip which contains batch/shell script and PeaZip Portable (for all systems), which performs the extraction automatically. This would be a standard .zip file so you can extract it without any special software. The batch file and shell script could share the same codebase using Batsch. This would be regenerated each time the above two archives change, with new checksums (see step 4).
  2. User would download all archives using Bittorrent or otherwise
  3. The user manually moves the 3 archives to the game directory
  4. The user manually extracts the installer.zip and runs it. The installer then checks the validity of the archives (just a CRC check), and checks if the user placed everything in the right place. If correct, it extracts the files and installs the game. This prevents user errors like the files being in the wrong place, or partially downloaded files.

I basically want to do it this way to move as much of the work to the server side, so that there is minimal chance for stuff to go wrong on the user side. It also makes the manual install much easier, since you effectively only have to download and extract 2 archives. Also, the only point which may trigger any antivirus software are running the batch file, and running the PeaZip executable.

Anyway, I wrote this idea up both for myself to figure out any logical errors, but also to see if you guys thought there was anything wrong with it. I've only considered this for the Umineko installer. The current system works OK (besides the downloads), but I would like to make it more universal across all operating systems.

edit: I realize you could make the 'install script' also download the files, but I guess I just prefer to manage downloading stuff myself.

drojf commented 6 years ago

@Petsnew has suggested to try using some sort of syncing software - rsync, subversion, even bittorrent - to just synchronize the sever's files to the client's files.

If we go this route, I would go with subversion since it's (easily) cross platform, and has branching support, so theoretically you could just select the branch option you want and download it directly from the server. It would also give us file version management for our assets 'automatically'.

ItaloKnox commented 6 years ago

Building the patch on the fly is extremely difficult. It requires a lot of server horsepower and even servers that are a lot stronger than what we are going to get (talking about Gitlab here) have trouble generating big archives for download. Having a file list and then making the installer download them (think of it as an MMO patcher) would also be a step back in terms of speed at least, that's because Higurashi and Umineko have thousands of really small files and you will never reach top speed when there's a queue the moves all the time. It also implies no compression so the patches would grow up in size. The same would apply to generating a file for download, we would have to remove compression in favor of fast compilation, that would end up being a store zip.

You see, there's absolutely no problem with the format we're running right now. In fact, what makes it difficult to update the build is actually Github. If we didn't have to split the archive, I could easily update the build in a couple minutes with minimal downtime. We also wouldn't have to push the files back to Github so this would simplify things even further.

The new installer we're using in Higurashi is really efficient so if we combine it with some changes in the .bat we could basically eliminate a lot (if not all) issues we currently have with the patches. I can add Umineko to it but I didn't at first because I was told you planned to do your own thing.

drojf commented 6 years ago

Building the patch on the fly is extremely difficult.

I meant to do it only once per each time we update the patch, not for every time it is served to the user.

I agree with you about the syncing idea, that would definitely put a large strain on the server....

I'll try and integrate the batch files into the main installer for now. I did get a c# version of the installer working, but it would be difficult to release updates for that installer (since you would need to re-compile the installer each time the c# source code is updated). So I'll just use the same method as the other installers.

ItaloKnox commented 6 years ago

The new installer is fully capable of running our .bat files so you only need to adapt the screen messages and some other stuff you might want to integrate into the UI. The best thing about it is that we don't get flagged anymore since the dependencies and bat files are downloaded only when you start the installation. I think it turned out to be a good solution for us.

About compiling the files, I see. I guess I misunderstood, then. We could indeed do that but I'm a little against merging some files (such as graphics and voices) when we could serve them separately with no issues. We can, though, serve a torrent with both files in a single download, I think that's a nice way of dealing with this.

The biggest "issue" right now is the executables and the updates, but at least the updates can be merged with the graphics and kept updated with little to no effort (supposing we're running our own server). We should encourage the installer more as it is more reliable than ever. Streamlining the manual installation, even more, is something I don't think we need since most installs come from the installer right now.

drojf commented 6 years ago

I added both game's batch files to the new installer, which wasn't too painful, but there was a bug which I spent a while fixing. In the new installer, if you close the installer while it is running, it doesn't terminate the batch process. The batch file will continue to run in the background until you close it from task manager.

I guess I just wanted the manual installation streamlined for the linux/mac users, since the new installer is Windows only. However @Petsnew kinda has that covered (for linux at least) when he updated the installer script the last few days. But as you say, they don't make up the majority of users.

ItaloKnox commented 6 years ago

That's great, I suppose Higurashi is still supported in your branch and we're not splitting the installer in two? I'm okay with either, though.

I also have a few concerns about Unix, especially Mac. Hopefully switching to a server might help us automate things a bit and reduce the times we need to update the installer scripts for all platforms.

By the way, what's our decision about this? I'm all in for migration, this could be pretty big for us. The new Higurashi chapter is slowly getting closer so we need to be ready to deal with this before the workload increases.

drojf commented 6 years ago

If you can do it now, go for it. Once you setup a PayPal.me I'd be happy to contribute some initial funding to get it going.

It's one universal installer, not split up. Vocal did a good job in running the batch files so not many changes were required.

ItaloKnox commented 6 years ago

Since PayPal.me is not available in my country, I set up a new account (and email) in the US using the address of a PO box I have in the US. It seems to work and looks fine, but I'm unsure if this could potentially be an issue. Our yearly spendings would be around $200 max (if we include a domain for the group - a smarter choice than leaving the bare IP), so it's not a lot of money moving to be really concerned about this, but I want to know what you guys think about it. If it's a red flag, then we need to find someone who can take over at least the setup process.

drojf commented 6 years ago

If you're ok with it, I could setup the paypal.me account?

I guess i'm just afraid that paypal will lock our funds or something like that...have you tried doing anything similar to this before? I don't have any experience either so I don't really know...

Tsumihoroboshi will be released soon so probably you won't have much time to setup the server, but at least we can get this sorted out as to which option we will take.

ItaloKnox commented 6 years ago

Sure, I could leave it to you but somehow after thinking about it all this time I think there is really no issue? The account seems fine, the link is working. Didn't get anything since the day it was created, I think we are clear.

I'm used to moving money with Paypal, never had any issues with money being locked. I know they usually lock down when you receive a lot of money, but since that's not our case I think we have nothing to fear.

Supposing we're ok with proceeding with what we have and we meet our quota by Tsumi's release I think I can have it set up just in time for the graphics patch release. This would also be a good chance to rework Higurashi's folder structure in the files.

drojf commented 6 years ago

Considering what you said, fine by me. so if enumag agrees, you just have to put up the paypal.me link?

ItaloKnox commented 6 years ago

That's right. After we get the donations for the year I'll make a page to disclose all the donations, the current balance, and the spendings. I'm just not sure if we should keep the link up all the time. Maybe we can leave it working but only rally for donations once a year? This might be safer in the long run.

drojf commented 6 years ago

I would just leave the link up all the time, unless we get way too much donations.