[minecraft-server] option to use forge

[danpolanco]

I'm going to try and figure this out and maybe fork and merge, but what do you think about having an argument to use a forge server? Or should that be a totally different image?

[danpolanco]

I can't find a Minecraft Forge version json file, so what might be easier is to prefer and only allow the recommended versions.

I'm not sure how to scrape that information... It would be nice if they supplied either a json or a URL redirect for recommended downloads.

[danpolanco]

Perhaps something like xidel would work.

Here is the html inspection for the version button pictured above:

That could get us all of the possible versions, and then we could generate a path from that perhaps? The data on the Minecraft Forge website is in a table.

I'm going to try out xidel and see what I can do with it.

Edit: There is also Scrapy, which depends on python.

[danpolanco]

What might be simpler is if I create a script that automatically generates a json file that you can consume. That would keep your Dockerfile, start.sh, etc. much cleaner. Let me know what you think.

Edit: To further clarify, I mean that the script I create wouldn't be inside your minecraft-server folder or anything like that. I would create a separate repository that had the script and the most up to date json file with version and download link.

[itzg]

I think the option for a Forge server is a fantastic idea. I waiting and watching (but then forgot) while they sorted our their 1.8 support. It was under that FML designation and was all a little odd. Packaging mods for FML 1.8 wasn't working for me, which added to my waning interest :)

Back to your follow up questions, yes, a nice, clean REST API on their part would be ideal. Scraping pages is not fun (what with HTML being not-XML), but I've resorted to that myself once or twice to get the job done.

The offline script to produce JSON is a tempting idea. I have pondered doing that for the vanilla versioning of this Minecraft image. For Chrome development I have been pleased with https://parse.com/ and was thinking of digging into their Cloud Code support.

With all that said, see if there's a way to script it within the container's startup, just so we can minimize adjacent dependencies. Depending on Python seems reasonable. I was half expecting it to already have snuck into my ubuntu-openjdk-7 image, but it's not. It would only add 16 MB itself:

The following extra packages will be installed:
  libpython-stdlib libpython2.7-minimal libpython2.7-stdlib python-minimal
  python2.7 python2.7-minimal
Suggested packages:
  python-doc python-tk python2.7-doc binutils binfmt-support
The following NEW packages will be installed:
  libpython-stdlib libpython2.7-minimal libpython2.7-stdlib python
  python-minimal python2.7 python2.7-minimal
0 upgraded, 7 newly installed, 0 to remove and 5 not upgraded.
Need to get 3734 kB of archives.
After this operation, 16.0 MB of additional disk space will be used.

BTW, I double checked and they have a non-adfly link alongside each of the download links, which is what the container would have to use.

Meanwhile, I'm going to poke around for a semi-RESTful interface into their file repo. I know bukkit has a nice once, but doesn't mean Forge does :)

[danpolanco]

Sounds great! I'm part way through a Scrapy script (doesn't work yet / still playing around):

import scrapy

class MinecraftVersion(scrapy.Item):
    url = scrapy.Field()

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LinkExtractor

class MinecraftForgeSpider(CrawlSpider):
    allowed_domains = ['minecraftforge.net']
    start_urls = ['http://files.minecraft.net']
    rules = [Rule(LinkExtractor(allow=['/minecraftforge/\d+\.\d+(\.d+)?', 'parse_minecraft_version']))]

    def parse_minecraft_version(self, response):
        minecraft_version = MinecraftVersion()
        minecraft_version['url'] = resonse.url
        return minecraft_version

[itzg]

I noticed they have "branch pages", such as http://files.minecraftforge.net/minecraftforge/1.8 , which might narrow the scraping needed.

[danpolanco]

Yup. I was able to get the two branches they have up:

[{"url": "http://files.minecraftforge.net/minecraftforge/1.8", "name": "http://files.minecraftforge.net/minecraftforge/1.8"},
{"url": "http://files.minecraftforge.net/minecraftforge/1.7.10", "name": "http://files.minecraftforge.net/minecraftforge/1.7.10"}]

[danpolanco]

Just so we don't lose any work: https://github.com/DanTheColoradan/minecraft-forge-scrapy.

[danpolanco]

I was able to scrape the Promotions portion of the Minecraft Forge downloads page. I can also easily get it to scrape other pages, but they can have quite a bit of information. It takes longer if I do more pages.

Here is a sample (pretty formatting mine):

[{
    "promotion": ["1.5.2-Latest"],
    "version": ["7.8.1.738"],
    "minecraft": ["1.5.2"],
    "downloads": {
        "Installer": "http://files.minecraftforge.net/maven/net/minecraftforge/forge/1.5.2-7.8.1.738/forge-1.5.2-7.8.1.738-installer.jar",
        "Universal": "http://files.minecraftforge.net/maven/net/minecraftforge/forge/1.5.2-7.8.1.738/forge-1.5.2-7.8.1.738-universal.zip",
         "Javadoc": "http://files.minecraftforge.net/maven/net/minecraftforge/forge/1.5.2-7.8.1.738/forge-1.5.2-7.8.1.738-javadoc.zip",
         "Src": "http://files.minecraftforge.net/maven/net/minecraftforge/forge/1.5.2-7.8.1.738/forge-1.5.2-7.8.1.738-src.zip"
    },
    "time": ["06/17/2013 10:36:00 AM"]
}]

The code is a bit complex right now since I used scrapy startproject but I think I should be able to squash it into one file.

Let me know what you think :)

Edit: I think it would be much nicer to get the "latest" and "recommended" for each version of Minecraft, but at least this is a start.

[danpolanco]

I feel like there should an easy way to get the "latest" and "recommended" info... They are using a script to determine which versions to display, but I haven't figured out how the script determines which are "latest" and recommended". Those tags have to be stored somewhere....

    <h1>Minecraft Forge Downloads</h1>
    <div class="mcversions">                                                     
      <script>                                                                   
        function display_relevent()                                              
        {                                                                        
          var opts = document.getElementById("view_version");                    
          var version = "all";                                                   
          for (var x = 0; x < opts.length; x++)                                  
          {                                                                      
            if (opts[x].selected)                                                
            {                                                                    
              version = opts[x].value;                                           
            }                                                                    
          }                                                                      
          for (var x = 0; x < opts.length; x++)                                  
          {                                                                      
            var display = opts[x].selected ? "block" : "none";                   
            if (version == "all")                                                
            {                                                                    
                display = "block";                                               
            }                                                                    
            var table = document.getElementById(opts[x].value + "_builds");      
            if (table)                                                           
            {                                                                    
              table.style.display = display;                                     
            }                                                                    
          }                                                                      
          var table = document.getElementById("promotions_table");               
          for (var i = 0; i < table.rows.length; i++)                            
          {                                                                      
            row = table.rows[i];                                                 
            data = row.cells[2].innerHTML;                                       
            if (version == "all" || data == "Minecraft")                         
            {                                                                    
              row.style.display = "table-row";                                   
            }                                                                    
            else                                                                 
            {                                                                    
              row.style.display = (data == version ? "table-row" : "none");      
            }                                                                    
          }                                                                      
        }                                                                        
      </script>  

[danpolanco]

I'm pretty sure I figured it out. It looks like they don't have "latest" and "recommended" for anything before 1.5.2. I'm going to see if I can just use the last build for any before 1.5.2.

[danpolanco]

Ok. So here is my new plan...

Problem: Anything before 1.5.2 does not have an installer and / or recommended build. Solution:

1. Start dictionary (Minecraft version = Forge source download link)
2. Fill all with Latest forge source link (i.e. Minecraft version 1.1 to 1.8)
3. Replace any that have Recommended builds (i.e. 1.5.2+)
4. Output json

You could then just grab the correct version of Forge for whatever Minecraft version asked for. Then, you would need to run the install.sh script to inject Forge into the Minecraft jar.

Working on this now...

[itzg]

I would actually recommend not bothering to support before 1.5.2 :) ; however, if you have a personal need to use a mod compiled against an older version, then go for it. Can always enhance it later, if needed.

Yeah, I poked around in their JavaScript. It does offer some hints, but they are rendering the actual version list server-side, so still needs to be scraped.

[itzg]

...did you have any thoughts about how users would install mod jars into a particular container? Since most mods are downloaded behind an adfly URL, a docker exec ... wget type approach won't always work. Otherwise, requiring attaching of /data to a host directory would probably be fine, right?

[danpolanco]

Hm. I don't have a need for anything pre-1.5.2. I mostly wanted to include it for the sake of completeness since it looks like you are able to get any version of Minecraft already?

And I hadn't given that thought yet! Attaching /data sounds ok to me.

[danpolanco]

I have finished up the script to a workable stage for you.

import scrapy
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LinkExtractor
from scrapy.selector import Selector

class MinecraftForgeDownload(scrapy.Item):
    url = scrapy.Field()
    promotion = scrapy.Field()
    minecraft = scrapy.Field()

class MinecraftForgeSpider(CrawlSpider):
    name = "MinecraftForgeSpider"
    allowed_domains = ["minecraftforge.net"]
    start_urls = ['http://files.minecraftforge.net']

    def parse(self, response):
        selector = Selector(response)
        rows = selector.xpath('//table[@id="promotions_table"]//tr')
        header = rows.pop(0)
        for row in rows:
            cells = row.xpath('td')
            minecraft = ''.join(cells[2].xpath('text()').extract())
            promotion = cells[0].xpath('text()').re('([a-zA-Z]+)')
            installer = cells[4].xpath('a[text()="Installer"]')
            url = installer.xpath('@href').re('http://adf.ly/\d+/(.+)')

            download = MinecraftForgeDownload()
            download['minecraft'] = minecraft
            download['promotion'] = promotion
            download['url'] = url
            yield download

This produces a json that you should be able to digest :) You can run the script with Scrapy installed:

scrapy runspider minecraftforge_spider.py -o latest.json

Edit: btw, it only does the promotions table from MinecraftForge, which means it is only for 1.5.2+

[danpolanco]

I'm thinking of formatting it as similar as I can to the amazon version you download. Would that be helpful?

[itzg]

Yeah, that would be great. Perhaps introduce new types, such as "forge-recommended" and "forge-latest", so it could look something like:

{ 
  "latest": {
    "snapshot": "1.8.2-pre7",
    "release": "1.8.3",
    "forge-recommended": "1.8-11.14.0.1299",
    "forge-latest": "1.8-11.14.1.1333"
  },
  "versions": [
    {
      "id": "1.8-11.14.0.1299",
      "time": "???",
      "releaseTime": "???",
      "type": "forge-recommended",
      "url": "http://files.minecraftforge.net/maven/net/minecraftforge/forge/1.8-11.14.0.1299/forge-1.8-11.14.0.1299-installer.jar"
    },
   ...

[danpolanco]

I was able to create a very similar layout for the Forge downloads:

[{
    "latest": {
        "forge_recommended": ["1.7.10"], 
        "forge_latest": ["1.8"]
    }, 
    "versions": [
            {
                "url": ["http://files.minecraftforge.net/maven/net/minecraftforge/forge/1.5.2-7.8.1.738/forge-1.5.2-7.8.1.738-installer.jar"], 
                "type": "forge_latest", 
                "id": ["1.5.2"], 
                "time": ["06/17/2013 10:36:00 AM"]
            }, 
            {
                "url": ["http://files.minecraftforge.net/maven/net/minecraftforge/forge/1.5.2-7.8.1.737/forge-1.5.2-7.8.1.737-installer.jar"], 
                "type": "forge_recommended", 
                "id": ["1.5.2"], 
                "time": ["06/15/2013 02:27:00 AM"]
            } 
    [...]
    }
}]

Would you like me to have python amalgamate what I produce with the Mojan json? I'm not sure how easy it'll be, but I can try if you like.

[danpolanco]

Whoops! I just need to fix the versions. I have the minecraft versions there instead of the forge versions.

[itzg]

Nah, the two separate, very extremely similar file formats will be just fine. The bootstrap script will need to decide vanilla or forge from the start anyway.

[danpolanco]

Great! Ok. How about the id?

Do you want:

"id": ["1.8-11.14.0.1299"]

or

"id": ["11.14.0.1299"]

In other words, do you want the minecraft version included in the id? I could also just make it a separate field:

"id":["11.14.0.1299"]
"minecraft":["1.8"]

A separate fields seems more flexible to me, but I'm not sure how you plan to implement this.

[itzg]

Since their id seems to be unique, I agree, let's go with separate fields to maximize flexibility.

[danpolanco]

Fantastic. Then I think most of what you need is in my repo:

The python Scrapy file is the following:

import scrapy
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors import LinkExtractor
from scrapy.selector import Selector

import re

class Forge(scrapy.Item):
    versions = scrapy.Field()
    latest = scrapy.Field()

class ForgeVersions(scrapy.Item):
    id = scrapy.Field()
    minecraft = scrapy.Field()
    type = scrapy.Field()
    time = scrapy.Field()
    url = scrapy.Field()

class ForgeLatest(scrapy.Item):
    forge_latest = scrapy.Field()
    forge_recommended = scrapy.Field()

class ForgeSpider(CrawlSpider):
    name = "ForgeSpider"
    allowed_domains = ["minecraftforge.net"]
    start_urls = ['http://files.minecraftforge.net']

    def parse(self, response):
        forge = Forge()
        forge['versions'] = []
        forge['latest'] = ForgeLatest()

        selector = Selector(response)
        rows = selector.xpath('//table[@id="promotions_table"]//tr')
        header = rows.pop(0)
        for row in rows:
            cells = row.xpath('td')

            id = cells[1].xpath('text()').extract()
            minecraft = cells[2].xpath('text()').extract()
            type = cells[0].xpath('text()')
            time = cells[3].xpath('text()')
            url = cells[4].xpath('a[text()="Installer"]/@href')

            #if has version
            has_version = re.match('(.+)\-.+', ''.join(type.extract()))
            if has_version:
                download = ForgeVersions()
                download['id'] = id
                download['minecraft'] = minecraft
                download['type'] = 'forge_' + ''.join(type.re('([a-zA-Z]+)')).lower()
                download['time'] = time.extract()
                download['url'] = url.re('http://adf.ly/\d+/(.+)')

                forge['versions'].append(download)
            else:
                is_recommended = re.match('Recommended', ''.join(type.extract()))
                if is_recommended:
                    download = ForgeLatest()
                    forge['latest']['forge_recommended'] = id
                else:
                    download = ForgeLatest()
                    forge['latest']['forge_latest'] = id

        return forge

My native language is C++ so feel free to fix my python code as needed. phew If you have an eta, let me know :) I'm quite excited. Also let me know if you need anything else. Once you incorporate the changes, I'll just delete my repo so we don't have extra info anywhere and so my github stays somewhat empty.

:beers:

[itzg]

How about you do a pull request with your script added under /minecraft-server. Then you'll get properly attributed for the contribution? I created a forge branch where you can aim the pull request.

I'll then follow up with the final changes, which I should be able to do sometime this week.

My native language is Java, so I won't be picky about your python :)

Thanks for all your help with this, BTW.

[danpolanco]

Sounds great! And no problem :+1:

[itzg]

Quick progress update:

I added installation of python and scrapy in this commit https://github.com/itzg/dockerfiles/commit/b4ec3cfb7112f9f207bdec6af969b275ab7f0ca8

...but I'm a little concerned about the increase in image size, where mc is the build against the forge branch:

REPOSITORY               TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
mc                       latest              5790d52f7022        4 seconds ago       669.7 MB
itzg/minecraft-server    latest              8a09745a5a4b        4 weeks ago         395.4 MB

I have been dabbling in go and might later play with their html parsing library :).

[danpolanco]

Wow! Yah. That's unacceptable. Out of curiosity, why go?

[itzg]

Good question :). It is very common in the Docker world and produces nice, statically linked binaries. I used to be a C/C++ developer, a long time ago, so there's a nostalgia factor too.

[danpolanco]

Kk :D Sounds like something I would be interested in then too. Have you tried the html parsing library yet? I might look at it in a bit.

[itzg]

I haven't tried it yet, but have been impressed with their other standard library/modules.

[danpolanco]

K. I'll give it a shot after I get some work done. Feel free to get it done before me though if you have free time.

[itzg]

Bingo! I was starting up Minecraft (with the Forge client) and noticed it stated a new version was available. I fired up Wireshark and found the conversation, below, with files.minecraftforge.net. So here's the magic JSON URL:

http://files.minecraftforge.net/maven/net/minecraftforge/forge/promotions_slim.json

GET /maven/net/minecraftforge/forge/promotions_slim.json HTTP/1.1
User-Agent: Java/1.8.0_31
Host: files.minecraftforge.net
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive

HTTP/1.1 200 OK
Server: nginx
Date: Wed, 04 Mar 2015 21:22:06 GMT
Content-Type: application/json
Content-Length: 1063
Last-Modified: Wed, 04 Mar 2015 19:36:43 GMT
Connection: keep-alive
ETag: "54f75ecb-427"
Accept-Ranges: bytes

{
  "homepage": "http://files.minecraftforge.net/minecraftforge/",
  "promos": {
    "1.5.2-latest": "7.8.1.738",
    "1.5.2-recommended": "7.8.1.737",
    "1.6.1-latest": "8.9.0.775",
    "1.6.2-latest": "9.10.1.871",
    "1.6.2-recommended": "9.10.1.871",
    "1.6.3-latest": "9.11.0.878",
    "1.6.4-latest": "9.11.1.965",
    "1.6.4-recommended": "9.11.1.965",
    "1.7.10-latest": "10.13.2.1291",
    "1.7.10-latest-1.7.10": "10.13.2.1307",
    "1.7.10-latest-new": "10.13.1.1216",
    "1.7.10-recommended": "10.13.2.1291",
    "1.7.10_pre4-latest-prerelease": "10.12.2.1149",
    "1.7.2-latest": "10.12.2.1147",
    "1.7.2-latest-mc172": "10.12.2.1161",
    "1.7.2-recommended": "10.12.2.1121",
    "1.8-latest": "11.14.1.1335",
    "1.8-latest-1.8": "11.14.0.1295",
    "1.8-recommended": "11.14.1.1334",
    "latest": "11.14.1.1335",
    "latest-1.7.10": "10.13.2.1307",
    "latest-1.8": "11.14.0.1295",
    "latest-mc172": "10.12.2.1161",
    "latest-new": "10.13.1.1216",
    "latest-prerelease": "10.12.2.1149",
    "recommended": "11.14.1.1334"
  }
}

[danpolanco]

Holy crap! hahah awesome!

[danpolanco]

Here is a sample link:

http://files.minecraftforge.net/maven/net/minecraftforge/forge/1.5.2-7.8.1.738/forge-1.5.2-7.8.1.738-installer.jar

So:

http://files.minecraftforge.net/maven/net/minecraftforge/forge/$minecraft-$forge/forge-$minecraft-$forge-installer.jar

Perhaps?

[itzg]

Yeah, that seems to be it. And then use their identifiers to find "latest", "recommended", etc

I haven't checked yet if we'll need the installer or universal jar.

[danpolanco]

I tried the installer on your Minecraft image and it started no issues.

The universal has a more complex setup:

"Open your minecraft.jar file and the universal zip file. Drop the contents of the zip file into minecraft.jar then delete the files in META-INF which begin with MOJANG.

To complete the forge installation - start minecraft and log in. Minecraft will then download some more instal files from http://files.minecraftforge.net/fmllibs/ and install them to .minecraft/lib. However, it will fail to find two files - bcprov-jdk15on-148.jar and scala-library.jar. For some reason these two files are on the files.minecraftforge.net but have been renamed scala-library.jar.stash and bcprov-jdk15on-148.jar.stash. So you need to manually download bcprov-jdk15on-148.jar.stash and scala-library.jar.stash to .minecraft/lib and remove the .stash extension from the file names. After doing this - restart minecraft and login again. The final forge setup phase should then complete.

If using a mod manager, place the forge zip under the jarmods section." Wiki

So I think the installer would be a better choice, yah?

[itzg]

Yep, installer will work out much better then. I vaguely remember it has a headless mode.

[itzg]

...sorry missed the first part of your last comment. Sounds like it's close then!

[danpolanco]

I think I'm getting close, but I'm having an issue with docker build. I'm not sure what is up, but I just wanted to let you know.

E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/c/cups/libcupsimage2_1.7.2-0ubuntu1.2_amd64.deb  404  Not Found [IP: 91.189.91.24 80]

[itzg]

I found I have to add an apt-get update on my images now.

[danpolanco]

Well, I'm not sure what's up, but if you get the chance, check out my forge branch. Docker seems to run it and then stop. I'm going to keep messing with it.

[danpolanco]

If I remove -e VERSION= or -e TYPE=, it looks like it starts working again...

[itzg]

I'm looking...but from here I usually launch the container running bash and fiddle with the script from there:

docker run -it --rm mc bash

and then

root@333b54269394:/data# bash -x /start-minecraft
+ '[' '!' -e /data/eula.txt ']'
+ cd /data
+ case $TYPE in
+ case $VERSION in
++ wget -O - https://s3.amazonaws.com/Minecraft.Download/versions/versions.json
++ jsawk -n 'out(this.latest.release)'
--2015-03-05 00:59:51--  https://s3.amazonaws.com/Minecraft.Download/versions/versions.json
Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.17.120
Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.17.120|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 21973 (21K) [application/octet-stream]
Saving to: 'STDOUT'

100%[===========================================================================================================>] 21,973      --.-K/s   in 0.05s

2015-03-05 00:59:51 (446 KB/s) - written to stdout [21973/21973]

+ export VANILLA_VERSION=1.8.3
+ VANILLA_VERSION=1.8.3
+ export SERVER=minecraft_server.1.8.3
+ SERVER=minecraft_server.1.8.3
+ '[' '!' -e minecraft_server.1.8.3 ']'
+ echo 'Downloading minecraft_server.1.8.3 ...'
Downloading minecraft_server.1.8.3 ...
+ wget -q https://s3.amazonaws.com/Minecraft.Download/versions/1.8.3/minecraft_server.1.8.3
+ '[' '!' -e server.properties ']'
+ '[' -n '' -a '!' -e ops.txt.converted ']'
+ '[' -n '' -a '!' -e server-icon.png ']'
+ exec java -Xmx1024M -Xms1024M -jar minecraft_server.1.8.3
Error: Unable to access jarfile minecraft_server.1.8.3

[danpolanco]

Thanks :+1:

[danpolanco]

That definitely helped. Now I'm able to debug. Quite a few problems.

[itzg]

Like missing the ".jar" suffix ;) I see you pushed the fix already.

[danpolanco]

Haha that was one xD Another is I'm not checking the vanilla version vs what version is on forge, so I get links that don't work:

http://files.minecraftforge.net/maven/net/minecraftforge/forge/1.8.3-11.14.1.1334/forge-1.8.3-11.14.1.1334-installer.jar

The 1.8.3 doesn't match the forge build.

[itzg]

Oh yeah, starting with 1.8, forge doesn't require matching up on the third version part, right?

[danpolanco]

If I understand your question correctly, I don't believe so.

Btw, should I keep going with fixing it? Or do you want to take over?

The debugging is going a lot faster now, so if you have other things to do, I can keep going.