search

dizzyd / mcdex

Minecraft Modpack Management
Apache License 2.0
74 stars 8 forks source link

Missing older minecraft versions for mods #28

Closed Magi1053 closed 6 years ago

Magi1053 commented 6 years ago

Not that I need it personally, but I was just querying some of the modfiles in the database, and I noticed for example in JEI that it only lists 25 files total for that mod, and all of them are from 1.12.2. The database doesn't have any files for older versions of minecraft for JEI.

image

dizzyd commented 6 years ago

Hi @Magi1053!

Yeah, this is the crawler trying to be nice and not go through every page. I'll take a look at making it more intelligent, but it might be a while since it would be a pretty big change. Do you think anyone else would actually use it?

Magi1053 commented 6 years ago

I suppose that depends. Do files in the database get pruned out if the crawler doesn't find them on the page? If they don't, then the crawler would only have to be modified one time to scan every page, and then when it runs daily updates it would only scan the first page. Since the program natively supports specifying an older minecraft version, it probably would be an issue if it didn't have mod files from that version.

LiaungYip commented 6 years ago

I think this can be done without using a web crawler.

Refer https://github.com/amcoder/Curse.RestProxy/wiki/Curse-Project-Feed :

Project Feed

The project feed is a set of bzip2-compressed json files that contains all of the add ons for a game.

For example, to get a list of all minecraft add ons, use:

http://clientupdate-v6.cursecdn.com/feed/addons/432/v10/complete.json.bz2

This will contain all mods, modpacks, maps, and resource packs hosted by Curse.

The Minecraft game id is 432.

The file complete.json.bz2 is a 5MB compressed file which expands to 96MB of JSON.

One entry, for the mod Immersive Engineering, is shown below.

Note that there is a section called GameVersionLatestFiles which appears to reference the most recent file available for each Minecraft version.

{
  "Id": 231951,
  "Name": "Immersive Engineering",
  "Authors": [
    {
      "Name": "BluSunrize",
      "Url": "https://www.curseforge.com/members/BluSunrize.aspx"
    },
    {
      "Name": "malte0811",
      "Url": "https://www.curseforge.com/members/malte0811.aspx"
    }
  ],
  "Attachments": [
    {
      "Description": "",
      "IsDefault": true,
      "ThumbnailUrl": "https://media.forgecdn.net/avatars/thumbnails/20/135/256/256/635707329671959611.png",
      "Title": "635707329671959611.png",
      "Url": "https://media.forgecdn.net/avatars/20/135/635707329671959611.png"
    }
  ],
  "WebSiteURL": "https://www.curseforge.com/minecraft/mc-mods/immersive-engineering",
  "GameId": 432,
  "Summary": "A retro-futuristic tech mod!",
  "DefaultFileId": 2559547,
  "CommentCount": 0,
  "DownloadCount": 20513849.0,
  "Rating": 0,
  "InstallCount": 0,
  "IconId": 1,
  "LatestFiles": [
    {
      "Id": 2330431,
      "FileName": "Immersive Engineering 0.10 - 42",
      "FileNameOnDisk": "ImmersiveEngineering-0.10-42.jar",
      "FileDate": "2016-09-16T06:25:05.58",
      "ReleaseType": 2,
      "FileStatus": 1,
      "DownloadURL": "https://files.forgecdn.net/files/2330/431/ImmersiveEngineering-0.10-42.jar",
      "IsAlternate": false,
      "AlternateFileId": 0,
      "Dependencies": [],
      "IsAvailable": true,
      "Modules": [
        {
          "Foldername": "META-INF",
          "Fingerprint": 672663953
        },
        {
          "Foldername": "blusunrize",
          "Fingerprint": 3189460725
        },
        {
          "Foldername": "cofh",
          "Fingerprint": 285800160
        },
        {
          "Foldername": "invtweaks",
          "Fingerprint": 783127171
        },
        {
          "Foldername": "vazkii",
          "Fingerprint": 2477178753
        },
        {
          "Foldername": "mcmod.info",
          "Fingerprint": 3648812616
        },
        {
          "Foldername": "pack.mcmeta",
          "Fingerprint": 2409430421
        },
        {
          "Foldername": "assets",
          "Fingerprint": 3325038261
        }
      ],
      "PackageFingerprint": 1547192360,
      "GameVersion": [
        "1.10.2"
      ]
    },
    {
      "Id": 2559547,
      "FileName": "Immersive Engineering 0.12 - 82",
      "FileNameOnDisk": "ImmersiveEngineering-0.12-82.jar",
      "FileDate": "2018-05-07T17:00:11.413",
      "ReleaseType": 1,
      "FileStatus": 2,
      "DownloadURL": "https://files.forgecdn.net/files/2559/547/ImmersiveEngineering-0.12-82.jar",
      "IsAlternate": false,
      "AlternateFileId": 0,
      "Dependencies": [
        {
          "AddOnId": 267602,
          "Type": 2
        }
      ],
      "IsAvailable": true,
      "Modules": [
        {
          "Foldername": "META-INF",
          "Fingerprint": 613926694
        },
        {
          "Foldername": "assets",
          "Fingerprint": 2695941184
        },
        {
          "Foldername": "blusunrize",
          "Fingerprint": 2239325091
        },
        {
          "Foldername": "invtweaks",
          "Fingerprint": 783127171
        },
        {
          "Foldername": "mcmod.info",
          "Fingerprint": 1448107709
        },
        {
          "Foldername": "pack.mcmeta",
          "Fingerprint": 2007758541
        }
      ],
      "PackageFingerprint": 3883997100,
      "GameVersion": [
        "1.12.2"
      ]
    }
  ],
  "Categories": [
    {
      "Id": 412,
      "Name": "Technology",
      "URL": "https:/www.curseforge.com/mc-mods/minecraft/category/technology"
    },
    {
      "Id": 424,
      "Name": "Cosmetic",
      "URL": "https:/www.curseforge.com/mc-mods/minecraft/category/cosmetic"
    },
    {
      "Id": 417,
      "Name": "Energy",
      "URL": "https:/www.curseforge.com/mc-mods/minecraft/category/technology-energy"
    },
    {
      "Id": 413,
      "Name": "Processing",
      "URL": "https:/www.curseforge.com/mc-mods/minecraft/category/technology-processing"
    }
  ],
  "PrimaryAuthorName": "BluSunrize",
  "ExternalUrl": null,
  "Status": 1,
  "Stage": 1,
  "DonationUrl": null,
  "PrimaryCategoryId": 412,
  "PrimaryCategoryName": "Technology",
  "PrimaryCategoryAvatarUrl": "https://media.forgecdn.net/avatars/6/27/635351494745973439.png",
  "Likes": 0,
  "CategorySection": {
    "ID": 6,
    "GameID": 432,
    "Name": "Mods",
    "PackageType": 6,
    "Path": "mods",
    "InitialInclusionPattern": ".",
    "ExtraIncludePattern": null
  },
  "PackageType": 6,
  "AvatarUrl": null,
  "GameVersionLatestFiles": [
    {
      "GameVesion": "1.10.2",
      "ProjectFileID": 2435559,
      "ProjectFileName": "Immersive Engineering 0.10 - 61",
      "FileType": 1
    },
    {
      "GameVesion": "1.10.2",
      "ProjectFileID": 2330431,
      "ProjectFileName": "Immersive Engineering 0.10 - 42",
      "FileType": 2
    },
    {
      "GameVesion": "1.11.2",
      "ProjectFileID": 2443553,
      "ProjectFileName": "Immersive Engineering 0.11 - 63",
      "FileType": 1
    },
    {
      "GameVesion": "1.12",
      "ProjectFileID": 2488329,
      "ProjectFileName": "Immersive Engineering 0.12 - 73",
      "FileType": 1
    },
    {
      "GameVesion": "1.12.1",
      "ProjectFileID": 2515333,
      "ProjectFileName": "Immersive Engineering 0.12 - 76",
      "FileType": 1
    },
    {
      "GameVesion": "1.12.2",
      "ProjectFileID": 2559547,
      "ProjectFileName": "Immersive Engineering 0.12 - 82",
      "FileType": 1
    },
    {
      "GameVesion": "1.7.10",
      "ProjectFileID": 2299019,
      "ProjectFileName": " Immersive Engineering v0.7.7",
      "FileType": 1
    },
    {
      "GameVesion": "1.7.10",
      "ProjectFileID": 2286655,
      "ProjectFileName": "Immersive Engineering v0.7.4",
      "FileType": 2
    },
    {
      "GameVesion": "1.8.9",
      "ProjectFileID": 2321292,
      "ProjectFileName": "Immersive Engineering 0.8 - 32",
      "FileType": 1
    },
    {
      "GameVesion": "1.8.9",
      "ProjectFileID": 2302876,
      "ProjectFileName": "Immersive Engineering 0.8 - 27",
      "FileType": 2
    }
  ],
  "IsFeatured": 0,
  "PopularityScore": 91815.8187,
  "GamePopularityRank": 12
},
dizzyd commented 6 years ago

Wowzers! That's a lot easier! Amazing find, @LiaungYip!

Let me dig into this, but it would probably make it possible for me to do proper dependency management as well as tracking all versions.

On a personal note, I'm getting married on Saturday, so it's probably going to be later next week before I get back to hacking. :)

LiaungYip commented 6 years ago

Congratulations! :tada:

dizzyd commented 6 years ago

I've been poking around this file a bit and we've got a problem - it looks like that file was last updated on June 8th, which means it's substantially out of date at this point. :/ I'm going to do a little tracing to see if the URL has changed, but it's likely this approach won't work after all.

Magi1053 commented 6 years ago

I tried doing some sniffing, and it looks like the twitch client now uses this api to get mods: https://addons-ecs.forgesvc.net/api/addon/search?gameId=432 The downside is that it requires an AuthenticationToken to use. If you want, I can dig some more to try to find the api request it uses to obtain this token.

dizzyd commented 6 years ago

Is is it the same file format? Or is this an API for searching?

Yes, any additional info you can gather on the API auth would be helpful, but I have a feeling that unless there's a way to get an API key, we're gonna have to depend on the crawler.

Magi1053 commented 6 years ago

https://www.twitch.tv/p/legal/terms-of-service/#7-license

(c) modifying or otherwise making any derivative uses of the Twitch Services or the Materials, or any portion of them; (d) use of any data mining, robots or similar data gathering or extraction methods; (e) downloading (except page caching) of any portion of the Twitch Services, the Materials, or any information contained in them, except as expressly permitted on the Twitch Services; or (f) any use of the Twitch Services or the Materials except for their intended purposes. Any use of the Twitch Services or the Materials except as specifically authorized in these Terms of Service, without the prior written permission of Twitch, is strictly prohibited and may violate intellectual property rights or other laws.

Magi1053 commented 6 years ago

I might be able to find a way for you to authenticate with the API for the purposes of generating the sql dump, but this licensing issue might get in the way. What are your thoughts on it @dizzyd?

dizzyd commented 6 years ago

Sigh

Yeah, this just isn't going to work. We're going to have to stick with the old way of doing things -- which works, just more manual labor for me. :)

Thanks @LiaungYip and @Magi1053 - I'll still take a run at deps, but it will be a while. :/

Magi1053 commented 6 years ago

I also just found this https://staging_cursemeta.dries007.net/docs

Looks like this is basically a wrapper for the curse api with baked in authentication and cached results up to an hour. (Prepare for slow load time) https://staging_cursemeta.dries007.net/api/v3/direct/addon/search?gameId=432&sectionId=6

Here are the available request parameters

gameId
sectionId
categoryId
gameVersion
index
pageSize
searchFilter
sort
sortDescending
Magi1053 commented 6 years ago

Dries007 handled all the licensing issues and claims to have been given permission by Twitch to create this api, so it should be safe to use internally for you to scan and use in your daily updates.

dizzyd commented 6 years ago

@Magi1053 - if you're ever on the Forge Discord, hit me up (I'm dizzyd).

dizzyd commented 6 years ago

OK - some good news. I've now worked out the details for tracking full deps and per-version latest files in an efficient way. I'm working on getting mcdex updated to use this info and hope to release in next week or so. :) Thanks @Magi1053 and @LiaungYip for all the suggestions. I will close out this ticket once the work is complete.

dizzyd commented 6 years ago

@Magi1053 and @LiaungYip - please give the latest mcdex a try. It should now have file versions for most Minecraft versions and full dep resolution! I'm sure there's bugs, but think it's mostly ready. :)