search

gorilla-devs / ferium

Fast and multi-source CLI program for managing Minecraft mods and modpacks from Modrinth, CurseForge, and GitHub Releases
Mozilla Public License 2.0
1.15k stars 49 forks source link

Specify a limit for the amount of simultaneous threads #370

Open ItsProfessional opened 1 year ago

ItsProfessional commented 1 year ago

Description

When upgrading a ton of modrinth mods, this error occurs. This only started to occur today, it was working fine yesterday.

Steps to reproduce

  1. Install a lot of mods from modrinth
  2. run "ferium upgrade"

Operating System

Linux

Are you using the latest version of ferium?

Additional Information

Command output:

$ ferium upgrade
Determining the Latest Compatible Versions

✓ Better Controls                             BetterControls-Fabric-1.20+v1.3.0.jar
✓ ARRP                                        arrp-0.8.0.jar
✓ Debrand                                     debrand-1.1.1.jar
✓ Elytra/Chestplate Swapper (Fabric)          elytra-chestplate-swapper-1.3.0-MC1.20.jar
✓ 'Slight' Gui Modifications                  slight-gui-modifications-2.6.1.jar
✓ Cloth Config API (Fabric/Forge)             cloth-config-11.1.106-fabric.jar
✓ Architectury API (Fabric/Forge)             architectury-9.1.12-fabric.jar
✓ AppleSkin                                   appleskin-fabric-mc1.20-2.5.0.jar
✓ Ash API                                     ash_api-quilt-3.0.2+1.20.1.jar
✓ Better Ping Display [Fabric]                BetterPingDisplay-Fabric-1.20.1-1.1.1.jar
✓ Architectury API                            architectury-9.1.12-fabric.jar
✓ Armor Chroma for Fabric                     armorchroma-1.2.6.jar
✓ CraterLib                                   CraterLib-fabric-1.20-1.0.2.jar
✓ Enhanced Attack Indicator                   enhanced_attack_indicator-1.0.4+1.20.jar
✓ Animatica                                   animatica-0.6+1.20.jar
✓ Boosted Brightness                          boosted-brightness-2.2.0+1.20.1.jar
✓ Better Suggestions                          better-suggestions-1.20.1-1.1.1.jar
✓ Clean F3                                    CleanF3-0.4.0-mc1.20.jar
✓ Better Eating Mod                           BetterEatingMod-0.4.0.jar
✓ Better Taskbar                              bettertaskbar-fabric-mc1.20-1.7.0.jar
✓ Capes                                       capes-1.5.2+1.20-fabric.jar
✓ CleanCut                                    cleancut-fabric-6.0+mc1.19.1-mc1.20.x.jar
✓ 3D Skin Layers                              3dskinlayers-fabric-1.5.4-mc1.20.1.jar
✓ bad packets                                 badpackets-fabric-0.4.1.jar
✓ ChatWalk                                    chatwalk-1.0.2.jar
✓ Better Resource Pack Sorting                better_resource_pack_sorting-mc1.20-1.0.0-build.3.jar
✓ Blanket                                     blanket-client-tweaks-1.1.4.jar
✓ AntiGhost                                   antighost-1.20-fabric0.83.0-1.1.5.jar
✓ Borderless Mining                           borderless-mining-1.1.8+1.20.1.jar
✓ Better Mount HUD                            bettermounthud-1.2.2.jar
✓ CompleteConfig                              completeconfig-2.5.0.jar
✓ Boat Item View                              Boat-Item-View-Fabric-1.20.1-0.0.5.jar
✓ ClearDespawn                                ClearDespawn-fabric-1.20.1-1.1.13.jar
✓ Continuity                                  continuity-3.0.0-beta.3+1.20.1.jar
✓ ClickThrough                                clickthrough-1.20-fabric0.83.0-0.4.1.jar
✓ Better Advancements                         BetterAdvancements-Fabric-1.20.1-0.3.2.161.jar
✓ Camera Utils                                camerautils-1.20.1-1.0.5.jar
✓ Blocked server bypass for Fabric            mc-blacklist-bypass-1.0.3-1.20.1.jar
✓ Command Block IDE                           command-block-ide-0.4.3.jar
✓ Delete Worlds To Trash                      DeleteWorldsToTrash-v8.0.0-1.20.1-Fabric.jar
✓ Better Sodium Video Settings Button         bsvsb-3.0.3.jar
✓ Compact Chat                                compact-chat-2.0.2.jar
✓ Better Recipe Book                          brb-1.8.0-fabric.jar
✓ Better Nether Map                           nethermap-3.0.0-1.20.1.jar
✓ Axolotl Buckets                             axolotlbuckets-1.2.0+1.20.jar
✓ CalcMod                                     calcmod-1.2.0.jar
✓ Disable Custom Worlds Advice                DisableCustomWorldsAdvice-4.1.jar
✓ Amecs                                       amecs-1.3.9+mc.1.20.1.jar
✓ e4mc                                        e4mc-4.0.1+1.19.4-fabric.jar
✓ Auth Me                                     authme-7.0.2+1.20.jar
✓ Cloth Config API                            cloth-config-11.1.106-fabric.jar
✓ Litematica                                  litematica-fabric-1.20.1-0.15.3.jar
✓ MaLiLib                                     malilib-fabric-1.20.1-0.16.2.jar
✓ Blur (Fabric)                               blur-3.1.0.jar
✓ MiniHUD                                     minihud-fabric-1.20.1-0.27.0.jar
✓ Nicephore (better screenshots) [Fabric]     nicephore-fabric-1.19.3-1.0.1.jar
✓ Dynamic Crosshair                           dynamiccrosshair-7.0.4+1.20-fabric.jar
✓ clientcommands                              clientcommands-2.8.3.jar
✓ [EMF] Entity Model Features                 entity_model_features_fabric_1.20-0.2.13.jar
✓ Debugify                                    Debugify-1.20.1+2.0.jar
✓ DashLoader                                  dashloader-5.0.0-beta.2+1.20.0.jar
✓ Chat Heads                                  chat_heads-0.10.20-fabric-1.20.jar
✓ BetterF3                                    BetterF3-7.0.1-Fabric-1.20.1.jar
✓ Carpet Extra                                carpet-extra-1.20-1.4.115.jar
✓ CIT Resewn                                  CITResewn-1.1.3+1.20.jar
✓ CreativeCore                                CreativeCore_FABRIC_v2.10.28_mc1.20.1.jar
✓ Better Runtime Resource Pack (BRRP)         brrp-fabric-1.0.1-1.20.1.jar
✓ Ambient Environment                         AmbientEnvironment-fabric-1.20.1-11.0.0.1.jar
✓ Carpet-Fixes                                carpet-fixes-1.20-1.16.1.jar
✓ Dynamic Crosshair Compat                    dynamiccrosshair-compat-3.3+1.20.1.jar
✓ Better Statistics Screen                    betterstats-2.10.3+1.20.1.jar
✓ Carpet                                      fabric-carpet-1.20-1.4.112+v230608.jar
✓ Enhanced Block Entities                     enhancedblockentities-0.9+1.20.jar
✓ FabricSkyboxes                              fabricskyboxes-0.7.2+mc1.20.1.jar
✓ Entity Culling                              entityculling-fabric-1.6.2-mc1.20.jar
✓ Exordium                                    exordium-fabric-1.1.4-mc1.20.1.jar
✓ Equipment Compare                           EquipmentCompare-1.20.1-fabric-1.3.7.jar
✓ Force-Port-Mod                              forceport-1.1.0.jar
✓ Fast Items                                  fastitems-1.0.0-mc1.20.1.jar
✓ GamemodeOverhaul                            gamemodeoverhaul-fabric-2.2.0+1.20.1.jar
✓ Fabrishot                                   fabrishot-1.10.0.jar
✓ Hide Hands                                  hidehands-1.20.1-4.0.jar
✓ Hold That Chunk                             holdthatchunk-2.0.1+1.19.jar
✓ FastQuit                                    fastquit-3.0.0-beta.2+1.20+.jar
✓ Game Menu Remove GFARB                      gamemenuremovegfarb-fabric-mc1.19.4-2.1.2.jar
✓ Held Item Info                              held-item-info-1.5.0.jar
✓ FerriteCore                                 ferritecore-6.0.0-fabric.jar
✓ Falling Leaves                              fallingleaves-1.15.2+1.20.1.jar
✓ Full Brightness Toggle                      fullbrightnesstoggle-1.20.1-3.0.jar
✓ FabricSkyBoxes Interop                      fsb-interop-1.3.4+mc1.20-build.47.jar
✓ Fadeless                                    fadeless-1.0.1.jar
✓ Ksyxis                                      Ksyxis-1.1.jar
✓ Inspecio                                    inspecio-1.8.1+1.20.jar
✓ Krypton                                     krypton-0.2.3.jar
✓ Fast Portals                                fast-portals-1.0.3.jar
✓ Methane                                     Methane - 2.1.jar
✓ MixinTrace                                  mixintrace-1.1.1+1.17.jar
✓ Make Minecraft Music Great Again            mmcmga-1.20.1-1.0.jar
✓ Lithium                                     lithium-fabric-mc1.20.1-0.11.2.jar
✓ Horse Stats Vanilla                         horse-stats-vanilla-4.3.0.jar
✓ Memory Usage Screen                         memoryusagescreen-fabric-mc1.20-1.9.0.jar
✓ InvMove                                     InvMove-1.20-0.8.4-Quilt.jar
✓ Indium                                      indium-1.0.25+mc1.20.1.jar
✓ LambDynamicLights                           lambdynamiclights-2.3.1+1.20.1.jar
✓ Improved Fire Overlay                       improved-fire-overlay-1.20.1-v3.jar
✓ [ETF] Entity Texture Features               entity_texture_features_fabric_1.20-4.5.1.jar
✓ Make Bubbles Pop                            make_bubbles_pop-0.2.0-fabric-mc1.19.4+.jar
✓ MidnightLib                                 midnightlib-quilt-1.4.1.1.jar
✓ Collective                                  collective-1.20.1-6.65.jar
✓ Mixin Conflict Helper                       MixinConflictHelper-1.2.0.jar
✓ Model Gap Fix                               modelfix-1.14-fabric.jar
✓ Iceberg                                     Iceberg-1.20.1-fabric-1.1.15.jar
✓ Log Begone                                  Log-Begone-Fabric-1.20.1-1.0.8.jar
✓ Enchantment Descriptions                    EnchantmentDescriptions-Fabric-1.20.1-17.0.4.jar
✓ Tweakeroo                                   tweakeroo-fabric-1.20.1-0.17.1.jar
✓ Name Pain                                   namepain-1.5.2 fabric-1.20.x.jar
✓ Monsters in the Closet                      monsters-in-the-closet-1.0.3+1.20.jar
✓ NBT Autocomplete                            NBTac-FABRIC-1.20.1-1.1.4.jar
✓ More Culling                                moreculling-1.20-0.18.1.jar
✓ MoreChatHistory                             morechathistory-1.19.1-1.1.1.jar
✓ Moonlight Lib                               moonlight-1.20-2.8.12-fabric.jar
✓ Freecam (Modrinth Edition)                  freecam-fabric-modrinth-1.2.1+1.20.jar
✓ Forge Config API Port                       ForgeConfigAPIPort-v8.0.0-1.20.1-Fabric.jar
error sending request for url (https://api.modrinth.com/v2/project/fM515JnW/version): http2 error: stream error received: refused stream before processing any application logic
ItsProfessional commented 1 year ago

Temporary workaround: I made a patch to add a 50ms delay between checking each mod 0001-sleep.patch.txt

theRookieCoder commented 1 year ago

This issue is caused by too many requests being made simultaneously, and is sometimes a result of a flaky internet connection. Instead of adding a delay, it should also be possible to decrease the number of threads created (75 by default). When resolving latest versions: https://github.com/gorilla-devs/ferium/blob/340470c9f4fec5030b8bb09c2530dd7a581c7d95/src/download.rs#L122 When downloading mods: https://github.com/gorilla-devs/ferium/blob/340470c9f4fec5030b8bb09c2530dd7a581c7d95/src/subcommands/upgrade.rs#L76

ItsProfessional commented 1 year ago

Can you add an option to change that hardcoded 75 value as a command line flag in the next version?

owosoup commented 1 year ago

Same issue here; it's made the whole program useless. How did you apply that temporary workaround? Did you recompile after editing the code?

owosoup commented 1 year ago

Nevermind, I figured it out! First time compiling something on rust :)

Gryzle commented 1 year ago

Can we expect a patched release soon? I installed using Homebrew and integrated it into automation and therefore can only use a release available from Homebrew.

Gryzle commented 1 year ago

I managed to build from source and limit the threads both in the source files and using the "-t" attribute, but am still getting the error ONLY when using "ferium upgrade." Exact command: ferium -t 1 upgrade

Are there any solutions to this? It can't even download one mod at a time. Listing mods with verbose works, but upgrading does not.

EDIT: I found that the current limit is 64 mods (ironic) from Modrinth. I'm not sure how to implement the cooldown, and limiting the threads does not work for me. I am on macOS (I know, calm down- it's what I have).

ItsProfessional commented 1 year ago

are you sure you're using the compiled build? your comment only shows "ferium", without any path. Which means (not sure about mac, but on linux) it will just use the one from your system path, i.e. from your package manager.

ItsProfessional commented 1 year ago

I found that the current limit is 64 mods

False. It only started happening for me on about 120 mods iirc.

Gryzle commented 1 year ago

For me, it was 64 mods, tested on two computers, both running macOS Ventura, one using the latest released version of ferium, and the other using a current compiled version.

Gryzle commented 1 year ago

are you sure you're using the compiled build? your comment only shows "ferium", without any path. Which means (not sure about mac, but on linux) it will just use the one from your system path, i.e. from your package manager.

I compiled it using Homebrew. I verified that it was using the latest version and modified thread limit I put in place of 25 directly in the code.