Edit: After looking at it again more closely, this is yet another race condition, similar to #755, but this time affecting Vanilla
assets.
This happens because minecraft/icons/minecraft.icns and icons/minecraft.icns both use the same remote path.
The download task will download the same file twice, and if it's unlucky, two threads will be writing to the same file on disk, resulting in a corrupt download.
~~Sometimes asset just fail to download correctly. Since it involves remote access, this can be expected.
On CI/CD servers this can be very annoying since it leads to false-positive build failures.~~
It would be nice for one of the following:
Use JDK11 HttpClient in HTTP/2 mode to support pipelining and remove the need to do 8 parallel downloads
After checking, resources.download.minecraft.net does not seem to support HTTP2 for some reason (S3?)
Make the parallel download count configurable, since this could be the issue in the CI/CD cloud environments (this is pure speculation)
Evaluate HTTP status codes correctly, this could explain the failed hash check (server error response written to file, instead of the actual asset)
status codes 400 and up should already lead to IOExceptions, it's unclear if the server uses some other weird status code to indicate rate-limiting(?)
Configurable retry count
Example issue from a GitHub Actions build:
Downloading: https://resources.download.minecraft.net/99/991b421dfd401f115241601b2b373140a8d78572 Asset: icons/minecraft.icns
Downloading: https://resources.download.minecraft.net/99/991b421dfd401f115241601b2b373140a8d78572 Asset: minecraft/icons/minecraft.icns
icons/minecraft.icns Hash failed.
minecraft/icons/minecraft.icns Failed.
java.io.FileNotFoundException: /home/runner/.gradle/caches/forge_gradle/assets/objects/99/991b421dfd401f115241601b2b373140a8d78572 (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at net.minecraftforge.gradle.common.util.HashFunction.hash(HashFunction.java:63)
at net.minecraftforge.gradle.common.task.DownloadAssets.lambda$run$0(DownloadAssets.java:70)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
shartte
commented
3 years ago
Edit: After looking at it again more closely, this is yet another race condition, similar to #755, but this time affecting Vanilla assets.
This happens because
minecraft/icons/minecraft.icnsandicons/minecraft.icnsboth use the same remote path. The download task will download the same file twice, and if it's unlucky, two threads will be writing to the same file on disk, resulting in a corrupt download.~~Sometimes asset just fail to download correctly. Since it involves remote access, this can be expected. On CI/CD servers this can be very annoying since it leads to false-positive build failures.~~
It would be nice for one of the following:Use JDK11 HttpClient in HTTP/2 mode to support pipelining and remove the need to do 8 parallel downloadsAfter checking, resources.download.minecraft.net does not seem to support HTTP2 for some reason (S3?)Make the parallel download count configurable, since this could be the issue in the CI/CD cloud environments (this is pure speculation)Evaluate HTTP status codes correctly, this could explain the failed hash check (server error response written to file, instead of the actual asset)status codes 400 and up should already lead to IOExceptions, it's unclear if the server uses some other weird status code to indicate rate-limiting(?)Configurable retry countExample issue from a GitHub Actions build: