Open Arcitec opened 2 weeks ago
Okay, I am now sure that the bug is that Bottles doesn't download new dependency hashes until you do the following routine:
I know this because the following process works. I just verified it:
This means that the real bug is that the hashes for the dependencies are not correctly updated in Bottles during the application lifetime. It must attempt to do an install (which probably fetches the new hashes), and then Bottles must be restarted to load the latest dependency download verification hashes.
Furthermore, you must try to install a dependency that frequently updates the binary on the server, so that the old hashes mismatch. VCRedist is a good candidate.
The reason I associated this bug with "newly created bottles" is that I only try to install dependencies after creating new bottles. ;)
So the fix is: After downloading the latest dependency hashes, LOAD THEM. ;)
Prior hash mismatch issues for vcredist2019
+ vcredist2022
were because their dependency manifests used a latest version download link instead of a version-specific download link. That was fixed in this PR last month:
https://github.com/bottlesdevs/dependencies/pull/259
Your hashes for the downloaded files on failed attempts seem to match the prior ones.
Bottles essentially clones https://github.com/bottlesdevs/dependencies on startup and uses that as its source of truth.
_You can override this by setting the LOCAL_DEPENDENCIES
environment variable to point to a local clone before starting Bottles in the same process. This is what's used to test dependency manifest changes._
In this case, it seems like Bottles' temp dependency installer cache is populated with an old binary. When Bottles tries to install vcredist2022
, it:
vcredist2022_x86.exe
and vcredist2022_x64.exe
files and re-uses them instead of re-downloading.So this seems to be a cache invalidation bug.
Relevant lines:
This seems to be the state on disk:
~/.local/share/bottles/temp/
├── vcredist2022_x86.exe (md5sum: 8457542fd4be74cb2c3a92b3386ae8e9)
└── vcredist2022_x64.exe (md5sum: 1d545507009cc4ec7409c1bc6e93b17b)
Fix probably requires changing the cache to be a content-addressable store instead where files have their hash (which should be switched from md5 to sha256 or something else to reduce the probability of collisions) somewhere in the path (either a parent directory name or in the file name).
Unsure if the maintainers are interested in reworking this given the current focus on Bottles Next.
@commiterate Ah thank you for your work investigating and fixing the hash URL issue.
I found something else that's super weird in this behavior.
Look at my "Troubleshooting Logs". I didn't notice it until now:
File [vcredist2022_x86.exe] already exists in temp, skipping. Source cksum: [822551b098e93c504de2c7865216928f] downloaded: [8457542fd4be74cb2c3a92b3386ae8e9]. Removing corrupted file [VC_redist.x86.exe].
File [vcredist2022_x64.exe] already exists in temp, skipping. Source cksum: [52f4f5a6adc24bce21dbceac2e2ad809] downloaded: [1d545507009cc4ec7409c1bc6e93b17b]. Removing corrupted file [VC_redist.x64.exe].
Anyway, I disagree that hashes would need to be part of the paths (content-addressable storage). Because Bottles lacks any way to install older versions of dependencies anyway.
All we really need to do to solve the bug is to change this:
File [vcredist2022_x64.exe] already exists in temp, skipping [the download and using local file].
To this:
File [vcredist2022_x64.exe] already exists in temp, but that local bad-boy has the wrong hash, so actually, nah, delete it and download it again!
Switching to a CAS is more to remove the current mechanism of dependency manifests needing to rename files to prevent collisions inside ~/.local/share/bottles/temp
.
With the way temp
and install_exe
/install_msi
work today, each dependency manifest needs to make sure its install_exe
/install_msi
step doesn't create file name collisions inside of temp
.
For example, both vcredist2019
and vcredist2022
download binaries that are originally named VC_redist.x86.exe
and VC_redist.x64.exe
.
If the manifest maintainer forgets to add a rename
directive in their install_exe
/install_msi
step, they'd clobber each other. That would produce a similar collision issue you're seeing, but instead of stale vcredist2022
installers creating collisions it would be up-to-date vcredist2019
installers creating collisions with vcredist2022
.
Invalidating the file in temp
and forcing a redownload would fix poisoned cache entries, but it can cause a lot of cache thrashing due to colliding file names. The cost of 100% cache misses isn't too much of an issue today since many installer binaries are only on the order of 10s to 100s of MBs, but it's a broken optimization nonetheless.
There's a slight problem with switching to a CAS since it changes checksums from optional to required in install_exe
/install_msi
. I wouldn't be surprised if there are installers which don't provide version-specific links and, for better UX at the expense of reproducibility and security, the dependency manifests will leave out a checksum as a result.
That's true, Bottles could just give each dependency recipe its own subdirectory, like temp/vcredist2022/VC_redist.x86.exe
to avoid clashes.
Describe the bug
Dependencies can never be installed in newly created Bottles without restarting Bottles first.
Update: See first comment below this. I found more details to track down the bug and have clarified.
To Reproduce
Package
Flatpak from Flathub
Distribution
Fedora Workstation 40
Debugging Information
Troubleshooting Logs
Log when it failed:
Log when you retry during the same failed session:
Log after Bottles restart, when it always succeeds:
Additional context
This is 100% reproducible in all new bottles. Leading me to believe that Bottles is hashing the wrong file or path when you have newly created a bottle.
Edit: Googling the downloaded hashes actually seems to bring up references to VCredist, so it seems more like Bottles is using the wrong hashes when verifying the downloads. But it's weird that it's only a problem for newly created bottles, and that it works after a Bottles restart.