search

dennwc / cas

Content Addressible Storage
Apache License 2.0
41 stars 3 forks source link

cas pull "corrupting" internal store #6

Closed gebi closed 7 months ago

gebi commented 7 months ago

Hi,

Sry if i understood something wrong, i'm trying to use cas to store downloaded artifacts, version them and later use them in some software (as more powerfull replacement for the typical ad-hoc downloader shellscripts you see everywhere).

what works on first execution

% cas pull floodgate-spigot.jar https://download.geysermc.org/v2/projects/floodgate/versions/latest/builds/latest/downloads/spigot
% cas sync floodgate-spigot.jar
% cas checkout floodgate-spigot.jar ./floodgate-spigot.jar

but after executing a second time cas checkout does not work anymore

% cas pull floodgate-spigot.jar https://download.geysermc.org/v2/projects/floodgate/versions/latest/builds/latest/downloads/spigot
floodgate-spigot.jar = sha256:d6f3fb960861d6560259f894bd514fca37195c086d7f2c6800c4783d8cde2216
% cas sync floodgate-spigot.jar
floodgate-spigot.jar -> sha256:d6f3fb960861d6560259f894bd514fca37195c086d7f2c6800c4783d8cde2216 (up-to-date)
% cas checkout floodgate-spigot.jar ./floodgate-spigot.jar
Error: blob: invalid ref
<help msg>
2024/03/17 19:54:41 blob: invalid ref
%

the problem seems to be, that a pull where nothing is updated creates a @type: cas:WebContent blob that has an empty ref.

% find .cas/blobs -type f -size -500 |xargs -n1 grep .
...
{
 "@type": "cas:WebContent",
 "url": "https://download.geysermc.org/v2/projects/floodgate/versions/latest/builds/latest/downloads/spigot",
 "ref": "sha256:4aca4a66a2641967dcc4b895dd1a7453f76b47c239e139f494a80c69066e55f1",
 "size": 11235940,
 "etag": "09b0c6b5cc19a1618c0b30ad13327890c",
 "ts": "2024-02-18T14:41:25Z"
}
{
 "@type": "cas:WebContent",
 "url": "https://download.geysermc.org/v2/projects/floodgate/versions/latest/builds/latest/downloads/spigot",
 "ref": "",
 "etag": "09b0c6b5cc19a1618c0b30ad13327890c",
 "ts": "2024-02-18T14:41:25Z"
}

environment

golang upstream installed via godeb install 1.22.1 % go version go version go1.22.1 linux/amd64 cas installed via: % go install github.com/dennwc/cas/cmd/cas@latest

questions

beside that, is there a way in cas to see the history / log of what file versions a pin had? (to get the old state back quickly in case something broke) or is it just grep/jq into the index objects?

dennwc commented 7 months ago

Thank you for reporting! Will look into it :+1:

beside that, is there a way in cas to see the history / log of what file versions a pin had?

There's no builtin way to do that currently.

An an alternative, you could get a ref from the current pin and set it to a different one. If your sync is regular you could also set two pins: one with the current date and the second one for the latest state.

dennwc commented 7 months ago

@gebi Done, fixed the bug. Please reinstall latest CAS version. It should work now!

In general, you don't need a second pull command. The first time you pull it records a URL of the source, so that later sync can use it. In your case it always re-downloads the file. Sync is smarter as it tried to check if the content has changed before downloading it.

gebi commented 7 months ago

awesome, thx! works now!