Use file name as archive description

[Jonny007-MKD]

I think it would be nice to use the file name (date, time, hash/rand, e.g. 20170309-140558-78a43) as archive description for all subordinated files.

These two files belong together and could be grouped by the description:

... |                         Archive Description                         |       Uploaded       | SHA256 tree hash | ...
... | /temp/iceshelf/20170309-140558-78a43.tar.gpg.sig.vol01+2.par2.sig   | 2017-03-09T14:21:32Z | 294e1fca1...     | ...
... | /temp/iceshelf/20170309-140558-78a43.tar.gpg.sig.vol03+4.par2.sig   | 2017-03-09T14:21:48Z | 6dd92125e...     | ...

[mrworf]

I'm not sure I understand, it will re-use the same vault, and all the files generated are prefixed with the unique id of the job. Latest version of iceshelf fixed that (previously, filelist.txt was not unique, but now it is)

[Jonny007-MKD]

Hi Henric, I would like to suggest to use the unique ID also as archive description (glacier-cmd upload V... --description "ID")

As I've seen right now on https://github.com/uskudnik/amazon-glacier-cmd-interface/blob/85ef4aa6dd58a36a77e4406366ddc27c7813cf5e/README.md, you can also pass several file names with 1 description

Am 12. März 2017 17:48:59 MEZ schrieb Henric Andersson notifications@github.com:

I'm not sure I understand, it will re-use the same vault, and all the files generated are prefixed with the unique id of the job. Latest version of iceshelf fixed that (previously, filelist.txt was not unique, but now it is)

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/mrworf/iceshelf/issues/6#issuecomment-285957249

[mrworf]

Ahh, oki :)

I'll see what I can do. Should be able to get that in today.

[Jonny007-MKD]

Great! Thanks :)

Am 12. März 2017 19:25:31 MEZ schrieb Henric Andersson notifications@github.com:

Ahh, oki :)

I'll see what I can do. Should be able to get that in today.

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/mrworf/iceshelf/issues/6#issuecomment-285963607

[mrworf]

Please try the latest version, it should use description AND remove the path from the filenames.

[Jonny007-MKD]

This version gives me a "unknown arguments" error from glacier-cmd and then lists the files. Seems like passing several file names is not supported!?

[mrworf]

Odd, try updating to the latest version of the glacier tool. I*ll look into it once I get home.

On 2017-03-13 14:05, Jonny007-MKD wrote:

This version gives me a "unknown arguments" error from glacier-cmd and then lists the files. Seems like passing several file names is not supported!?

-- You are receiving this because you commented. Reply to this email directly, view it on GitHub [1], or mute the thread [2].

*

Links:

[1] https://github.com/mrworf/iceshelf/issues/6#issuecomment-286243583 [2] https://github.com/notifications/unsubscribe-auth/ABw0IoQG_okoXtRCvTaaSLJjcaCfoEacks5rla-xgaJpZM4MaT4D

[Jonny007-MKD]

I checked it and I'm on master and up-to-date for glacier-cmd. perhaps we cannot specify --description and/or --name more than once and the error message is wrong?

Am 13. März 2017 22:09:28 MEZ schrieb Henric Andersson notifications@github.com:

Odd, try updating to the latest version of the glacier tool. I*ll look into it once I get home.

On 2017-03-13 14:05, Jonny007-MKD wrote:

This version gives me a "unknown arguments" error from glacier-cmd and then lists the files. Seems like passing several file names is not supported!?

-- You are receiving this because you commented. Reply to this email directly, view it on GitHub [1], or mute the thread [2].

*

Links:

[1] https://github.com/mrworf/iceshelf/issues/6#issuecomment-286243583 [2]

https://github.com/notifications/unsubscribe-auth/ABw0IoQG_okoXtRCvTaaSLJjcaCfoEacks5rla-xgaJpZM4MaT4D

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/mrworf/iceshelf/issues/6#issuecomment-286244575

[mrworf]

You're right, only works once 😞 so for now it's gone and it's only going to set the description.

[mrworf]

Latest version should work now.

[Jonny007-MKD]

Hey, it is indeed working :) Now my inventory looks like this:

QGxSSyaOpEtOnT9UUQlw | 20170328-100633-16524 | 2017-03-28T11:58:18Z |
dKmaRnlUwFiIZYy_Wb_w | 20170328-100633-16524 | 2017-03-28T11:58:20Z | 
SOMqueK7vJS19zpsCIgQ | 20170328-100633-16524 | 2017-03-28T11:58:20Z | 
mNpDtkWDMJT7kX5mg_sg | 20170328-100633-16524 | 2017-03-28T11:58:21Z | 
RxpNX9cp84dxj4TYMlLQ | 20170328-100633-16524 | 2017-03-28T11:58:21Z | 
cNiqEgBfOUxD8iJSWGuQ | 20170328-100633-16524 | 2017-03-28T11:58:22Z | 

Do you know whether Glacier remembers the file extension? Or can we afterwards automatically figure out which item is what?

[mrworf]

Odd, there should be a description AND a filename, not just the description

[Jonny007-MKD]

That's what I thought, too. In my inventory I've got the columns

• Archive ID
• Description
• Uploaded (timestamp)
• SHA256 tree hash
• Size

[mrworf]

If you can, try downloading the smallest one and see if you get the actual filename. Otherwise we have a slight issue. Should be able to reconstitute the extension based on header in file, but I'd rather not do that

[Jonny007-MKD]

Here is what I tried:

$ glacier-cmd ... download Music-Backup 61ZVotZte1tIZF_J0waSzobzVfFmH-Zj0eThIN42f2Q9LKTHtR5Chk-wdAZbDJenGQFFaE-CVqCMSYXBeQXlkOmtk_JMfWqklX3CHDhL4ZJ7do_HQVJI4GmNpDtkWDMJT7kX5mg_sg
334d4a393a19356b740e36227013a2069839066c  20170328-100633-16524.tar.bz2.gpg.sig.par2.sig
2abedfe7072338b2c48d532f49ada07cf4f66f91  20170328-100633-16524.tar.bz2.gpg.sig.vol07+8.par2.sig
87129da51fd2d5907ba8b3f1040d697e9fe9b3fe  20170328-100633-16524.tar.bz2.gpg.sig.vol03+4.par2.sig
2c13150e0b9fb51a04bd7fd937174e2406de8790  20170328-100633-16524.json.gpg.asc
[...]
Wrote 731.0 bytes. Rate 2.48 KB/s.00%). Rate 3.50 MB/s, average 2.50 KB/s, ETA 06:34:13.

Seems like I downloaded the list with filenames, whoever created it. So manually we could download all files with the same description, parse this special file and use the hashes to get the file names.

[mrworf]

That's actually iceshelf. To avoid exposing what's in your archive but still quickly check it's "OK" and that you have all files, it creates a .lst file which can be used with shasum. But I'm none too happy with the loss of filenames. That was not the intention of this change (beats himself over the head for not confirming this).

After some more digging, it seems that description and filename are one and the same here and most likely it's just for glacier-cmd's own benefit it has both. This makes it quite useless though. I always assumed the filename would be stored separately, but I guess not.

So should I revert to oiriginal behavior? I can remove the path of the file and just store the name itself, and since it's all prefixed with the ID of the archive, it should still be fine.

Thoughts?

[Jonny007-MKD]

I totally agree with you, and I'm fine with your suggestion :) The path is quite useless, but with the filename we get the ID and the extension(s).

[mrworf]

Alright, cool, I'll fix that tonight (running on PST time here) so expect an update in ~10hrs.

Btw, if you want a more direct communication, send me an email and I'll invite you to the slack channel

[Jonny007-MKD]

Any news on this? :)

[mrworf]

Sorry for lack of update, I'm just running a test to confirm that it does indeed upload the expected data to glacier before I push the change. Should be done in a few hours (waiting for inventory ...zzzzz)

[mrworf]

So I'm having some issues right now getting any sensible results from this. I see my files being uploaded, but I cannot make glacier-cmd's inventory call actually reflecting this. Even when I force it with --refresh it still lists old content from 2015. Have you seen this issue?

[Jonny007-MKD]

Yes, sometimes I only get the old inventory, even if I see that the new inventory job completed. But after a while glacier-cmd forgets about the old job...

[mrworf]

depressing :( oh well, it does work, I've seen it upload and issue the correct commands, so I'm just going to cleanup and then push it.

[mrworf]

I wish github had a messaging system. I have a slack for this already, if you send me your email to github@sensenet.nu I'll invite you.

[mrworf]

Alright, it works, and I also found the issue in glacier-cmd (for inventory), so it might be time to fork and maintain that too since it doesn't seem like the original owner have time anymore.

[Jonny007-MKD]

Great, thanks! :) If you fork glacier-cmd I'll happily follow you!

[mrworf]

We'll see :) Might do it for this fix at least. Closing this issue now.