search

uskudnik / amazon-glacier-cmd-interface

Command line interface for Amazon Glacier
MIT License
374 stars 100 forks source link

inconsistency between search results and inventory output #13

Closed jose1711 closed 11 years ago

jose1711 commented 11 years ago

hello, do you have any explanation for this? i am using amazon-glacier-cmd all times and have uploaded both full backup and manifest.

$ ./glacier.py search duplic Region Vault Filename Archive ID us-east-1 ireland duplicity-full.20120906T215539Z.manifest.gpg somehash $ ./glacier.py inventory ireland | grep dupli duplicity-full-signatures.20120906T215539Z.sigtar.gpg 2012-09-07T07:26:36Z 1103441939 somehash

offlinehacker commented 11 years ago

Can you more precisely describe the problem?

jose1711 commented 11 years ago

i uploaded two files at different times: ireland duplicity-full.20120906T215539Z.manifest.gpg and duplicity-full-signatures.20120906T215539Z.sigtar.gpg. for some strange reason, manifest did make it into the simpleDB but did not make it into glacier (i tried inventory command several times during a period of 2 days) and vice versa for full-signatures file. i have no idea what could've caused this.

offlinehacker commented 11 years ago

Will take a look, this software is still in development, so there must be a bug.

jose1711 commented 11 years ago

i should probably add that iirc one of the files has been uploaded using an older version of amazon-glacier. but even the older version had the ability to send to simpleDB.

offlinehacker commented 11 years ago

I'm testing with identic file names this right now(invertory retrival in progress), will report.

jose1711 commented 11 years ago

thank you. in the meantime please correct me if i am wrong, but as soon as the results of inventory query are displayed - they should also be inserted into simpleDB, if not already there, correct? or: how could i fix this inconsistency using amazon-glacier-cmd?

offlinehacker commented 11 years ago

I'm sorry currently bookkeeping code is buggy and somehow incomplete. Looks like we save info about archived files in simpleDB and then we more or less forget about that. One day next week i will take time and finish this. Currently i can say only one thing, when you upload archive it's metadata gets saved and that's basicly it. When you retrive inventory, we put info about your retrival into SimpleDB(and i don't know why do we actually do that).

The problem is also we currently don't have tests. And that's currently more important, because more untested code means more problems latter, like you can see by yourself. Will try to fix that ASOAP.

offlinehacker commented 11 years ago

Sorry my bad, actually we delete entry from simpleDB when we delete archive, i still need to check what happens when we retrive inventory.

uskudnik commented 11 years ago

We save the list, but do not update the simpledb cache yet - thats when I ran out of time and what will be one of first updates. On Sep 11, 2012 3:03 PM, "Jaka Hudoklin" notifications@github.com wrote:

Sorry my bad, actually we delete entry from simpleDB when we delete archive, i still need to check what happens when we retrive inventory.

— Reply to this email directly or view it on GitHubhttps://github.com/uskudnik/amazon-glacier-cmd-interface/issues/13#issuecomment-8458745.

offlinehacker commented 11 years ago

+1

jose1711 commented 11 years ago

aaah.. i think i have an explanation for half of the story. when bookkeeping was detected as unset (https://github.com/uskudnik/amazon-glacier-cmd-interface/issues/12) the sigtar did not make it into simpleDB (only to glacier). the reason why manifest has not been archived to glacier but is stored into simpleDB is still a mystery

offlinehacker commented 11 years ago

There's one way to fix such inconsistencies. We can put the same data that we put into SimpleDB into description of archive. This way we can sync both(having command line option to do sync). This could be on option that user decide to use or not.

What I actually mean is that glacier archive description would be able to fix/recover SimpleDB entries and fix inconsistencies. What do you think?

jose1711 commented 11 years ago

i would definitely welcome the sync option..

uskudnik commented 11 years ago

At the time of writing I can't check the code right now since github appears to be down.

One reason for this could be that upload failed but instead of raising exception we still proceeded and saved it - or alternatively, we uploaded the message to simpledb before upload data and didn't delete after exception has been raised.

As per my status: I get my new laptop tomorrow and I hope to be operational by the weekend.

Cheers, Urban

On Tue, Sep 11, 2012 at 5:15 PM, jose1711 notifications@github.com wrote:

aaah.. i think i have an explanation for half of the story. when bookkeeping was detected as unset ( https://github.com/uskudnik/amazon-glacier-cmd-interface/issues/12) the sigtar did not make it into simpleDB (only to glacier). the reason why manifest has not been archived to glacier but is stored into simpleDB is still a mystery

— Reply to this email directly or view it on GitHubhttps://github.com/uskudnik/amazon-glacier-cmd-interface/issues/13#issuecomment-8463686.

wvmarle commented 11 years ago

I think this is pretty much a duplicate of #26. Close this one?

uskudnik commented 11 years ago

I believe so too. Will close for now.