search

splunk / splunk-shuttl

Splunk app for archive management, including HDFS support.
Apache License 2.0
36 stars 19 forks source link

Integration testing with glacer #94

Closed petterik closed 11 years ago

petterik commented 11 years ago

Make sure that the glacier backend works by shuttling and thawing buckets to it.

So far shuttling has been tested, but not thawing, because it should take about ~4 hours to test.

Klevmarken commented 11 years ago

Did a single node test with 100MB buckets and CSV files in the size range of 1-7GB. Tested shuttling and thawing. Test passed; completed in 4 hours in the minute.

Klevmarken commented 11 years ago

Persisted 10+ SPLUNK_BUCKETS_TGZ to glacier. Thawed 10 SPLUNK_BUCKETS_TGZ. The thawed buckets have the following structure: /thaweddb/db_number_number/SPLUNK_BUCKET_TGZ. The following is an example of the bucket content:

^?^H^@^Y??P^@^C?? ^TE?ǻzzf29^H?? "??????c8"W??CA?%2Lf:Ɋ@f&̊A?Vߊ"(?ȥ"?"^P??UQ?UY?S^D^Q^D/N?å^K^T?^EV^Vدf?\<^E]??ϯ??H??? ????߮S^Y?^U?$ L^A^RÊ^K?$ L??????R,^R??ss^Y3b?G]་??F?＋^B??iȫU TCի^EU yC^V4]^Q$ͫ^Y??}?^?r^L^U|?㫋?^W%Y??$ ҁLE?!ӑ^YȑzȑLd}d^Cd^V?!2^[ّ^H?;?eȑˑ??M?W ?"?D6C? s?W!?#[ ["[![#ۑ ?F?^^y^M?-?^]2^O?DV=ӫBK@?H $ ?/ȊW??!G?D?^U? ?^F?M?j?^Z?Z?:?z?^F?[ȍȷ?????? ? ߷EnEnC?? ܁|^?^A?C?Gȏ?;?????{?{???? ?E~?܏<?<?<??^\?^E?K?Wȯ???G?߯ ??KV?ƊɊˊԊęi^Z^\;EQ?^D5ۙ&I??(ٙ6Y^Q$]ndUe0`?M??tZ???Z(4It_ݙ^_t?9e^4d?DQ^L?϶^Y?!JZ#?&??f˶^Pk^[???P#Ѷ{ֶR$

Klevmarken commented 11 years ago

You also have to spam on thaw in order to get several buckets to thaw. Would be good if all none thawed buckets would be thawed.

petterik commented 11 years ago

Made a test experience this. Created a failing test. Made it pass. Everything should work now with #105.

Klevmarken commented 11 years ago

Re-thawed 6 buckets. The following is the example content of one of them.

[eserv@sfeserv32 db_1350878964_1346986424_98]$ ls 1350853564-1350850852-6764658681876345039.tsidx 1350853575-1350852089-8830571220069019855.tsidx 1350853579-1350852101-2813574556680220879.tsidx 1350853601-1350852166-6172856512550039759.tsidx 1350853652-1350852177-2949141887958675663.tsidx 1350853658-1350852179-7802544362247712975.tsidx 1350853662-1350852180-7011217180788289743.tsidx 1350853668-1350852183-950934870622761167.tsidx 1350853678-1350852186-5257931596924743887.tsidx 1350853684-1350852186-5950445440762671311.tsidx 1350853693-1350852189-5256067568233377999.tsidx 1350853730-1350852192-6552306664014377167.tsidx 1350853731-1350852195-8244448981782917327.tsidx 1350853733-1350852198-5928484461229404367.tsidx 1350853738-1350852204-7517414153619796175.tsidx 1350853739-1350109021-2210595383506137295.tsidx 1350853748-1350852244-5124924517126271183.tsidx 1350853750-1350852252-7759907762338820303.tsidx 1350853751-1350508528-6307184599734710479.tsidx 1350853751-1350852255-2550076092572528847.tsidx 1350853757-1350852262-4844436090108539087.tsidx 1350853760-1350851350-5324010355968471247.tsidx 1350853769-1350852290-2808675506529005775.tsidx 1350877344-1350852096-6300649609425350863.tsidx 1350877379-1350852146-5359760512689139919.tsidx 1350877497-1350852286-3620211140476301519.tsidx 1350877516-1350852103-3180114624863301839.tsidx 1350877520-1349593200-265556466407076047.tsidx 1350877551-1350852170-8993647305777246415.tsidx 1350877674-1350221440-5055285449237491919.tsidx 1350877695-1350852292-4287086524933107919.tsidx 1350877696-1347693247-1378815498518556879.tsidx 1350877696-1350852126-3651657757146247375.tsidx 1350877698-1350852141-1612611798463181007.tsidx 1350877704-1350852174-6412265789289689295.tsidx 1350877706-1350852205-5627456757572858063.tsidx 1350877740-1350852201-5655537545811882191.tsidx 1350877852-1350791008-42899189914431695.tsidx 1350877902-1346986424-4679207942317174991.tsidx 1350878071-1350852224-1195654526531563727.tsidx 1350878537-1350457026-2992062531234705253.tsidx 1350878846-1350852175-998211538449887439.tsidx 1350878964-1350852287-1683377096971284687.tsidx bucket_info.csv Hosts.data optimize.result rawdata Sources.data SourceTypes.data splunk-autogen-params.dat splunk-need-optimize.dat Strings.data

Does everything look ok?

petterik commented 11 years ago

Beautiful :+1: