search

calrissian / accumulo-recipes

Recipes & cookbooks for Accumulo.
http://www.calrissian.org
Apache License 2.0
37 stars 18 forks source link

Pig loaders/aggregators/accumulators for each of the stores. #49

Open cjnolet opened 11 years ago

cjnolet commented 11 years ago

Many of these stores can be streamed into Map/Reduce. This also means they can be streamed/processed into Pig.

eawagner commented 11 years ago

+1

cjnolet commented 11 years ago

I have written MetricsStorage, EventStorage and EntityStorage modules for this. They are pretty simple and I'll copy them down when I get some time.

eawagner commented 10 years ago

I would close this and open individual tickets for each of the stores that are left.

jf-hopkins commented 10 years ago

Hi Corey! Did you ever get a chance to check in the modules you referenced above: EventStorage and EntityStorage? I'm just getting started with Pig stores and I'm sure your work would get me off to a great start!

cjnolet commented 10 years ago

Hey John,

Nice hearing from you! I did check in three loaders- Metrics, Events, and Entities. I have store funcs up here on my day job but I need to rewrite them down here for Calrissian so right now there is the ability to load data but not store it easily. I'd like to, at the very least, get an EventStoreFunc so that we could put analytics results back into the event store.

The loaders are located here:

store/entity-store/src/main/java/org/calrissian/accumulorecipes/entitystore/pig store/event-store/src/main/java/org/calrissian/accumulorecipes/eventstore/pig store/feature-store/src/main/java/org/calrissian/accumulorecipes/featurestore/pig

Unfortunately, we need better documentation for actually using them. We are using the Calrissian loaders now in our own Analytics Framework on my day job. Let me know if you have any questions about using them and I'll try to add some documentation as we go.

jf-hopkins commented 10 years ago

Hi Corey,

Yes -- I saw that you had done a lot of work with the loaders. That's awesome(!), because I'm very interested in those too. I won't make any promises about if/when, but we (on my end) are going to need to create some documentation for our day jobs also, and as the work will probably involve both loaders and stores, I'll pass that documentation on to you if it's allowed.

No worries about the store code not being available yet. I know you're busy! This is another timing issue on my side -- we may need to just go ahead and start writing our own very soon, so we'll start from scratch if we must.

fyi -- I'm working on the same project as Alan.

Kind Regards, -John

G2 Incorporated cell: 321-246-4828

On Fri, Jul 11, 2014 at 11:07 AM, Corey J. Nolet notifications@github.com wrote:

Hey John,

Nice hearing from you! I did check in three loaders- Metrics, Events, and Entities. I have store funcs up here on my day job but I need to rewrite them down here for Calrissian so right now there is the ability to load data but not store it easily. I'd like to, at the very least, get an EventStoreFunc so that we could put analytics results back into the event store.

The loaders are located here:

store/entity-store/src/main/java/org/calrissian/accumulorecipes/entitystore/pig

store/event-store/src/main/java/org/calrissian/accumulorecipes/eventstore/pig

store/feature-store/src/main/java/org/calrissian/accumulorecipes/featurestore/pig

Unfortunately, we need better documentation for actually using them. We are using the Calrissian loaders now in our own Analytics Framework on my day job. Let me know if you have any questions about using them and I'll try to add some documentation as we go.

— Reply to this email directly or view it on GitHub https://github.com/calrissian/accumulo-recipes/issues/49#issuecomment-48742103 .