search

GoogleCloudPlatform / appengine-mapreduce

A library for running MapReduce jobs on App Engine
https://github.com/GoogleCloudPlatform/appengine-mapreduce/wiki/1-MapReduce
Apache License 2.0
234 stars 109 forks source link

Java - Specifying all namespaces or one namespace #92

Open aswath-veersoft opened 8 years ago

aswath-veersoft commented 8 years ago

Hi, we have lot of namespaces in the datastore, and we want to change the schema of some KINDS. For this, we want to specify all the namespaces or one namespace as a parameter.

Currently, if the Map has to be run for all namespaces, then we have to create n no of jobs. After this, we have to monitor, if everything was fine or not by the jobid. This is very very inconvenient.

aswath-veersoft commented 8 years ago

anyway you can make the mapreduce easy for namespaces by specified wildcards or passing a list of namespaces.

aswath-veersoft commented 8 years ago

Any update on this.

aswath-veersoft commented 8 years ago

Any update on this.

aozarov commented 8 years ago

The MR Datastore input source is based on the datastore API which does not support cross name space queries. Indeed, it could have started with a metadata query to get all the name spaces and then for each namespace create a separate query to iterate over but I am not aware of any plans to do that so far. However, doing that in the application level and tying all the individual MR jobs in one pipeline job (using the AE Pipeline API - see this example) should not be too bad and allow the namespaces to be processed in parallel.

aswath-veersoft commented 8 years ago

Hi, Thanks. We don't know much about the new mapreduce. We are using namespaces for 5 years, and we are lacking a good way to do schema change. More java examples with namespaces will encourage teams to use mapreduce and also adopt namespaces. The dataflow team also lacks good support for namespaces. Namespaces support and tooling is always poor and is like an after-thought process with adjustments.
My 2 cents Thanks.

aswath-veersoft commented 8 years ago

Hi, if there are 10k namespaces, then we should start 10k jobs and monitor them. Instead, if there is one job, that can run all the namespaces specified, that is more elegant.

Thanks

aozarov commented 8 years ago

@aswath-veersoft I completely agree with you. It is just a matter of priority. As this project is open-source any contribution are welcome! Until then the only viable option that I can think of is using AE pipelines (as suggested before) to tie all the jobs together problematically (and then you can use the Pipeline UI to monitor them as one pipeline).

aswath-veersoft commented 8 years ago

Any news of supporting multiple namespaces in the mapreduce. It should be out of the box solution.
Looking forward to Google supporting this issue and make it easy for developers to use namespaces in the appengine applications.
Without proper support from all sides, namespaces is a very bad addition, and developers are loosers.

Thanks.

aswath-veersoft commented 8 years ago

Any help on this? Thanks.

ludoch commented 8 years ago

Hi, Sorry, we have no plans working on this but we do accept contributions from the community. Thanks.

aswath-veersoft commented 8 years ago

It is so bad on the appengine team to ignore the easy use of namespaces in a mapreduce. You are penalizing the small developers using namespaces by asking them to learn more and write more code. Why is namespaces support ignored?

aswath-veersoft commented 7 years ago

Contributions? Developers are looking for help and more support with examples on this mapreduce. Don't understand, why Google makes it opensource and pushes the onus on to the community. Its hardly 4-5 years you introduced mapreduce with a big annoucements and blogs, and now Google is already retracting on its support.