Store map data in the data store in a compact, encoded form

MapQuest / mapquest-osm-server

A scalable OpenStreetMap API server built over a distributed key/value store.

Other

19 stars 4 forks source link

Store map data in the data store in a compact, encoded form #9

Open jkoshy opened 13 years ago

jkoshy commented 13 years ago

Store data in the data store encoded using protobufs. If done right, this change has the potential to reduce the storage requirements for the back end.

noblepaul commented 13 years ago

Try avro. It has the most frugal encoding.

jkoshy commented 13 years ago

A portability hurdle: building Avro from source requires that the build machine have java, ruby, php, python, c and c++ all installed on it, in addition to Apache ant and mvn. This is a bit much to expect from the typical developer!

I cannot also find pre-built packages for the Linux based OSes that I can point people to.

I had a quick look around in the Avro Python source tree: the build requires ant but this is used only to set the value of the AVRO_VERSION variable.

noblepaul commented 13 years ago

which linux are you looking for? You may shoot a mail to the user mailing list. I'm sure somebody would have built for that On 18 May 2011 15:58, "jkoshy" < reply@reply.github.com> wrote:

jkoshy commented 13 years ago

I searched for prebuilt packages for Ubuntu or Debian; packaged Avro libraries have not reached those repositories yet.

The download page for Avro has source code only; nothing in pre-packaged form.

Anyhow, for now I have forked the Avro tree, let me see if an alternative build scheme is easy to implement...

jkoshy commented 13 years ago

The jkoshy-build-rework branch of my fork of Avro now supports the following features:

The build.sh script works with a stock /bin/sh.
The build.sh script supports a -l LANG option, that allows builds to be restricted to language LANG.
The Python module can be built entirely using the standard installation tools for Python (i.e., distutils), without the need for an intermediate pre-processing step by ant.

noblepaul commented 13 years ago

This looks like a useful feature in general. You may contribute it back to avro itself . Raise an issue here

jkoshy commented 13 years ago

FYI, the following patches have been submitted to the Avro project:

AVRO-831 Enhance the Python distutils control file setup.py to build Avro/Python without a pre-processing step.
AVRO-829 Enhance the build.sh script to support building a subset of the available language bindings
AVRO-827 Use portable shell syntax for build.sh
(This is a minor patch which I submitted mostly to figure out how to use Apache's issue tracker.)

jkoshy commented 13 years ago

FYI: Google Protocol Buffers vs Apache Avro.

jkoshy commented 13 years ago

Noting another interesting project, for the record: MsgPack.