search

dominictarr / json-select

MIT License
43 stars 3 forks source link

leveldb ideas #1

Open max-mapper opened 10 years ago

max-mapper commented 10 years ago

re: I'm thinking about a json-select like interface to leveldb where you can treat the whole database like it's just one single enormous json file

so one downside of geojson is that some feature collections may have a million features in them. to store this in leveldb would require breaking up each feature into a row (or even more than that)

I have been thinking for a little while about how to store large json in leveldb, shooting for the roughly 1-100kb sweet spot leveldb row size

what about taking this:

{ 
  "type": "FeatureCollection",
  "features": [
    { "type": "Feature",
      "geometry": {"type": "Point", "coordinates": [102.0, 0.5]},
      "properties": {"prop0": "value0"}
    }
  ]
}

giving it a UUID of say abc123, and storing this in leveldb:

abc123.type = FeatureCollection
abc123.features[0].type = Feature
abc123.features[0].geometry.type = Point
abc123.features[0].coordinates[0] = 102.0
abc123.features[0].coordinates[1] = 0.5
abc123.features[0].properties.prop0 = value0

this way (or some variant but you get the idea) json-select could implement its query api on top of leveldb (I think)

dominictarr commented 10 years ago

hmm, interesting... the same api could be wrapped around that too, although it would be more cumbersome if you had smallish json objects.