developmentseed / planet-stream

Stream the planet!
BSD 3-Clause "New" or "Revised" License
31 stars 9 forks source link

Kinesis cleanup #7

Closed matthewhanson closed 8 years ago

matthewhanson commented 8 years ago

Reorganization of the planet-stream repo including

kamicut commented 8 years ago

This pull request has some great ideas. However I think that planet-stream should be similar to a unix utility that outputs JSON delimited data, that you can include in other unix pipelines. You can also include planet-stream as a node module in your code and call '.onValue' on the initialized stream. By separating concerns, we can test planet-stream separately from kinesis.

Using kinesis as an output with planet-stream is an example of its use, and not the main output of the project. I would keep the simulator and refactor it to also output to STDOUT and STDERR. Here's an example project that has a bin, lib and can be included as a node_module https://github.com/mapbox/osm-fetch

In this case, there would be two bin: planet-stream [args] planet-stream-simulator [args]

The kinesis work would stay in the /examples/ folder, or be moved to another repo.

kamicut commented 8 years ago

@matthewhanson thanks. I'm thinking that the binary should have an argument toggle (--hashtags) to either get all data or filter. I'll work on that.