GlobalFishingWatch / gpsdio

AIS I/O with Python, dictionaries, and the GPSd AIVDM schema.
Other
6 stars 3 forks source link

Allow gpsdio.open() and drivers to operate on already open files #118

Closed geowurster closed 9 years ago

geowurster commented 9 years ago

Closes #104.

@pwoods25443 we may find that this does not completely close #104 once we actually test it, but its a good start, and worth implementing anyway.

Supports syntax like:

import gzip
import gpsdio

with gzip.open(path) as gz, gpsdio.open(gz, compression='GZIP', driver=<name>) as src:
    print(next(src))

Which we need for Luigi's file targets:

import luigi
from luigi.contrib.gcs import GCSTarget
import gpsdio

class Convert(luigi.Task):
    infile = luigi.Parameter()
    outfile = luigi.Parameter()

    def input(self):
        return GCSTarget(self.infile)

    def output(self):
       return GCSTarget(self.outfile)

    def run(self):
        with gpsdio.open(self.input.open(), driver='NewlineJSON') as src,
                 gpsdio.open(self.output.open('w'), 'w', driver='MsgPack', compression='GZIP') as dst:
            for msg in src:
                dst.write(msg)

Not very pretty, but it works.