duneanalytics / dune-client

A framework for interacting with Dune Analytics' officially supported API service
Apache License 2.0
85 stars 22 forks source link

Add Append to FileIO #37

Closed bh2smith closed 1 year ago

bh2smith commented 1 year ago

This will look a lot like the write method, but CSVs will require special treatment (so we don't add the column headers again).

    @skip_empty
    def _append(self, data: list[DuneRecord], name: str, ftype: FileType) -> None:
        with open(self._filepath(name, ftype), "a", encoding=self.encoding) as out_file:
            ftype.write(out_file, data)

Also want a @skip_empty decorator on _write and _append

Something like this: but I couldn't get the types right:

def skip_empty(func):
    """Decorator used on write methods"""

    def write_wrapper(self, data, name, ftype):
        if len(data) == 0:
            logger.info(f"Nothing to write to {name}... skipping")
            return
        func(self, data, name, ftype)

    return write_wrapper
bh2smith commented 1 year ago

this actually works perfectly for python >= 3.10

    # Decorator to skip write on empty data sets.
    # TODO - this cool decorator only works for python >= 3.10
    # @staticmethod
    def _skip_empty(write_like: WriteLikeSignature) -> WriteLikeSignature:
        def wrapper(
            self: FileIO, data: List[DuneRecord], name: str, ftype: FileType
        ) -> None:
            if len(data) == 0:
                logger.info(f"Nothing to write to {name}... skipping")
                return
            write_like(self, data, name, ftype)

        return wrapper

Unfortunately it fails for any earlier version with

 TypeError: 'staticmethod' object is not callable

Here is an issue about it:

https://bugs.python.org/issue43682