search

ynqa / pandavro

Apache Avro <-> pandas DataFrame
MIT License
134 stars 32 forks source link

Pandas kwargs #10

Closed dkaslovsky closed 5 years ago

dkaslovsky commented 5 years ago

For the use case where one wants to construct a pandas.DataFrame using only a subset of avro fields (columns), it is more efficient to pass the columns kwarg to the DataFrame.from_records constructor rather than to load all fields into memory and filter. Similarly, one might wish to specify fields (columns) to exclude, although this use case is of secondary concern since it is unlikely that such an exclude list would be large enough to realize significant efficiency gains.

This PR exposes the kwargs of the DataFrame.from_records constructor in the read_avro/from_avro functions for this purpose.

ynqa commented 5 years ago

@dkaslovsky Thanks for your improvements! LGTM!