fix read_avro kwargs + cleanup tests

marctorsoc commented 2 years ago

This PR solves two issues:

In a previous PR, the columns kwarg of from_records was promoted to read_avro to save RAM. However, this meant that all rows were read, ignoring nrows. Same for exclude to exclude columns. Here this is fixed
Tests were flaky and not working for me locally. This was due to an unspecified timezone, which makes avro dump as UTC but then when reading it infers what it can. I hypothesize this might be working since the tests are run in UTC located machines, but I hope this will work now for any place.

Let me know your thoughts

marctorsoc commented 2 years ago

@ynqa and @ruben-trdj for exposure

deleted commented 1 year ago

FWIW, I can confirm from local testing (Ubuntu 20.04) that:

tests on yqna:master fail with my system locale set to local time and pass when I set my locale to UTC
tests on this branch pass regardless of system locale.

ynqa commented 1 year ago

Once I thought to use https://github.com/spulec/freezegun for mocking or something, but it is also enough now. Thanks.

ynqa / pandavro