search

contiamo / contiamo-client-python

MIT License
2 stars 2 forks source link

Upload large dataframes in chunks #24

Closed ian-contiamo closed 6 years ago

ian-contiamo commented 6 years ago

@nagautier @bmwilly can you please double-check the formula I used to slice the data? We need to be absolutely certain that we do not skip (or repeat) a single row!!! I'm 99.9% certain of what I did, even in edge cases, but I'd like to add a few more 9s ;)

ian-contiamo commented 6 years ago

@bmwilly The test that fails is unrelated to this commit. It does not fail for me locally. Not sure how to address this.

bmwilly commented 6 years ago

For posterity, the tests were failing because they're hitting a live server. See discussion issue: https://github.com/contiamo/contiamo-client-python/issues/26

bmwilly commented 6 years ago

lgtm pending the small imports improvement ;)

ian-contiamo commented 6 years ago

@bmwilly can you check this commit? https://github.com/contiamo/contiamo-client-python/pull/24/commits/5b8c0731d8e2f3aaf856916000e784252d03acad

It makes chunking testable. And the code a bit nicer (the slicing is abstracted from the main function).

bmwilly commented 6 years ago

It makes chunking testable. And the code a bit nicer (the slicing is abstracted from the main function)

good stuff!