search

scraperwiki / spreadsheet-download-tool

A ScraperWiki plugin for downloading data from a box as a CSV or Excel spreadsheet
BSD 2-Clause "Simplified" License
2 stars 1 forks source link

Write CSV file when Excel fails. #54

Closed drj11 closed 10 years ago

drj11 commented 10 years ago

Didn't change the Excel module, so now fails when > 65535 rows are written, but should continue writing the CSV file.

drj11 commented 10 years ago

@pwaller I totally take those comments on board. but... what do you suggest?

pwaller commented 10 years ago

Even a log.warning() would be something. At the minimum it has to alert the developer they did something wrong when they typo'ed a method name inside write_csv_row, for instance.

Ideally, I would also perhaps mark that file as bad and delete it, replace it with "MALFORMED CONTENT" or something, because with the current setup a user could end up with partial data but actually some rows were failing for an unknown reason.

pwaller commented 10 years ago

s/partial data/partial data which seems good/g

drj11 commented 10 years ago

The exception is now logged which is I think the most minimal useful thing.

I think in the future we probably should do better error reporting and things like removing half-written files. But the current structure where both files are written simultaneously makes it difficult (it's difficult to raise an exception at the right point to get CsvOutput to remove its output; it's difficult to separate the errors from the two output files and present them to the user sensibly).

What we have in this pull request is (a bit) better than master without it. I think.

frabcus commented 10 years ago

What's the status of this pull request, what's it actually do?

pwaller commented 10 years ago

IIRC it was superceded by #56.