Provide a full row of data to csv.Sniffer().sniff() so it can effectively detect the correct dialect. MFR reads a small amount of data from the file and recursively read more until either the following happens.
Nothing available. Return the empty string since there is no new line characters in this file at all.
The to-sniff size goes over max render size allowed. Return TabularRenderError. However, this is more like a failsafe since oversized file should never reach the sniffer in the first place.
Any type of new line characters found, trim (if necessary) and send the the full first row to the sniff.
Please see the video captures in the tickets as a quick demo.
Updated existing and added new unit tests
Side effects
Memory usage for sniffing depends on the size of the first row of the file. In the worst case, it could sniff at most 10MB. Please note that we don't have enough statistical data on how many bytes the first row of a .csv file takes on average. However, files larger than 10MB have already failed the size check by the renderer before the sniffing starts.
Given that for CSV, we needs to read the full file into memory for rendering anyway and the partial sniff data is always deleted after use, I don't think it will be a problem.
[ ] For CR: maybe add a sentry alert when the sniffing size goes over 512KB (or some other size) so we know how often this happens and how effective this algorithm actually is.
Ticket
https://openscience.atlassian.net/browse/ENG-1809
Purpose
Improve .csv dialect sniffing.
Changes
Provide a full row of data to
csv.Sniffer().sniff()so it can effectively detect the correct dialect. MFR reads a small amount of data from the file and recursively read more until either the following happens.TabularRenderError. However, this is more like a failsafe since oversized file should never reach the sniffer in the first place.Updated existing and added new unit tests
Side effects
Memory usage for sniffing depends on the size of the first row of the file. In the worst case, it could sniff at most 10MB. Please note that we don't have enough statistical data on how many bytes the first row of a
.csvfile takes on average. However, files larger than 10MB have already failed the size check by the renderer before the sniffing starts.Given that for CSV, we needs to read the full file into memory for rendering anyway and the partial sniff data is always deleted after use, I don't think it will be a problem.
QA Notes
TBD
Deployment Notes
N / A