wireservice / csvkit

A suite of utilities for converting to and working with CSV, the king of tabular file formats.
https://csvkit.readthedocs.io
MIT License
6k stars 605 forks source link

agate: strings are interpreted as datetimes #683

Closed vtselfa closed 7 years ago

vtselfa commented 7 years ago

CSVLook seems to interpret as dates some strings that it should not. This is using last version (1.0.0) from github, branch master.

%cat test.txt          
core,app
5,05_leslie3d_base
6,06_povray_base
4,04_h264ref_base
3,03_astar_base
2,02_lu.C
7,07_cg.C
1,01_mcf_base
0,00_xalancbmk_base

test.txt

% csvlook test.txt
| core |        app |
| ---- | ---------- |
|    5 | 0501-01-01 |
|    6 | 0601-01-01 |
|    4 | 0401-01-01 |
|    3 | 0301-01-01 |
|    2 | 0201-01-01 |
|    7 | 0701-01-01 |
|    1 | 0101-01-01 |
|    0 | 0001-01-01 |

As can be seen, the strings in the second column have been replaced by strange dates.

jpmckinney commented 7 years ago

Try with --no-inference.

vtselfa commented 7 years ago

It doesn't happen with --no-inference.

jpmckinney commented 7 years ago

agate shouln't consider text like 05_leslie3d_base or 20160506175330-3168-01 to be a date.

Can reproduce #607 with

echo '{"results":[{"id":"20160506175330-3168-01","changes":[{"rev":"1-8bda66017d962508e51ac5061557635b"}],"doc":{"_id":"20160506175330-3168-01","_rev":"1-8bda66017d962508e51ac5061557635b","year":"2016","month":"05","day":"06","hour":"17","minute":"53","second":"30","imagebox":"18x94+428+180","alchemy":{"text":"person","score":"0.845535"},"visual":{"image":"20160506175330-3168-01.jpg","scores":[{"classifier_id":"Mixed_Color","name":"Mixed_Color","score":0.964822},{"classifier_id":"Stove","name":"Stove","score":0.923713},{"classifier_id":"Archery","name":"Archery","score":0.836994},{"classifier_id":"Dish_Washer","name":"Dish_Washer","score":0.835787},{"classifier_id":"Barber_Shop","name":"Barber_Shop","score":0.825716},{"classifier_id":"Full_Body","name":"Full_Body","score":0.824954},{"classifier_id":"Musical_Instrument","name":"Musical_Instrument","score":0.817304},{"classifier_id":"Beauty_Salon","name":"Beauty_Salon","score":0.808518},{"classifier_id":"Clothing_Store","name":"Clothing_Store","score":0.784035},{"classifier_id":"Distillery","name":"Distillery","score":0.67477},{"classifier_id":"Sweet_Treat","name":"Sweet_Treat","score":0.588916},{"classifier_id":"Beer","name":"Beer","score":0.538915},{"classifier_id":"Store","name":"Store","score":0.536244},{"classifier_id":"Yellow","name":"Yellow","score":0.526932}]}}}],"last_seq":"10457-g1AAAAPreJy90lFqAjEQBuCgFj2B4EOpUgRfXNyscWef9CbtTMZFZF3Brs96E3uT9ib1IMIaE2XpmynUlwkE8n_8TDIhRGtRZzFg0uvNfMakAloNGbPhR4E544ZloLP1ljEvgnxeZOZFDQW9lGW5XNRJiKfnlblrYiRZqtQj6rcr73CpayZNK3pg6bGOmZE80vwr0-xCv1X0q6XViDGJ_t46vId-v9C7iu5YegIEmIw90vxb5w0zxd4cRj9c-Z7lSUEEIP-1ueM_Hf_l-MbJ8iGmxDE8oP2343-u7fvuy6WaRqQf0P7o-Nvu25ZPATRA6JG4PAMPEEFw","pending":0}' | in2csv -k results
onyxfish commented 7 years ago

Tracked upstream.

onyxfish commented 7 years ago

Fixed upstream.