Add "Values are unique" to --csv

[tacman]

It would be valuable to me to know if the all the values are unique. There is a number of unique values, and at the of the csvstat command there is

 Row count: 1856

csvstat data/subtitles_day.tsv 
  1. "IDSubtitle"

    Type of data:          Number
    Contains null values:  False
    Unique values:         1856
    Smallest value:        9,747,231
    Largest value:         9,749,339
    Sum:                   18,092,851,467
    Mean:                  9,748,303.592
    Median:                9,748,352.5
    StDev:                 628.279
    Most common values:    9,747,231 (1x)
                           9,747,232 (1x)
                           9,747,233 (1x)
                           9,747,234 (1x)
                           9,747,235 (1x)

That's lost with --csv, along with the frequency, so there's no easy way to know if the values are unique. For my purposes, I'm trying to find the primary key from a set of files, so knowing that the values are unique would be enormously helpful.

csvstat data/subtitles_day.tsv --csv | csvjson | jq

 {
    "column_id": 1,
    "column_name": "IDSubtitle",
    "type": "Number",
    "nulls": false,
    "unique": 1856,
    "min": "9,747,231",
    "max": "9,749,339",
    "sum": 18092851467,
    "mean": 9748303.592,
    "median": 9748352.5,
    "stdev": 628.279,
    "len": null,
    "freq": "9747231, 9747232, 9747233, 9747234, 9747235"
  },

If the frequency count were included in the "freq" key, I could parse that and see if the top one was just 1, but adding "Values are unique" would be better. Of course, to determine primary key I'd also check "Contains null values".

[jpmckinney]

In HEAD, I instead added a "Non-null values" statistic (also appears in the --csv output). This information is useful for this use case as well as others.

You can thus compare non-null values to unique values. Note that if the column contains nulls, then NULL counts as one additional unique value.

[tacman]

Thanks!

I've been installing this via "sudo apt install csvkit" but I think I need a ppm in order to get the latest version. Is one available?

I've had trouble following the installation instructions on Ubuntu via pip.

[jpmckinney]

I only manage the PyPI package. Packages in Linux distributions are created independently.

I'll make a new release of the PyPI package shortly.