Deprecation and suggested replacement(s) of of `unique:` property is not a one-to-one functionality

[jfy133]

We noticed that the functionality provided by the removed of items: property does not actually get replaced by the uniqueItems and uniqueEntries fields.

• items: allowed for a single 'column' of a TSV to be validated that everything was unique, independent of any other column.
• uniqueItems: forces all columns to be unique, not particular ones
• uniqueEntries requires specific combinations of columns to be unique together (not a single column).

We can almost get around this with placing a allOf field at the top level of the schema, and listing each column that should be independently validated with uniqueEntries, such as:

    "allOf": [
        { "uniqueEntries": ["id"] },
        { "uniqueEntries": ["fastq_dna"] },
        { "uniqueEntries": ["fastq_aa"] }
    ]

However if any of those columns violate uniqueness, the error message reports all of the columns being non-unique in independent errors (and still saying 'combinations'

    * --input (test_duplicate_id.csv): Validation of file failed:
        -> Entry 2: Detected non-unique combination of the following fields: [id]
        -> Entry 2: Detected non-unique combination of the following fields: [fastq_dna]
        -> Entry 2: Detected non-unique combination of the following fields: [fastq_aa]
        -> Value does not match against the schemas at indexes [0, 1, 2]

Example schema:

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "$id": "https://raw.githubusercontent.com/nf-core/createtaxdb/master/assets/schema_input.json",
    "title": "nf-core/createtaxdb pipeline - params.input schema",
    "description": "Schema for the file provided with params.input",
    "type": "array",
    "items": {
        "type": "object",
        "properties": {
            "id": {
                "type": ["string", "integer"],
                "pattern": "^\\S+$",
                "errorMessage": "Sequence reference name must be provided and cannot contain spaces",
                "meta": ["id"]
            },
            "taxid": {
                "type": "integer",
                "errorMessage": "Please provide a valid taxonomic ID in integer format",
                "meta": ["taxid"]
            },
            "fasta_dna": {
                "anyOf": [
                    {
                        "type": "string",
                        "format": "file-path",
                        "pattern": "^\\S+\\.(fasta|fas|fa|fna)(\\.gz)?$"
                    },
                    {
                        "type": "string",
                        "maxLength": 0
                    }
                ],
                "uniqueItems": true,
                "exists": true,
                "format": "file-path"
            },
            "fasta_aa": {
                "anyOf": [
                    {
                        "type": "string",
                        "format": "file-path",
                        "pattern": "^\\S+\\.(fasta|fas|fa|faa)(\\.gz)?$"
                    },
                    {
                        "type": "string",
                        "maxLength": 0
                    }
                ],
                "uniqueItems": true,
                "exists": true,
                "format": "file-path"
            }
        },
        "required": ["id", "taxid"],
        "anyOf": [{ "required": ["fasta_dna"] }, { "required": ["fasta_aa"] }]
    },
    "allOf": [{ "uniqueEntries": ["id"] }, { "uniqueEntries": ["fastq_dna"] }, { "uniqueEntries": ["fastq_aa"] }]
}

Example broken csv:

id,taxid,fasta_dna,fasta_aa
Severe_acute_respiratory_syndrome_coronavirus_2,2697049,https://raw.githubusercontent.com/nf-core/test-datasets/createtaxdb/data/fasta/sarscov2.fasta,https://raw.githubusercontent.com/nf-core/test-datasets/createtaxdb/data/fasta/sarscov2.faa
Severe_acute_respiratory_syndrome_coronavirus_2,26970499,https://github.com/nf-core/test-datasets/blob/modules/data/genomics/prokaryotes/bacteroides_fragilis/genome/genome.fna.gz,https://raw.githubusercontent.com/nf-core/test-datasets/createtaxdb/data/fasta/sarscov2.faa
Haemophilus_influenzae,727,https://raw.githubusercontent.com/nf-core/test-datasets/createtaxdb/data/fasta/haemophilus_influenzae.fna.gz,

Where there is one duplicate in id and one duplicate in fasta_aa

[LilyAnderssonLee]

I had the same issue in taxprofiler. I would like to define the these columns are unique: fastq_1, fastq_2, fasta, and the combination of sample and run_accession.

schema_input.json: schema_input.json

samplesheet.csv with duplicates: samplesheet.csv

Error message:

ERROR ~ Validation of pipeline parameters failed!

 -- Check '.nextflow.log' file for details
The following invalid input values have been detected:

* --input (https://raw.githubusercontent.com/nf-core/test-datasets/taxprofiler/samplesheet.csv): Validation of file failed:
        -> Entry 7: Detected non-unique combination of the following fields: [fastq_1]
        -> Entry 4: Detected non-unique combination of the following fields: [fastq_2]
        -> Entry 3: Detected non-unique combination of the following fields: [fasta]
        -> Value does not match against the schemas at indexes [0, 1, 2]

[jfy133]

Done!