Add checksum-based duplicate file checks for landing zone uploads

[mikkonie]

Requested by @holtgrewe. A user may accidentally upload a file to a landing zone that already exists in the project's sample repository, but with a different file name and/or under a different collection. Currently, this is not checked so duplicates can be uploaded. This should be preventable, at least for large files.

This task requires #1846 to be done first.

Spec (To Be Refined)

• Add default Django settings for warn level and deny level
  • LANDINGZONES_DUPLICATE_SIZE_DENY
    • Int, in KB (same for the other one)
    • Default = 100000
  • LANDINGZONES_DUPLICATE_SIZE_WARN
    • Default = 1000
• Add landingzones app settings to override defaults
  • duplicate_size_deny
    • scope = PROJECT
    • type = BOOLEAN
    • user_modifiable = True
    • default = settings.LANDINGZONES_DUPLICATE_SIZE_DENY
  • duplicate_size_warn
    • Same as above, but with the default naturally pointing for the other Django setting
  • TODO: Write out labels and descriptions
• Add multi-setting validation for project-specific settings
  • Both values can be 0
  • If neither are 0, WARN cannot be larger than DENY
• Add allow_duplicates boolean field to the LandingZone model
  • Default=False
  • If set True, this will opt out from the check
• Add allow_duplicates parameter for landing_zone_move taskflow
• Add logic to landing_zone_move taskflow
  • (If allow_duplicates==False, skip all of the following)
  • In flow setup, get file listing of project sample repository using taskflowbackend
    • TODO: Check if existing taskflowbackend list output has to be massaged in some way. Are checksums included?
    • We need checksums and file size. File names and even paths are irrelevant
    • Only query for this once, hence it has to happen in the setup
  • After creating and validating checksums in the VALIDATING phase, check for dupes
    • TODO: Write a detailed spec for the new taskflow task to be added
    • This should be a batch task
    • Pass landing zone file list and checksum/size list to task
    • For each non-md5 file in the landing zone, go through list and search for a checksum match
    • IF found AND deny limit > 0 AND file size exceeds deny limit, THEN store file path in deny message
    • ELIF found AND warn limit > 0 AND file size exceeds warn limit, THEN store file path in warning message
    • TODO: How to store these within a task? Modify lists provided from the flow as args, probably?
  • After looping through all files
    • IF denied files were found, do not move and return with failed status, also include warnings
    • ELSE if warnings, add them into the zone success status
  • This has to be done for both "validate only" and zone moving modes of this flow
    • Hence, I believe this tasks requires doing #1846 first

Tasks

• [ ] Add Django settings
• [ ] Add app settings
• [ ] Add app settings validation
• [ ] Add LandingZone.allow_duplicates field and taskflow parameter
• [ ] Add batch duplicate check task
• [ ] Add logic to landing_zone_move flow
• [ ] Add task tests
• [ ] Add flow tests
• [ ] Add landingzones view tests
• [ ] Add documentation

[Nicolai-vKuegelgen]

In addition to the file_size check it might be worthwhile to have different behaviour based on file type: i.e. only apply the check to specific file types (like fastq.gz / bam); or only raise WARN but not DENY for certain file types.

However, in practice the file size distinction will likely already catch most cases where duplicate files are not uploaded in error (like log files), so this might not be a necessary addition.