search

broadinstitute / single_cell_portal_core

Rails/Docker application for the Broad Institute's single cell RNA-seq data portal
https://singlecell.broadinstitute.org
BSD 3-Clause "New" or "Revised" License
62 stars 26 forks source link

Enable remote CSFV bypass, omit .gz check for RDS files (SCP-5821) #2146

Closed eweitz closed 3 weeks ago

eweitz commented 3 weeks ago

This refines validation for Seurat data, and lets admins disable client-side validation for files uploaded via bucket path.

Impact

Previously, Single Cell Portal did not account for a convention in the R community to omit .gz extensions from gzip-compressed RDS files. Incidentally, this revealed a need to allow staff to occasionally disable client-side validation (CSFV) for remote files.

Now, RDS files can be gzipped or not, and won't throw any gzip-related error message. And admins can now disable CSFV for remote files, like they already can for local files.

Demo

Here's how it looks.

Enable remote CSFV bypass

https://github.com/user-attachments/assets/c16332d1-b86a-491a-b34d-f142fae48acc

Omit .gz check for RDS files

https://github.com/user-attachments/assets/47a6ff9a-c7da-4b1a-8c59-fb905b537c15

Test

An automated test verifies omitting the .gz check for RDS files. Optionally, to manually test:

  1. Set "clientside_validation" feature flag to "false"
  2. Go to AnnData upload UI for a study with no files
  3. Click "Use bucket path"
  4. Paste "invalid_disease_id.h5ad" into input
  5. Confirm no error message (if you get warning file doesn't exist, upload to bucket from test/test_data/anndata)
  6. Set "clientside_validation" feature flag to "true"
  7. Upload to bucket this file: test/test_data/validation/missing_gz_extension.rds
  8. Confirm no error message

This satisfies SCP-5821.

codecov[bot] commented 3 weeks ago

Codecov Report

Attention: Patch coverage is 33.33333% with 4 lines in your changes missing coverage. Please review.

Project coverage is 69.57%. Comparing base (b6abf58) to head (efb3522). Report is 4 commits behind head on development.

Files with missing lines Patch % Lines
app/javascript/lib/validation/validate-file.js 0.00% 3 Missing and 1 partial :warning:
Additional details and impacted files [![Impacted file tree graph](https://app.codecov.io/gh/broadinstitute/single_cell_portal_core/pull/2146/graphs/tree.svg?width=650&height=150&src=pr&token=HMWE5BO2a4&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=broadinstitute)](https://app.codecov.io/gh/broadinstitute/single_cell_portal_core/pull/2146?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=broadinstitute) ```diff @@ Coverage Diff @@ ## development #2146 +/- ## =============================================== + Coverage 69.55% 69.57% +0.02% =============================================== Files 331 331 Lines 27936 27942 +6 Branches 2438 2440 +2 =============================================== + Hits 19430 19441 +11 + Misses 8353 8347 -6 - Partials 153 154 +1 ``` | [Files with missing lines](https://app.codecov.io/gh/broadinstitute/single_cell_portal_core/pull/2146?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=broadinstitute) | Coverage Δ | | |---|---|---| | [...javascript/lib/validation/validate-file-content.js](https://app.codecov.io/gh/broadinstitute/single_cell_portal_core/pull/2146?src=pr&el=tree&filepath=app%2Fjavascript%2Flib%2Fvalidation%2Fvalidate-file-content.js&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=broadinstitute#diff-YXBwL2phdmFzY3JpcHQvbGliL3ZhbGlkYXRpb24vdmFsaWRhdGUtZmlsZS1jb250ZW50Lmpz) | `87.84% <100.00%> (+0.13%)` | :arrow_up: | | [app/javascript/lib/validation/validate-file.js](https://app.codecov.io/gh/broadinstitute/single_cell_portal_core/pull/2146?src=pr&el=tree&filepath=app%2Fjavascript%2Flib%2Fvalidation%2Fvalidate-file.js&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=broadinstitute#diff-YXBwL2phdmFzY3JpcHQvbGliL3ZhbGlkYXRpb24vdmFsaWRhdGUtZmlsZS5qcw==) | `63.21% <0.00%> (-3.05%)` | :arrow_down: | ... and [2 files with indirect coverage changes](https://app.codecov.io/gh/broadinstitute/single_cell_portal_core/pull/2146/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=broadinstitute)
eweitz commented 3 weeks ago

The build failure is a false positive that will be fixed by #2143:

Total elapsed time: 26 minutes, 31 seconds

### FAILURES and ERRORS ###
*** Yarn test failures ***
  FAIL test/js/lib/validate-anndata.test.js
      ✕ Parses AnnData headers (448 ms)
      ✕ Reports AnnData with valid headers as valid (114 ms)
      ✕ Reports AnnData with invalid headers as invalid (224 ms)
      ✕ Parses AnnData rows and reports invalid ontology IDs (257 ms)
  FAIL test/js/lib/validate-anndata.test.js

*** Rails test failures ***
Exiting with code: 1