search

whole-tale / wt-design-docs

MIT License
5 stars 9 forks source link

v1.1rc1 testing (Chrome/MacOS) #166

Closed craig-willis closed 2 years ago

craig-willis commented 2 years ago

Test Plan

End-to-End Test

Issue: Globus transfer fails because globus-sdk is missing and keychain.json isn't configured -- also no indication that this needs to happen (https://github.com/whole-tale/whole-tale/issues/115) Issue: Error on DERIVA zip (https://github.com/whole-tale/whole-tale/issues/114) Note: Tale export/import doesn't retain permissions (e.g., execute bit on run.sh)

Splash page

Issue: TOS points to https://dashboard.test.wholetale.org/undefined/tos now noted in https://github.com/whole-tale/ngx-dashboard/issues/255. Note: Should we acknowledge Cornell, UNC, UMich?

Authentication

Issue: ORCID login fails with 500 error. Issue with Client ID not respecting requests from test.wholetale.org? Login is working on stage. (https://github.com/whole-tale/girder_wholetale/issues/459)

Issue: https://github.com/whole-tale/ngx-dashboard/issues/258. I must not have tested the fix thoroughly, because the create fails on the backend with RuntimeError: Lookup for "https:%2F%2Fdoi.org%2F10.5281%2Fzenodo.820575" failed with: Failed to interpret "https:%2F%2Fdoi.org%2F10.5281%2Fzenodo.820575" in any meaningful way. This suggests the URL isn't being correctly decoded. And there is no error except in the console. (https://github.com/whole-tale/ngx-dashboard/issues/258)

Navigation

Tale Dashboard

Preconditions:

Notes:

Managing Data

Preconditions

Issue: Missing DERIVA

Run Tale

Preconditions:

Test case: Missing recorded run from menu list

Issue: File context menu no longer works after upload file but works during upload. Context menu doesn't work after after rename. https://github.com/whole-tale/ngx-dashboard/issues/259

Issue: UI does not refresh after move or copy to workspace, but file is copied. Note: Why can't I see files from public tales (e.g., LIGO)? What constitutes an accessible tale?

Test case: Need to fix bdbag validation steps. Issue: Cannot export public tale -- "message": "Write access denied for tale 59f0b91584b7920001b46f2e (user 59fb6165f7e8790001da4e8b)." Disable menu as we do for publish https://github.com/whole-tale/ngx-dashboard/issues/260

Versions

Note: I always force new version since it's checked. When wouldn't I? (Doc issue or hide checkmark)

Recorded Runs

Settings

Issue: Need to remove dev2, which no longer exists.

Issue: Need to change CN to cn-sandbox.test.dataone.org (migration?). cn-stage-2.test.dataone.org/cn/v2 is redirecting. https://github.com/whole-tale/girder_wholetale/pull/528

Tale Creation

Issue: Migration required for R 4.0.2 due to old WTRSTUDIO* env vars. Issue: Migration step to add environments

Issue: Repeated by different build errors, latest appears to be due to memory limits (I deployed too small of test instances?)

#42 260.3 Solving environment: ...working... Killed
#42 ERROR: executor failed running [/bin/sh -c conda install --quiet -p ${NB_PYTHON_PREFIX} -y 'pyarrow' &&     conda clean --all -f -y]: runc did not terminate sucessfully
------
> [35/39] RUN conda install --quiet -p /srv/conda/envs/notebook -y 'pyarrow' &&     conda clean --all -f -y:
------
failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: rpc error: code = Unknown desc = failed to build LLB: executor failed running [/bin/sh -c conda install --quiet -p ${NB_PYTHON_PREFIX} -y 'pyarrow' &&     conda clean --all -f -y]: runc did not terminate sucessfully

Issue: Build error during install of toolboxes under R2020b. Investigating.

Issue: Can we just remove OpenRefine? It's a waste of time to even test. How many openrefine Tales are there?

Analyze in Whole Tale

These test cases cover potential situations that can occur when importing datasets from Dataverse.

Issue? This test case only works if you don't have a tale with the dataset registered already https://github.com/whole-tale/girder_wholetale/issues/529 Test case: Category is social science

Issue: Test case only works if environment matches exact name. This has alway been fragile.

Issue?: During register, dataset appears as subfolder in workspace, but is eventually removed Issue: Import fails with I/O error https://github.com/whole-tale/girder_wholetale/issues/530

Test case: Update CN Issue: proveit dataset registers as a single file "FK2K075M25". Need to investigate (Girder logs WARNING: No object was found in the index for doi:10.5072/FK2K075M25.)

Tale metadata tests

The purpose of these tests are to confirm that the metadata files (manifest.json, environment.json, LICENSE) we generate are correct.

import urllib.parse urllib.parse.uses_relative.append('arcp') urllib.parse.uses_netloc.append('arcp')

g = Graph().parse(source="manifest.json", format="json-ld") print(g.serialize(format='turtle', indent=2).decode())



* [ ] environment.json
1. Open the archive from the manifest.json test
1. Navigate to `metadata/environment.json`
1. Open the girder web API
1. Locate and GET the Image that the Tale used
1. Confirm that the `config` section from girder matches the `config` section in `environment.json`

* [ ] LICENSE
1. Open the archive from the manifest.json test
1. Open the `LICENSE` file
1. Navigate to Run > metadata
1. Confirm that the opened `LICENSE` file matches the selected license on the metadata page
1. Change the Tale's license
1. Export the Tale
1. Confirm that the `LICENSE` changed appropriately

## Tale Sharing Tests

These tests are centered around the feature for sharing Tales with other Whole Tale
users. To run these tests, register at least one other account, referred to as User B.

* [x] Case 1: Sharing with 'Can View' permissions
  1. Create a new Tale
  1. Add data to the workspace and add at least one dataset
  1. Navigate to the 'Share' tab
  1. Share the Tale with User B
  1. Confirm that the default permission is 'Can View'
  1. Switch the account to User B
  1. Navigate to the 'Shared with Me' tab on the Browse page
  1. Confirm that you see the Tale from step 1
  1. Confirm that you can't edit the Workspace, External Data, or metadata
  1. Unshare tale with User B
  1. Note modal displays that tale has been unshared.

* [x] Case 2: Sharing with 'Can Edit' permissions
  1. Create a new Tale
  1. Add data to the workspace and add at least one dataset
  1. Navigate to the 'Share' tab
  1. Share the Tale with User B
  1. Set the permissions to 'Can Edit'
  1. Switch the account to User B
  1. Navigate to the 'Shared with Me' tab on the Browse page
  1. Confirm that you see the Tale from step 1
  1. Remove one of the files from the Workspace
  1. Add a dataset to External Data
  1. Change a field in the metadata (title, description, etc)
  1. Switch to User A
  1. Confirm that the changes made to the Tale persisted

* [x] Case 3: Edit sync modal
  1. Login as User A and User B in separate browser sessions
  1. As User B, select "Shared with me"
  1. As User A, share a tale with User B; as User B confirm the tale appears
  1. As User B, select "View" on the shared tale
  1. As User A and B edit the tale metadata
  1. As User A change title, author, category, description; save changes
  1. As User B confirm Tale edited modal, select Yes, confirm updates to metadata

## Publishing tests

* [x] Case 1: No Accounts Configured
  1. Disconnect all of your external accounts
  1. Open a Tale that you own
  1. Click the ellipsis dropdown on the Run page
  1. Click `Publish tale...`
  1. Confirm that your are notified to link accounts on the settings page
  1. Click `Go to Settings`
  1. Confirm you are re-directed to settings

* [ ] Case 2: Publishing to DataONE
  1. Connect to DataONE (dev server)
  1. Open a Tale that you own
  1. Add some data to it
  1. Click `Publish tale...`
  1. Confirm that the publishing modal opens
  1. Confirm that the third party that you are connected to appears in the dropdown
  1. Select the third party
  1. Click `Publish`
  1. Once complete, navigate to the metadata page
  1. Confirm that the published location is present
  1. Visit the published location
  1. Confirm that you see the following files plus yours
      1. `metadata.xml`
      1. `LICENSE`
      1. `README.md`
      1. `manifest.json`
      1. `environment.json`
  1. After publishing to DataONE, use girder to query the Tale
  1. Ensure that the 'pid' field in the 'publishInfo' object has an identifier in a doi format

**Issue:** Can't even select dev server to publish to.

* [ ] Case 3: Re-publish Tale to DataONE
1. Launch a Tale that has been published before
1. Take note of the `Published Location` in Run > metadata
1. Re-publish the Tale to the third party
1. Confirm that the `Published Location` has changed to the correct package landing page

## Zenodo integration tests

* [x] Register Zenodo data
1. On a new or existing tale, select "Files" > "External Data"
1. Select "+" > "Web (DOI or URL)"
1. Search for Concept DOI `doi:10.5281/zenodo.16384`
1. Confirm "Belmont Forum..." dataset is found on Zenodo with suffix `ver_1`
1. Search for URL  `https://zenodo.org/record/1172960#.XidIvlNKh24`
1. Confirm "Belmont Forum..." dataset is found on Zenodo with suffix `ver_2`
1. Search for DOI `doi:10.5281/zenodo.1172960`
1. Confirm "Belmont Forum..." dataset is found on Zenodo with suffix `ver_2`
1. Search for Concept DOI `doi:10.5281/zenodo.608984`
1. Confirm "Belmont Forum..." dataset is found on Zenodo with suffix `ver_2`
1. Select Register
1. Select "+" > "WT Data Catalog"
1. Add "Belmot Forum..." dataset to Tale
1. Run Tale and confirm contents on file system match Zenodo

The register tests the following cases.
   * Datasets with multiple files
   * Datasets with multiple versions
   * Concept DOI 

* [x] Zenodo dataset via AiWT
1. Open https://dashboard.test.wholetale.org/browse?uri=https%3A%2F%2Fdoi.org%2F10.5281%2Fzenodo.820575&name=Automotive%20Sensor%20Data
1. Confirm Source Data URL and Title match above URL and "READ ONLY" is selected
1. Select JupyterLab environment
1. Select 'Create New Tale'
1. Confirm dataset is mounted read-only, citation and related ids are as expected,
1. Run IDE and use terminal to `ls ../data/Auto*` to confirm zipfile is not extracted

**Test case:** Could setup test tale with valid MD5 check, etc.

The register tests the following cases.
   * Dataset with single file (zipfile)

* [ ] Zenodo Binder via AiWT
1. Open https://dashboard.test.wholetale.org/browse?uri=https%3A%2F%2Fdoi.org%2F10.5281%2Fzenodo.3242073&environment=JupyterLab&name=Simple%20requirements.txt%20based%20example%20for%20repo2docker&asTale=true
1. Confirm Source Data and Title match URL and "READ WRITE" is selected
1. Select 'Create Tale'
1. Confirm image builds and notebook can run

**Note:** Image build starts with step 22/install-miniforge? But why?
**Issue:** Intermittent 503 from https://keyserver.ubuntu.com/. Need to retest.

* [x] Zenodo Tale via AiWT
1. Open https://girder.test.wholetale.org/api/v1/integration/zenodo?doi=10.5281%2Fzenodo.490449&resource_server=sandbox.zenodo.org
1. Confirm Water Tale is created. 
1. Confirm citation on Metadata view
1. Confirm DOI matches above
1. Use the Swagger UI to confirm the `publishInfo` field on the tale object contains pid, repository, repository_id, uri
1. Run the tale and and notebook, confirm it works as expected (produces glorious map of Texas) 

* [ ] Publish/import to/from Zenodo Sandbox
1. Follow instructions above to configure your API key for Zenodo sandbox 
1. Create your own Tale and publish to sandbox.zenodo.org
1. Confirm publishInfo contains Sandbox entry
1. Delete tale
1. Import tale from Zenodo
1. Confirm publishInfo
1. Modify the tale
1. Re-publish to sandbox.zenodo.org
1. Delete tale
1. Import tale from Zenodo
1. Confirm publishInfo
1. Download and extract the exported Tale
1. Execute the `run-local.sh` command to run the Tale locally

## Git Integration Tests

* [x] Compose Tale from Git Repository
Covered by other tests
1. Select the Create New Tale button on the 'Browse' page
1. Select the `Create Tale from Git Repository` option
1. Paste a link to a Git repository in the correct field
1. Add a name and pick a compute environment
1. Create the Tale
1. Confirm notification progress and success
1. Confirm that the git repository was cloned to the `workspace/` root
1. Run the tale
1. Confirm interaction via git tools (may require environment customization)

* [x] Connect to Git
1. Create a new empty tale
1. Select "Connect to Git Repository" menu item from Tale menu
1. Confirm notification progress and success
1. Once complete, navigate to the Tale workspace and confirm that the repository has been cloned

## Other tests

* [ ] Citations
1. Add files from multiple external datasets (e.g.: doi:10.5281/zenodo.3381658, doi:10.5065/D6862DM8, doi:10.5063/F10K26VR)
1. Confirm `Datasets Used` citations and related identifiers are updated

### Event Notification Tests
* [x] Tale sharing/unsharing
1. Login as user1 and user2 in separate browser sessions
1. As user2, select "Shared with me"
1. As user1, share a tale with user2; as user2 confirm the tale appears
1. As user2, select "View" on the shared tale
1. As user1 and 2 edit the tale metadata
1. As user1 save changes; as user2 confirm Tale edited modal
1. As user1, unshare the tale; as user2 confirm Tale unshared modal
1. As user1, re-share with user2
1. As user2, select "Shared with Me"
1. As user1, delete the shared tale; as user2 confirm that the tale disappears

* [ ] Import tale
1. Import a tale from Github, Zenodo, and as Binder
1. Confirm metadata updates accordingly after import completes

## Vocabulary Tests

* [ ] Vocabularies
1. Go to https://vocabularies.test.wholetale.org/wt/1.0/index.html
1. Confirm that it matches https://whole-tale.github.io/serialization-format/wt/1.0/

**Issue:** Oops! Need to update vocab for 1.1... PR needs to be merged?

## Forward Authentication Test
* [x] Forward authentication
1. Start a tale
1. Confirm that the tale is accessible via iframe and popout
1. Right click popout icon and copy URL (this is necessary to get JupyterLab token)
1. Open an incognito window and paste the copied URL
1. Confirm that you are prompted for authentication and can access the tale once authenciated
1. Login as a user who does not have permission to access the tale
1. Confirm that they receive an access denied errror

## Changing the Environment
* [ ] Changing the environment
1. Create a new JupyterLab tale
1. Run the Tale, confirm JupyterLab IDE displays
1. Edit metadata, change environment to "RStudio"
1. Confirm icon changes in title bar and Environment section
1. Stop/run tale or rebuild/restart and confirm RStudio IDE displays

## Idle Instances
* [ ] Idle instance culling
1. Create and start a MATLAB tale and a JupyterLab tale
1. Confirm both instances start and IDEs are accessible
1. Close all IDE tabs/windows
1. Wait for 2 hours and 24 hours respectively
1. Confirm both instances are stopped automatically
craig-willis commented 2 years ago

Identified issues:

Test case issues:

Notes:

craig-willis commented 2 years ago

On to RC2...