search

whole-tale / wt-design-docs

MIT License
5 stars 9 forks source link

v1.0rc3 testing (on OSX Catalina / Chrome 90.0.4430.212) #155

Closed craig-willis closed 3 years ago

craig-willis commented 3 years ago

Test Plan

https://github.com/whole-tale/wt-design-docs/issues/new?template=TEST_PLAN.md

Note: For all tests, repeat for supported browser/OS combinations.

Preconditions:

Splash page

Authentication

Navigation

Tale Dashboard

Preconditions:

Managing Data

Preconditions

Run Tale

Preconditions:

FAIL: fetch.txt has incorrect path

Settings

Tale Creation

Analyze in Whole Tale

Task 1: Importing a Dataset from Dataverse

These test cases cover potential situations that can occur when importing datasets from Dataverse.

Testing Steps:

  1. Remove all running Tale instances
  2. Navigate to https://girder.stage.wholetale.org/api/v1/integration/dataverse?datasetPid=doi%3A10.7910%2FDVN%2F3MJ7IR&siteUrl=https%3A%2F%2Fdataverse.harvard.edu
  3. Confirm that the Tale title reads as Replication Data for: "Agricultural Fires and Health at Birth"
  4. Confirm that the only item in the Input data section matches the uri with Data Source appended
  5. Confirm that no environment is selected
  6. Confirm that the Create New Tale button is disabled
  7. Select an environment
  8. Confirm that READ/WRITE is selected
  9. Click Create New Tale
  10. Confirm registration notification progress and success
  11. Confirm that you are redirected to the Metadata page and "Run Tale" is enabled
  12. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  13. Confirm that the data exists under Tale Workspace
  14. Confirm that the Tale's category is science

Task 2: Importing a Dataset from DataONE

These test cases cover different scenarios that may happen when importing datasets from DataONE.

This test covers the ideal case where dataset importing has all of the needed information to properly run. This includes the dataset location, title, and selected environment.

Testing Steps:

  1. Remove all running Tale instances
  2. Navigate to https://girder.stage.wholetale.org/api/v1/integration/dataone?uri=https%3A%2F%2Fsearch.dataone.org%2Fview%2Fdoi%3A10.18739%2FA2VQ2S94D&title=Fire%20influences%20on%20forest%20recovery%20and%20associated%20climate%20feedbacks%20in%20Siberian%20Larch%20Forests%2C%20Russia&environment=RStudio
  3. Confirm that the Tale title reads as Fire influences on forest recovery and associated climate feedbacks in Siberian Larch Forests, Russia
  4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
  5. Confirm that Rstudio is selected in the Environments widget
  6. Confirm that the Create New Tale button is enabled
  7. Confirm READ ONLY is selected
  8. Click Create New Tale
  9. Confirm that you are redirected to the Metadata tab
  10. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  11. Confirm that the data exists in the Tale under External Data
  12. Confirm that the Tale's category is science
  13. Confirm that the Datasets Used and Related Identifiers are updated to reflect the dataset

Links:

  1. "proveit": https://dashboard.stage.wholetale.org/browse?api=https%3A%2F%2Fcn-stage-2.test.dataone.org%2Fcn%2Fv2&name=Dataone%20Dataset&uri=https://dev.nceas.ucsb.edu/view/doi:10.5072/FK2K075M25
  2. "Alpha Helix 2001...": https://dashboard.stage.wholetale.org/browse?api=http%3A%2F%2Fcn-stage.test.dataone.org%2Fcn%2Fv2&name=Dataone%20Dataset&uri=https://test.arcticdata.io/view/doi:10.5065/D6N014NC

Testing Steps:

  1. Remove all running Tale instances
  2. Navigate to each of the two links above
  3. Select an environment
  4. Click Create New Tale
  5. Confirm that the correct data exists in the Tale under External Data

FAIL: proveit import fails; Alpha Helix 2001 does not get correct title. DataONE config problem? Or due to CN change?

Task 3: Shared Behavior

These cases show errors that are common between Dataverse and DataONE. Although they are tested using DataONE URIs, the errors should be the same when replacing it with a Dataverse URI.

This test covers the case where the name query parameter is missing. The expected behavior is that the uri query parameter is used instead.

Testing Steps:

  1. Remove all running Tale instances
  2. Navigate to https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdataverse.harvard.edu%2Fapi%2Faccess%2Fdatafile%2F3323458
  3. Confirm that the Tale title reads as https://dataverse.harvard.edu/api/access/datafile/3323458
  4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
  5. Confirm that no Environment is selected
  6. Confirm that the Create New Tale button is disabled
  7. Select an environment
  8. Click Create New Tale
  9. Confirm that the notification bar appears & properly updates
  10. Confirm that you are redirected to the Metadata tab
  11. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  12. Confirm that the data exists in the Tale under External Data

FAIL: Title does not match URI. Does this really matter?

This test covers the case where a user imports data from an external source and selects READ/WRITE in the Create Tale Modal.

Testing Steps (Variant 1):

  1. Remove all running Tale instances
  2. Navigate to https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdataverse.harvard.edu%2Fapi%2Faccess%2Fdatafile%2F3323458
  3. Confirm that the Tale title matches the URI above
  4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
  5. Confirm that the Create New Tale button is disabled
  6. Select an environment
  7. Select READ/WRITE
  8. Click Create New Tale
  9. Confirm that the notification bar appears & properly updates
  10. Confirm that you are redirected to the run page
  11. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  12. Confirm that the data exists in the Tale Workspace

Testing Steps (Variant 2):

  1. Remove all running Tale instances
  2. Navigate to https://dashboard.stage.wholetale.org/browse?api=https%3A%2F%2Fdev.nceas.ucsb.edu%2Fknb%2Fd1%2Fmn%2Fv2&environment=JupyterLab&name=proveit&uri=doi%3A10.5072%2FFK27P92Z55
  3. Confirm that the Tale title matches the dataset
  4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
  5. Confirm that the Create New Tale button is disabled
  6. Select READ/WRITE
  7. Click Create New Tale
  8. Confirm that the notification bar appears & properly updates
  9. Confirm that you are redirected to the run page
  10. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  11. Confirm that the data exists in the Tale Workspace

Tale metadata tests

The purpose of these tests are to confirm that the metadata files (manifest.json, environment.json, LICENSE) we generate are correct.

import urllib.parse urllib.parse.uses_relative.append('arcp') urllib.parse.uses_netloc.append('arcp')

g = Graph().parse(source="manifest.json", format="json-ld") print(g.serialize(format='turtle', indent=2).decode())



* [x] environment.json
1. Open the archive from the manifest.json test
1. Navigate to `metadata/environment.json`
1. Open the girder web API
1. Locate and GET the Image that the Tale used
1. Confirm that the `config` section from girder matches the `config` section in `environment.json`

* [x] LICENSE
1. Open the archive from the manifest.json test
1. Open the `LICENSE` file
1. Navigate to Run > metadata
1. Confirm that the opened `LICENSE` file matches the selected license on the metadata page
1. Change the Tale's license
1. Export the Tale
1. Confirm that the `LICENSE` changed appropriately

## Tale Sharing Tests

These tests are centered around the feature for sharing Tales with other Whole Tale
users. To run these tests, register at least one other account, referred to as User B.

* [x] Case 1: Sharing with 'Can View' permissions
  1. Create a new Tale
  1. Add data to the workspace and add at least one dataset
  1. Navigate to the 'Share' tab
  1. Share the Tale with User B
  1. Confirm that the default permission is 'Can View'
  1. Switch the account to User B
  1. Navigate to the 'Shared with Me' tab on the Browse page
  1. Confirm that you see the Tale from step 1
  1. Confirm that you can't edit the Workspace, External Data, or metadata

* [x] Case 2: Sharing with 'Can Edit' permissions
  1. Create a new Tale
  1. Add data to the workspace and add at least one dataset
  1. Navigate to the 'Share' tab
  1. Share the Tale with User B
  1. Set the permissions to 'Can Edit'
  1. Switch the account to User B
  1. Navigate to the 'Shared with Me' tab on the Browse page
  1. Confirm that you see the Tale from step 1
  1. Remove one of the files from the Workspace
  1. Add a dataset to External Data
  1. Change a field in the metadata (title, description, etc)
  1. Switch to User A
  1. Confirm that the changes made to the Tale persisted

## Publishing tests

* [x] Case 1: No Accounts Configured
  1. Disconnect all of your external accounts
  1. Open a Tale that you own
  1. Click the ellipsis dropdown on the Run page
  1. Click `Publish tale...`
  1. Confirm that your are notified to link accounts on the settings page
  1. Click `Go to Settings`
  1. Confirm you are re-directed to settings

* [x] Case 2: Publishing to DataONE
  1. Connect to DataONE (dev server)
  1. Open a Tale that you own
  1. Add some data to it
  1. Click `Publish tale...`
  1. Confirm that the publishing modal opens
  1. Confirm that the third party that you are connected to appears in the dropdown
  1. Select the third party
  1. Click `Publish`
  1. Once complete, navigate to the metadata page
  1. Confirm that the published location is present
  1. Visit the published location
  1. Confirm that you see the following files plus yours
      1. `metadata.xml`
      1. `LICENSE`
      1. `README.md`
      1. `manifest.json`
      1. `environment.json`
  1. After publishing to DataONE, use girder to query the Tale
  1. Ensure that the 'pid' field in the 'publishInfo' object has an identifier in a doi format

* [x] Case 3: Re-publish Tale to DataONE
1. Launch a Tale that has been published before
1. Take note of the `Published Location` in Run > metadata
1. Re-publish the Tale to the third party
1. Confirm that the `Published Location` has changed to the correct package landing page

## Zenodo integration tests

* [x] Register Zenodo data
1. On a new or existing tale, select "Files" > "External Data"
1. Select "+" > "Web (DOI or URL)"
1. Search for Concept DOI `doi:10.5281/zenodo.16384`
1. Confirm "Belmont Forum..." dataset is found on Zenodo with suffix `ver_1`
1. Search for URL  `https://zenodo.org/record/1172960#.XidIvlNKh24`
1. Confirm "Belmont Forum..." dataset is found on Zenodo with suffix `ver_2`
1. Search for DOI `doi:10.5281/zenodo.1172960`
1. Confirm "Belmont Forum..." dataset is found on Zenodo with suffix `ver_2`
1. Search for Concept DOI `doi:10.5281/zenodo.608984`
1. Confirm "Belmont Forum..." dataset is found on Zenodo with suffix `ver_2`
1. Select Register
1. Select "+" > "WT Data Catalog"
1. Add "Belmot Forum..." dataset to Tale
1. Run Tale and confirm contents on file system match Zenodo

The register tests the following cases.
   * Datasets with multiple files
   * Datasets with multiple versions
   * Concept DOI 

* [x] Zenodo dataset via AiWT
1. Open https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdoi.org%2F10.5281%2Fzenodo.820575&name=Automotive%20Sensor%20Data
1. Confirm Source Data URL and Title match above URL and "READ ONLY" is selected
1. Select JupyterLab environment
1. Select 'Create New Tale'
1. Confirm dataset is mounted read-only, citation and related ids are as expected,
1. Run IDE and use terminal to `ls ../data/Auto*` to confirm zipfile is not extracted

The register tests the following cases.
   * Dataset with single file (zipfile)

* [x] Zenodo Binder via AiWT
1. Open https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdoi.org%2F10.5281%2Fzenodo.3242073&environment=JupyterLab&name=Simple%20requirements.txt%20based%20example%20for%20repo2docker&asTale=true
1. COnfirm Source Data and Title match URL and "READ WRITE" is selected
1. Select 'Create Tale'
1. Confirm image builds and notebook can run

* [x] Zenodo Tale via AiWT
1. Open https://girder.stage.wholetale.org/api/v1/integration/zenodo?doi=10.5281%2Fzenodo.490449&resource_server=sandbox.zenodo.org
1. Confirm Water Tale is created. 
1. Confirm citation on Metadata view
1. Confirm DOI matches above
1. Use the Swagger UI to confirm the `publishInfo` field on the tale object contains pid, repository, repository_id, uri
1. Run the tale and and notebook, confirm it works as expected (produces glorious map of Texas) 

* [x] Publish/import to/from Zenodo Sandbox
1. Follow instructions above to configure your API key for Zenodo sandbox 
1. Create your own Tale and publish to sandbox.zenodo.org
1. Confirm publishInfo contains Sandbox entry
1. Delete tale
1. Import tale from Zenodo
1. Confirm publishInfo
1. Modify the tale
1. Re-publish to sandbox.zenodo.org
1. Delete tale
1. Import tale from Zenodo
1. Confirm publishInfo
1. Download and extract the exported Tale
1. Execute the `run-local.sh` command to run the Tale locally

## Versioning Tests

* [x] Tale Versioning
1. View a Tale that you own
1. Expand Tale Version panel (on the right)
1. Create a New Version: `init`
1. Modify metadata, upload a file, and attach a dataset to the Tale
1. Create another new version: `modified`
1. While viewing Run > Files > External Data, restore from `init` - you should see no datasets
1. Rename the `modified` Version to `something else`
1. In the same view, restore back to `something else` - you should see the dataset you added
1. Run the tale. Confirm the versions are available under `../versions` and that the contents including metadata are as expected
1. View Info for a version
1. Rename a version. Confirm rename in container.
1. Export a version. Confirm exported version matches selected version.
1. Delete a version. Confirm deletion in container.

* [x] Copy versions
1. Login as a user with view-only permissions of a tale with versions.
1. Run the tale and confirm versions are copied

* [x] Publish/export version
1. Create a new tale
1. Upload files but do not create a version
1. Publish the tale
1. Note that there is a new version in the version panel
1. Confirm that the data files in the published artifact are correct
1. Repeat for export
1. Confirm manifest contents including vocabulary

## Git Integration Tests

* [x] Compose Tale from Git Repository
1. Select the Create New Tale button on the 'Browse' page
1. Select the `Create Tale from Git Repository` option
1. Paste a link to a Git repository in the correct field
1. Add a name and pick a compute environment
1. Create the Tale
1. Confirm notification progress and success
1. Confirm that the git repository was cloned to the `workspace/` root
1. Run the tale
1. Confirm interaction via git tools (may require environment customization)

* [x] Connect to Git
1. Create a new empty tale
1. Select "Connect to Git Repository" menu item from Tale menu
1. Confirm notification progress and success
1. Once complete, navigate to the Tale workspace and confirm that the repository has been cloned

## Other tests

* [x] Citations
1. Add files from multiple external datasets (e.g.: doi:10.5281/zenodo.3381658, doi:10.5065/D6862DM8, doi:10.5063/F10K26VR)
1. Confirm `Datasets Used` citations and related identifiers are updated

### Event Notification Tests
* [x] Tale sharing/unsharing
1. Login as user1 and user2 in separate browser sessions
1. As user2, select "Shared with me"
1. As user1, share a tale with user2; as user2 confirm the tale appears
1. As user2, select "View" on the shared tale
1. As user1 and 2 edit the tale metadata
1. As user1 save changes; as user2 confirm Tale edited modal
1. As user1, unshare the tale; as user2 confirm Tale unshared modal
1. As user1, re-share with user2
1. As user2, select "Shared with Me"
1. As user1, delete the shared tale; as user2 confirm that the tale disappears

* [ ] Import tale
1. Import a tale from Github, Zenodo, and as Binder
1. Confirm metadata updates accordingly after import completes

FAIL: Notification progress seems to stop for import AinWT from Zenodo

## Vocabulary Tests

* [x] Vocabularies
1. Go to https://vocabularies.stage.wholetale.org/wt/1.0/index.html
1. Confirm that it matches https://whole-tale.github.io/serialization-format/wt/1.0/

## Forward Authentication Test
* [x] Forward authentication
1. Start a tale
1. Confirm that the tale is accessible via iframe and popout
1. Right click popout icon and copy URL (this is necessary to get JupyterLab token)
1. Open an incognito window and paste the copied URL
1. Confirm that you are prompted for authentication and can access the tale once authenciated
1. Login as a user who does not have permission to access the tale
1. Confirm that they receive an access denied errror

## Changing the Environment
* [x] Changing the environment
1. Create a new JupyterLab tale
1. Run the Tale, confirm JupyterLab IDE displays
1. Edit metadata, change environment to "RStudio"
1. Confirm icon changes in title bar and Environment section
1. Stop/run tale or rebuild/restart and confirm RStudio IDE displays
craig-willis commented 3 years ago

Issues:

craig-willis commented 3 years ago

Notes/thoughts: