whole-tale / wt-design-docs

MIT License
5 stars 9 forks source link

v0.9rc3 testing (v0.9rc3-0-g3a6bf5c on OSX/Chrome) #133

Open bodom0015 opened 4 years ago

bodom0015 commented 4 years ago

Test Plan

https://github.com/whole-tale/wt-design-docs/issues/new?template=test_plan.md

Note: For all tests, repeat for supported browser/OS combinations.

Preconditions:

Authentication

Navigation

Browse

Preconditions:

STAGE: LIGO Tale missing from dashboard.stage.wholetale.org - could it be marked as private by mistake?

TEST: Filter is no longer reset

STAGE: LIGO Tale missing from dashboard.stage.wholetale.org - could it be marked as private by mistake?

STAGE: LIGO Tale missing from dashboard.stage.wholetale.org - could it be marked as private by mistake?

Manage

Preconditions

BUG: Needed refresh after deleting datasets to see newly registered ones

BUG: Searching for this DOI/URL only returned No matching results found.

Run

Preconditions:

BUG: Adding too many datasets pushes select-data-modal's Submit button of the screen

BUG: Deleting a single file from a folder cleared multiple files from view - this has already been fixed in master

BUG: Rename allows duplicates, but a refresh reveals that saving has failed BUG: "Copy to Workspace" threw a stack trace when attempting to move/copy a specific instance of a file - unclear what caused the error: Failed {"message": "OSError: OSError('Specified file does not exist: /collection/WholeTale Workspaces/WholeTale Workspaces/5e39bf2c8bec16c2663dd8d5/1m.dat',)", "trace": ["<FrameSummary file /girder/girder/api/rest.py, line 630 in endpointDecorator>", "<FrameSummary file /girder/girder/api/rest.py, line 1224 in POST>", "<FrameSummary file /girder/girder/api/rest.py, line 967 in handleRoute>", "<FrameSummary file /girder/girder/api/access.py, line 63 in wrapped>", "<FrameSummary file /girder/girder/api/describe.py, line 709 in wrapped>", "<FrameSummary file /girder/girder/api/v1/resource.py, line 374 in copyResources>", "<FrameSummary file /girder/girder/models/item.py, line 445 in copyItem>", "<FrameSummary file /girder/girder/events.py, line 314 in trigger>", "<FrameSummary file /girder/plugins/wt_home_dir/server/init.py, line 80 in handler>", "<FrameSummary file /girder/plugins/wt_home_dir/server/lib/EventHandlers.py, line 214 in run>", "<FrameSummary file /girder/plugins/wt_home_dir/server/lib/EventHandlers.py, line 199 in assertIsValidFile>"], "type": "internal"}

Settings

BUG: http://dataverse.harvard.org/ is down - cannot generate or add API keys. Validation error message is blank in this case

Tale Creation

BUG: Invalid metadata prevents deleting Tale { "field": "id", "message": "No such item: 59aeb3f246a83d0001ab6771", "type": "validation" }

STAGE: Clearing out old datasets from staging may break some existing Tales - need to manually remove datasets before Girder allows you to delete

Analyze in Whole Tale

Task 1: Importing a Dataset from Dataverse

These test cases cover potential situations that can occur when importing datasets from Dataverse.

Testing Steps:

  1. Remove all running Tale instances
  2. Navigate to https://girder.stage.wholetale.org/api/v1/integration/dataverse?datasetPid=doi%3A10.7910%2FDVN%2F3MJ7IR&siteUrl=https%3A%2F%2Fdataverse.harvard.edu
  3. Confirm that the Tale title reads as Replication Data for: "Agricultural Fires and Health at Birth"
  4. Confirm that the only item in the Input data section matches the uri with Data Source appended
  5. Confirm that no environment is selected
  6. Confirm that the Create New Tale and Launch button is disabled
  7. Select an environment
  8. Confirm that READ/WRITE is selected
  9. Click Create New Tale and Launch
  10. Confirm that you are redirected to the run page
  11. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  12. Confirm that the data exists under Tale Workspace
  13. Confirm that the Tale's category is science

Task 2: Importing a Dataset from DataONE

These test cases cover different scenarios that may happen when importing datasets from DataONE.

This test covers the ideal case where dataset importing has all of the needed information to properly run. This includes the dataset location, title, and selected environment.

Testing Steps:

  1. Remove all running Tale instances
  2. Navigate to https://girder.stage.wholetale.org/api/v1/integration/dataone?uri=https%3A%2F%2Fsearch.dataone.org%2Fview%2Fdoi%3A10.18739%2FA2VQ2S94D&title=Fire%20influences%20on%20forest%20recovery%20and%20associated%20climate%20feedbacks%20in%20Siberian%20Larch%20Forests%2C%20Russia&environment=RStudio
  3. Confirm that the Tale title reads as Fire influences on forest recovery and associated climate feedbacks in Siberian Larch Forests, Russia
  4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
  5. Confirm that Rstudio is selected in the Environments widget
  6. Confirm that the Create New Tale and Launch button is enabled
  7. Confirm READ ONLY is selected
  8. Click Create New Tale and Launch
  9. Confirm that you are redirected to the run page
  10. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  11. Confirm that the data exists in the Tale under External Data
  12. Confirm that the Tale's category is science

Links:

  1. https://dashboard.stage.wholetale.org/browse?api=https%3A%2F%2Fdev.nceas.ucsb.edu%2Fknb%2Fd1%2Fmn%2Fv2&name=Dataone Dataset&uri=https://dev.nceas.ucsb.edu/view/doi:10.5072/FK2K075M25
  2. https://dashboard.stage.wholetale.org/browse?api=https://test.arcticdata.io/metacat/d1/mn/v2&name=Dataone Dataset&uri=https://test.arcticdata.io/view/doi:10.5065/D6N014NC

Testing Steps:

  1. Remove all running Tale instances
  2. Navigate to each of the two links above
  3. Select an environment
  4. Click Create New Tale and Launch
  5. Confirm that the correct data exists in the Tale under External Data

TEST: Space(s) in test URL(s) should be replaced with %20

Task 3: Shared Behavior

These cases show errors that are common between Dataverse and DataONE. Although they are tested using DataONE URIs, the errors should be the same when replacing it with a Dataverse URI.

This test covers the case where the name query parameter is missing. The expected behavior is that the uri query parameter is used instead.

Testing Steps:

  1. Remove all running Tale instances
  2. Navigate to https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdataverse.harvard.edu%2Fapi%2Faccess%2Fdatafile%2F3323458
  3. Confirm that the Tale title reads as https://dataverse.harvard.edu/api/access/datafile/3323458
  4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
  5. Confirm that no Environment is selected
  6. Confirm that the Create New Tale and Launch button is disabled
  7. Select an environment
  8. Click Create New Tale and Launch
  9. Confirm that the notification bar appears & properly updates
  10. Confirm that you are redirected to the run page
  11. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  12. Confirm that the data exists in the Tale under External Data

This test covers the case where a user imports data from an external source and selects READ/WRITE in the Create Tale Modal.

Testing Steps (Variant 1):

  1. Remove all running Tale instances
  2. Navigate to https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdataverse.harvard.edu%2Fapi%2Faccess%2Fdatafile%2F3323458
  3. Confirm that the Tale title matches the URI above
  4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
  5. Confirm that the Create New Tale and Launch button is disabled
  6. Select an environment
  7. Select READ/WRITE
  8. Click Create New Tale and Launch
  9. Confirm that the notification bar appears & properly updates
  10. Confirm that you are redirected to the run page
  11. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  12. Confirm that the data exists in the Tale Workspace

Testing Steps (Variant 2):

  1. Remove all running Tale instances
  2. Navigate to https://dashboard.stage.wholetale.org/browse?api=https%3A%2F%2Fdev.nceas.ucsb.edu%2Fknb%2Fd1%2Fmn%2Fv2&environment=JupyterLab&name=proveit&uri=doi%3A10.5072%2FFK27P92Z55
  3. Confirm that the Tale title matches the dataset
  4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
  5. Confirm that the Create New Tale and Launch button is disabled
  6. Select READ/WRITE
  7. Click Create New Tale and Launch
  8. Confirm that the notification bar appears & properly updates
  9. Confirm that you are redirected to the run page
  10. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  11. Confirm that the data exists in the Tale Workspace

Tale metadata tests

The purpose of these tests are to confirm that the metadata files (manifest.json, environment.json, LICENSE) we generate are correct.

Publishing tests

Zenodo integration tests

The register tests the following cases.

The register tests the following cases.

  1. Open https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdoi.org%2F10.5281%2Fzenodo.3242073&environment=JupyterLab&name=Simple%20requirements.txt%20based%20example%20for%20repo2docker&asTale=true
  2. COnfirm Source Data and Title match URL and "READ WRITE" is selected
  3. Select 'Create and Launch Tale'
  4. Confirm image builds and notebook can run
  1. Open https://girder.stage.wholetale.org/api/v1/integration/zenodo?doi=10.5072%2Fzenodo.490450&resource_server=sandbox.zenodo.org
  2. Confirm Water Tale is created.
  3. Confirm citation on Metadata view
  4. Confirm DOI matches above
  5. Use the Swagger UI to confirm the publishInfo field on the tale object contains pid, repository, repository_id, uri
  6. Run the tale and and notebook, confirm it works as expected (produces glorious map of Texas)

BUG: Notebook cell throws an error - No Texas :(

AttributeError Traceback (most recent call last)

in 12 13 output_notebook() ---> 14 palette.reverse() 15 16 # Choose a state to plot AttributeError: 'tuple' object has no attribute 'reverse' * [x] Publish/import to/from Zenodo Sandbox 1. Follow instructions above to configure your API key for Zenodo sandbox 1. Create your own Tale and publish to sandbox.zenodo.org 1. Confirm publishInfo contains Sandbox entry 1. Delete tale 1. Import tale from Zenodo 1. Confirm publishInfo 1. Modify the tale 1. Re-publish to sandbox.zenodo.org 1. Delete tale 1. Import tale from Zenodo 1. Confirm publishInfo 1. Download and extract the exported Tale 1. Execute the `run-local.sh` command to run the Tale locally ### Regression tests * [x] Internal-state problems ([#326](https://github.com/whole-tale/dashboard/pull/326)) 1. Open a non-incognito browser tab, log into the UI and navigate to the `/manage` view 1. Create a folder (e.g. `test123`) and click down into it 1. Open an incognito browser tab, login to the UI again, and navigate to the `/manage` view 1. In the incognito tab, delete the `test123` folder and close the incognito tab 1. Refresh your non-incognito browser tab 1. You should **not** be stuck on a blue screen 1. You should **not** need to manually refresh the page 1. You should be taken to the `/manage` view of your Home folder 1. NOTE: Your last refresh may take a couple of seconds longer than usual * [x] Invalid instance ID ([#333](https://github.com/whole-tale/dashboard/pull/333)) 1. Manually change the path suffix in the address bar to `/run/ThisIsObviouslyNotARealID` 1. You should be brought to `/browse` page instead * [x] Files don't overflow Manage page ([#410](https://github.com/whole-tale/dashboard/pull/410)) 1. Navigate to the manage page 2. Register enough/upload enough data to the point where you need to scroll through files on the right hand side 3. Make sure that the files don't overflow the container, also check in Firefox * [x] Handle read only Tales ([#406](https://github.com/whole-tale/dashboard/pull/406)) 1. Login to the WholeTale Dashboard 2. View a Tale that allows you to read, but not write 3. Navigate to Run > Files > Tale Workspaces * You should see the Tale's Workspace folders/files are listed here * You should see that (+) has been greyed out, disallowing writes 4. Navigate to Run > Files > External Data * You should see the Tale's External Datasets are listed here * You should see that (+) has been greyed out, disallowing writes
bodom0015 commented 4 years ago

Results (see above for context)

PR for test plan improvements: https://github.com/whole-tale/wt-design-docs/pull/134

Key

Xarthisius commented 4 years ago
  • BUG: dataverse.harvard.org is down - cannot generate or add API keys. Validation error message is blank in this case

Where did you see this? It's .edu not .org That's why it's not there.

Xarthisius commented 4 years ago
  • BUG: dataverse.harvard.org is down - cannot generate or add API keys. Validation error message is blank in this case

Where did you see this? It's .edu not .org That's why it's not there.

I thought this went in for v0.9rc3, but it has not. Sorry!

bodom0015 commented 4 years ago

Ah, that makes sense :+1: I probably should've noticed that hehe

It was when connecting an external account via API key.. I was using the link in the modal to set link me off to each server and set up API keys, but the last Dataverse one didn't work.