search

whole-tale / wt-design-docs

MIT License
5 stars 9 forks source link

v1.0rc2 testing (on OSX Catalina / Chrome 90.0.4430.93) #153

Closed craig-willis closed 3 years ago

craig-willis commented 3 years ago

Test Plan

https://github.com/whole-tale/wt-design-docs/issues/new?template=TEST_PLAN.md

Note: For all tests, repeat for supported browser/OS combinations.

Preconditions:

Splash page

Authentication

NOTE: Post auth re-display of splash page is unexpected.

Navigation

TEST: Minor test case inconsistency re: "report a problem" should be "bug" icon

Tale Dashboard

Preconditions:

TEST: Minor test inconsistency with search for "Ligo" instead of "LIGO"

FAIL: Interact displays by default (known) TEST: Test case inconsistencies. Step 4 should be Public Tales and step 5.4 should be My Tales

FAIL: Redirected to interact, not metadata on copy on launch FAIL: Redirected to metadata on view after run (expected interact)

Managing Data

Preconditions

TEST: Need to correct step for opening search modal and titles for additional URLs would be helpful NOTE: Updated DASHBOARD_URL to point to prod_cn on stage for this to work.

FAIL: https://github.com/whole-tale/girder_wholetale/pull/481

Run Tale

Preconditions:

TEST: Test case inconsistencies with release

NOTE: Home folder is not present on public tale? Test case should state select owned tale since previous was public FAIL: Creating folder with duplicate name fails with 500 (https://github.com/whole-tale/virtual_resources/pull/14?) TEST: File operations test could use some details

FAIL: 500 on create duplicate folder name (https://github.com/whole-tale/virtual_resources/pull/14) NOTE: Downloaded folder contains files but not folder (expected Girder behavior, but possibly note in test case) NOTE: Select Data workspace sorted by ? Not Tale Title FAIL: Copy from tale workspace requires manual refresh

Settings

NOTE: API Key link does not provide documentation. Consider adding docs describing what to do to. I think a general RTD link for all repos would suffice. FAIL: Sandox.zenodo.org repo "Get from links to zenodo.org

NOTE: Stage has dev2.dataverse.org listed

FAIL: DataONE auth fails, results in empty key

Tale Creation

TODO: Needs rebuild test case NOTE: Need to decide what to do about different RStudio versions

TEST: Revise test case

Analyze in Whole Tale

Task 1: Importing a Dataset from Dataverse

These test cases cover potential situations that can occur when importing datasets from Dataverse.

Testing Steps:

  1. Remove all running Tale instances
  2. Navigate to https://girder.stage.wholetale.org/api/v1/integration/dataverse?datasetPid=doi%3A10.7910%2FDVN%2F3MJ7IR&siteUrl=https%3A%2F%2Fdataverse.harvard.edu
  3. Confirm that the Tale title reads as Replication Data for: "Agricultural Fires and Health at Birth"
  4. Confirm that the only item in the Input data section matches the uri with Data Source appended
  5. Confirm that no environment is selected
  6. Confirm that the Create New Tale button is disabled
  7. Select an environment
  8. Confirm that READ/WRITE is selected
  9. Click Create New Tale
  10. Confirm notification progress and success
  11. Confirm that you are redirected to the run page
  12. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  13. Confirm that the data exists under Tale Workspace
  14. Confirm that the Tale's category is science

Task 2: Importing a Dataset from DataONE

These test cases cover different scenarios that may happen when importing datasets from DataONE.

This test covers the ideal case where dataset importing has all of the needed information to properly run. This includes the dataset location, title, and selected environment.

Testing Steps:

  1. Remove all running Tale instances
  2. Navigate to https://girder.stage.wholetale.org/api/v1/integration/dataone?uri=https%3A%2F%2Fsearch.dataone.org%2Fview%2Fdoi%3A10.18739%2FA2VQ2S94D&title=Fire%20influences%20on%20forest%20recovery%20and%20associated%20climate%20feedbacks%20in%20Siberian%20Larch%20Forests%2C%20Russia&environment=RStudio
  3. Confirm that the Tale title reads as Fire influences on forest recovery and associated climate feedbacks in Siberian Larch Forests, Russia
  4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
  5. Confirm that Rstudio is selected in the Environments widget
  6. Confirm that the Create New Tale button is enabled
  7. Confirm READ ONLY is selected
  8. Click Create New Tale
  9. Confirm that you are redirected to the run page
  10. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  11. Confirm that the data exists in the Tale under External Data
  12. Confirm that the Tale's category is science

Links:

  1. https://dashboard.stage.wholetale.org/browse?api=https%3A%2F%2Fcn-stage-2.test.dataone.org%2Fcn%2Fv2&name=Dataone%20Dataset&uri=https://dev.nceas.ucsb.edu/view/doi:10.5072/FK2K075M25
  2. https://dashboard.stage.wholetale.org/browse?api=http%3A%2F%2Fcn-stage.test.dataone.org%2Fcn%2Fv2&name=Dataone%20Dataset&uri=https://test.arcticdata.io/view/doi:10.5065/D6N014NC

Testing Steps:

  1. Remove all running Tale instances
  2. Navigate to each of the two links above
  3. Select an environment
  4. Click Create New Tale
  5. Confirm that the correct data exists in the Tale under External Data

Task 3: Shared Behavior

These cases show errors that are common between Dataverse and DataONE. Although they are tested using DataONE URIs, the errors should be the same when replacing it with a Dataverse URI.

This test covers the case where the name query parameter is missing. The expected behavior is that the uri query parameter is used instead.

Testing Steps:

  1. Remove all running Tale instances
  2. Navigate to https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdataverse.harvard.edu%2Fapi%2Faccess%2Fdatafile%2F3323458
  3. Confirm that the Tale title reads as https://dataverse.harvard.edu/api/access/datafile/3323458
  4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
  5. Confirm that no Environment is selected
  6. Confirm that the Create New Tale button is disabled
  7. Select an environment
  8. Click Create New Tale
  9. Confirm that the notification bar appears & properly updates
  10. Confirm that you are redirected to the Metadata page
  11. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  12. Confirm that the data exists in the Tale under External Data

FAIL: Tale title is blank

This test covers the case where a user imports data from an external source and selects READ/WRITE in the Create Tale Modal.

Testing Steps (Variant 1):

  1. Remove all running Tale instances
  2. Navigate to https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdataverse.harvard.edu%2Fapi%2Faccess%2Fdatafile%2F3323458
  3. Confirm that the Tale title matches the URI above
  4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
  5. Confirm that the Create New Tale button is disabled
  6. Select an environment
  7. Select READ/WRITE
  8. Click Create New Tale
  9. Confirm that the notification bar appears & properly updates
  10. Confirm that you are redirected to the run page
  11. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  12. Confirm that the data exists in the Tale Workspace

FAIL: Tale title is blank

Testing Steps (Variant 2):

  1. Remove all running Tale instances
  2. Navigate to https://dashboard.stage.wholetale.org/browse?api=https%3A%2F%2Fdev.nceas.ucsb.edu%2Fknb%2Fd1%2Fmn%2Fv2&environment=JupyterLab&name=proveit&uri=doi%3A10.5072%2FFK27P92Z55
  3. Confirm that the Tale title matches the dataset
  4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
  5. Confirm that the Create New Tale button is disabled
  6. Select READ/WRITE
  7. Click Create New Tale
  8. Confirm that the notification bar appears & properly updates
  9. Confirm that you are redirected to the run page
  10. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  11. Confirm that the data exists in the Tale Workspace

Tale metadata tests

The purpose of these tests are to confirm that the metadata files (manifest.json, environment.json, LICENSE) we generate are correct.

Tale Sharing Tests

These tests are centered around the feature for sharing Tales with other Whole Tale users. To run these tests, register at least one other account, referred to as User B.

Publishing tests

FAIL: Can't publish to DataONE

NOTE: Tested for Zenodo

Zenodo integration tests

The register tests the following cases.

The register tests the following cases.

  1. Open https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdoi.org%2F10.5281%2Fzenodo.3242073&environment=JupyterLab&name=Simple%20requirements.txt%20based%20example%20for%20repo2docker&asTale=true
  2. COnfirm Source Data and Title match URL and "READ WRITE" is selected
  3. Select 'Create Tale'
  4. Confirm image builds and notebook runs without error
  1. Open https://girder.stage.wholetale.org/api/v1/integration/zenodo?doi=10.5072%2Fzenodo.490450&resource_server=sandbox.zenodo.org
  2. Confirm Water Tale is created.
  3. Confirm citation on Metadata view
  4. Confirm DOI matches above
  5. Use the Swagger UI to confirm the publishInfo field on the tale object contains pid, repository, repository_id, uri
  6. Run the tale and and notebook, confirm it works as expected (produces glorious map of Texas)

FAILS: Bokeh

Versioning Tests

FAIL: "This Tale was updated by another user." popup on unshared tale

Git Integration Tests

Other tests

Event Notification Tests

FAIL: Fields do not update for user2 (only displayed title, not field)

Vocabulary Tests

Forward Authentication Test

Changing the Environment

FAIL: Restart after rebuild fails with:

[2021-05-14 17:31:48,211: ERROR/ForkPoolWorker-5] Task gwvolman.tasks.update_container[f3e4857c-d2db-4905-8afd-26b342ae829c] raised unexpected: RuntimeError('Restarting the Tale failed with "update paused due to failure or early termination of task 3r2t2ngowx0u9ovshd0b5afee"')
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/girder_worker/task.py", line 148, in __call__
    results = super(Task, self).__call__(*_t_args, **_t_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  File "/gwvolman/gwvolman/tasks.py", line 269, in update_container
    raise RuntimeError(
RuntimeError: Restarting the Tale failed with "update paused due to failure or early termination of task 3r2t2ngowx0u9ovshd0b5afee"

docker service ps tmp-ixzecwnwcme4 --no-trunc
m0wy6hhav6y65np77dcouwqxn    \_ tmp-ixzecwnwcme4.1   registry.stage.wholetale.org/5f71f410a4f22e103392e760/1620934770@sha256:055d72e9feeaef67084798e5cc27e6569ecaf2f10aa04778c5bf000541751f02   wt-stage-02         Shutdown            Failed 1 second ago     "starting container failed: OCI runtime create failed: container_linux.go:367: starting container process caused: exec: "jupyter": executable file not found in $PATH: unknown"
craig-willis commented 3 years ago

Summary of issues found/encountered. Most of these are minor.

Other observations (not necessarily issues)