TEST: Need to correct step for opening search modal and titles for additional URLs would be helpful
NOTE: Updated DASHBOARD_URL to point to prod_cn on stage for this to work.
[x] Dataverse
Register dataset doi:10.7910/DVN/SZ9YXZ
Confirm "Replication Data for: Direct Democracy and Women's Political Engagement" from Dataverse succeeds
Launch (Copy and Run) the Ligo Tale and wait for the instance to start
Confirm you are redirected to the Run page
Confirm that the Run panel displays the Tale icon, title, and author
Confirm that the Run page displays the Interact, Files, Metadata tabs
Select the "..." menu.
Confirm that menu options exist for "Rebuild Tale", "Restart Tale", "Save Tale Version", "Duplicate Tale", "Publish Tale", "Export Tale", "Connect to Git Repository..."
Confirm that the "View Fullscren" link displays the Run panel in full screen
TEST: Test case inconsistencies with release
[x] Interact tab
Select the "Interact" tab
Confirm Jupyter environment is displayed in iframe
Select "popout" icon. Confirm Jupyter is opened in new Tab
[x] Files tab
Select "Files" tab
You should see a vertical navigation menu with three tabs: "Home", "External Data", and "Tale Workspace".
[ x] Metadata tab - Owned Tale
Open a Tale that you own
Navigate to the metadata page
Add multiple datasets that have DOIs to the Tale
Navigate to Run > Metadata
Valid environments should be listed in the Environment dropdown menu
Valid licenses should be listed in the License dropdown
Confirm that you see Created by <your_name> under Authors
Confirm that your Tale has the CC4 license assigned to it
Confirm that the Tale's Published Location reads This Tale has not been published
Confirm that there are citations for the external datasets
Click the "Edit" button to edit the metadata fields
Change the name of the Tale, Category, Description, Illustration, Public, License, and add multiple authors
Select Save. Confirm that you see a notification telling you that the Tale was saved.
Refresh the page. Your changes should be preserved between page refreshes, confirming that they are being properly persisted.
Confirm that the new Tale authors are displayed on the Tale card
[x] Metadata tab - Non-Owned Tale
Open a Tale that you don't own
Navigate to the metadata page
Note that there isn't an 'Edit' button present
[ ] Home
Select Files > Home. Confirm that your home directory displays
Create a folder and upload a file to the folder
Confirm file operations:
Copy, rename, remove, download, Move To...
NOTE: Home folder is not present on public tale? Test case should state select owned tale since previous was public
FAIL: Creating folder with duplicate name fails with 500 (https://github.com/whole-tale/virtual_resources/pull/14?)
TEST: File operations test could use some details
[x] External data
Precondition: Register one or more Datasets (e.g. doi:10.18739/A29G5GD0V)
Confirm notification progress and success
Confirm "External Data" list is initially empty
Confirm (+) button ("Select Data modal") behavior
Confirm that "WT Data Catalog" shows all registered datasets
Confirm that "My External Data" filters to only show your registered datasets
Select a dataset
Confirm that clicking "Add Selected" adds the dataset to the "Selected Data" column
Confirm that "Remove Selected" data to dissociate it with a Tale
Using the dataset above, add the dataset to the Tale, by clicking "Save"
Confirm that the Tale's "External Data" list matches what was saved
Remove the dataset from the Tale
Confirm "External Data" list matches what was saved
Select a dataset but then click out of the modal or click "Cancel"
Confirm that the changes weren't saved
[ ] Workspaces
Confirm create folder
Confirm upload file
Select "+" button has options "New folder", "Upload file", Import Tale Data..."
Folder operations
Confirm context menu Copy, Rename, Download, Remove
Confirm rename folder
Confirm download folder (this should result in a zip file)
Confirm remove folder
File operations
Upload a file
Confirm context menu Copy, Rename, Download, Remove
Confirm rename folder
Confirm download file
Confirm remove file
Select "Select Data..."
Note vertical navigation with Tale Workspaces
Copy to Workspace from another workspace
Move to workspace from another workspace
FAIL: 500 on create duplicate folder name (https://github.com/whole-tale/virtual_resources/pull/14)
NOTE: Downloaded folder contains files but not folder (expected Girder behavior, but possibly note in test case)
NOTE: Select Data workspace sorted by ? Not Tale Title
FAIL: Copy from tale workspace requires manual refresh
[x] Files - Non-Owned Tale
Open a Tale that you don't have write permissions to
Attempt to add data to the workspace and external data
Note that the "+" icons are greyed out, preventing you from doing so
[x] Export Tale
View a Tale that you own
Select "Export Tale" from Tale menu
Confirm that a download starts for the ZIP archive
Open the archive
Confirm bagit format and contents
Run the exported tale and confirm that it builds and runs as expected
Settings
[x] Default State
Navigate to the Settings page
Confirm there are empty settings for DataONE, Dataverse, and Zenodo
[ ] Connect to Zenodo
Click "Connect Account" in the Zenodo section
Confirm that the connection modal opens
Confirm that sandbox.zenodo.org and zenodo.org are listed
Enter your API key for one of the repositories
Click "Connect"
Confirm that you see "Authorized on "
Select disconnect, confirmation window should appear
Repeat test with invalid key, confirm error message appears
Repeat for each Zenodo server
NOTE: API Key link does not provide documentation. Consider adding docs describing what to do to. I think a general RTD link for all repos would suffice.
FAIL: Sandox.zenodo.org repo "Get from links to zenodo.org
[x] Connect to Dataverse
Click "Connect Account" in the Dataverse section
Confirm that the connection modal opens
Enter your API key for one of the repositories
Click "Connect"
Confirm that you see "Authorized on "
Select disconnect, confirmation window should appear
Repeat test with invalid key, confirm error message appears
Repeat for each Dataverse server
NOTE: Stage has dev2.dataverse.org listed
[ ] Connect to DataONE
Click "Connect Account" under the DataONE repository
Confirm that you're redirected to the DataONE login
Log in with ORCID
Confirm that you're redirected back to Whole Tale
Confirm that you see "Authorized" under the DataONE server
Confirm that the "Connect Account" Button is disabled
Confirm that the Tale title reads as Replication Data for: "Agricultural Fires and Health at Birth"
Confirm that the only item in the Input data section matches the uri with Data Source appended
Confirm that no environment is selected
Confirm that the Create New Tale button is disabled
Select an environment
Confirm that READ/WRITE is selected
Click Create New Tale
Confirm notification progress and success
Confirm that you are redirected to the run page
Confirm that the Tale name matches the Tale Name in the Create Tale Modal
Confirm that the data exists under Tale Workspace
Confirm that the Tale's category is science
Task 2: Importing a Dataset from DataONE
These test cases cover different scenarios that may happen when importing datasets from DataONE.
[x] Case 1: Ideal Scenario:
This test covers the ideal case where dataset importing has all of the needed information
to properly run. This includes the dataset location, title, and selected environment.
Confirm that the Tale title reads as Fire influences on forest recovery and associated climate feedbacks in Siberian Larch Forests, Russia
Confirm that the only item in the Selected data section matches the uri with Data Source appended
Confirm that Rstudio is selected in the Environments widget
Confirm that the Create New Tale button is enabled
Confirm READ ONLY is selected
Click Create New Tale
Confirm that you are redirected to the run page
Confirm that the Tale name matches the Tale Name in the Create Tale Modal
Confirm that the data exists in the Tale under External Data
Confirm that the Tale's category is science
[x] Case 2: Testing Different DataONE Deployments
The Analyze in Whole Tale feature is deployed across a few DataONE test sites. This test ensures Whole Tale has the ability to import
data from each of them.
Confirm that the correct data exists in the Tale under External Data
Task 3: Shared Behavior
These cases show errors that are common between Dataverse and DataONE. Although they are tested using DataONE URIs, the errors should be the same when replacing it with a Dataverse URI.
[ ] Case 1: Missing Dataset Title Query Param:
This test covers the case where the name query parameter is missing. The expected behavior is
that the uri query parameter is used instead.
Confirm that the only item in the Selected data section matches the uri with Data Source appended
Confirm that the Create New Tale button is disabled
Select READ/WRITE
Click Create New Tale
Confirm that the notification bar appears & properly updates
Confirm that you are redirected to the run page
Confirm that the Tale name matches the Tale Name in the Create Tale Modal
Confirm that the data exists in the Tale Workspace
Tale metadata tests
The purpose of these tests are to confirm that the metadata files (manifest.json, environment.json, LICENSE) we generate are correct.
[x] manifest.json
Create or view a Tale that you own
Add a dataset to the Tale
Add at least one author
Add at least one file to the workspace
Export the Tale (as BagIt)
Open the manifest.json file under metadata/
Confirm that the author is listed under schema:author
Confirm that you are listed under createdBy
Confirm that the file(s) in the workspace are properly listed in aggregates
Confirm that the external dataset is referenced in aggregates
Confirm that the dataset is also listed under Datasets
[x] environment.json
Open the archive from the manifest.json test
Navigate to metadata/environment.json
Open the girder web API
Locate and GET the Image that the Tale used
Confirm that the config section from girder matches the config section in environment.json
[x] LICENSE
Open the archive from the manifest.json test
Open the LICENSE file
Navigate to Run > metadata
Confirm that the opened LICENSE file matches the selected license on the metadata page
Change the Tale's license
Export the Tale
Confirm that the LICENSE changed appropriately
Tale Sharing Tests
These tests are centered around the feature for sharing Tales with other Whole Tale
users. To run these tests, register at least one other account, referred to as User B.
[x] Case 1: Sharing with 'Can View' permissions
Create a new Tale
Add data to the workspace and add at least one dataset
Navigate to the 'Share' tab
Share the Tale with User B
Confirm that the default permission is 'Can View'
Switch the account to User B
Navigate to the 'Shared with Me' tab on the Browse page
Confirm that you see the Tale from step 1
Confirm that you can't edit the Workspace, External Data, or metadata
[x] Case 2: Sharing with 'Can Edit' permissions
Create a new Tale
Add data to the workspace and add at least one dataset
Navigate to the 'Share' tab
Share the Tale with User B
Set the permissions to 'Can Edit'
Switch the account to User B
Navigate to the 'Shared with Me' tab on the Browse page
Confirm that you see the Tale from step 1
Remove one of the files from the Workspace
Add a dataset to External Data
Change a field in the metadata (title, description, etc)
Switch to User A
Confirm that the changes made to the Tale persisted
Publishing tests
[x] Case 1: No Accounts Configured
Disconnect all of your external accounts
Launch a Tale that you own
Click the ellipsis dropdown on the Run page
Click Publish tale...
Confirm that your are notified to link accounts on the settings page
Click Go to Settings
Confirm you are re-directed to settings
[] Case 2: Publishing to a Third Party
Connect to a third party provider (make sure it's a dev server)
Open a Tale that you own
Add some data to it
Click Publish tale...
Confirm that the publishing modal opens
Confirm that the third party that you are connected to appears in the dropdown
Select the third party
Click Publish
Once complete, navigate to the metadata page
Confirm that the published location is present
Visit the published location
Confirm that you see the following files plus yours
metadata.xml
LICENSE
README.md
manifest.json
environment.json
Repeat for each third party provider
a. After publishing to DataONE, use girder to query the Tale
b. Ensure that the 'pid' field in the 'publishInfo' object has an identifier in a doi format
FAIL: Can't publish to DataONE
[x] Case 3: Re-published Tale
Launch a Tale that has been published before
Take note of the Published Location in Run > metadata
Re-publish the Tale to the third party
Confirm that the Published Location has changed to the correct package landing page
NOTE: Tested for Zenodo
Zenodo integration tests
[x] Register Zenodo data
Select Manage > Data tab
Select "+". Confirm "Search for Data Sources" modal displays
Search for Concept DOI doi:10.5281/zenodo.16384
Confirm "Belmont Forum..." dataset is found on Zenodo with suffix ver_1
Search for URL https://zenodo.org/record/1172960#.XidIvlNKh24
Confirm "Belmont Forum..." dataset is found on Zenodo with suffix ver_2
Search for DOI doi:10.5281/zenodo.1172960
Confirm "Belmont Forum..." dataset is found on Zenodo with suffix ver_2
Search for Concept DOI doi:10.5281/zenodo.608984
Confirm "Belmont Forum..." dataset is found on Zenodo with suffix ver_2
Select Register
Confirm dataset appears in Manage Data list
Add to tale, run tale and list data dir to confirm contents match Zenodo version
Confirm that the tale is accessible via iframe and popout
Open an incognito window and access the tale URL
Confirm that you are prompted for authentication and can access the tale once authenticated
Login as a user who does not have permission to access the tale
Confirm that they receive an access denied error
Changing the Environment
[ ] Changing the environment
Create a new JupyterLab tale
Run the Tale, confirm JupyterLab IDE displays
Edit metadata, change environment to "RStudio"
Confirm icon changes in title bar and Environment section
Stop/run tale or rebuild/restart and confirm RStudio IDE displays
FAIL: Restart after rebuild fails with:
[2021-05-14 17:31:48,211: ERROR/ForkPoolWorker-5] Task gwvolman.tasks.update_container[f3e4857c-d2db-4905-8afd-26b342ae829c] raised unexpected: RuntimeError('Restarting the Tale failed with "update paused due to failure or early termination of task 3r2t2ngowx0u9ovshd0b5afee"')
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 412, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/girder_worker/task.py", line 148, in __call__
results = super(Task, self).__call__(*_t_args, **_t_kwargs)
File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 704, in __protected_call__
return self.run(*args, **kwargs)
File "/gwvolman/gwvolman/tasks.py", line 269, in update_container
raise RuntimeError(
RuntimeError: Restarting the Tale failed with "update paused due to failure or early termination of task 3r2t2ngowx0u9ovshd0b5afee"
docker service ps tmp-ixzecwnwcme4 --no-trunc
m0wy6hhav6y65np77dcouwqxn \_ tmp-ixzecwnwcme4.1 registry.stage.wholetale.org/5f71f410a4f22e103392e760/1620934770@sha256:055d72e9feeaef67084798e5cc27e6569ecaf2f10aa04778c5bf000541751f02 wt-stage-02 Shutdown Failed 1 second ago "starting container failed: OCI runtime create failed: container_linux.go:367: starting container process caused: exec: "jupyter": executable file not found in $PATH: unknown"
~We should have a set of test repos for each IDE similar to matlab and stata to ease testing~
Why can't I copy/paste text from External Data?
Since I can't sort, My Tales should be listed by last updated?
Uploaded file not displayed for shared user without refresh
Version created on export could have name indicating it was created during export (e.g., EXPORT-DATE)
Sometimes I can't get the version menu to appear, no errors in console
Staging Water Tale fails due to unpinned Bokeh -- fix or replace in test
I shouldn't be able to restart a stopped tale
Need to think about how to clarify the rebuild/restart requirement. Maybe a notification when a change is detected (i.e., you've changed your image or changed the r2d config, these changes will not be reflected until you rebuild/restart).
Author names are separated by " ," instead of ", "
One I dismiss notifications, I can't get back to the status or log of a building tale. Would be nice to be able to get that info
Current OpenRefine version is 3.4.1; Apache Spark 3.1.1; JupyterLab 3.0.15; R 4.0.5; Just to say, we might want to go out with more recent versions of IDEs? Our openrefine is using xenial base?
Test Plan
https://github.com/whole-tale/wt-design-docs/issues/new?template=TEST_PLAN.md
Note: For all tests, repeat for supported browser/OS combinations.
Preconditions:
Splash page
Authentication
NOTE: Post auth re-display of splash page is unexpected.
[x] Basic logout flow
[x] Return-route for non-logged in users
Navigation
TEST: Minor test case inconsistency re: "report a problem" should be "bug" icon
Tale Dashboard
Preconditions:
Assumes production Tales present (e.g., LIGO, materials, etc).
No running instances
[x] General
[x] Search
TEST: Minor test inconsistency with search for "Ligo" instead of "LIGO"
FAIL: Interact displays by default (known) TEST: Test case inconsistencies. Step 4 should be Public Tales and step 5.4 should be My Tales
FAIL: Redirected to interact, not metadata on copy on launch FAIL: Redirected to metadata on view after run (expected interact)
Managing Data
Preconditions
Empty home directory
No registered data
[x] Register General
[x] Register DataONE data
10.5065/D6862DM8
TEST: Need to correct step for opening search modal and titles for additional URLs would be helpful NOTE: Updated DASHBOARD_URL to point to prod_cn on stage for this to work.
[x] Dataverse
[ ] Globus/MDF
FAIL: https://github.com/whole-tale/girder_wholetale/pull/481
Run Tale
Preconditions:
No running Tale instances
[x] General
TEST: Test case inconsistencies with release
[x] Interact tab
[x] Files tab
[ x] Metadata tab - Owned Tale
Environment
dropdown menuLicense
dropdownCreated by <your_name>
underAuthors
Published Location
readsThis Tale has not been published
[x] Metadata tab - Non-Owned Tale
[ ] Home
NOTE: Home folder is not present on public tale? Test case should state select owned tale since previous was public FAIL: Creating folder with duplicate name fails with 500 (https://github.com/whole-tale/virtual_resources/pull/14?) TEST: File operations test could use some details
[x] External data
doi:10.18739/A29G5GD0V
)[ ] Workspaces
FAIL: 500 on create duplicate folder name (https://github.com/whole-tale/virtual_resources/pull/14) NOTE: Downloaded folder contains files but not folder (expected Girder behavior, but possibly note in test case) NOTE: Select Data workspace sorted by ? Not Tale Title FAIL: Copy from tale workspace requires manual refresh
[x] Files - Non-Owned Tale
[x] Export Tale
Settings
[x] Default State
[ ] Connect to Zenodo
NOTE: API Key link does not provide documentation. Consider adding docs describing what to do to. I think a general RTD link for all repos would suffice. FAIL: Sandox.zenodo.org repo "Get from links to zenodo.org
NOTE: Stage has dev2.dataverse.org listed
FAIL: DataONE auth fails, results in empty key
/user/me
endpoint to confirm tokens are still presentTale Creation
TODO: Needs rebuild test case NOTE: Need to decide what to do about different RStudio versions
[x] Compose Jupyter Notebook Tale
[x] Compose JupyterLab Tale
[x] Compose JupyterLab with Spark
[x] Compose MATLAB Desktop Tale
multiplicative_arima_example.m
, confirm outputs[x] Compose MATLAB Jupyter Kernel Tale
multiplicative_arima_example.ipynb
, confirm outputs matchmultiplicative_arima_example.html
[x] Compose STATA Desktop Tale
example.do
, confirm outputs[x] Compose STATA Jupyter Kernel Tale
example.ipynb
, confirm outputs matchexample.html
[ ] Compose OpenRefine Tale
[x] Too many instances
TEST: Revise test case
Analyze in Whole Tale
Task 1: Importing a Dataset from Dataverse
These test cases cover potential situations that can occur when importing datasets from Dataverse.
Testing Steps:
Replication Data for: "Agricultural Fires and Health at Birth"
Input data
section matches the uri withData Source
appendedCreate New Tale
button is disabledCreate New Tale
science
Task 2: Importing a Dataset from DataONE
These test cases cover different scenarios that may happen when importing datasets from DataONE.
This test covers the ideal case where dataset importing has all of the needed information to properly run. This includes the dataset location, title, and selected environment.
Testing Steps:
Fire influences on forest recovery and associated climate feedbacks in Siberian Larch Forests, Russia
Selected data
section matches the uri withData Source
appendedRstudio
is selected in the Environments widgetCreate New Tale
button is enabledCreate New Tale
science
Analyze in Whole Tale
feature is deployed across a few DataONE test sites. This test ensures Whole Tale has the ability to import data from each of them.Links:
Testing Steps:
Create New Tale
Task 3: Shared Behavior
These cases show errors that are common between Dataverse and DataONE. Although they are tested using DataONE URIs, the errors should be the same when replacing it with a Dataverse URI.
This test covers the case where the name query parameter is missing. The expected behavior is that the uri query parameter is used instead.
Testing Steps:
https://dataverse.harvard.edu/api/access/datafile/3323458
Selected data
section matches the uri withData Source
appendedCreate New Tale
button is disabledCreate New Tale
FAIL: Tale title is blank
This test covers the case where a user imports data from an external source and selects READ/WRITE in the Create Tale Modal.
Testing Steps (Variant 1):
Selected data
section matches the uri withData Source
appendedCreate New Tale
button is disabledREAD/WRITE
Create New Tale
FAIL: Tale title is blank
Testing Steps (Variant 2):
Selected data
section matches the uri withData Source
appendedCreate New Tale
button is disabledREAD/WRITE
Create New Tale
Tale metadata tests
The purpose of these tests are to confirm that the metadata files (manifest.json, environment.json, LICENSE) we generate are correct.
[x] manifest.json
manifest.json
file undermetadata/
schema:author
createdBy
aggregates
aggregates
Datasets
[x] environment.json
metadata/environment.json
config
section from girder matches theconfig
section inenvironment.json
[x] LICENSE
LICENSE
fileLICENSE
file matches the selected license on the metadata pageLICENSE
changed appropriatelyTale Sharing Tests
These tests are centered around the feature for sharing Tales with other Whole Tale users. To run these tests, register at least one other account, referred to as User B.
[x] Case 1: Sharing with 'Can View' permissions
[x] Case 2: Sharing with 'Can Edit' permissions
Publishing tests
[x] Case 1: No Accounts Configured
Publish tale...
Go to Settings
[] Case 2: Publishing to a Third Party
Publish tale...
Publish
metadata.xml
LICENSE
README.md
manifest.json
environment.json
FAIL: Can't publish to DataONE
Published Location
in Run > metadataPublished Location
has changed to the correct package landing pageNOTE: Tested for Zenodo
Zenodo integration tests
doi:10.5281/zenodo.16384
ver_1
https://zenodo.org/record/1172960#.XidIvlNKh24
ver_2
doi:10.5281/zenodo.1172960
ver_2
doi:10.5281/zenodo.608984
ver_2
The register tests the following cases.
The register tests the following cases.
publishInfo
field on the tale object contains pid, repository, repository_id, uriFAILS: Bokeh
run-local.sh
command to run the Tale locallyVersioning Tests
init
modified
init
- you should see no datasetsmodified
Version tosomething else
something else
- you should see the dataset you added../versions
and that the contents including metadata are as expectedFAIL: "This Tale was updated by another user." popup on unshared tale
[x] Copy versions
[x] Publish/export version
Git Integration Tests
[x] Compose Tale from Git Repository
Create Tale from Git Repository
optionworkspace/
root[x] Connect to Git
Other tests
Datasets Used
citations and related identifiers are updatedEvent Notification Tests
FAIL: Fields do not update for user2 (only displayed title, not field)
Vocabulary Tests
Forward Authentication Test
Changing the Environment
FAIL: Restart after rebuild fails with: