search

whole-tale / wt-design-docs

MIT License
5 stars 9 forks source link

v1.0rc2 testing (v1.0rc2-091f09f4 on OSX Catalina / Chrome 90.0.4430.85) #152

Open bodom0015 opened 3 years ago

bodom0015 commented 3 years ago

Test Plan

https://github.com/whole-tale/wt-design-docs/issues/new?template=TEST_PLAN.md

Note: For all tests, repeat for supported browser/OS combinations.

Preconditions:

Splash page

Authentication

Navigation

Tale Dashboard

Preconditions:

FAIL: Interact page displayed by default, instead of Metadata

Currently, clicking on View leads to interact, while clicking the Tale's title will bring you to metadata. This is probably too obscure / confusing, so I can change these to point to metadata and match the test case.

Also noting that this is inconsistent with "Running Tales", where the two destinations are swapped. I think it definitely makes sense for this to be consistent across cards, at the very least.

FAIL: Redirected to Interact, instead of Metadata (likely due to previous test failure)

Noting that copyOnLaunch does redirect to the run view, but does not explicitly specify a tab. We can add a queryParam here to specify an explicit tab destination, if it would make sense. Interact seems like the sensible choice to me, since a user would only encounter copyOnLaunch when trying to actually Run the Tale, which for me is enough to indicate that a user wants to Interact with it.

Managing Data

Preconditions

FAIL: Dataset name is incorrect, data appears invalid (none of these 5 produced correct results for me on stage)

FAIL: Found 0 results

Run Tale

Preconditions:

TEST: latest now used, instead of stable

TEST: Reorder test steps to reflect reality - e.g. need to click edit before seeing dropdowns, "Created By" no longer listed under authors (see https://github.com/whole-tale/ngx-dashboard/pull/157), etc

FAIL: Copy does not work properly if file is inside of a subfolder of Home folder (copy is placed in the root of the Home folder, UI state is mismatched and invalid) FAIL: minor note that I still encounter https://github.com/whole-tale/ngx-dashboard/issues/8

FAIL: Copy folder fails with a 500: {"message": "Folder 'folder' already exists at /tmp/data/workspaces/5/5bdb63616141bc0001cc1b59/folder", "type": "girder"} TEST: No test steps to exercise File/Folder "Move-To..." FAIL: UI not automatically refreshed after "copy to workspace" or "move to workspace" completed

FAIL: Export worked, but running locally appeared to hang and never finished:

#18 17.81 
#18 17.81 Preparing transaction: ...working... done
#18 18.15 Executing transaction: ...working... WARNING conda.core.envs_manager:register_env(52): Unable to register environment. Path not writable or missing.
#18 20.27   environment location: /srv/conda
#18 20.27   registry file: /root/.conda/environments.txt
#18 20.27 done
#18 20.77 installation finished.
#18 20.78 time: 14.719
#18 20.78 + export PATH=/srv/conda/bin:/srv/conda/envs/notebook/bin:/srv/conda/bin:/srv/npm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
#18 20.78 + PATH=/srv/conda/bin:/srv/conda/envs/notebook/bin:/srv/conda/bin:/srv/npm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
#18 20.78 + conda config --system --append channels defaults
#18 20.91 + conda config --system --set auto_update_conda false
#18 21.06 + conda config --system --set show_channel_urls true
#18 21.23 + echo 'update_dependencies: false'
#18 21.23 + conda config --system --set channel_priority flexible
#18 21.36 + conda install -y mamba==0.6.1
#18 21.73 Collecting package metadata (current_repodata.json): ...working... done
#18 29.92 Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
#18 29.92 Collecting package metadata (repodata.json): ...working... #18 29.92 Collecting package metadata (repodata.json): ...working... 
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/repo2docker/__main__.py", line 411, in main
    r2d.start()
  File "/usr/lib/python3.8/site-packages/repo2docker/app.py", line 778, in start
    self.build()
  File "/usr/lib/python3.8/site-packages/repo2docker/app.py", line 764, in build
    raise docker.errors.BuildError(l["error"], build_log="")
docker.errors.BuildError: #18 29.92 Collecting package metadata (repodata.json): ...working...

Settings

FAIL: Unable to login/signup for demo.dataverse.org FAIL: The following request timed out: POST https://girder.stage.wholetale.org/api/v1/account/dataverse/key?resource_server=dev2.dataverse.org&key=MY_REAL_API_KEY_VALUE&key_type=apikey:

{"message": "ConnectionError: ConnectionError(MaxRetryError(\"HTTPSConnectionPool(host='dev2.dataverse.org', port=443): Max retries exceeded with url: /api/users/token (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f8bcc566c40>: Failed to establish a new connection: [Errno 110] Connection timed out'))\"))", "trace": ["<FrameSummary file /girder/girder/api/rest.py, line 630 in endpointDecorator>", "<FrameSummary file /girder/girder/api/rest.py, line 1230 in POST>", "<FrameSummary file /girder/girder/api/rest.py, line 970 in handleRoute>", "<FrameSummary file /girder/girder/api/access.py, line 63 in wrapped>", "<FrameSummary file /girder/girder/api/describe.py, line 709 in wrapped>", "<FrameSummary file /girder/plugins/wholetale/server/rest/account.py, line 339 in addAccountKey>", "<FrameSummary file /girder/plugins/wholetale/server/lib/dataverse/auth.py, line 16 in verify>", "<FrameSummary file /girder/venv/lib/python3.9/site-packages/requests/api.py, line 76 in get>", "<FrameSummary file /girder/venv/lib/python3.9/site-packages/requests/api.py, line 61 in request>", "<FrameSummary file /girder/venv/lib/python3.9/site-packages/requests/sessions.py, line 542 in request>", "<FrameSummary file /girder/venv/lib/python3.9/site-packages/requests/sessions.py, line 655 in send>", "<FrameSummary file /girder/venv/lib/python3.9/site-packages/requests/adapters.py, line 516 in send>"], "type": "internal"}

Tale Creation

TEST: There are two different RStudio images, so one is missing a test case Tale failed to Build the first time, but worked when trying again

#4 sha256:c797c633effb523ebfce4b651d3e8cbdc155fdfee8bdc7f1658d9be0a63dca97 238.87MB / 444.78MB 23.3sHTTPError: 502 Server Error: Bad Gateway for url: https://girder.stage.wholetale.org/api/v1/job/6089dfee22bad715637ae558
  File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/girder_worker/task.py", line 148, in __call__
    results = super(Task, self).__call__(*_t_args, **_t_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  File "/gwvolman/gwvolman/tasks.py", line 433, in build_tale_image
    ret = _build_image(
  File "/gwvolman/gwvolman/utils.py", line 356, in _build_image
    print(line.decode('utf-8').strip())
  File "/usr/local/lib/python3.8/dist-packages/girder_worker/utils.py", line 158, in write
    self._write_func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/girder_worker/utils.py", line 259, in write
    self._flush()
  File "/usr/local/lib/python3.8/dist-packages/girder_worker/utils.py", line 237, in _flush
    req.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)

This is my default env

Tale built and ran successfully

It builds, I guess? I don't know anything about Spark, so not sure how to test this

TEST: Are these the Content Security Policy settings in the browser? How do I check these? Need more steps here FAIL: MATLAB could not run the script:

Screen Shot 2021-04-28 at 5 41 24 PM

TEST: Duplicated test case FAIL: MATLAB could not run the script:

Screen Shot 2021-04-28 at 5 41 24 PM

FAIL: %help summarize hangs, everything else up to that and after appear to work properly WARN: warning appears in example notebook towards the end

NOTE: A newer version of stata_kernel exists. Run

    pip install stata_kernel --upgrade

to install the latest version.

WARN: Newer version available It builds and runs, I guess? I don't know anything about OpenRefine, so not sure how else to test this.. additional steps needed

Analyze in Whole Tale

Task 1: Importing a Dataset from Dataverse

These test cases cover potential situations that can occur when importing datasets from Dataverse.

Testing Steps:

  1. Remove all running Tale instances
  2. Navigate to https://girder.stage.wholetale.org/api/v1/integration/dataverse?datasetPid=doi%3A10.7910%2FDVN%2F3MJ7IR&siteUrl=https%3A%2F%2Fdataverse.harvard.edu
  3. Confirm that the Tale title reads as Replication Data for: "Agricultural Fires and Health at Birth"
  4. Confirm that the only item in the Input data section matches the uri with Data Source appended
  5. Confirm that no environment is selected
  6. Confirm that the Create New Tale button is disabled
  7. Select an environment
  8. Confirm that READ/WRITE is selected
  9. Click Create New Tale
  10. Confirm notification progress and success
  11. Confirm that you are redirected to the run page
  12. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  13. Confirm that the data exists under Tale Workspace
  14. Confirm that the Tale's category is science

FAIL: Not automatically redirected FAIL: Tale creation/import hangs at "Initializing..." FAIL: Tale is automatically launched with AiWT cases (is this expected?)

Task 2: Importing a Dataset from DataONE

These test cases cover different scenarios that may happen when importing datasets from DataONE.

This test covers the ideal case where dataset importing has all of the needed information to properly run. This includes the dataset location, title, and selected environment.

Testing Steps:

  1. Remove all running Tale instances
  2. Navigate to https://girder.stage.wholetale.org/api/v1/integration/dataone?uri=https%3A%2F%2Fsearch.dataone.org%2Fview%2Fdoi%3A10.18739%2FA2VQ2S94D&title=Fire%20influences%20on%20forest%20recovery%20and%20associated%20climate%20feedbacks%20in%20Siberian%20Larch%20Forests%2C%20Russia&environment=RStudio
  3. Confirm that the Tale title reads as Fire influences on forest recovery and associated climate feedbacks in Siberian Larch Forests, Russia
  4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
  5. Confirm that Rstudio is selected in the Environments widget
  6. Confirm that the Create New Tale button is enabled
  7. Confirm READ ONLY is selected
  8. Click Create New Tale
  9. Confirm that you are redirected to the run page
  10. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  11. Confirm that the data exists in the Tale under External Data
  12. Confirm that the Tale's category is science

FAIL: Not automatically redirected FAIL: Strange delay before Tale goes into Initializing... FAIL: Tale creation/import hangs at "Initializing..." WARN: Tale is automatically launched with AiWT cases (is this expected?)

Links:

  1. https://dashboard.stage.wholetale.org/browse?api=https%3A%2F%2Fcn-stage-2.test.dataone.org%2Fcn%2Fv2&name=Dataone%20Dataset&uri=https://dev.nceas.ucsb.edu/view/doi:10.5072/FK2K075M25
  2. https://dashboard.stage.wholetale.org/browse?api=http%3A%2F%2Fcn-stage.test.dataone.org%2Fcn%2Fv2&name=Dataone%20Dataset&uri=https://test.arcticdata.io/view/doi:10.5065/D6N014NC

Testing Steps:

  1. Remove all running Tale instances
  2. Navigate to each of the two links above
  3. Select an environment
  4. Click Create New Tale
  5. Confirm that the correct data exists in the Tale under External Data

WARN: No convenient way (via the UI) to abort while "starting" a Tale FAIL: AiWT cases may no longer respect the instance limit:

Screen Shot 2021-04-29 at 9 08 23 AM

Task 3: Shared Behavior

These cases show errors that are common between Dataverse and DataONE. Although they are tested using DataONE URIs, the errors should be the same when replacing it with a Dataverse URI.

This test covers the case where the name query parameter is missing. The expected behavior is that the uri query parameter is used instead.

Testing Steps:

  1. Remove all running Tale instances
  2. Navigate to https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdataverse.harvard.edu%2Fapi%2Faccess%2Fdatafile%2F3323458
  3. Confirm that the Tale title reads as https://dataverse.harvard.edu/api/access/datafile/3323458
  4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
  5. Confirm that no Environment is selected
  6. Confirm that the Create New Tale button is disabled
  7. Select an environment
  8. Click Create New Tale
  9. Confirm that the notification bar appears & properly updates
  10. Confirm that you are redirected to the run page
  11. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  12. Confirm that the data exists in the Tale under External Data

FAIL: Tale name not defaulted to uri

This test covers the case where a user imports data from an external source and selects READ/WRITE in the Create Tale Modal.

Testing Steps (Variant 1):

  1. Remove all running Tale instances
  2. Navigate to https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdataverse.harvard.edu%2Fapi%2Faccess%2Fdatafile%2F3323458
  3. Confirm that the Tale title matches the URI above
  4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
  5. Confirm that the Create New Tale button is disabled
  6. Select an environment
  7. Select READ/WRITE
  8. Click Create New Tale
  9. Confirm that the notification bar appears & properly updates
  10. Confirm that you are redirected to the run page
  11. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  12. Confirm that the data exists in the Tale Workspace

FAIL: Not redirected to run page

Testing Steps (Variant 2):

  1. Remove all running Tale instances
  2. Navigate to https://dashboard.stage.wholetale.org/browse?api=https%3A%2F%2Fdev.nceas.ucsb.edu%2Fknb%2Fd1%2Fmn%2Fv2&environment=JupyterLab&name=proveit&uri=doi%3A10.5072%2FFK27P92Z55
  3. Confirm that the Tale title matches the dataset
  4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
  5. Confirm that the Create New Tale button is disabled
  6. Select READ/WRITE
  7. Click Create New Tale
  8. Confirm that the notification bar appears & properly updates
  9. Confirm that you are redirected to the run page
  10. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  11. Confirm that the data exists in the Tale Workspace

Tale metadata tests

The purpose of these tests are to confirm that the metadata files (manifest.json, environment.json, LICENSE) we generate are correct.

FAIL: No "Datasets" section (but there is "wt:usesDataset"? are these the same?)

FAIL: LICENSE file in exported Tale always has CC 4.0 text, even after changing, saving, and re-exporting

Tale Sharing Tests

These tests are centered around the feature for sharing Tales with other Whole Tale users. To run these tests, register at least one other account, referred to as User B.

Publishing tests

Zenodo: PASS DataONE: FAIL - "Not enough segments" error returned when publishing.. re-authed with DataONE to refresh my token, but same behavior. Cannot publish to DataONE.

Zenodo integration tests

The register tests the following cases.

TEST: There is no Manage > Data tab TEST: Cannot browse contents of External Data via the UI, not sure how to verify contents

The register tests the following cases.

FAIL: No /data (or /home or /workspace) mounted to notebook, cannot run notebook

  1. Open https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdoi.org%2F10.5281%2Fzenodo.3242073&environment=JupyterLab&name=Simple%20requirements.txt%20based%20example%20for%20repo2docker&asTale=true
  2. COnfirm Source Data and Title match URL and "READ WRITE" is selected
  3. Select 'Create Tale'
  4. Confirm image builds and notebook can run

FAIL: Image failed to build, cannot view build logs {"message": "Read access denied for job 608b1fc622bad715637af6e5 (user 608b1a8f22bad715637af658).", "type": "access"} (iirc, this also happened during review, but we couldn't figure out what had happened)

  1. Open https://girder.stage.wholetale.org/api/v1/integration/zenodo?doi=10.5072%2Fzenodo.490450&resource_server=sandbox.zenodo.org
  2. Confirm Water Tale is created.
  3. Confirm citation on Metadata view
  4. Confirm DOI matches above
  5. Use the Swagger UI to confirm the publishInfo field on the tale object contains pid, repository, repository_id, uri
  6. Run the tale and and notebook, confirm it works as expected (produces glorious map of Texas)

FAIL: Publish works, import hangs on "Copying files to workspace" (noting that the workspace in my case is completely empty: https://sandbox.zenodo.org/record/804485)

Versioning Tests

FAIL: Datasets / Metadata changes are not preserved when renaming a version. Oddly, Tale Workspace was preserved correctly.

FAIL: Versions are copied, but "created" timestamps are not preserved

Git Integration Tests

Overlaps with "Compose XX Tale" Tale tests, whose Git capabilities functioned as expected :+1:

WARN: Log output says "Tale created" - this is likely a copy paste error

Other tests

Event Notification Tests

FAIL: See AiWT Test Cases - too many import cases broken, and I suspect that those failing cases will affect this one

Vocabulary Tests

FAIL: https://vocabularies.stage.wholetale.org/wt/1.0/index.html only returns 404 page not found

Forward Authentication Test

FAIL: I am correctly prompted for login when accessing (for example) https://tmp-zf2ypkftc9yu.stage.wholetale.org/login?next=%2Ftree, but after authing I see this view indicating that I require access to the container logs to continue:

Screen Shot 2021-04-29 at 4 53 57 PM