search

whole-tale / wt-design-docs

MIT License
5 stars 9 forks source link

v0.9rc2 testing (v0.9rc2-2-gc9c4b5e on OSX/Chrome) #129

Open bodom0015 opened 4 years ago

bodom0015 commented 4 years ago

Test Plan

https://github.com/whole-tale/wt-design-docs/issues/new?template=test_plan.md

Note: For all tests, repeat for supported browser/OS combinations.

Preconditions:

Authentication

Navigation

Browse

Preconditions:

TESTCASE: Search filtering appears to work, but the original LIGO Tale missing from master list on https://dashboard.stage.wholetale.org

TESTCASE: Add a step to "View Logs" while awaiting Tale image build

Manage

Preconditions

XXX: Had to manually refresh after removing datasets - possibly https://github.com/whole-tale/dashboard/issues/427?

XXX: Searching for this DOI/dataset yielded an error message: No matching results found.

Run

Preconditions:

TESTCASE: There is now only one option for exporting a Tale - wording could be adjusted slightly to convey this

TESTCASE: Missing confirmation of "Move To..." - how should this work? XXX: Needed manual refresh in between copying and deleting - we should attempt to refresh the data after each operation (e.g. copy, delete, etc)

XXX: Needed manual refresh in between copying and deleting - we should attempt to refresh the data after each operation (e.g. copy, delete, etc)

TESTCASE: I was not asked for confirmation, the download just started up

Settings

XXX: "Connect Account" should disable when no new resource_servers can be offered

XXX: "Connect Account" should disable when no new resource_servers can be offered

Tale Creation

TESTCASE: Which Jupyter? All of them? (Current assumption is to try to run all of them) XXX: I attempted to build all types of Jupyter Tales and received an error message (Error: Building image) from both Jupyter Notebook and Jupyter Classic - . I am unable to view the logs of the Tale to debug further

Analyze in Whole Tale

Task 1: Importing a Dataset from Dataverse

These test cases cover potential situations that can occur when importing datasets from Dataverse.

Testing Steps:

  1. Remove all running Tale instances
  2. Navigate to https://girder.stage.wholetale.org/api/v1/integration/dataverse?datasetPid=doi%3A10.7910%2FDVN%2F3MJ7IR&siteUrl=https%3A%2F%2Fdataverse.harvard.edu
  3. Confirm that the Tale title reads as Replication Data for: "Agricultural Fires and Health at Birth"
  4. Confirm that the only item in the Input data section matches the uri with Data Source appended
  5. Confirm that no environment is selected
  6. Confirm that the Create New Tale and Launch button is disabled
  7. Select an environment
  8. Confirm that READ WRITE is selected
  9. Click Create New Tale and Launch
  10. Confirm that you are redirected to the run page
  11. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  12. Confirm that the dataset contents exists under External Data Tale Workspace
  13. Confirm that the Tale's category is science

TESTCASE: No "External Data" appeared, even after Tale was running. Dataset also did not appear under "Manage" after Tale was running (same as in #125). I believe this is a typo in the test case, as READ/WRITE shouldn't result in any "External Data"

Task 2: Importing a Dataset from DataONE

These test cases cover different scenarios that may happen when importing datasets from DataONE.

This test covers the ideal case where dataset importing has all of the needed information to properly run. This includes the dataset location, title, and selected environment.

Testing Steps:

  1. Remove all running Tale instances
  2. Navigate to https://girder.stage.wholetale.org/api/v1/integration/dataone?uri=https%3A%2F%2Fsearch.dataone.org%2Fview%2Fdoi%3A10.18739%2FA2VQ2S94D&title=Fire%20influences%20on%20forest%20recovery%20and%20associated%20climate%20feedbacks%20in%20Siberian%20Larch%20Forests%2C%20Russia&environment=RStudio
  3. Confirm that the Tale title reads as Fire influences on forest recovery and associated climate feedbacks in Siberian Larch Forests, Russia
  4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
  5. Confirm that Rstudio is selected in the Environments widget
  6. Confirm that the Create New Tale and Launch button is enabled
  7. Confirm READ ONLY is selected
  8. Click Create New Tale and Launch
  9. Confirm that you are redirected to the run page
  10. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  11. Confirm that the data exists in the Tale under External Data
  12. Confirm that the Tale's category is science

Links:

  1. https://dashboard.stage.wholetale.org/browse?api=https%3A%2F%2Fdev.nceas.ucsb.edu%2Fknb%2Fd1%2Fmn%2Fv2&name=Dataone%20Dataset&uri=https://dev.nceas.ucsb.edu/view/doi:10.5072/FK2K075M25
  2. https://dashboard.stage.wholetale.org/browse?api=https://test.arcticdata.io/metacat/d1/mn/v2&name=Dataone%20Dataset&uri=https://test.arcticdata.io/view/doi:10.5065/D6N014NC

Testing Steps:

  1. Remove all running Tale instances
  2. Navigate to each of the two links above
  3. Select an environment
  4. Click Create New Tale and Launch
  5. Confirm that the correct data exists in the Tale under External Data

Task 3: Shared Behavior

These cases show errors that are common between Dataverse and DataONE. Although they are tested using DataONE URIs, the errors should be the same when replacing it with a Dataverse URI.

This test covers the case where the name query parameter is missing. The expected behavior is that the uri query parameter is used instead.

Testing Steps:

  1. Remove all running Tale instances
  2. Navigate to https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdataverse.harvard.edu%2Fapi%2Faccess%2Fdatafile%2F3323458
  3. Confirm that the Tale title reads as https://dataverse.harvard.edu/api/access/datafile/3323458
  4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
  5. Confirm that no Environment is selected
  6. Confirm that the Create New Tale and Launch button is disabled
  7. Select an environment
  8. Click Create New Tale and Launch
  9. Confirm that the notification bar appears & properly updates
  10. Confirm that you are redirected to the run page
  11. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  12. Confirm that the data exists in the Tale under External Data

This test covers the case where a user imports data from an external source and selects READ/WRITE in the Create Tale Modal.

Testing Steps:

  1. Remove all running Tale instances
  2. Navigate to https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdataverse.harvard.edu%2Fapi%2Faccess%2Fdatafile%2F3323458 or https://dashboard.stage.wholetale.org/browse?api=https%3A%2F%2Fdev.nceas.ucsb.edu%2Fknb%2Fd1%2Fmn%2Fv2&environment=JupyterLab&name=proveit&uri=doi%3A10.5072%2FFK27P92Z55
  3. Confirm that the Tale title matches the dataset
  4. Confirm that the only item in the Selected data section matches the uri with Data Source appended
  5. Confirm that the Create New Tale and Launch button is disabled
  6. Select an environment
  7. Select READ/WRITE
  8. Click Create New Tale and Launch
  9. Confirm that the notification bar appears & properly updates
  10. Confirm that you are redirected to the run page
  11. Confirm that the Tale name matches the Tale Name in the Create Tale Modal
  12. Confirm that the data exists in the Tale Workspace

TESTCASE: https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdataverse.harvard.edu%2Fapi%2Faccess%2Fdatafile%2F3323458 does not provide a title, so step 3 may not be accurate for that case TESTCASE: https://dashboard.stage.wholetale.org/browse?api=https%3A%2F%2Fdev.nceas.ucsb.edu%2Fknb%2Fd1%2Fmn%2Fv2&environment=JupyterLab&name=proveit&uri=doi%3A10.5072%2FFK27P92Z55 provides an environment, so step 7 is not necessary for that case XXX: https://dashboard.stage.wholetale.org/browse?api=https%3A%2F%2Fdev.nceas.ucsb.edu%2Fknb%2Fd1%2Fmn%2Fv2&environment=JupyterLab&name=proveit&uri=doi%3A10.5072%2FFK27P92Z55 yields only a 404.html in the Tale Workspace.. possibly a bug, but perhaps this is expected?

Tale metadata tests

The purpose of these tests are to confirm that the metadata files (manifest.json, environment.json, LICENSE) we generate are correct.

Publishing tests

XXX: Zenodo Sandbox worked fine, but DataONE Dev did not - after fixing the description and the authors, I was unable to publish with the following error message:

ERROR An error was encountered while publishing your Tale.
'size'

....... wat?

Zenodo integration tests

The register tests the following cases.

The register tests the following cases.

  1. Open https://dashboard.stage.wholetale.org/browse?uri=https%3A%2F%2Fdoi.org%2F10.5281%2Fzenodo.3242073&environment=JupyterLab&name=Simple%20requirements.txt%20based%20example%20for%20repo2docker&asTale=true
  2. COnfirm Source Data and Title match URL and "READ WRITE" is selected
  3. Select 'Create and Launch Tale'
  4. Confirm image builds and notebook can run
  1. Open https://girder.stage.wholetale.org/api/v1/integration/zenodo?doi=10.5072%2Fzenodo.490450&resource_server=sandbox.zenodo.org
  2. Confirm Water Tale is created.
  3. Confirm citation on Metadata view
  4. Confirm DOI matches above
  5. Use the Swagger UI to confirm the publishInfo field on the tale object contains pid, repository, repository_id, uri
  6. Run the tale and and notebook, confirm it works as expected (produces glorious map of Texas)

TESTCASE: Cannot use the Water Tale for this, must create your own Tale - Failed to get the deposition (id=490450). Server returned: 403 Client Error: FORBIDDEN for url: https://sandbox.zenodo.org/api/deposit/depositions/490450 { "message": "You don't have the permission to access the requested resource. It is either read-protected or not readable by the server.", "status": 403 }. Perhaps the imported tale already contains some publishInfo that is being reused? TESTCASE: Clarification needed here. I know how to export, but not sure I quite grasp how to "download and run local" - do you mean run the Docker image on my local machine?

Regression tests

I'll compile these down into a more-easily digestible list soon

bodom0015 commented 4 years ago

Results

Major: Test Failures (XXX)

Minor: Test Case Issues (TESTCASE)

PR for test plan improvements forthcoming

bodom0015 commented 4 years ago

PR for test plan improvements: https://github.com/whole-tale/wt-design-docs/pull/130

Outstanding Issues:

Open Questions:

ThomasThelen commented 4 years ago
bodom0015 commented 4 years ago

Thank you for clarifying, @ThomasThelen

I just reran the builds for Jupyter Classic and Jupyter Notebook and this time they completed just fine.

Looking back, I'm not sure which was which but it looks like a 502: Bad Gateway was hit during both of those builds? Perhaps Girder was somehow restarted during these builds? Unclear what might have happened, but it seems to be working now...

From https://girder.stage.wholetale.org/#job/5e59902e351c4c0da4b48872:

Step 51/76 : RUN R --quiet -e "install.packages('shiny', repos='https://mran.microsoft.com/snapshot/2020-02-26', method='libcurl')"
---> Running in 024c93a1c194
> install.packages('shiny', repos='https://mran.microsoft.com/snapshot/2020-02-26', method='libcurl')
Installing package into ‘/srv/rlibs’
(as ‘lib’ is unspecified)
also installing the dependencies ‘BH’, ‘magrittr’, ‘httpuv’, ‘xtable’, ‘sourcetools’, ‘later’, ‘promises’, ‘fastmap’

    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...

CC       src/libuv_la-fs-poll.lo
CC       src/libuv_la-idna.loHTTPError: 502 Server Error: Bad Gateway for url: https://girder.stage.wholetale.org/api/v1/job/5e59902e351c4c0da4b48872
  File "/usr/local/lib/python3.5/dist-packages/celery/app/trace.py", line 382, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/girder_worker/task.py", line 148, in __call__
    results = super(Task, self).__call__(*_t_args, **_t_kwargs)
  File "/usr/local/lib/python3.5/dist-packages/celery/app/trace.py", line 641, in __protected_call__
    return self.run(*args, **kwargs)
  File "/gwvolman/gwvolman/tasks.py", line 437, in build_tale_image
    ret = _build_image(cli, tale_id, image, tag, temp_dir, REPO2DOCKER_VERSION)
  File "/gwvolman/gwvolman/utils.py", line 324, in _build_image
    print(line.decode('utf-8').strip())
  File "/usr/local/lib/python3.5/dist-packages/girder_worker/utils.py", line 158, in write
    self._write_func(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/girder_worker/utils.py", line 259, in write
    self._flush()
  File "/usr/local/lib/python3.5/dist-packages/girder_worker/utils.py", line 237, in _flush
    req.raise_for_status()
  File "/usr/local/lib/python3.5/dist-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)

From https://girder.stage.wholetale.org/#job/5e598fb7351c4c0da4b48855:

Step 57/64 : RUN chown -R ${NB_USER}:${NB_USER} ${REPO_DIR}
---> Running in 5cf07978e98a
Removing intermediate container 5cf07978e98aHTTPError: 502 Server Error: Bad Gateway for url: https://girder.stage.wholetale.org/api/v1/job/5e598fb7351c4c0da4b48855
  File "/usr/local/lib/python3.5/dist-packages/celery/app/trace.py", line 382, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/girder_worker/task.py", line 148, in __call__
    results = super(Task, self).__call__(*_t_args, **_t_kwargs)
  File "/usr/local/lib/python3.5/dist-packages/celery/app/trace.py", line 641, in __protected_call__
    return self.run(*args, **kwargs)
  File "/gwvolman/gwvolman/tasks.py", line 437, in build_tale_image
    ret = _build_image(cli, tale_id, image, tag, temp_dir, REPO2DOCKER_VERSION)
  File "/gwvolman/gwvolman/utils.py", line 324, in _build_image
    print(line.decode('utf-8').strip())
  File "/usr/local/lib/python3.5/dist-packages/girder_worker/utils.py", line 158, in write
    self._write_func(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/girder_worker/utils.py", line 259, in write
    self._flush()
  File "/usr/local/lib/python3.5/dist-packages/girder_worker/utils.py", line 237, in _flush
    req.raise_for_status()
  File "/usr/local/lib/python3.5/dist-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)