search

IQSS / dataverse

Open source research data repository software
http://dataverse.org
Other
878 stars 490 forks source link

Payara 6: ensure API tests are executing #9329

Closed pdurbin closed 1 year ago

pdurbin commented 1 year ago

Small chunk from the big Payara 6 issue:

We have https://jenkins.dataverse.org/job/IQSS-Dataverse-Payara6/ set up already so I think we can use it as a starting point.

I'll defer to @donsizemore for a size estimate.

donsizemore commented 1 year ago

I've made enough attempts that I feel confident in reporting "different" behavior:

The Ansible URI module consistently reports:

"status": 404, "url": "http://localhost:8080/api/admin/externalTools"

which was not resolved by a 45-second sleep after the last service start, so when I disabled externaltool installation as a test, things moved on to:

"status": 403, "url": "http://localhost:8080/api/admin/settings/BuiltinUsers.KEY"

These same problems don't present on develop over Payara-5.2022.3, nor do they present by the time I log in to the instance and call these endpoints manually with curl.

I don't find anything helpful in the Payara log, I don't blame SELinux or an out-of-memory condition, at least that I can find.

donsizemore commented 1 year ago

Potentially related https://github.com/payara/Payara/issues/6145

you can sometimes get a HTTP 403 Foprbidden status code if the server is under load.

pdurbin commented 1 year ago

Narrowed scope to executing. 10 for providing dev help to Don. Thank you Don!

donsizemore commented 1 year ago

current state of integration tests on Payara 6.2023.2 running @poikilotherm's Payara6 branch which he just merged with develop (post-5.13): 403 Forbidden trying to set the user management secret:

"close", "content_language": "", "content_length": "1078", "content_type": "text/html", "elapsed": 0, "msg": "Status code was 403 and not [200]: HTTP Error 403: Forbidden", "redirected": false, "server": "Payara Server 6.2023.2 #badassfish", "status": 403, "url": "http://localhost:8080/api/admin/settings/BuiltinUsers.KEY", "x_frame_options": "SAMEORIGIN", "x_powered_by": "Servlet/6.0 JSP/3.1 (Payara Server 6.2023.2 #badassfish Java/Red Hat, Inc./11)"

I don't get the same error setting this on payara6.odum.unc.edu, currently running Oliver's branch based off of develop post-5.12. May be useful to rebuild, redeploy and re-test on that box, but the error is (twice now) reproducible on a clean installation. Unfortunately Payara logs absolutely nothing when it returns this 403.

donsizemore commented 1 year ago

With Oliver's help, I'm now blaming asynchronous startup for these 403 Forbiddens. For a long time Ansible waited for port 8009 (AJP) to become available, then in troubleshooting these errors I added an additional 15, then 30 seconds. Once Oliver pointed out which log entry meant Dataverse was officially, really open for business, I saw that we need 86 seconds on a t3a.large AWS instance - so 120 seconds for plenty of wiggle room. Will continue testing, but the essential workflow is in place.

Oliver is also testing in containers, which he hopes to make available using a GitHub runner, and is achieving different results depending on the system resources he affords the Payara container.

donsizemore commented 1 year ago

Integration tests are running in Jenkins on Oliver's Payara6 branch. Current results:

"[ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 24.564 s <<< FAILURE! - in edu.harvard.iq.dataverse.api.DownloadFilesIT", "[ERROR] edu.harvard.iq.dataverse.api.DownloadFilesIT.downloadFilenameUtf8  Time elapsed: 1.033 s  <<< FAILURE!", "java.lang.AssertionError: ", "Expected status code <201> doesn't match actual status code <400>.", "", "\tat edu.harvard.iq.dataverse.api.DownloadFilesIT.downloadFilenameUtf8(DownloadFilesIT.java:426)", "", "[INFO] ", "[INFO] Results:", "[INFO] ", 
"[ERROR] Failures: ", "[ERROR]   DatasetsIT.testArchivalStatusAPI:3021 expected:<200> but was:<403>", 
"[ERROR]   DatasetsIT.testFileChecksum:1310 expected:<201> but was:<400>", 
"[ERROR]   DatasetsIT.testPrivateUrl:903 expected:<201> but was:<400>", 
"[ERROR]   DataversesIT.testMalformedFacetQueryString:305", 
"[ERROR]   DownloadFilesIT.downloadFilenameUtf8:426 Expected status code <201> doesn't match actual status code <400>.", "", 
"[ERROR]   InReviewWorkflowIT.testCuratorSendsCommentsToAuthor:206 XML path error.summary doesn't match.", "Expected: Couldn't update dataset edu.harvard.iq.dataverse.engine.command.exception.IllegalCommandException: Dataset cannot be edited due to In Review dataset lock.", "  Actual: Filename could not be extracted from Content-Disposition: Expected separator ';' instead of '='", "", 
"[ERROR]   SearchIT.testGeospatialSearchInvalid:1267 Expected status code <400> doesn't match actual status code <500>.", "", 
"[ERROR]   SwordIT.testCreateAndDeleteDatasetInRoot:437 Expected status code <201> doesn't match actual status code <400>.", "",
"[ERROR]   SwordIT.testCreateDataverseCreateDatasetUploadFileDownloadFileEditTitle:250 XML path error.summary doesn't match.", "Expected: user user60a5e34c user60a5e34c is not authorized to modify dataset with global ID doi:10.5072/FK2/9KJJRE", "  Actual: Filename could not be extracted from Content-Disposition: Expected separator ';' instead of '='", "", 
"[ERROR]   SwordIT.testDeleteFiles:793 expected:<201> but was:<400>", 
"[ERROR] Errors: ", "[ERROR]   SearchIT.testDatasetThumbnail:422 » IllegalArgument Cannot get property 'dataF..."