Failed deposit into ORA4-PRD-SYMP: uuid_1355d1d9-91a0-4914-b9aa-caf0c1b6823f, 1086623

tomwrobel commented 4 years ago

At 2020-02-10 18:26:50 an attempt was made to deposit publication 1086623 in ORA. The user was Oxris user JOANNA OSTAPKOWICZ

https://oxris.ox.ac.uk/viewobject.html?cid=1&id=1086623
https://ora4-prd-symp.bodleian.ox.ac.uk/concern/journal_articles/uuid_1355d1d9-91a0-4914-b9aa-caf0c1b6823f
duplicate created during redeposit: uuid_5e31ae1f-8fed-436e-b2cf-98b56c8ab902

The situation and potential problems:

An object is sitting in ORA4 SYMP without binary files and with a 'Draft' status.
The object has not been sent to review
Oxris has no visible link between the ORA record and the Oxris record

Potential problems:

A deposit failed, this is a bad thing but there may be a good reason
We have Draft objects in ORA4 Symp from where deposits failed. We want to know if these can be safely deleted
We don't know what happens when redeposit is attempted

What we need to know:

[x] if re-deposit of a file is attempted, will this update the Draft object or create a duplicate? ANSWER: it will create a duplicate
[x] can 'Draft' status objects be safely left in ORA4-SYMP for ever (or until we need to clean them up? ANSWER: they can be cleaned up after a given amount of time, as redeposit will create duplicates and harvest might create bad links
[ ] can we walk the user through a re-deposit? We need to know why deposit failed

Notes

This action resulted in a partial failure. The metadata was deposited correctly, but the attempt to deposit a binary file failed.

Reminder: there are three SWORD2 actions involved in depositing a publications record with a file: create ORA object, create ORA fileset with binary data, update fileset metadata. Only the final action is accompanied by an 'in progress=false' flag, this marks the work as 'Submitted' in the ORA4 system triggers a push to ORA4 Review

The full Oxris trace is below. I don't know what caused this error, but we should probably investigate? If she wants support, I'm happy to help walk her through what she did in order to debug.

Error: Handling a webpage request
An error occurred whose cause was related to an external system. Request made to /rt2depositaction.html?com=deposit using Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.3 Safari/605.1.15.
System component: Website. Level: 5. User: Dr JOANNA OSTAPKOWICZ. Elements Version: 5.18.0.2494.

Exception on machine: NetBIOS name "BSPWIN685", OS "Microsoft Windows NT 6.3.9600.0" (64-bit)
Software instance: Default, Software Version: 5.18.0.2494, Commit Hash: 92537a5ecec13d10f5a26b6cd311f584b96a3380
Command "C:\Windows\SysWOW64\inetsrv\w3wp.exe -ap "DefaultAppPool" -v "v4.0" -l "webengine4.dll" -a \\.\pipe\iisipm5da94cf7-856b-48c1-a87a-5c6788e6693b -h "C:\inetpub\temp\apppools\DefaultAppPool\DefaultAppPool.config" -w "" -m 0 -t 20 -ta 0" (32-bit)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Log entry ID: 0c90383f-38a9-44fc-9de9-21078eb5e19d
Logged at: 18:26:54.222 10/02/20

Level 1:
-------
Exception type: Symplectic.Elements.DataEntities.WebsiteExternalEnvironmentException
Message: An error occurred whose cause was related to an external system.
Stack trace:
at Symplectic.Elements.Website.Application.Handlers.Rt2DepositActionPage.ProcessRequest(DepositFilesAndFormInfo) in elements\Website.Application\Handlers\Rt2DepositActionPage.cs:line 135, char 5
at Symplectic.Elements.Website.Application.BaseHandlers.XmlOutputHandler<THandler, TDefaultPageInfo>+PostHandlerInfo<TFormInfo, TResponse>.Execute(THandler) in elements\Website.Application\BaseHandlers\XmlOutputHandler.cs:line 313, char 6
at Symplectic.Elements.Website.Application.BaseHandlers.XmlOutputHandler<THandler, TDefaultPageInfo>.InnerProcessRequest() in elements\Website.Application\BaseHandlers\XmlOutputHandler.cs:line 654, char 5

Level 2:
-------
Exception type: Symplectic.Elements.DataEntities.Sync.SyncException
Message: Processing of the HTTP request resulted in an exception. Please see the HTTP response returned by the 'Response' property of this exception for details.
Stack trace:
at Symplectic.Elements.Sync.Interop.XwalkInOutDataSourceClient.PerformDeposit(PublicationDepositInfo, GroupedProgressMonitor) in elements\Sync.Interop\XwalkDataSourceClient.cs:line 304, char 5
at Symplectic.Elements.Website.Application.Handlers.Rt2DepositActionPage.PerformDepositAndFetch(SourceInfo, IPublicationView, IPublicationViewRecord, List<FileToDepositMetadata>, IEnumerable<FileToDeposit>, DepositSupplementaryInfo, GroupedProgressMonitor) in elements\Website.Application\Handlers\Rt2DepositActionPage.cs:line 197, char 5
at Symplectic.Elements.Website.Application.Handlers.Rt2DepositActionPage.ProcessRequest(DepositFilesAndFormInfo) in elements\Website.Application\Handlers\Rt2DepositActionPage.cs:line 98, char 5

Level 3:
-------
Exception type: System.Web.Http.HttpResponseException
Message: Processing of the HTTP request resulted in an exception. Please see the HTTP response returned by the 'Response' property of this exception for details.
Stack trace:
at Symplectic.Foundation.Http.SympHttpClient+<SendAsync>d__16.MoveNext() in libraries\Symplectic.Foundation\Http\SympHttpClient.cs:line 365, char 6
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task)
at Symplectic.Foundation.Http.SympHttpClient+<GetBufferedResponseAsync>d__14<TContent>.MoveNext() in libraries\Symplectic.Foundation\Http\SympHttpClient.cs:line 276, char 10
at Symplectic.Foundation.Concurrency.AggregateExceptionHandling.RethrowFirstInnerNonAggregateExeption(AggregateException) in libraries\Symplectic.Foundation\Concurrency\AggregateExceptionHandling.cs:line 25, char 4
at Symplectic.Foundation.Concurrency.AggregateExceptionHandling.UnwrapResultingAggregateExceptions<TResult>(Func<TResult>) in libraries\Symplectic.Foundation\Concurrency\AggregateExceptionHandling.cs:line 40, char 5
at Symplectic.Foundation.Http.SympHttpClient.GetBufferedResponse<TContent>(HttpRequestMessage, Nullable<TimeSpan>) in libraries\Symplectic.Foundation\Http\SympHttpClient.cs:line 256, char 4
at Symplectic.Elements.Sync.Interop.Hyrax.HyraxSword2Client.UploadFile(String, String, FileToDeposit, Boolean) in elements\Sync.Interop\Hyrax\HyraxSword2Client.cs:line 144, char 5
at Symplectic.Elements.Sync.Interop.Hyrax.HyraxDataSourceClient.UploadFiles(String, IEnumerable<FileToDeposit>, XwalkOutXmlInterpreter, Int32) in elements\Sync.Interop\Hyrax\HyraxDataSourceClient.cs:line 242, char 5
at Symplectic.Elements.Sync.Interop.Hyrax.HyraxDataSourceClient.PerformInitialDeposit(InitialPublicationDepositInfo, XwalkOutXmlInterpreter, GroupedProgressMonitor) in elements\Sync.Interop\Hyrax\HyraxDataSourceClient.cs:line 221, char 5
at Symplectic.Elements.Sync.Interop.XwalkInOutDataSourceClient.PerformDeposit(PublicationDepositInfo, GroupedProgressMonitor) in elements\Sync.Interop\XwalkDataSourceClient.cs:line 288, char 6

HttpResponseException-specific information:
    Request Method: POST
    Request URL: https://ora4-prd-symp.bodleian.ox.ac.uk/sword/collections/default/works/uuid_1355d1d9-91a0-4914-b9aa-caf0c1b6823f/file_sets
    Response Status Code: 400
    Response Reason Phrase: Bad Request

mrdsaunders commented 4 years ago

I have contacted her to offer help and get more details.

mrdsaunders commented 4 years ago

Attempted redeposit - failed again. @tomwrobel so it's failing the Sword call to update the fileset metadata (https://ora4-prd-symp.bodleian.ox.ac.uk/sword/collections/default/works/uuid_1355d1d9-91a0-4914-b9aa-caf0c1b6823f/file_sets? Does the fileset exist at that point or just the ORA object?

Exception on machine: NetBIOS name "BSPWIN685", OS "Microsoft Windows NT 6.3.9600.0" (64-bit)
Software instance: Default, Software Version: 5.18.0.2494, Commit Hash: 92537a5ecec13d10f5a26b6cd311f584b96a3380
Command "C:\Windows\SysWOW64\inetsrv\w3wp.exe -ap "DefaultAppPool" -v "v4.0" -l "webengine4.dll" -a \\.\pipe\iisipmb111975d-bba0-40fe-a2dd-23dd41632808 -h "C:\inetpub\temp\apppools\DefaultAppPool\DefaultAppPool.config" -w "" -m 0 -t 20 -ta 0" (32-bit)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Log entry ID: ea70f3e4-25a2-4d31-9376-3948531c4b74
Logged at: 11:59:04.853 11/02/20

Level 1:
-------
Exception type: Symplectic.Elements.DataEntities.WebsiteExternalEnvironmentException
Message: An error occurred whose cause was related to an external system.
Stack trace:
at Symplectic.Elements.Website.Application.Handlers.Rt2DepositActionPage.ProcessRequest(DepositFilesAndFormInfo) in elements\Website.Application\Handlers\Rt2DepositActionPage.cs:line 135, char 5
at Symplectic.Elements.Website.Application.BaseHandlers.XmlOutputHandler<THandler, TDefaultPageInfo>+PostHandlerInfo<TFormInfo, TResponse>.Execute(THandler) in elements\Website.Application\BaseHandlers\XmlOutputHandler.cs:line 313, char 6
at Symplectic.Elements.Website.Application.BaseHandlers.XmlOutputHandler<THandler, TDefaultPageInfo>.InnerProcessRequest() in elements\Website.Application\BaseHandlers\XmlOutputHandler.cs:line 654, char 5

Level 2:
-------
Exception type: Symplectic.Elements.DataEntities.Sync.SyncException
Message: Processing of the HTTP request resulted in an exception. Please see the HTTP response returned by the 'Response' property of this exception for details.
Stack trace:
at Symplectic.Elements.Sync.Interop.XwalkInOutDataSourceClient.PerformDeposit(PublicationDepositInfo, GroupedProgressMonitor) in elements\Sync.Interop\XwalkDataSourceClient.cs:line 304, char 5
at Symplectic.Elements.Website.Application.Handlers.Rt2DepositActionPage.PerformDepositAndFetch(SourceInfo, IPublicationView, IPublicationViewRecord, List<FileToDepositMetadata>, IEnumerable<FileToDeposit>, DepositSupplementaryInfo, GroupedProgressMonitor) in elements\Website.Application\Handlers\Rt2DepositActionPage.cs:line 197, char 5
at Symplectic.Elements.Website.Application.Handlers.Rt2DepositActionPage.ProcessRequest(DepositFilesAndFormInfo) in elements\Website.Application\Handlers\Rt2DepositActionPage.cs:line 98, char 5

Level 3:
-------
Exception type: System.Web.Http.HttpResponseException
Message: Processing of the HTTP request resulted in an exception. Please see the HTTP response returned by the 'Response' property of this exception for details.
Stack trace:
at Symplectic.Foundation.Http.SympHttpClient+<SendAsync>d__16.MoveNext() in libraries\Symplectic.Foundation\Http\SympHttpClient.cs:line 365, char 6
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task)
at Symplectic.Foundation.Http.SympHttpClient+<GetBufferedResponseAsync>d__14<TContent>.MoveNext() in libraries\Symplectic.Foundation\Http\SympHttpClient.cs:line 276, char 10
at Symplectic.Foundation.Concurrency.AggregateExceptionHandling.RethrowFirstInnerNonAggregateExeption(AggregateException) in libraries\Symplectic.Foundation\Concurrency\AggregateExceptionHandling.cs:line 25, char 4
at Symplectic.Foundation.Concurrency.AggregateExceptionHandling.UnwrapResultingAggregateExceptions<TResult>(Func<TResult>) in libraries\Symplectic.Foundation\Concurrency\AggregateExceptionHandling.cs:line 40, char 5
at Symplectic.Foundation.Http.SympHttpClient.GetBufferedResponse<TContent>(HttpRequestMessage, Nullable<TimeSpan>) in libraries\Symplectic.Foundation\Http\SympHttpClient.cs:line 256, char 4
at Symplectic.Elements.Sync.Interop.Hyrax.HyraxSword2Client.UploadFile(String, String, FileToDeposit, Boolean) in elements\Sync.Interop\Hyrax\HyraxSword2Client.cs:line 144, char 5
at Symplectic.Elements.Sync.Interop.Hyrax.HyraxDataSourceClient.UploadFiles(String, IEnumerable<FileToDeposit>, XwalkOutXmlInterpreter, Int32) in elements\Sync.Interop\Hyrax\HyraxDataSourceClient.cs:line 242, char 5
at Symplectic.Elements.Sync.Interop.Hyrax.HyraxDataSourceClient.PerformInitialDeposit(InitialPublicationDepositInfo, XwalkOutXmlInterpreter, GroupedProgressMonitor) in elements\Sync.Interop\Hyrax\HyraxDataSourceClient.cs:line 221, char 5
at Symplectic.Elements.Sync.Interop.XwalkInOutDataSourceClient.PerformDeposit(PublicationDepositInfo, GroupedProgressMonitor) in elements\Sync.Interop\XwalkDataSourceClient.cs:line 288, char 6

HttpResponseException-specific information:
    Request Method: POST
    Request URL: https://ora4-prd-symp.bodleian.ox.ac.uk/sword/collections/default/works/uuid_5e31ae1f-8fed-436e-b2cf-98b56c8ab902/file_sets
    Response Status Code: 400
    Response Reason Phrase: Bad Request

tomwrobel commented 4 years ago

Do you have a copy of the file for debugging? We can try to replicate on QA

tomwrobel commented 4 years ago

@mrdsaunders worth noting that it's file deposit that is failing, not fileset metadata. The calls are as follows:

Create object via POST https://ora4-prd-symp.bodleian.ox.ac.uk/sword/collections/default/works (new object uuid is returned in the response)
Deposit binary file via POST https://ora4-prd-symp.bodleian.ox.ac.uk/sword/collections/default/works/uuid_{uuid}/file_sets (file set id is returned in response)
Update binary file metadata via PUT https://ora4-prd-symp.bodleian.ox.ac.uk/sword/collections/default/works/uuid_{uuid}/file_sets/{file_set_id}

It's failing on a POST call to https://ora4-prd-symp.bodleian.ox.ac.uk/sword/collections/default/works/uuid_5e31ae1f-8fed-436e-b2cf-98b56c8ab902/file_sets, so we know it's step 2 that's the problem

mrdsaunders commented 4 years ago

Do you have a copy of the file for debugging? We can try to replicate on QA

Ostapkowicz_et_al_NewWorld‘Souvenirs’.docx

tomwrobel commented 4 years ago

I've tried on QA and can't replicate I've deposited the file. @mrdsaunders can I try on PRD as an impersonator?

mrdsaunders commented 4 years ago

Yes no problem @thomas-wrobel

tomwrobel commented 4 years ago

Tried again, and it passed. The file has been deposited

@mrdsaunders want to tell the user. This looks like a transient error, I suspect it was to do with a badly cached directory but we'll never know!

tomwrobel commented 4 years ago

Please close when you're happy

mrdsaunders commented 4 years ago

Thanks @thomas-wrobel. I have notified user.

bodleian / ora_data_model