Implement VBMS pagination API for connect VBMS gem

[sharonwarner]

Sentry link:

https://sentry.ds.va.gov/department-of-veterans-affairs/caseflow/issues/5194/activity/

Notes

Ask Anya Roltsch to forward the email regarding the API

AC

• Implement VBMS pagination API for connect VBMS gem
• Create additional tickets as necessary to use it in caseflow. This is estimated as a 5, but could definitely be more.

[lomky]

Sneaking mute avoider. Another example. Muted again. https://sentry.ds.va.gov/department-of-veterans-affairs/caseflow/issues/5147/

[monfresh]

What is the reason for continuing to send this (and other known VBMS errors) to Sentry only to mute them? Why not stop sending them to Sentry forever?

[monfresh]

In this case, this seems like a VBMS issue. We are not posting any documents to VBMS. This error happens when we request documents for a particular appeal. It sounds like some appeals have a large result set that VBMS is unable to send back as a response? Searching for DocumentTooBig in Slack, I don't see any discussions about this. I will start one.

[monfresh]

According to this related issue, there is a VBMS pagination API we could be using: https://github.com/department-of-veterans-affairs/caseflow/issues/10665#issuecomment-494138528

[pkarman]

Email thread summary about this problem:

From: Hoang, Thuy [USA] Sent: Friday, August 23, 2019 10:40 AM To: Nemanja Djuric ; Williams, Cameron [USA] ; Mrozinski, Jared F. (BAH) Cc: Carlson, Nate [USA] Subject: Re: [External] paging API

I’m guessing if Caseflow is still asking about this that our changes to the pagination code did > > not fix this issue. I’ll get Thomas to test it and create a defect again.

From: Nemanja Djuric Date: Friday, August 23, 2019 at 10:33 AM Subject: Re: [External] paging API

Hello, Is there any update on findPagedDocumentSeriesReferences request returning EFSERR1045?

From: Hoang, Thuy [USA] Sent: Thursday, April 25, 2019 9:29 AM Subject: Re: [External] paging API

Hey Nemo,

We’re still investigating this issue to try to reproduce it in our lower environments. It’s probable that we have some scenarios with data that we do not have in the lower environments that is different from PROD. We’ve created a defect to work it but any code changes will likely not go out and be available to you until the next release.

If you can set your page size with a property and use a lower page size then I would recommend that you do that for now so that you can move forward with your development.

From: Nemanja Djuric Date: Thursday, April 25, 2019 at 7:56 AM Subject: Re: [External] paging API

Hello Thuy,

Any updates on this?

Please let me know. Our effort on introducing pagination is on hold until this is resolved.

Also, please let me know if you need any inputs from our side.

From: Nemanja Djuric Sent: Monday, April 22, 2019 12:08 PM Subject: Re: [External] paging API

I'm testing against Production environment and yes - I got the same error with other two file numbers.

I can provide file numbers if you'd like to go through logs on your end, or you if you're aware of one that it's proven to work, you can send it over (secure channel) and I can find out if I'm doing something wrong on my end.

From: Hoang, Thuy [USA] Sent: Monday, April 22, 2019 11:26 AM Subject: Re: [External] paging API

Hey Nemo,

Which environment are you testing against? Have you tried other veterans or is this the only one that is giving you problems?

We’re currently investigating and can’t reproduce this issue in our lower environments.

From: Nemanja Djuric Date: Friday, April 19, 2019 at 2:48 PM Subject: Re: [External] paging API

Hi Thuy,

I did try 4,999 with the same number (should have mentioned that), I got the same error. It did work for set of 3,000 (the total result count is 12,471).

Let me know if you would like to try the same search.

From: Hoang, Thuy [USA] Sent: Friday, April 19, 2019 2:36 PM Subject: Re: [External] paging API

Hey Nemo,

Can you try pageSize of 4999 and see if that works for you? I just tried the startIndex = 0 and pageSize = 5000 and was able to get results in our DEV environment. I then tried to set the pageSize to 5001 and got the EFSERR1045 error message as expected.

There are not any calls that will return more than 5000 records currently. The payload would probably get too large to send over the network if we allowed more than that.

The <!--0 to 5000 repetitions:--> is for the metadata elements. For example I just tried to > search for files with a specific subject so I used search criteria that looked like this

<v5:metadata key="subject">
   <v5:value>some subject</v5:value>
</v5:metadata>

From: Nemanja Djuric Date: Friday, April 19, 2019 at 1:43 PM Subject: Re: [External] paging API

I have findPagedDocumentSeriesReferences request from below returning EFSERR1045. How is this possible?

Also, is there an option argument to get all the documents even if there more than 5,000 with one client call? In your example request I see which made me think something like this might exist after all.

The only documentation I have is SCD, and there's no detailed information on this.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:upload="http://service.efolder.vbms.vba.va.gov/eFolderUploadService" xmlns:read="http://service.efolder.vbms.vba.va.gov/eFolderReadService" xmlns:v5="http://vbms.vba.va.gov/cdm/document/v5" xmlns:cdm="http://vbms.vba.va.gov/cdm" xmlns:xop="http://www.w3.org/2004/08/xop/include">

  <soapenv:Body>
    <read:findPagedDocumentSeriesReferences>
      <read:searchCriteria>
        <v5:veteran fileNumber="XXXXXXXXX"/>
      </read:searchCriteria>
      <read:pagingCriteria>
        <v5:pageSize>5000</v5:pageSize>
        <v5:startIndex>0</v5:startIndex>
      </read:pagingCriteria>
    </read:findPagedDocumentSeriesReferences>
  </soapenv:Body>
</soapenv:Envelope>

<ns4:code>EFSERR1045</ns4:code>
<ns4:message>Requested result set exceeds acceptable size.</ns4:message>

From: Hoang, Thuy [USA] Sent: Thursday, April 18, 2019 4:38 PM Subject: Re: [External] paging API

Hey Nemo,

The major version field is a required field on the WSDL so for the version information it will just be defaulted to “1” for calls that only return the current version of the document. If you need the actual major version number I would suggest using the getDocumentSeriesReference call.
We can certainly populate this field in future revisions of the web service to display the major version if that is something you need. If you call getDocumentSeriesReference you’ll be able to see that the version of the document that is returned for the other calls are the most recent version.

It's possible that the SCD does not have this level of detail. We’ll make sure to add more descriptions for the method calls in the next version of the SCD.

From: Nemanja Djuric Date: Thursday, April 18, 2019 at 2:51 PM Subject: Re: [External] paging API

Thuy,

So reading through this it seems that both FindDocumentVersionReference and FindPagedDocumentSeriesReferences should return the most recent version of document? This contradicts that with FindPagedDocumentSeriesReferences we get version 1 vs version 2 (with FindDocumentVersionReference).

Also, if you can please let me know where can I find info from below since the manual I have (eFolder v1.0 Service Contract Document, Release 15.0) does not have information on returned versions.

From: Hoang, Thuy [USA] Sent: Thursday, April 18, 2019 1:48 PM Subject: Re: [External] paging API

Hey Nemo,

The series reference calls will be different than the version reference calls. The Series Reference Id keeps track of all the different versions of a particular document. Each version of the document will have a version ID.

The calls will return data as followed:

findPagedDocumentSeriesReference

Returns all the veteran’s series and the most recent version of the document in the series. This will match on the veteran information and also the metadata that you will send in the request body.

findDocumentSeriesReference

Same as findPagedDocumentSeriesReference but without any pagination. If the criteria for this call returns more than 5000 records, an EFSERR1045 error will be returned. The message you will get will be “Requested result set exceeds acceptable size.”

getDocumentSeriesReference

Will retrieve ALL versions of the document in a series by the document series id

findDocumentVersionReference

Finds the most current version of the documents that matches the criteria (veteran information and metadata) from the request body.

getDocumentVersionReference

Retrieves the most current version of a document by the documentVersionRefId

From: Nemanja Djuric Date: Wednesday, April 17, 2019 at 3:03 PM Subject: Re: [External] paging API

Hi Thuy,

Thanks for the sample, it was helpful.

I noticed that FindDocumentVersionReference and FindPagedDocumentSeriesReferences responses for the same file number are different in a sense that they have couple of documents with the same id but different version numbers (major value). Do you know why is this? I can provide exact details through a secure channel if needed.

Also, beside accuracy, our other focus is performance - any insider tips on benchmark performance difference, if any, between FindPagedDocumentSeriesReferences and FindDocumentVersionReference.

I ran some benchmark tests (3 x 1,000 sample) and it seems that FindPagedDocumentSeriesReferences performs on average 5% better.

Which hints toward using FindPagedDocumentSeriesReferences exclusively in our Efolder eXpress application (vs FindDocumentVersionReference for the most part and FindPagedDocumentSeriesReferences in cases where sets are over 5,000 only) .

Any thoughts on this? Thanks, Nemo

From: Hoang, Thuy [USA] Sent: Monday, April 15, 2019 1:01 PM Subject: Re: [External] paging API

Hey Nemo,

Here is a sample SOAP request body for the findPagedDocumentSeriesReferences call. Let me know if this works for you. This is the same SOAP body that I used to test the call in SoapUI.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:efol="http://service.efolder.vbms.vba.va.gov/eFolderReadService" xmlns:v5="http://vbms.vba.va.gov/cdm/document/v5">

   <soapenv:Header/>
   <soapenv:Body>
      <efol:findPagedDocumentSeriesReferences>
         <!--Optional:-->
         <efol:searchCriteria>
            <!--0 to 5000 repetitions:-->
            <v5:metadata key="">
               <v5:value/>
            </v5:metadata>
            <!--Optional:-->
            <v5:veteran edipi="" fileNumber="replaceMe" ssn="" bin=""/>
         </efol:searchCriteria>
         <efol:pagingCriteria>
            <v5:pageSize>5000</v5:pageSize>
            <v5:startIndex>1</v5:startIndex>
         </efol:pagingCriteria>
      </efol:findPagedDocumentSeriesReferences>
   </soapenv:Body>
</soapenv:Envelope>

From: Nemanja Djuric Date: Friday, April 12, 2019 at 1:33 PM Subject: [External] paging API

I'm an engineer over at Caseflow and we're trying to take advantage of VBMS paging API.

However, findPagedDocumentSeriesReference does not get recognized

Message part {http://service.efolder.vbms.vba.va.gov/eFolderReadService}findPagedDocumentSeriesReference was not recognized.  (Does it exist in service WSDL?)

We're using it the same way as for other eFolderReadService calls like findDocumentSeriesReference and findDocumentVersionReference

Any ideas what we're doing wrong here?

Thanks, Nemo

P.S. Is there a way to get a document count from VBMS without querying all documents?

[pkarman]

Example in prod: User has trouble downloading folder for Veteran with 13k+ documents. PID: 3978641

[major]

Happy Holidays from @major. 😀

(You mentioned my username accidentally in one of the messages above and GitHub notified me by email.)

[pkarman]

Being implemented in https://github.com/department-of-veterans-affairs/connect_vbms/pull/241

Closing this ticket in favor of #13047