search

zowe / zowe-explorer-vscode

Visual Studio Code Extension for Zowe, which lets users interact with z/OS Data Sets, Unix System Services, and Jobs on a remote mainframe instance. Powered by Zowe SDKs.
Eclipse Public License 2.0
159 stars 89 forks source link

Filtering large datasets doesn't return expected results #2978

Open antonkc opened 4 days ago

antonkc commented 4 days ago

Greetings,

The problem

Trying to Retrieve members with a filter in populous datasets doesn't return the expected results. The magnitude is around ten thousand for the particular dataset where the problem was detected.

Issue 1270 looks similar, but this issue doesn't talk about filters.

Error recreation

Recreation requires a dataset with thousands of member. I will name such dataset "TEST.DS". Assume this dataset has an even spread of members in alphabetical ordering.

Result is:

No results are returned.

Expected result is:

Results are returned and start with "Q"

Next is a recreation of the error in our environment: filter_dataset-error

Ancillary tests

Under hundred members dataset

Assume dataset "LIGHT.DS" with less than 100 members.

Result is:

Results are returned and start with "Q".

Curl call

Assume dataset "TEST.DS". More than a thousand members. this dataset has an even spread of members in alphabetical ordering.

Use the following curl (replace what is needed):

curl -X 'GET' '{{mainframehost}}/rseapi/api/v1/datasets/search/PDSMembers?filter={{TEST.DS}}&memberPattern=Q%2A' -H 'accept: application/json' -H 'Authorization: {{auth}}' -H 'Number-Of-Results: 50000'

Result is:

Results are returned and start with "Q"

Recreation in our environment: filter_dataset-curl

Miscelaneous data

When we call the server through VS Code, the server registered this entry:

--------------- - - [---------------------] "GET /rseapi/api/v1/datasets/search/PDSMembers?filter=CMN.ABASE.CPY&memberPattern=* HTTP/1.1" 200

Otherwise, with curl, the server entry is:

--------------- - - [---------------------] "GET /rseapi/api/v1/datasets/search/PDSMembers?filter=CMN.ABASE.CPY&memberPattern=Q* HTTP/1.1" 200

We believe the filtering is happening client-side in the explorer and that is what is causing the bug since there is a limit of items fetched.

github-actions[bot] commented 4 days ago

Thank you for creating a bug report. We will investigate the bug and evaluate its impact on the product. If you haven't already, please ensure you have provided steps to reproduce the bug and as much context as possible.