search

ansible-collections / ibm_zos_core

Red Hat Ansible Certified Content for IBM Z
76 stars 44 forks source link

[Bug] zos_find - does not find VSAM file open with DISP of OLD #643

Open EdwardMcCarthy opened 1 year ago

EdwardMcCarthy commented 1 year ago

Bug description

I had this task

- name: Check if a VSAM files exist
  ibm.ibm_zos_core.zos_find:
    patterns:
      - CICSUSER.SC00CIC1.EXMPCAT
    resource_type: cluster
  register: cnvResult

- name:  result 1
  debug:
    msg: "{{ cnvResult }}"

When VSAM file open in CICS with DISP of SHR get expected result as VSAM file does exist:

ok: [zceeZosLpar] => { "msg": { "changed": false, "data_sets": [ { "name": "CICSUSER.SC00CIC1.EXMPCAT", "type": "CLUSTER" } ], "examined": 1, "failed": false, "matched": 1 } }

But when VSAM file open in CICS with DISP of OLD get

ok: [zceeZosLpar] => { "msg": { "changed": false, "data_sets": [], "examined": 0, "failed": false, "matched": 0 } }

which is clearly not correct

Playbook verbosity output

feb-16-cicsVsamFile-Disp-OLD-ansible-log.txt

Contents of ansible.cfg

[defaults]
forks = 25
stdout_callback = yaml
callback_whitelist = profile_tasks

[callback_profile_tasks]
sort_order = none
task_output_limit=400

[ssh_connection]
pipelining = True

Contents of the inventory

n/a

Contents of group_vars or host_vars

n/a

Ansible version

ansilbe --version 

shows

ansible [core 2.14.1]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/ansible/library']
  ansible python module location = /usr/local/lib/python3.10/dist-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = True

zOAU version

/usr/lpp/IBM/zoautil/bin>./zoaversionhelper
2022/12/06 20:44:00 CUT V1.2.2 ee30137 2525

IBM z/OS Ansible core Version

v1.4.0

IBM ZOAU version

v1.2.2

z/OS version

2.5

Ansible module

zos_find

ddimatos commented 1 year ago

@EdwardMcCarthy - Thank you for opening an issue and for your patience. I was not able to reproduce this on Ansible z/OS core 1.4.0 or 1.5.0-beta.1.

Could you try this with a ZOAU command, note that with ZOAU vls:

  0         At least one dataset was found matching a dataset-pattern
  1         No datasets were found.

For example, if its not found , the module will fail because a non-zero rc of 1 will be returned.

Try ZOAU vls:

    - name: ZOAU vls vsam
      ansible.builtin.shell: vls ESDS.DATA.SET
      register: result

Successful example for when you try it:

ok: [zvm] => {
    "result": {
        "changed": true,
        "cmd": "vls ESDS.DATA.SET",
        "delta": "0:00:01.294500",
        "end": "2023-03-26 05:45:24.746259",
        "failed": false,
        "msg": "",
        "rc": 0,
        "start": "2023-03-26 05:45:23.451759",
        "stderr": "",
        "stderr_lines": [],
        "stdout": "ESDS.DATA.SET\nESDS.DATA.SET.DATA",
        "stdout_lines": [
            "ESDS.DATA.SET",
            "ESDS.DATA.SET.DATA"
        ]
    }
}

Not successful example for when you try it, notice the non-zero rc from ZOAU vls:

fatal: [zvm]: FAILED! =>
{
   "changed":true,
   "cmd":"vls ESDS.DATA.FOO",
   "delta":"0:00:01.135326",
   "end":"2023-03-26 05:45:46.273975",
   "msg":"non-zero return code",
   "rc":1,
   "start":"2023-03-26 05:45:45.138649",
   "stderr":"",
   "stderr_lines":[

   ],
   "stdout":"",
   "stdout_lines":[

   ]
}

With zos_find I tried both tasks as DISP=SHR and DISP=OLD and had no problem, both returned:

ok: [zvm] => {
    "result": {
        "changed": false,
        "data_sets": [
            {
                "name": "ESDS.DATA.SET",
                "type": "CLUSTER"
            }
        ],
        "examined": 1,
        "failed": false,
        "matched": 1
    }
}
EdwardMcCarthy commented 1 year ago

I created a new Docker Image to run Ansible in, now have this level

ansible --version ansible [core 2.14.5] config file = /etc/ansible/ansible.cfg configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /usr/local/lib/python3.10/dist-packages/ansible ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections executable location = /usr/local/bin/ansible python version = 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] (/usr/bin/python3) jinja version = 3.1.2 libyaml = True

/root/.ansible/collections/ansible_collections

Collection Version

ibm.ibm_zos_cics 1.0.5 ibm.ibm_zos_core 1.6.0-beta.1 ibm.ibm_zosmf 1.4.0

Add the ZOAU vls task

Still see same issue

Uploading may-22-cics-check-ok-debug.txt, this was run when VSAM file not open in CICS or anywhere, ibm_zos_core.zos_find finds the file

Uploading may-22-cics-check-fail-debug.txt, this was run when VSAM file now open in CICS with DISP OLD, zos_find says not found

in both zOAU vls finds the file

EdwardMcCarthy commented 1 year ago

may-22-cics-check-fail-debug.txt may-22-cics-check-ok-debug.txt

EdwardMcCarthy commented 1 year ago

can see in those files

ZOAU_HOME=/usr/lpp/IBM/zoautil

now have

./zoaversionhelper 2023/05/05 21:09:40 CUT V1.2.3.2 7e00691c 2576 PH54319

ddimatos commented 1 year ago

Thanks @EdwardMcCarthy - I did not notice the update, regardless capacity was reach this quarter so I put in Q3 to be investigated.

fernandofloresg commented 12 months ago

Finally got to the issue, apparently when zos_find executes a vls -l command to get the vsam data sets. When the data set is in DISP=SHR or not in use this is the vls -l output:

bash-4.3# vls -l OMVSADM.KSDS.VS
OMVSADM.KSDS.VS                              2023/09/20 000000
OMVSADM.KSDS.VS.DATA                         2023/09/20 000000
OMVSADM.KSDS.VS.INDEX                        2023/09/20 000000

whereas when using vls -l when a data set is opened with DISP=OLD, this is the result:

IBMUSER:/u/ibmuser>vls -l IBMUSER.REGIONS.CICS01.EXMPCAT
IBMUSER.REGIONS.CICS01.EXMPCAT.DATA          2023/08/25 T50734
IBMUSER.REGIONS.CICS01.EXMPCAT.INDEX         2023/08/25 T50734

So when looking for cluster type it cannot find it, but even if is opened with DISP=OLD, if resource_type is selected as INDEX or DATA zos_find does find the specified resource type.

vls call is executed in line 424 rc, out, err = _vls_wrapper(pattern, details=True)

fernandofloresg commented 12 months ago

Created zoau issue NAZARE-10146

richp405 commented 6 months ago

This behavior is based on zOS behavior (not necessarily zoau). When disp=OLD in a dd statement, the program requesting this disposition gets EXCLUSIVE control over the destination file.

In the case of VSAM (and pdse) this means many directory style/catalog queries against that dataset will not return a value, until the program asking for disp=old terminates.

From https://www.ibm.com/docs/en/zos/3.1.0?topic=definition-status-subparameter

OLD Indicates that the data set exists before this step and that this step requires exclusive (unshared) use of the data set. If you specify DISP=OLD for an output tape data set and (1) the data set is not protected by RACF® or a password or (2) the data set has no expiration date, the system does not verify the data set name in the header label.

Because there is an issue where vls can still find the file, I'll check to see if anything else changed, but this looks like an issue coming from zOS

richp405 commented 5 months ago

I see a code difference between zos_find and vls. zos_find calls listvtoc first, and it appears to get a 'no files' response, whereas vls finds the file (I've duplicated the latter).

I'll put one other question over to a zoau developer, to make sure, but I believe the listvtoc is trying to access file attributes, which can't be done if disp=old is set from a different process.

EdwardMcCarthy commented 5 months ago

Hi

There are ways to list what datasets there are that show all the datasets even if any of those datasets are allocated with DISP of OLD

Ansible should be able to show all datasets that exist even if they are allocated with DISP of OLD

Regards

Edward McCarthy

Senior zStack Technical Specialist Asia/Pacific IBMers Value Dedication to every client's success Innovation that matters - for the company and for the world

Trust and personal responsibility in all relationships Location Code: CAGR

Phone: +61 2 6212 1137 Fax: +61 2 6124 2155

Mobile: +61 411 254 783

Email / Sametime: @.**@.>

To infinity and beyond

From: Rich Parker @.> Sent: Wednesday, March 27, 2024 2:46 AM To: ansible-collections/ibm_zos_core @.> Cc: EDWARD MCCARTHY @.>; Mention @.> Subject: [EXTERNAL] Re: [ansible-collections/ibm_zos_core] [Bug] zos_find - does not find VSAM file open with DISP of OLD (Issue #643)

I see a code difference between zos_find and vls. zos_find calls listvtoc first, and it appears to get a 'no files' response, whereas vls finds the file (I've duplicated the latter). I'll put one other question over to a zoau developer, to ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. Report Suspicious https://us-phishalarm-ewt.proofpoint.com/EWT/v1/AdhS1Rd-!-ZFRWZ5ZeETxVl8SZW8KOwee17lcjKBz4DgCxuuZOoyrReHYVbADYhPeTAsVHPgffaQ8ywB_h3XRsr2mYBCdnElh7ucv74iNvpuR4WoLfRPqvCVH1h4qYcL-7qM$ ‌ ZjQcmQRYFpfptBannerEnd

I see a code difference between zos_find and vls. zos_find calls listvtoc first, and it appears to get a 'no files' response, whereas vls finds the file (I've duplicated the latter).

I'll put one other question over to a zoau developer, to make sure, but I believe the listvtoc is trying to access file attributes, which can't be done if disp=old is set from a different process.

— Reply to this email directly, view it on GitHubhttps://github.com/ansible-collections/ibm_zos_core/issues/643#issuecomment-2020793041, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APAPVRYTS2AUNS55KINTVYLY2GJ35AVCNFSM6AAAAAAU5WFCWGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRQG44TGMBUGE. You are receiving this because you were mentioned.Message ID: @.**@.>>

richp405 commented 5 months ago

@EdwardMcCarthy ,

I agree, and I appreciate your patience in this matter. Yes, vls shows there is a way to do it. I'm trying to find out why we're not doing that first. This may be code that wasn't updated, or there may be a reason that wasn't documented. As soon as I find out for certain, I will relay it to you.

richp405 commented 5 months ago

Will continue this into 1.6: I'm trying to validate if we can change the order of dataset inquiry with zoau and core team dev's.

EdwardMcCarthy commented 5 months ago

Ok

Read through the comments in this git

To me seems to be bug in way the zoau vls cmd works

It should return all datasets regardless of how they are allocated – in my view

Regards

Edward McCarthy

Senior zStack Technical Specialist Asia/Pacific IBMers Value Dedication to every client's success Innovation that matters - for the company and for the world

Trust and personal responsibility in all relationships Location Code: CAGR

Phone: +61 2 6212 1137 Fax: +61 2 6124 2155

Mobile: +61 411 254 783

Email / Sametime: @.**@.>

To infinity and beyond

From: Rich Parker @.> Sent: Thursday, March 28, 2024 12:33 AM To: ansible-collections/ibm_zos_core @.> Cc: EDWARD MCCARTHY @.>; Mention @.> Subject: [EXTERNAL] Re: [ansible-collections/ibm_zos_core] [Bug] zos_find - does not find VSAM file open with DISP of OLD (Issue #643)

Will continue this into 1. 6: I'm trying to validate if we can change the order of dataset inquiry with zoau and core team dev's. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. Report Suspicious https://us-phishalarm-ewt.proofpoint.com/EWT/v1/AdhS1Rd-!-ZFRWZ5ZeETxVl8SZW8KOwee17lcjKBz4DgCxuuZOoyrReHYVbADYhPeTAsVHPgffaR8yyA_59Vxsr2mYBCdnAVA-TlGms6EFvTi9fWZjZ8fetoIrPhUKNidXn4$ ‌ ZjQcmQRYFpfptBannerEnd

Will continue this into 1.6: I'm trying to validate if we can change the order of dataset inquiry with zoau and core team dev's.

— Reply to this email directly, view it on GitHubhttps://github.com/ansible-collections/ibm_zos_core/issues/643#issuecomment-2022781314, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APAPVRZKAFUC5F54YAAP7IDY2LDANAVCNFSM6AAAAAAU5WFCWGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRSG44DCMZRGQ. You are receiving this because you were mentioned.Message ID: @.**@.>>

fernandofloresg commented 1 day ago

Hi @EdwardMcCarthy are you still facing this issue ? Are you using the same ibm_zos_core/zoau version?

EdwardMcCarthy commented 20 hours ago

Hi

I have not tried testing this again since last time I tested and updated the git link.

Has any code been changed in zoau or Ansible so that this should work ok now ? If so what version etc ?

Regards

Edward McCarthy

Senior zStack Technical Specialist Asia/Pacific IBMers Value Dedication to every client's success Innovation that matters - for the company and for the world

Trust and personal responsibility in all relationships Location Code: CAGR

Phone: +61 2 6212 1137 Fax: +61 2 6124 2155

Mobile: +61 411 254 783

Email / Sametime: @.**@.>

To infinity and beyond

From: Fernando Flores @.> Sent: Wednesday, September 18, 2024 1:55 AM To: ansible-collections/ibm_zos_core @.> Cc: EDWARD MCCARTHY @.>; Mention @.> Subject: [EXTERNAL] Re: [ansible-collections/ibm_zos_core] [Bug] zos_find - does not find VSAM file open with DISP of OLD (Issue #643)

Hi @EdwardMcCarthy are you still facing this issue ? Are you using the same ibm_zos_core/zoau version? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned. Message ID: ansible-collections/ibm_zos_core/issues/643/2356325425@ github. com

Hi @EdwardMcCarthyhttps://github.com/EdwardMcCarthy are you still facing this issue ? Are you using the same ibm_zos_core/zoau version?

— Reply to this email directly, view it on GitHubhttps://github.com/ansible-collections/ibm_zos_core/issues/643#issuecomment-2356325425, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APAPVR2WDG4UZKVOJMVUZY3ZXBGE5AVCNFSM6AAAAAAU5WFCWGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJWGMZDKNBSGU. You are receiving this because you were mentioned.Message ID: @.**@.>>