Problems with chaining the build of several programs with several compilation steps

[FALLAI-Denis]

Hi,

In our Cobol.groovy script we want to process a stream of COBOL programs, and building each program requires several steps:

1. MVSJob.start()
2. MVSExec.exetute() for COBOL pre-compilation MDECK(NOCOMPILE) option to expand COPYBOOKs ; SYSMDECK output is for DB2 pre-compilation
3. MVSExec.exetute() for DB2 pre-compilation: we don't want to use the DB2 coprocessor; SYSCIN output is for COBOL compilation
4. MVSExec.exetute() for COBOL compilation ; SYSMDECK is not used but mandatory, SYSLIN output is for linkedit
5. MVSExec.exetute() for linkedit
6. MVSJob.stop()

We encounter many problems with BPXWDYN which manages the file allocations of each step (MVSExec / new DDStatement() ).

These problems come from the fact that the COBOL compiler is called twice for each COBOL program iteration, and that the DDNAMES are therefore used twice, and in particular problems with the SYSMDECK ddname.

First problem: For a file created in a MVSExec.exetute() to be retrieved in a next step, it seems imperative to use the .pass(true) option when creating the file. This works well for first program iteration, but failed for second program iteration: BPXWDYN error 1702 on SYSMDECK ddname. The problem is that the declared .pass(true) file remains allocated and on the next program iteration. The solution found was to add the REUSE option when creating the SYSMDECK file. In your Cobol.groovy sample, you also use .pass(true) on the SYSLIN ddname of the COBOL compilation, without using REUSE, and this script works for us...

Second problem: After solving firts problem, on the iteration for the second COBOL programs, the Db2 pre-compilation step failed because the SYSMDECK created during the COBOL pre-compilation is unusable: BPXWDYN error 1708. We have not yet managed to find the origin of the problem. Apparently it is the SYSMDECK created at the real COBOL compilation stage of the first iteration, which must interfere with the SYSMDECK created at the COBOL pre-compilation stage of the second iteration.

Problems don't appear with other COBOL compiler ddnames, they're not declared .pass(true) (except SYSLIN but we haven't managed to do 2 full iterations yet, and it's possible we have also a problem with this file).

[dennis-behm]

Hi @FALLAI-Denis , thanks for reaching out.

with the .pass(true) option, temporary datasets remain allocated for the context of the MVSJob execution. There is no need to add the DDStatement again for any following MVSExecs.

If you would like to understand what happened under the covers, I am recommending to use the log4j2 setting and activate tracing to gain all allocations including frees.

Are you familiar with the DDNames API to pass temporary datasets to different DDnames in subsequent steps? The Assembler.groovy has this logic, while it has the cics + db2 precompile translators in it.

You can find a sample of the db2 precompile step which also uses the DDNames API in my fork of zAppBuild. Hope this gives you some pointers.

While you are planning to invoke the Compiler twice ( which I have not done yet ), which require the same DDNames, what about running two MVSJobs, so that the initial allocations are freed after the execution of the Cobol pre-compilation step? This works as long as you store the intermediate expanded source on a dataset.

For dynamic allocation issues - I assume you are aware of the IBM Docs page on DYNALLOC

[FALLAI-Denis]

Hi @dennis-behm

Thanks for your answer.

with the .pass(true) option, temporary datasets remain allocated for the context of the MVSJob execution. There is no need to add the DDStatement again for any following MVSExecs.

I consulted Assembler.groovy, I don't understand how setDdnames works... apparently it allows to change the ddnames expected by the program to reuse the ddnames of a previous step? (IBM DBB API documentation is not very verbose).

Is there any relation to the ddnamelist as described here Starting the compiler from an assembler program?

[dennis-behm]

Yes, it allows to change the dd names. The link you found is correct, but I don't know how this is done in the background of DYNALLOC.

I am typically using this table in the SCLM redbook, Page 747.

[FALLAI-Denis]

Thanks, I also found ddnamelists in reference manuals:

Db2 V12. https://www.ibm.com/docs/en/db2-for-zos/12?topic=procedures-dd-name-list-format

Cobol V6R1 to V6R4. https://www.ibm.com/docs/en/cobol-zos/6.4?topic=zos-starting-compiler-from-assembler-program

HLASM V1R6. https://www.ibm.com/docs/en/hla-and-tf/1.6?topic=zos-invoking-assembler-dynamically

Binder z/OS 2.4. https://www.ibm.com/docs/en/zos/2.4.0?topic=facilities-invoking-binder-program-from-another-program

Do we have to use this technique to pass files between MVSExec steps? It's complicated... and not natural compared to the habits taken with the JCL.

[dennis-behm]

Hi @FALLAI-Denis, this is for temporary datasets only, which need to be passed in under an alternate DD name. If you use persistent datasets, you might not need the alternate DD name, while you just point tot he dataset name rather the temporary allocation.

In JCL you would have provided a &&TEMPOBJ to refer back to the previous different temporary dataset allocation. However in the build scripts here, you can manage multiple variations in a single script - which leads to a lot of flexibility, but also a more complex setup.

[dennis-behm]

@FALLAI-Denis , I think Gil has worked with you to implement the above steps using the ddname APIs to pass on temporary files between the steps using the DYNALLOC macro. If you would like to share the solution here, this would be highly welcome by the community.

[FALLAI-Denis]

Hi @dennis-behm,

Yes, I work with Gil P and other people from IBM France on the implementation of IBM DBB on our site. They do a good job.

Our technical context is complicated. Our problem concerned the implementation of a concatenation "before": reuse the file written on SYSLIN ddname at the COBOL step, but use it by concatenate it behind a new file always on the SYSLIN ddname of linkedi step, which is not planned in the API. But in fact the solution was found by playing with the file allocation list of the COBOL step by writing on another ddname than SYSLIN, which then makes it possible to freely use the SYSLIN ddname at the linkedit step.

This difficulty is not attributable to Zappbuild, but to the IBM DBB API. Once it has been understood that IBM DBB actually made use of JZOS, which itself makes use of BPXWDYN, and once it has been understood that everything was happening in a single task and that the MVSExec object was actually wrapper to CALL a program within this single task, and that the files allocated at a MVSExec step remain potentially allocated for the following MVSExec steps, things become clearer, but not necessarily simpler.

This internal behavior of the DBB API is not documented... or I could not find this documentation, not even in JZOS. Fortunately I have a little experience of BPXWDYN which I used to do real dynamic management of files in COBOL processing, (decide on the file name at runtime), which allowed me to understand how it works from IBM DBB.

And we achieve our goals because the precompilation, compilation and linkedit utilities allow you to substitute the ddnames at runtime. If we used other programs, including programs developed by ourselves, I think we would not be able to manage these substitutions of ddnames.

We have to lose all your habits related to classic JCL and completely rethink the sequence of steps in the compilation / linkedit process according to the IBM DBB API.

I think there is an effort to be made on the documentation of the operation of the IBM DBB API, with real use cases.

[FALLAI-Denis]

For information, for the sequence of our precompilation steps before the real COBOL compilation, we came across the COBOL compiler bug described here : PH48619: COMPILER IGNORES THE ALTERNATE DDNAME LIST AND ONLY READS SOURCE FROM THE SYSIN DD WHEN THE TEST COMPILER OPTION IS USED We don't have the fix on our site...

The use of the IBM DBB API is already not simple, but if in addition we come across bugs in the COBOL compiler, that does not simplify our task... ;-)

[FALLAI-Denis]

Hi,

Solved by adaptation of our COBOL.groovy script.