search

zowe / zowe-cli

Zowe CLI
Eclipse Public License 2.0
113 stars 88 forks source link

zowe files download fails with Error 500 but reports succesfull download. #566

Open jguillaumes opened 4 years ago

jguillaumes commented 4 years ago

Client version:

 zowe --version
2.31.0

I've observed this with relatively big files. Console output:

Data set downloaded successfully.
Destination: ./data/cmnt22.unl

But the last register (line) in the file contains:

ACD 100806      Error 500: java.io.IOException: Error response could not be sent to client.  See log for details.

(The ACD 100806 is part of the record being transmitted when the problem happened)

This is the content of the SYSPRINT file in the z/OS job:

[15:24:36.163443] ServletDispatcher[I]: Received request: GET /ds/T8864.U0G.CMNT22
 [15:24:36.166498] ServletDispatcher[I]: z/OSMF transaction: tx000000000000DB8C
 [15:24:36.168191] ServletDispatcher[I]: Invoking Servlet: SELECT  PGM(IZUGGTDS) PARM(-LI,t -q 393231 -r 0x3e44a298 )
 [15:26:27.306967] servletMain[E]: Unable to send exception
 [15:26:27.307028] TsoServletException[E]: TsoServletException - CATEGORY/RC/REAS=0x2/0x4/0x5 ServletMsgQ::sendOutputChunk(Message
 &, bool) illegal state
 [15:26:27.307044] TsoServletException[E]: TsoServletException - Details:
 [15:26:27.307051] TsoServletException[E]: Output_Sent
 [15:26:27.307056] TsoServletException[E]:   1 3E6007A8+00000628 ServletMsgQ::sendOutputChunk(Message&,bool)
   2 3E5D77C8+0000013E ServletOutputStream::sendMessage(bool)
   3 3E60E0C0+000004DC TsoServlet::sendException(TsoServletException&)
   4 3E60D1A0+0000033A TsoServlet::run()
   5 3E60CB78+00000386 main
   6 0BACC928+0000127E CEEVROND
   7 3DD6AB90+000000B4 EDCZHINV
   8 0B964410+000001C6 CEEBBEXT

I understand a long download can fail because of network problems (it shoud not, anyway!), my main compaint is the client part believes the download has succeed and appends the error message to the downloaded file!

tucker01 commented 4 years ago

This is interesting, if z/OS is reporting an HTTP 500 then the CLI should fail with an error (which is what normally occurs in this scenario). We will try to reproduce.

George0013 commented 4 years ago

Hi everyone,

I have the same problem, I am trying to download a file, the command responds satisfactorily however in the destination the file is cut and ends with: Error 500: java.io.IOException: Error response could not be sent to client. See log for details.

George0013 commented 4 years ago

I forgot to mention.....we've .... zowe --version (2.30.0), Is there any solution?.

George0013 commented 4 years ago

Hi,

Sorry for the insistence, but there is some solution for this issue?. We are doing a PoC and we would need to download a large file.

Regards

ghost commented 4 years ago

To confirm that the issue still exists, please upgrade to the newest version of the lts-incremental distribution of Zowe CLI (version 2.36.1).

If the issue still persists there, please try the active development version of Zowe CLI (version 6.6.3). This version should have improved support for downloading large files.

George0013 commented 4 years ago

Thank you.

George0013 commented 4 years ago

Hi:

I installed the last one that Zowe CLI found from a local zowe-cli-package-1.7.1 package. npm --version 6.13.4 zowe --version 2.32.1

The same problem: The command responds satisfactorily however in the destination the file is cut and ends with: Error 500: java.io.IOException: The error response could not be sent to the client. See record for more details.

George0013 commented 4 years ago

Hi,

I just tried with zowe-cli-package-2.x-ACTIVE-DEVELOPMENT \npm>zowe --version 6.0.1

.... And the problem continues.

George0013 commented 4 years ago

Hi,

Sorry, again.

As we said, we are doing a PoCO and would need to download a large file. Will it be fixed soon?

Thanks and sorry for the insistence.

Regards

George0013 commented 4 years ago

Another thing, do you know from what size it fails?.

Thanks.

Alexandru-Dumitru commented 4 years ago

Hi @George0013,

Could you please provide the following details? I will try to replicate it. File size: binary / text: origin (USS / Data Set):

Thanks

George0013 commented 4 years ago

Hi,

Of course,

Data Set Name . . . . : SYST.TEMP.WLMXGB
Allocated tracks . : 4,345
Allocated extents . : 3
Device type . . . . : 3390
Organization . . . : PS Current Utilization
Record format . . . : FB Used tracks . . . . : 4,345
Record length . . . : 100 Used extents . . . : 3
Block size . . . . : 27900
1st extent tracks . : 22
Secondary tracks . : 15000 Data set name type : LARGE SMS Compressible . : NO

D:\APPSYS\ZOWE\npm>zowe zos-files --zosmf-profile CTA1POCZOWE dl ds "SYST.TEMP.WLMXGB" Data set downloaded successfully. Destination: syst/temp/wlmxgb.txt

George0013 commented 4 years ago

Hi,

Additionally, I have seen in the mainframe side the following error: ÝERROR ~ SRVE0777E: Exception thrown by application class 'com.ibm.zoszmf.restfiles.TsoServletProxy.sendErrorResponse:394' java.io.IOException: Error response could not be sent to client. See log for details.at com.ibm.zoszmf.restfiles.TsoServletProxy.sendErrorResponse(TsoServletProxy..java:394) at com.ibm.zoszmf.restfiles.TsoServletProxy.service(TsoServletProxy.java:165) at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)

And the task cancels: IZUFPROC IZUFPROC - ABEND=S222 U0000 REASON=00000000 637

Tt does not seem to depend on the size because in several downloads of exactly the same file cuts at different points and even sometimes it finishs correctly.

Regards

Alexandru-Dumitru commented 4 years ago

Hi @George0013

I don't have any conclusive findings yet regarding your specific error, but I wanted to update you on my testing progress:

data set used:

Allocated tracks . : 19,400 Allocated extents . : 16 Device type . . . . : 3390 Data class . . . . . : USER Organization . . . : PS Record format . . . : FB Used tracks . . . . : 19,400 Record length . . . : 100 Used extents . . . : 16 Block size . . . . : 27900 1st extent tracks . : 4400 Secondary tracks . : 1000 Data set name type : LARGE SMS Compressible . : NO

In Windows this translates to about 250mb text file.

download failed:

[8336:0000022EA3260D30] 956803 ms: Mark-sweep 1238.3 (1245.5) -> 1238.3 (1244.5) MB, 165.9 / 0.0 ms (average mu = 0.533, current mu = 0.000) last resort GC in old space requested [8336:0000022EA3260D30] 956987 ms: Mark-sweep 1238.3 (1244.5) -> 1238.3 (1244.5) MB, 184.1 / 0.0 ms (average mu = 0.322, current mu = 0.000) last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

0: ExitFrame [pc: 000001911535C5C1]
1: StubFrame [pc: 000001911534E17D]

Security context: 0x036cb349e6e9 2: replace [0000036CB34905E9](this=0x015925a82201 <Very long string[200640711]>,0x005328c04f41 <JSRegExp <String[9]: ([^\r])\n>>,0x03e184171d31 <String[4]\: $1\r\n>) 3: processNewlines [0000019B1ABAE509] [C:\Users\xxxxxx\AppData\Roaming\npm\node_modules\@brightside\core\node_modules\@brightside\imperative\l...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory 1: 00007FF7D345DD8A v8::internal::GCIdleTimeHandler::GCIdleTimeHandler+4506 2: 00007FF7D3438886 node::MakeCallback+4534 3: 00007FF7D3439200 node_module_register+2032 4: 00007FF7D37530DE v8::internal::FatalProcessOutOfMemory+846 5: 00007FF7D375300F v8::internal::FatalProcessOutOfMemory+639 6: 00007FF7D3939804 v8::internal::Heap::MaxHeapGrowingFactor+9620 7: 00007FF7D3937CCB v8::internal::Heap::MaxHeapGrowingFactor+2651 8: 00007FF7D3A61B2B v8::internal::Factory::AllocateRawWithImmortalMap+59 9: 00007FF7D3A6449D v8::internal::Factory::NewRawOneByteString+77 10: 00007FF7D3AA93D3 v8::internal::interpreter::BytecodeArrayIterator::done+4563 11: 00007FF7D3B0968A v8::internal::WasmJs::Install+124170 12: 00007FF7D3B0BF11 v8::internal::WasmJs::Install+134545 13: 00007FF7D3B105CC v8::internal::WasmJs::Install+152652 14: 000001911535C5C1

download succeeded:

NOTE: Current Zowe LTS version (2.x) is using buffers for file management, but in Latest version (6.x) we switched to streams. Soon, Zowe v6.8.2 will become LTS and streams will be the default.

Also I have NodeJS v10.16.3

Maybe we need to look as this from z/OSMF perspective. I will try to talk with our sys progs.

If you have time, could you try to download it using Postman or cURL tools?

Thanks.

George0013 commented 4 years ago

Hi,

I agree. It seems that the problem could be in the ZOSMF

And about downloading the file in other ways ..., the download is not important, as I said we are doing a PoC about ZOWE and what we want is to do it through ZOWE / ZOSMF.

regards

Alexandru-Dumitru commented 4 years ago

This I understand. The goal was to scope the error, if it really resides in the CLI or z/OSMF. If it's z/OSMF only, you should get the same error while calling the z/OSMF APIs directly.

George0013 commented 4 years ago

Hi,

You're right. I tried to run itcalling the z/OSMF APIs directly (/zosmf/restfiles/ds/SYST.TEMP.WLMXGB) in the browser and and I get the same error "Error 500: java.io.IOException: Error response could not be sent to client. See log for details" at the end of the file.

George0013 commented 4 years ago

Hi,

I've opened a z/OSMF restfile issue at IBM., i will tell you the conclusions.

Regards