When FTP transfer mode is set to AUTO or ASCII, download is in ASCII but upload is in IMAGE

[zippy1981]

The FTP server in question is PUB1.DE an iSeries server running OS/400. Accounts are available for free and I'll gladly assist reproducing the issue.

My transfer mode is set to AUTO. The file on the server is EBCIDIC encoded, but if I perform an ASCII GET or PUSH it will translate the file to and from ASCII. If I select one of these files (a source member in the parlance of the server OS) and hit F4, an ASCII download is initialted, and the file is translated. However, when I save the file in the editor, it does an IMAGE or binary transfer and the operating system cannot read the resulting file.

. 2016-01-22 22:53:17.246 Copying 1 files/directories to local directory "C:\Users\zippy\AppData\Local\Temp\F3TD9A5.tmp"
. 2016-01-22 22:53:17.261   PrTime: Yes; PrRO: No; Rght: rw-r--r--; PrR: No (No); FnCs: N; RIC: 퇘; Resume: N (102400); CalcS: Yes; Mask: 
. 2016-01-22 22:53:17.261   TM: M; ClAr: No; RemEOF: No; RemBOM: No; CPS: 0; NewerOnly: No; InclM: ; ResumeL: 0
. 2016-01-22 22:53:17.261   AscM: *.*html; *.htm; *.txt; *.php; *.php3; *.cgi; *.c; *.cpp; *.h; *.pas; *.bas; *.tex; *.pl; *.js; .htaccess; *.xtml; *.css; *.cfg; *.ini; *.sh; *.xml; *.MBR
. 2016-01-22 22:53:17.277 File: '/QSYS.LIB/ZIPPY19811.LIB/QRPGLESRC.FILE/HELLO.MBR' [2016-01-23T09:45:00.000Z] [204]
. 2016-01-22 22:53:17.277 Copying "/QSYS.LIB/ZIPPY19811.LIB/QRPGLESRC.FILE/HELLO.MBR" to local directory started.
. 2016-01-22 22:53:17.277 Ascii transfer mode selected.
. 2016-01-22 22:53:17.277 Starting download of /QSYS.LIB/ZIPPY19811.LIB/QRPGLESRC.FILE/HELLO.MBR
> 2016-01-22 22:53:17.277 TYPE A
< 2016-01-22 22:53:17.461 200 Representation type is ASCII nonprint.
> 2016-01-22 22:53:17.461 PASV
< 2016-01-22 22:53:17.508 227 Entering Passive Mode (178,249,3,21,40,78).
> 2016-01-22 22:53:17.577 RETR HELLO.MBR
< 2016-01-22 22:53:17.808 150 Retrieving member HELLO in file QRPGLESRC in library ZIPPY19811.
. 2016-01-22 22:53:17.877 Trying reuse main TLS session ID
. 2016-01-22 22:53:17.877 ../filezilla/TransferSocket.cpp(1127): TLS layer changed state from none to connected   caller=0x049f4848
. 2016-01-22 22:53:17.930 Session ID reused
. 2016-01-22 22:53:17.930 TLS connection established
< 2016-01-22 22:53:18.046 250 File transfer completed successfully.
. 2016-01-22 22:53:18.046 ../filezilla/TransferSocket.cpp(1127): TLS layer changed state from connected to closed   caller=0x049f4848
. 2016-01-22 22:53:18.061 Download successful
. 2016-01-22 22:53:18.061 Transfer done: 'HELLO.MBR' [204]
. 2016-01-22 22:53:43.184 Copying 1 files/directories to remote directory "/QSYS.LIB/ZIPPY19811.LIB/QRPGLESRC.FILE"
. 2016-01-22 22:53:43.184   PrTime: Yes; PrRO: No; Rght: rw-r--r--; PrR: No (No); FnCs: N; RIC: 푈; Resume: S (102400); CalcS: Yes; Mask: *.*
. 2016-01-22 22:53:43.184   TM: B; ClAr: No; RemEOF: No; RemBOM: No; CPS: 0; NewerOnly: No; InclM: ; ResumeL: 0
. 2016-01-22 22:53:43.184   AscM: *.*html; *.htm; *.txt; *.php; *.php3; *.cgi; *.c; *.cpp; *.h; *.pas; *.bas; *.tex; *.pl; *.js; .htaccess; *.xtml; *.css; *.cfg; *.ini; *.sh; *.xml
. 2016-01-22 22:53:43.200 File: 'C:\Users\zippy\AppData\Local\Temp\F3TD9A5.tmp\HELLO.MBR' [2016-01-23T03:53:43.168Z] [199]
. 2016-01-22 22:53:43.200 Copying "HELLO.MBR" to remote directory started.
. 2016-01-22 22:53:43.200 Binary transfer mode selected.
. 2016-01-22 22:53:43.200 Starting upload of \\?\C:\Users\zippy\AppData\Local\Temp\F3TD9A5.tmp\HELLO.MBR
> 2016-01-22 22:53:43.200 TYPE I
< 2016-01-22 22:53:43.329 200 Representation type is binary IMAGE.
> 2016-01-22 22:53:43.382 PASV
< 2016-01-22 22:53:43.500 227 Entering Passive Mode (178,249,3,21,69,207).
> 2016-01-22 22:53:43.567 STOR HELLO.MBR
< 2016-01-22 22:53:43.825 150 Sending file to member HELLO in file QRPGLESRC in library ZIPPY19811.
. 2016-01-22 22:53:43.879 Trying reuse main TLS session ID
. 2016-01-22 22:53:43.879 ../filezilla/TransferSocket.cpp(1127): TLS layer changed state from none to connected   caller=0x04a05aa8
. 2016-01-22 22:53:43.982 Session ID reused
. 2016-01-22 22:53:43.982 TLS connection established
< 2016-01-22 22:53:44.098 250 File transfer completed successfully.
. 2016-01-22 22:53:44.229 Upload successful
. 2016-01-22 22:53:44.229 Transfer done: 'HELLO.MBR' [199]

[VictorVG]

IBM EBCIDIC code page does not match the encoding tables with ASCII, and to avoid future problems of her conversion, activate the transfer of all files as Binary. When you send a file as a Pline/Text FTP adds the symbol of the End of File (EOF, CtrlZ) which leads to its deterioration.

All transactions conversion of downloaded files when working with the IBM OS/360 - OS/370 - AS/400 - AIX - OS/390 must be carried out only as a local copy there is a nonzero probability that the intermediate nodes of the network and do try to "correct errors" if the files are transmitted as Plain/Text. It all depends on their configuration. If the file is transferred in BINARY mode, the intermediate network nodes do not change it, and no data is lost.

[zippy1981]

@VictorVG Are you saying the server will choke on the EOF character when converting back from ASCII to EBCIDIC?

I see two problems with directly editing EBCIDIC files at the moment"

1. The editor initially displays the EBCIDIC files as ASCII encoded, so I have to change the encoding manually. If I could set the default encoding for the FTP configuration, that would solve that.
2. The files don't have a newline, but are padded with spaces so each line is exactly 80 characters. Is there a way to make the editor chop them up correctly?

[VictorVG]

In our case, if we pass the remote server command "type image" (see. RFC959) command to "get/mget file / files "the FTP protocol does not change its / their structure considering binary files, and you have log (see. timestamp. 01/22/2016 22: 53: 17.277) worth recording. "01/22/2016 22: 53: 17.277 Ascii transfer mode selected" - files are transferred in ASCII mode in which the end of each file appended to the symbol EOF, symbols of Cr (Lf ) transfer lines do not change.

What we need to solve the problem?

1) Set up a connection to the BINARY mode the files came "as is" - today NetBox no support other code pages other than 866/1251/20866/65001, so all you need to perform a text-editor;

2) The task of reformatting lines simple enough and can be easily solved or Lua script or plugin AutoWrap distribution of Far Manager.

3) The problem of conversion is achieved either as "open text editor and save to change the encoding of ShiftF2" TextConv or plugin which is able to change the encoding, line breaks, replace the tabs with spaces and remove blank lines at the end of the file.

In other editors their money, but there is no reason to use for this simple editors like ee, ed, edit, notepad - they do not have many tools as they are designed to solve basic problems, and viewing of text.

[zippy1981]

You brought up some great points, and someone else brought up another good point, that if I ever have source members that aren't US English, ASCII transfer will lose data.

I think its best if I configure ESC to load .MBR files as wrapped at 80 charcters and EBCIDIC encoded.

[VictorVG]

Just upload, such as binary files and will not lose data. Why did you decide that the inside of the container are stored text data to be recoded? And if your assumption is incorrect? After all, unless you have received from a remote host file and not seeing the structure you do not know what's there. This in Windows from the existence of a set of conventional rules that the suffix "extension" definitely compares the contents of the file and the application handler, and UNIX, IBM OS/360 (OS/370, OS/390), this rule was not, and hopefully never will be. There is the concept of "access rights" - RWX - Read -Write-Execute and "Protocol" defines the application processor, depending on the structure of the file, and he can have any name which is part of the suffix name is used to uniquely identify a particular data set .

[zippy1981]

Why did you decide that the inside of the container are stored text data to be recoded?

Two reasons.

First of all, by design of the FTP server on OS/400, when you download with ASCII, it converts the members from EBCIDIC to ASCII. When you upload it converts from ASCII to EBCDIC, removes the newlines, and pads each line to 80 characters.

Secondly, the first time I opened the file after BINARY transfer and set encoding to EBCDIC I got all the text on one line. and lots of spaces. After ASCII transfer I got newlines and no spaces.

[VictorVG]

Two reason it is certainly powerful,:) but what about the loss of data? You yourself hopelessly corrupt files willed designated, he alleged properties and then refuse to knowingly use safe methods to test your conclusions. If the file uses a non-ASCII encoding, the same character "!" in character sets HP EBCDIC, IBM EBCDIC, EBCDIC AT & T have different values: ASCII - 0x21, HP EBCDIC - 0x4F, IBM EBCDIC - 0x5A, EBCDIC AT & T - 0x5A, and he is not the only plus to the second part of the table with addresses from 0x80 is national symbols and that you are there to encode one case is known ...

Just use VEDIT for EBCDIC files.:)