SFTPv3Client.closeFile(fh): "The server sent an invalid id field."

[GoogleCodeExporter]

Exception thrown when closing open file handle.

What steps will reproduce the problem?
1. Create client.
2. Perform file transfer using filehandler.
3. client.closeFile(fh)
(4. client.close())

Backtrace:
java.io.IOException: The server sent an invalid id field.
    at ch.ethz.ssh2.SFTPv3Client.expectStatusOKMessage(SFTPv3Client.java:550)
    at ch.ethz.ssh2.SFTPv3Client.closeHandle(SFTPv3Client.java:299)
    at ch.ethz.ssh2.SFTPv3Client.closeFile(SFTPv3Client.java:1569)

Original issue reported on code.google.com by sky.ri...@gmail.com on 21 Jun 2011 at 1:28

[GoogleCodeExporter]

If you are using the SFTPInputStream introduced in the head of the repository? 
It automatically closes the handle for you when the stream is closed.

Original comment by dkocher@sudo.ch on 21 Jun 2011 at 6:50

[GoogleCodeExporter]

Original comment by dkocher@sudo.ch on 24 Jun 2011 at 11:56

• Changed state: Invalid

[GoogleCodeExporter]

Actually I'm using SFTPv3FileHandle fh:

fh = client.openFileRO(remotefile);

I don't know what it does internally. However, if what you say is the case 
here, then at least it should not throw an exception when doing the following:

...
   (I/O operation)
...

try {
   if (!fh.isClosed())
      client.closeFile(fh);
} catch (IOException e) {
  e.printStackTrace();
}

So, I think the fh.isClosed() check doesn't work correctly, or something else 
is wrong.

Original comment by sky.ri...@gmail.com on 27 Jun 2011 at 2:06

[GoogleCodeExporter]

Original comment by dkocher@sudo.ch on 27 Jun 2011 at 9:48

• Changed state: Accepted

[GoogleCodeExporter]

Can you post the full source code so I can try to replicate the issue here.

Original comment by dkocher@sudo.ch on 28 Jun 2011 at 9:58

[GoogleCodeExporter]

Here is the complete method:

    /**
     * Copy file on remote server to the local filesystem via SFTP. Assumes that
     * the connection is established and authenticated.
     * @param localfile name of the destination file on the local machine
     * @param remotefile name of the source file on the remote server
     * @throws IOException
     */
    @SuppressWarnings("unused")
    private void SFTPDownload(String localfile, String remotefile) throws IOException {
        //System.out.println("Download " + remotefile + " to " + localfile + ".");

        SFTPv3FileHandle fh = null;
        FileOutputStream fos = null;
        SFTPv3Client client = null;
        this.setBytesTransferred(0);

        /* Create a SFTP client */
        try {
            client = new SFTPv3Client(this.conn);
        } catch (IOException e) {
            throw new IOException("Error creating SFTP client; Cause: " + e.getMessage());
        }

        /* Open the remote file read-only */
        try {
            fh = client.openFileRO(remotefile);
        } catch (IOException e) {
            throw new IOException("Could not open remote file for reading; Cause: " + e.getMessage());
        }

        /* Get the length of the remote file */
        long len = 0;
        try {
            SFTPv3FileAttributes attr = client.stat(remotefile);
            len = attr.size;
        } catch (IOException e) {
            throw new IOException("Could not retrieve length of remote file; Cause: " + e.getMessage());
        }
        //System.out.println(remotefile + ": File length is " + len);

        /* Open the local file for writing, but delete any existing file first */
        File lf = new File(localfile);
        if (lf.exists() && ! lf.delete()) {
            throw new IOException("Could not delete already existing local file.");
        }
        try {
            fos = new FileOutputStream(lf);
        } catch (Exception  e) {
            throw new IOException("Could not open local file for writing; Cause: " + e.getMessage());
        }

        /* Perform copy operation */
        for (long offset = 0; offset < len; offset += SFTP_READ_MAX_BYTES) {
            byte[] data = new byte[SFTP_READ_MAX_BYTES];
            int count;
            try {
                count = client.read(fh, offset, data, 0, SFTP_READ_MAX_BYTES);
                //System.out.println("Read " + count + " bytes.");
            } catch (IOException e) {
                throw new IOException("Error reading remote file; Cause: " + e.getMessage());
            }
            if (count != -1) {
                try {
                    fos.write(data, 0, count);
                    this.increaseBytesTransferred(count);
                    //System.out.println("Wrote " + count + " bytes.");
                } catch (IOException e) {
                    throw new IOException("Error writing local file; Cause: " + e.getMessage());
                }
            }
        }

        /* Close open files */
        try {
            if (!fh.isClosed()) {
                client.closeFile(fh);
            }
        } catch (IOException e) {
            /* We do not care if closing the remote file is unsuccessful, as
             * we're going to close the channel anyway. However, we still need
             * to take care that this won't stop us completing the transfer.
             */
            System.out.println("Error closing file: " + e.getMessage());
            e.printStackTrace();
        } finally {
            client.close();
        }
        try {
            fos.close();
        } catch (IOException e) {
            throw new IOException("Error closing local file; Cause: " + e.getMessage());
        }
    }

Original comment by sky.ri...@gmail.com on 28 Jun 2011 at 1:08

[GoogleCodeExporter]

You should call SFTPv3Client#read until it returns -1. 

Original comment by dkocher@sudo.ch on 6 Jul 2011 at 8:07

[GoogleCodeExporter]

It might be simpler to use the SFTPInputStream class and use standard java.io. 
to copy between streams. 

Original comment by dkocher@sudo.ch on 6 Jul 2011 at 8:08

[GoogleCodeExporter]

Because of request parallelism, for small files more requests are being sent 
than necessary and the status of each request must be read before attempting to 
close the file.

Original comment by dkocher@sudo.ch on 6 Jul 2011 at 8:09

[GoogleCodeExporter]

Another workaround would be to set request parallelism to zero but that would 
impact performance for large file transfers. See 
SFTPv3Client#setRequestParallelism.

Original comment by dkocher@sudo.ch on 6 Jul 2011 at 8:10

[GoogleCodeExporter]

As of r46 your sample code should work. Pending read status is emptied when 
closing the handle.

Original comment by dkocher@sudo.ch on 6 Jul 2011 at 8:41

• Changed state: Fixed

[GoogleCodeExporter]

I wrote the code using the old beta version, before I found the project here on 
google code. Back then, it worked without any error, and I didn't want to 
rewrite it, since the methods were still there. I'll consider the 
SFTPInputStream alternative, though.

Thank you for your suggestions and the fix.

Original comment by sky.ri...@gmail.com on 6 Jul 2011 at 10:34