Is there a way in RequestServer to determine the max packet size

[cchantep]

In RequestServer it could be useful to know the max packet size to implement custom handler (for Put operation). For know it seems that on OSX it's 32768 but would be nicer to have a clean/portable way to determine that.

[drakkan]

Hi,

please take a look here. The pkg/sftp limit is 256KB. It will refuse to accept bigger packets. So in your WriterAt implementation you can receive at most 256KB buffers

[cchantep]

On osx, each packet passed to WriteAt is at most 32768 bytes.

[drakkan]

On Linux is the same, it is the client that sets the packet size. When used as client pkg/sftp use 32768 as default too.

[cchantep]

Then it would be nice to make available the detected packet size from the current session at hook level.

[drakkan]

Then it would be nice to make available the detected packet size from the current session at hook level.

I don't understand sorry, you already have the size for each packet. A strange client might send packets of different sizes, and the final packet will almost always have a different size.

[cchantep]

The last packet is another story but the max size of packet in current session is useful

[drakkan]

There is no concept of max packet per session, is the client that decide the packet size, this library accepts 256*1024 bytes as max packet size.

Can you please point me to the relevant SSH/SFTP specs? Maybe I'm missing something, thanks

[cchantep]

Apart from using the offset / sessionMaxPacketSize there is no way to determine the absolute order (aka index starting from 0) of the current packet a custom WriteAt provided by hook. Some logic to write the received packet require that.

[puellanivis]

Every WriteAt will have the offset tracked properly by the calls to WriteAt(), and the packet size of the data to write tracked by len(b).

I just don’t understand by what you mean by “absolute order” when… oh wait, I think I just got it. You want to be able to order the packets not just the offset? I don’t understand why that would be necessary. And writers could write these packets in any order of sizes, they don’t always need to be the same length as the maximum packet size.

For instance, a client could write per buffered line of output, rather than waiting until it had a completely full buffer then transmitting.

I don’t think what you want is possible.

[cchantep]

The len(b) is the size of the current buffer, not the expected max packet size. And yes, absolute order may be required, as where you write in WriteAt may require it and not support the very same offset as reference.

[drakkan]

Hi, you can convert the WriteAt to a sequential writer using for example pipeat or reordering the packet yourself based on the offset in a similar way. The server cannot limit the packet size, it can only refuse to handle too big packets, where too big is quite arbitrary

[cchantep]

The use case is not to pipe and the offset by itself is not sufficient to reorder. It only makes possible to determine whether the current offset is greater than the last processed one, not that the current one it's the next one as there is no guarantee another packet must be processed between.

[drakkan]

I don't know your use case, maybe you should better explain it.

Anyway, the offset is enough to reorder the received packets if you accumulate them until you have no "holes". This is what pipeat does.

Another option is to set SftpServerWorkerCount to 1 here: this way reads/writes will become sequential: the client will decide the packet order in this case (a client could send in parallel anyway)

[puellanivis]

I’m sorry, but there’s just no way that the SFTP protocol will allow us to obtain the feature you are looking for. A client could send any data it wants in any particular arbitrary order, with any number of offsets and packet sizes.

We have a ton of extensive code to coördinate multiple parallel requests in WriteTo and ReadFrom. There is no way to get you an absolute order, except by, as drakkan noted, keeping track of how much has been written in total, and delaying the writing of packets until you receive a packet at lastWrite.offset + lastWrite.length.

The SFTP protocol itself does not provide any sequential writing/reading, it is always “WriteAtandReadAt`, it is, in this way, similar to a raw file-system store, and ensuring the consistency of your data is a problem left for the client and server to negotiate.

[drakkan]

There is nothing we can do here, sorry