This introduces a new feature: packet splitting. This effectively splits files bigger than the maximum size allowed by OpenDHT (which is around 56 KiB) and spreads the pieces in packets published evenly on the DHT. Let's consider the following regarding this new approach:
The paste code (used for sharing the pasted blob) now incorporates a new field for encoding the number of packets the file was split into. The format is now described as follows:
This suggests 4 possible code formats. The size for each field is given in the following table:
Data
Size
Size (in characters)
LOCATION_CODE
32 bits
8 hex chars
NPACKETS
8 bits
2 hex chars
PWD
32 bits
8 hex chars
For now this change is not backward compatible with prior version such as 0.3.3 or even the latest master.
[ ] implement backward compatibility by parsing the field NPACKETS or not.
While new versions of dpaste will support old code formats, it obviously won't be magically the case for the other way around except for values of size less than 56KiB (non splitted values) and without AES encryption.
As said above, the file is split in packets spread on evenly on the DHT. While publishing all packets around a same area on the DHT could improve network performance (subsequent requests would resolve faster after the first one), OpenDHT's rate limiting could interfere with the capacity of a node to publish the whole file it looks to publish.
The time issue mentioned above is in part due to the multiple gets/puts done sequentially inside Bin.
[ ] Parallelize the multiple get/put operations.
Tests have also been added to verify good behaviour of different new functions.
N.B: OpenDHT advertises 64KiB as its maximum value size, but it doesn't take serializing into account so the effective maximum size is rather around 56 KiB.
As this change does break in some way compatibility with previous versions, this change should be part of the 1.0.0 release to indicate the possible break.
This introduces a new feature: packet splitting. This effectively splits files bigger than the maximum size allowed by OpenDHT (which is around 56 KiB) and spreads the pieces in packets published evenly on the DHT. Let's consider the following regarding this new approach:
The paste code (used for sharing the pasted blob) now incorporates a new field for encoding the number of packets the file was split into. The format is now described as follows:
This suggests 4 possible code formats. The size for each field is given in the following table:
LOCATION_CODE
NPACKETS
PWD
For now this change is not backward compatible with prior version such as 0.3.3 or even the latest master.
NPACKETS
or not.While new versions of dpaste will support old code formats, it obviously won't be magically the case for the other way around except for values of size less than 56KiB (non splitted values) and without AES encryption.
Tests have also been added to verify good behaviour of different new functions.
N.B: OpenDHT advertises 64KiB as its maximum value size, but it doesn't take serializing into account so the effective maximum size is rather around 56 KiB.