"Value too large for defined data type" when dictionary file is > 4GB

[GoogleCodeExporter]

What steps will reproduce the problem?
1. File larger than 4GB
2.
3.

What is the expected output? What do you see instead?
normal operation

What version of the product are you using? On what operating system?
0.2

Please provide any additional information below.

Original issue reported on code.google.com by jeff.gus...@gmail.com on 9 Oct 2008 at 10:13

[GoogleCodeExporter]

Thanks for the report, Jeff!

open-vcdiff currently uses 32-bit integers to represent addresses and offsets.  
That 
representation causes the file size limitation you encountered.  The solution 
will be 
to convert to using 64-bit integers uniformly.  I will look into doing so in a 
future 
version of open-vcdiff.

Original comment by openvcd...@gmail.com on 9 Oct 2008 at 10:26

• Changed state: Accepted
• Added labels: OpSys-All, Type-Enhancement, Usability
• Removed labels: Type-Defect

[GoogleCodeExporter]

To expand upon my earlier comment, open-vcdiff was not originally designed with 
very
large input sets in mind, but rather as a tool for implementing the SDCH 
protocol for
moderately-sized HTTP responses.  In that context, 2GB was seen as an ample 
limit.

I would like to make open-vcdiff as useful as possible for as many people as
possible.  Generalizing it to handle very large input and output files (so that 
it
can be applied, for example, to revision control of huge text files) will be a 
good
step towards that goal.

Side note: the use of std::string objects for internal storage restricts data 
sizes
to std::string::max_size(), independently of whether 32-bit or 64-bit integers 
are
used for addresses and offsets.

Original comment by openvcd...@gmail.com on 11 Oct 2008 at 12:22

[GoogleCodeExporter]

Original comment by openvcd...@gmail.com on 24 Oct 2008 at 12:26

• Added labels: Priority-Low
• Removed labels: Priority-Medium

[GoogleCodeExporter]

Where is this on the roadmap?  Will it be done anytime soon?

Original comment by john.b.s...@gmail.com on 29 Oct 2010 at 8:46

[GoogleCodeExporter]

Thanks for your question.  This requested feature is not actively under 
development.  We welcome code contributions if anyone is interested in updating 
open-vcdiff to use 64-bit offsets.

Original comment by openvcd...@gmail.com on 15 Nov 2010 at 7:01

[GoogleCodeExporter]

Any update on this? This would be handy functionality.

Original comment by gcon...@google.com on 14 Aug 2013 at 1:07