nyurik / timeseriesdb

Automatically exported from code.google.com/p/timeseriesdb
GNU General Public License v3.0
13 stars 6 forks source link

Appending Data to CompressedSeriesFile Exception #4

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago
When appending Data to a compressedSeries file I get the following exception 
(below).

It appears that when appending, and checking the last index, the TryGetIndex 
method has a hang up when it does a YieldFixedSize with a -1. The 
CalcMaxItemsInBlock is called with a byteBufSize of 40, and the maxSize is 50, 
resulting in a -1 value.

ArgumentOutOfRangeException
<0
Parameter name: bufferSize
Actual value was -1.

   at NYurik.FastBinTimeseries.Buffer`1..ctor(Int32 bufferSize) in C:\Users\Karl\Visual Studio\External Projects\timeseriesdb\FastBinTimeseries\Buffer.cs:line 39
   at NYurik.FastBinTimeseries.BufferProvider`1.<YieldFixedSize>d__6.MoveNext() in C:\Users\Karl\Visual Studio\External Projects\timeseriesdb\FastBinTimeseries\BufferProvider.cs:line 85
   at NYurik.FastBinTimeseries.BinCompressedSeriesFile`2.TryGetIndex(Int64 blockIndex, Int64 cachedFileCount, TInd& index, Boolean inReverse) in C:\Users\Karl\Visual Studio\External Projects\timeseriesdb\FastBinTimeseries\BinCompressedSeriesFile.cs:line 501
   at NYurik.FastBinTimeseries.BinCompressedSeriesFile`2.get_LastIndex() in C:\Users\Karl\Visual Studio\External Projects\timeseriesdb\FastBinTimeseries\BinCompressedSeriesFile.cs:line 236
   at NYurik.FastBinTimeseries.BinCompressedSeriesFile`2.AppendData(IEnumerable`1 bufferStream, Boolean allowFileTruncation) in C:\Users\Karl\Visual Studio\External Projects\timeseriesdb\FastBinTimeseries\BinCompressedSeriesFile.cs:line 306
   at Tanius.Brontosaurus.Lib.Tapes.MultiContractIndexedBinFile`1.SaveContractsDelta(List`1 contracts, String dataFile, String indexFile) in C:\Users\Karl\Visual Studio\Tanius.Brontosaurus\Tanius.Brontosaurus.Lib\Tapes\MultiContractIndexedBinFile.cs:line 208
   at Tanius.Brontosaurus.Converter.Program.Main(String[] args) in C:\Users\Karl\Visual Studio\Tanius.Brontosaurus\Tanius.Brontosaurus.Converter\Program.cs:line 38
   at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
   at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

Original issue reported on code.google.com by kar...@gmail.com on 23 Feb 2012 at 7:42

GoogleCodeExporter commented 9 years ago
the file already had data and you are appending more? could you provide a short 
test case please. thx

Original comment by yuriastrakhan on 23 Feb 2012 at 7:47

GoogleCodeExporter commented 9 years ago
Right - the file already had data, and I'm appending more. It appears that the 
last block has only 40 bytes in it, while the method CalcMaxItemsInBlock was 
called with a MaxItemsByteSize of 50 and a minItemByteSize of 5. This results 
in a MaxItemsInBlock of -1 (40 - 50) / 5 + 1 = -1.

I'm assuming each block must begin with a write that uses the MaxItemsByteSize, 
somehow a block is being written to that is shorter than this MaxItemsByteSize.

It is quite reporoducable, but my test case uses a several gb file. I will see 
if I can reproduce with a simple test case. Let me know if you can understand 
from the details I've sent.

Original comment by kar...@gmail.com on 23 Feb 2012 at 7:52

GoogleCodeExporter commented 9 years ago
I kinda understand where the problem is, but i still could use a unit test of 
some sorts so that i can reliably test it and make sure any similar bugs are 
fixed. Could you create a simple one like that? It shouldn't require a 
pre-existing data file.

Original comment by yuriastrakhan on 23 Feb 2012 at 8:30

GoogleCodeExporter commented 9 years ago
Changing CalcMaxItemsInBlock method to simply be blockSize / _minItemByteSize 
appears to solve the issue. I did several tests that showed the first record 
written to a block can often be much less than the maxByte size, which appears 
to be causing this issue.

Original comment by kar...@gmail.com on 23 Feb 2012 at 9:05

GoogleCodeExporter commented 9 years ago
I have changed some writing code, hard to tell if I have fixed your issues.
Can provide a simple unit test for this? For examples, see WritableFeedTests.cs

Original comment by yuriastrakhan on 1 Mar 2012 at 8:02

GoogleCodeExporter commented 9 years ago
Please see the attached unit test which causes this exception.

If you modify CalcMaxItemsInBlock to "return 1 + (blockSize - _maxItemByteSize) 
/ _minItemByteSize;" within BinCompressedSeriesFile.cs, the tests pass.

Original comment by kar...@gmail.com on 2 Mar 2012 at 8:25

Attachments:

GoogleCodeExporter commented 9 years ago
fixed, thanks!

Original comment by yuriastrakhan on 2 Mar 2012 at 10:18