search

peter-lawrey / HugeCollections-OLD

Huge Collections for Java using efficient off heap storage
273 stars 51 forks source link

problem using SharedHashMapBuilder ... #36

Closed Cotton-Ben closed 10 years ago

Cotton-Ben commented 10 years ago

Hi,

I fetched/merged into my clone'd workspaces all updates from each of my upstream views of the various OpenHFT parent repo modules.

I then re-ran my Test at https://github.com/Cotton-Ben/HugeCollections/blob/master/collections/src/test/java/net/openhft/collections/fromdocs/com/jpmorgan/pingpong_latency/PingPongPlayerLeft.java

via 

java  org.junit.runner.JUnitCore net.openhft.collections.fromdocs.com.jpmorgan.pingpong_latency.PingPongPlayerLeft

Since this update, when the following line #18 in my Test is executed:

SharedHashMap<String, BondVOInterface> shmLeft = new SharedHashMapBuilder()
                .generatedValueType(true)
                .entrySize(320)
                .create(
                        new File("/dev/shm/BondPortfolioSHM"),
                        String.class,
                        BondVOInterface.class
                );

This test now fails with the following exception:

Time: 0.107
There was 1 failure:
1) bondExample(net.openhft.collections.fromdocs.com.jpmorgan.pingpong_latency.PingPongPlayerLeft)
java.lang.IllegalArgumentException: invalid size: 343048307875968
    at net.openhft.lang.io.MappedStore.<init>(MappedStore.java:54)
    at net.openhft.lang.io.MappedStore.<init>(MappedStore.java:49)
    at net.openhft.collections.AbstractVanillaSharedHashMap.createMappedStoreAndSegments(VanillaSharedHashMap.java:148)
    at net.openhft.collections.VanillaSharedHashMap.<init>(VanillaSharedHashMap.java:45)
    at net.openhft.collections.SharedHashMapBuilder.create(SharedHashMapBuilder.java:264)
    at net.openhft.collections.fromdocs.com.jpmorgan.pingpong_latency.PingPongPlayerLeft.bondExample(PingPongPlayerLeft.java:18)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.junit.runners.Suite.runChild(Suite.java:127)
    at org.junit.runners.Suite.runChild(Suite.java:26)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:138)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:117)
    at org.junit.runner.JUnitCore.runMain(JUnitCore.java:96)
    at org.junit.runner.JUnitCore.runMainAndExit(JUnitCore.java:47)
    at org.junit.runner.JUnitCore.main(JUnitCore.java:40)

FAILURES!!!
Tests run: 1,  Failures: 1

Should I expect the way that line #18 above is coded in my Test to work as before without exception? Any recommendation for remedy?

peter-lawrey commented 10 years ago

It is possible there have been changes in the format. Can you delete the old files first? We need to get to a point where old formats are supported.

On 26 June 2014 20:53, Ben Cotton notifications@github.com wrote:

Hi,

I fetched/merged into my clone'd workspaces all updates from each of my upstream views of the various OpenHFT parent repo modules.

I then re-ran my Test at https://github.com/Cotton-Ben/HugeCollections/blob/master/collections/src/test/java/net/openhft/collections/fromdocs/com/jpmorgan/pingpong_latency/PingPongPlayerLeft.java

via

java org.junit.runner.JUnitCore net.openhft.collections.fromdocs.com.jpmorgan.pingpong_latency.PingPongPlayerLeft

Since this update, when the following line #18 https://github.com/OpenHFT/HugeCollections/pull/18 in my Test is executed:

SharedHashMap<String, BondVOInterface> shmLeft = new SharedHashMapBuilder() .generatedValueType(true) .entrySize(320) .create( new File("/dev/shm/BondPortfolioSHM"), String.class, BondVOInterface.class );

This test now fails with the following exception:

Time: 0.107 There was 1 failure: 1) bondExample(net.openhft.collections.fromdocs.com.jpmorgan.pingpong_latency.PingPongPlayerLeft) java.lang.IllegalArgumentException: invalid size: 343048307875968 at net.openhft.lang.io.MappedStore.(MappedStore.java:54) at net.openhft.lang.io.MappedStore.(MappedStore.java:49) at net.openhft.collections.AbstractVanillaSharedHashMap.createMappedStoreAndSegments(VanillaSharedHashMap.java:148) at net.openhft.collections.VanillaSharedHashMap.(VanillaSharedHashMap.java:45) at net.openhft.collections.SharedHashMapBuilder.create(SharedHashMapBuilder.java:264) at net.openhft.collections.fromdocs.com.jpmorgan.pingpong_latency.PingPongPlayerLeft.bondExample(PingPongPlayerLeft.java:18) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runners.Suite.runChild(Suite.java:127) at org.junit.runners.Suite.runChild(Suite.java:26) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runner.JUnitCore.run(JUnitCore.java:160) at org.junit.runner.JUnitCore.run(JUnitCore.java:138) at org.junit.runner.JUnitCore.run(JUnitCore.java:117) at org.junit.runner.JUnitCore.runMain(JUnitCore.java:96) at org.junit.runner.JUnitCore.runMainAndExit(JUnitCore.java:47) at org.junit.runner.JUnitCore.main(JUnitCore.java:40)

FAILURES!!! Tests run: 1, Failures: 1

Should I expect the way that line #18 https://github.com/OpenHFT/HugeCollections/pull/18 above is coded in my Test to work as before without exception? Any recommendation for remedy?

— Reply to this email directly or view it on GitHub https://github.com/OpenHFT/HugeCollections/issues/36.

Cotton-Ben commented 10 years ago

Not really sure what you mean by "Can you delete the old files first?"

Cotton-Ben commented 10 years ago

Ah! I got it 

rm -fr /dev/shm/*

did the trick!! ;-)

Thanks Peter ...

peter-lawrey commented 10 years ago

Like I said, you should be able to read old file with newer versions of SHM. We are in the process of documenting and solidifying the format so this issue doesn't arise.

I am also working on ways to make data changes more robust with self-describing data formats. I have some plans to support both zero-copy in place data structures which also support data format changes.

On 26 June 2014 21:13, Ben Cotton notifications@github.com wrote:

Closed #36 https://github.com/OpenHFT/HugeCollections/issues/36.

— Reply to this email directly or view it on GitHub https://github.com/OpenHFT/HugeCollections/issues/36#event-135612661.

Cotton-Ben commented 10 years ago

As always, great work!

I am also working on ways to make data changes more robust with self-describing data formats

Oh man ... you've got our minds percolating now (also, as always).

peter-lawrey commented 10 years ago

You can have a format which allows random access, but also has self describing data embedded (which doesn't change of course) so it can still be read if the format changes or you don't have access to such information.

On 26 June 2014 21:24, Ben Cotton notifications@github.com wrote:

As always, great work!

I am also working on ways to make data changes more robust with self-describing data formats

Oh man ... you've got our minds percolating now (also, as always).

— Reply to this email directly or view it on GitHub https://github.com/OpenHFT/HugeCollections/issues/36#issuecomment-47268396 .