search

GiraffaFS / giraffa

Giraffa FileSystem (Slack: giraffa-fs.slack.com)
https://giraffa.ci.cloudbees.com
Apache License 2.0
17 stars 6 forks source link

Upgrade to HBase 1.0.1 #122

Closed octo47 closed 9 years ago

octo47 commented 9 years ago

We need to upgrade to recent HBase version.

octo47 commented 9 years ago

A bit on what is there:

  1. reapplied #114, because it didn't work on MacOSX.
  2. reapplied rolled back HBase changes from #94
  3. Added fixes from #94 comments

Tests only first test works under maven, next tests won't initialize, they try to use strange zk port. After fixing LocatedBlock initialisation got following exception.

15/05/06 12:57:02 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:63137, sessionid = 0x14d2916093f000c, negotiated timeout = 40000
15/05/06 12:57:02 ERROR ipc.RpcServer: Unexpected throwable object 
java.lang.ArrayIndexOutOfBoundsException: -1
    at java.util.ArrayList.elementData(ArrayList.java:418)
    at java.util.ArrayList.get(ArrayList.java:431)
    at org.apache.giraffa.INode.getLocatedFileStatus(INode.java:103)
    at org.apache.giraffa.hbase.NamespaceProcessor.create(NamespaceProcessor.java:354)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:369)
    at org.apache.giraffa.hbase.ClientNamenodeProtocolServerSideCallbackTranslatorPB.create(ClientNamenodeProtocolServerSideCallbackTranslatorPB.java:216)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$1.create(ClientNamenodeProtocolProtos.java)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol.callMethod(ClientNamenodeProtocolProtos.java)
    at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:6154)
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1692)
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1674)
octo47 commented 9 years ago

It seems main problem now is the fact, that org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel instantiate its own org.apache.hadoop.hbase.client.RegionServerCallable which is org.apache.hadoop.hbase.client.RetryingCallable. So any our IOExceptions actually will be retried. That is how org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel#callExecService method works. So we need to wrap our exceptions and return just regular Message with embedded (may be as bytes) actual NameNodeProtocol reply object.

octo47 commented 9 years ago

One possible solution can be by encoding exception into the string. Looks quite hacky, but unless there will be more clean solution built into HBase there no clean way to intercept instance of retryable callable. So here is how it can be done: 0b96e49186cae3cd1ef476865f093e1e3acc9a75

octo47 commented 9 years ago

More changes. I filed two jiras hbase which is affect our implementation: we need to pass barely buildable response messages. I filed jira https://issues.apache.org/jira/browse/HBASE-13646 and made patch for that. Another problem is retries in serviceCall for coprocessor, it makes any failure will be retried forever. https://issues.apache.org/jira/browse/HBASE-13647. For now I fixed that with setting property in tests to some small value like 5seconds.

shvachko commented 9 years ago

Can we upgrade to HBase 1.0 without any changes to Giraffa code?

Fixing HBase is great! Doesn't help our upgrade though.

octo47 commented 9 years ago
  1. Yes, it will compile.
  2. Yes, it can pass tests, just need to create minimal protobuf object which will not throw exceptions that something missing.
milandesai commented 9 years ago

Hi Andrei, the reason the Exceptions weren't working was that it is not possible to configure an interceptor for the RegionCoprocessorRpcChannel. The configuration properties added in core-site for the interceptor are actually for HConnectionImplementation so they are no good. Instead, we can keep our GiraffaRpcRetryingCallerFactory class from trunk but change the logic to simply call the super with an instance of our FileSystemExceptionInterceptor. Now the interceptor will be loaded and exceptions will work. I pushed the change onto your branch just for demonstration, you will need to merge Issue 94 and revert some of your previous commits to get it to work. I tried on my machine and all of TestExceptionHandling passed.

milandesai commented 9 years ago

I have made the proto file updates, response fixes, and other changes on Issue 94, so you should revert your changes back to your main upgrade commit and merge Issue 94. You should also merge in Issue 130. Also FYI, the Jasper changes you made to the pom break the web ui build for me, the code no longer compiles.

shvachko commented 9 years ago

Andrey, looks like that Milan suggested a way to handle exceptions with current code. So could you please update the HBase upgrade branch, so that it minimizes changes required for upgrade only.

milandesai commented 9 years ago

Yes, we can get Exceptions to work without the interceptor stuff by keeping the existing GiraffaRpcRetryingCallerFactory and GiraffaRpcRetryingCaller; we just need to modify the parameters in the method calls to get it to compile. I'll go ahead and revert the interceptor related stuff since those were my changes. Then @octo47 you just need to modify the above classes to make them compile, and Exceptions should work. I will introduce the interceptor changes later in a different issue.

octo47 commented 9 years ago

@milandesai why we still need interceptor. commit 019ac48d3c393e3c5b724bf1c83b38e377fa1580 solves that by encoding original exception and throwing DontRetryIOE. actually all tests are passed, the only problem in two tests TestGiraffaFS and TestGiraffaFSNegative they can't close fs. (there could help RpcRetryable, but quite possible to wrap close() exceptins with DontRetryIOE)

octo47 commented 9 years ago

that was why tests didn't finished correctly #134

octo47 commented 9 years ago

yeah, that was premature... tests run in IDE, somehow they don't run under maven.

milandesai commented 9 years ago

We don't need the interceptor, but we also don't need to encode exceptions if we just use the existing logic - which is why I reverted all interceptor related changes. It you fix the compilation errors in the two classes mentioned in my previous comment, exceptions should work.

On May 13, 2015, at 8:56 AM, Andrey Stepachev notifications@github.com wrote:

yeah, that was premature... tests run in IDE, somehow they don't run under maven.

— Reply to this email directly or view it on GitHub.

octo47 commented 9 years ago

Ok, @milandesai. It compiles and part of tests are passed. Let's merge that so you can fix rest of the tests.

shvachko commented 9 years ago

Let's avoid force pushes, please. Checked with Milan, he is grooming your branch now. Andrey, why do you need jasper changes?

octo47 commented 9 years ago

Without jasper changes I have an exception when trying to build: https://gist.github.com/octo47/37e3af28fcc7c070746b

milandesai commented 9 years ago

TestGiraffaUpgrade is one of the two tests that are still failing (the other being TestGiraffaCLI). The problem is that we have to use the new offline image viewer, which is class OfflineImageViewerPB. But there is no more "Indented" processing option, which means we have to use "XML". So we need to xml parse the output to get the INode information. Working on that right now.

milandesai commented 9 years ago

I was able to fix TestGiraffaUpgrade without resorting to the XML parser by forcing the FSImage to be saved in legacy format. There was some additions and formatting changes even in the legacy OIV with the Indented parser so the test needed to be slightly reformatted. It passes now, moving on to TestGiraffaCLI.

shvachko commented 9 years ago

So, I replaced testHDFSConf.xml with the one from HDFS 2.5.1 and TestGiraffaCLI is passing. But something else is failing in the build framework after that. LMK if you know what it could be:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.13:test (default-test) on project giraffa-core: Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:2.13:test failed: There was an error in the forked process
[ERROR] org.apache.maven.surefire.testset.TestSetFailedException: java.lang.ArrayIndexOutOfBoundsException: 0; nested exception is java.lang.ArrayIndexOutOfBoundsException: 0
[ERROR] java.lang.ArrayIndexOutOfBoundsException: 0
[ERROR] at org.apache.maven.surefire.report.SmartStackTraceParser.rootIsInclass(SmartStackTraceParser.java:176)
[ERROR] at org.apache.maven.surefire.report.SmartStackTraceParser.getString(SmartStackTraceParser.java:131)
[ERROR] at org.apache.maven.surefire.common.junit4.JUnit4StackTraceWriter.smartTrimmedStackTrace(JUnit4StackTraceWriter.java:73)
[ERROR] at org.apache.maven.surefire.booter.ForkingRunListener.encode(ForkingRunListener.java:328)
[ERROR] at org.apache.maven.surefire.booter.ForkingRunListener.encode(ForkingRunListener.java:312)
[ERROR] at org.apache.maven.surefire.booter.ForkingRunListener.toString(ForkingRunListener.java:258)
[ERROR] at org.apache.maven.surefire.booter.ForkingRunListener.testError(ForkingRunListener.java:131)
[ERROR] at org.apache.maven.surefire.common.junit4.JUnit4RunListener.testFailure(JUnit4RunListener.java:111)
[ERROR] at org.junit.runner.notification.RunNotifier$4.notifyListener(RunNotifier.java:100)
[ERROR] at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:41)
[ERROR] at org.junit.runner.notification.RunNotifier.fireTestFailure(RunNotifier.java:97)
[ERROR] at org.junit.internal.runners.model.EachTestNotifier.addFailure(EachTestNotifier.java:25)
[ERROR] at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:83)
[ERROR] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
[ERROR] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
[ERROR] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
[ERROR] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
[ERROR] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
[ERROR] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
[ERROR] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
[ERROR] at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
[ERROR] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
[ERROR] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
[ERROR] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
[ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[ERROR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR] at java.lang.reflect.Method.invoke(Method.java:601)
[ERROR] at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:208)
[ERROR] at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:158)
[ERROR] at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:86)
[ERROR] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
[ERROR] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:95)
octo47 commented 9 years ago

looks like https://jira.codehaus.org/browse/SUREFIRE-967, 

For some whatever strange reason, WebSphere creates an exception chain, in which not all exceptions in the chain have a stack trace array filled in. (I have heard rumors of JVMs loosing stack traces in exceptions under some strange conditions).

In our case it can be thrown exception with no stacktrace in it. It is fixed in 2.14.1+ of surefire. We can use more recent version of surefire to fix this.

shvachko commented 9 years ago

Thanks for the hint, Andrey. In updated issue-122 branch I upgraded surefire to 2.16, fixed few issues with TestGiraffaCLI, including reducing passing bar to 86% from 94%. We should fix some test cases in a follow-up issue. Like spaces in file path should not be a big problem. Now all tests pass, but the branch needs merging with trunk.

shvachko commented 9 years ago

So for the upgrade I reverted issue #132 from trunk, reverted Issue94 branch back to pre #132 commit. Then merged branch Issue94 into trunk, then merged issue-122 branch on top of it. The tests are passing. Will commit. Next will be

shvachko commented 9 years ago

Committed. Thank you Andrey and Milan.