search

ceph / cephfs-hadoop

cephfs-hadoop
GNU Lesser General Public License v2.1
57 stars 47 forks source link

The config of "ceph.root.dir" seems not work #35

Open dengquan opened 7 years ago

dengquan commented 7 years ago

I think if config "ceph.root.dir", hadoop should use it as root dir. For example, when config <property> <name>fs.default.name</name> <value>ceph://192.0.0.1:6789/</value> </property> <property> <name>fs.defaultFS</name> <value>ceph://192.0.0.1:6789/</value> </property> <property> <name>ceph.root.dir</name> <value>/hadoop-dir</value> </property>

I think when visit path "ceph://192.0.0.1:6789:/123",it should enter /hadoop-dir/123 on cephfs. But it seems not work like this. When we run hbase on cephfs, if config the ceph.root.dir not use "/", it will throw some exception. Is it a bug or I misunderstand it ？

gregsfortytwo commented 7 years ago

This sounds like a bug, but nobody's actively maintaining the Hadoop plugin right now. :(

This is probably just in the config wrapping so you may be able to hunt down the issue without much trouble.

dotnwat commented 7 years ago

The parameter ceph.root.dir is the mount point directory (https://github.com/ceph/ceph/blob/master/src/include/cephfs/libcephfs.h#L241). When you set it to "/" or the default, then things work OK? If you set ceph.root.dir to /hadoop-dir you'll probably need to manually make that directory before mounting the hadoop file system. Can you provide the exception?

dengquan commented 7 years ago

yes, when ceph.root.dir is "/", it works ok. when it is "/hadoop-dir" which I have made the directory before running， the exception when I runhbase master start as follows: 2017-07-27 09:35:37,254 FATAL [pre1-hmaster1:16000.activeMasterManager] master.HMaster: Failed to become active master java.io.IOException: Invalid argument at com.ceph.fs.CephMount.native_ceph_get_pool_id(Native Method) at com.ceph.fs.CephMount.get_pool_id(CephMount.java:1000) at org.apache.hadoop.fs.ceph.CephTalker.get_pool_id(CephTalker.java:321) at org.apache.hadoop.fs.ceph.CephFileSystem.getPoolReplication(CephFileSystem.java:308) at org.apache.hadoop.fs.ceph.CephFileSystem.selectDataPool(CephFileSystem.java:327) at org.apache.hadoop.fs.ceph.CephFileSystem.create(CephFileSystem.java:449) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:928) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:806) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:795) at org.apache.hadoop.hbase.util.FSUtils.setVersion(FSUtils.java:727) at org.apache.hadoop.hbase.util.FSUtils.setVersion(FSUtils.java:705) at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:444) at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:153) at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:128) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:693) at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:189) at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:1803) at java.lang.Thread.run(Thread.java:745) 2017-07-27 09:35:37,255 FATAL [pre1-hmaster1:16000.activeMasterManager] master.HMaster: Unhandled exception. Starting shutdown. java.io.IOException: Invalid argument at com.ceph.fs.CephMount.native_ceph_get_pool_id(Native Method) at com.ceph.fs.CephMount.get_pool_id(CephMount.java:1000) at org.apache.hadoop.fs.ceph.CephTalker.get_pool_id(CephTalker.java:321) at org.apache.hadoop.fs.ceph.CephFileSystem.getPoolReplication(CephFileSystem.java:308) at org.apache.hadoop.fs.ceph.CephFileSystem.selectDataPool(CephFileSystem.java:327) at org.apache.hadoop.fs.ceph.CephFileSystem.create(CephFileSystem.java:449) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:928) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:806) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:795) at org.apache.hadoop.hbase.util.FSUtils.setVersion(FSUtils.java:727) at org.apache.hadoop.hbase.util.FSUtils.setVersion(FSUtils.java:705) at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:444) at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:153) at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:128) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:693) at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:189) at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:1803) at java.lang.Thread.run(Thread.java:745) 2017-07-27 09:35:37,255 INFO [pre1-hmaster1:16000.activeMasterManager] regionserver.HRegionServer: STOPPED: Unhandled exception. Starting shutdown.

dotnwat commented 7 years ago

Since native_ceph_get_pool_id will just lookup the name of a pool, the error seems to suggest that either the pool name is invalid, or the mount context is invalid. The later would be inline with a problem mount on a sub directory, which seemed most likely. Given how much work has been done on file system stuff in the 1-2 years, I wouldn't be surprised if there are some slightly different semantics / return values etc... How badly do you need the sub-dir mounting?

taodd commented 7 years ago

I have just pulled a request to fix this issue