HADOOP-COS

功能说明

Hadoop-COS实现了以腾讯云 COS 作为底层文件系统运行上层计算任务的功能，支持使用Hadoop、Spark以及Tez等处理存储在腾讯云COS对象存储系统上的数据。

使用限制

只适用于 COS V5 版本

使用环境

系统环境

Linux 或 Windows 系统

软件依赖

Hadoop-2.6.0及以上版本

NOTE：

1. 目前hadoop-cos已经正式被Apache Hadoop-3.3.0官方集成：https://hadoop.apache.org/docs/r3.3.0/hadoop-cos/cloud-storage/index.html。
2. 在Apache Hadoop-3.3.0 之前版本或CDH集成Hadoop-cos jar 包后，需要重启NameNode才能加载到jar包。
3. 需要编译具体Hadoop版本的jar包可更改pom文件中hadoop.version进行编译。

安装方法

获取 hadoop-cos 分发包及其依赖

下载地址：hadoop-cos release

安装hadoop-cos

1. 将hadoop-cos-{hadoop.version}-x.x.x.jar和cos_api-bundle-5.x.x.jar 拷贝到 $HADOOP_HOME/share/hadoop/tools/lib下。

NOTE: 根据hadoop的具体版本选择对应的jar包，若release中没有提供匹配版本的jar包，可自行通过修改pom文件中hadoop版本号，重新编译生成。

2. 修改 hadoop_env.sh 在 $HADOOP_HOME/etc/hadoop目录下，进入 hadoop_env.sh，增加如下内容，将 cosn 相关 jar 包加入 Hadoop 环境变量：

for f in $HADOOP_HOME/share/hadoop/tools/lib/*.jar; do
  if [ "$HADOOP_CLASSPATH" ]; then
    export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
  else
    export HADOOP_CLASSPATH=$f
  fi
done

使用方法

HADOOP配置

修改 $HADOOP_HOME/etc/hadoop/core-site.xml，增加 COS 相关用户和实现类信息，例如：

<configuration>

    <property>
        <name>fs.defaultFS</name>
        <value>cosn://examplebucket-1250000000000</value>
    </property>

    <property>
        <name>fs.cosn.credentials.provider</name>
        <value>org.apache.hadoop.fs.auth.SimpleCredentialProvider</value>
        <description>

            This option allows the user to specify how to get the credentials.
            Comma-separated class names of credential provider classes which implement
            com.qcloud.cos.auth.COSCredentialsProvider:

            1.org.apache.hadoop.fs.auth.SessionCredentialProvider: Obtain the secretId and secretKey from the URI:cosn://secretId:secretKey@example-1250000000000/;
            2.org.apache.hadoop.fs.auth.SimpleCredentialProvider: Obtain the secret id and secret key
            from fs.cosn.userinfo.secretId and fs.cosn.userinfo.secretKey in core-site.xml;
            3.org.apache.hadoop.fs.auth.EnvironmentVariableCredentialProvider: Obtain the secret id and secret key
            from system environment variables named COS_SECRET_ID and COS_SECRET_KEY.

            If unspecified, the default order of credential providers is:
            1. org.apache.hadoop.fs.auth.SessionCredentialProvider
            2. org.apache.hadoop.fs.auth.SimpleCredentialProvider
            3. org.apache.hadoop.fs.auth.EnvironmentVariableCredentialProvider
            4. org.apache.hadoop.fs.auth.SessionTokenCredentialProvider
            5. org.apache.hadoop.fs.auth.CVMInstanceCredentialsProvider
            6. org.apache.hadoop.fs.auth.CPMInstanceCredentialsProvider
            7. org.apache.hadoop.fs.auth.EMRInstanceCredentialsProvider
        </description>
    </property>

    <property>
        <name>fs.cosn.userinfo.secretId</name>
        <value>xxxxxxxxxxxxxxxxxxxxxxxxx</value>
        <description>Tencent Cloud Secret Id</description>
    </property>

    <property>
        <name>fs.cosn.userinfo.secretKey</name>
        <value>xxxxxxxxxxxxxxxxxxxxxxxx</value>
        <description>Tencent Cloud Secret Key</description>
    </property>

    <property>
        <name>fs.cosn.bucket.region</name>
        <value>ap-xxx</value>
        <description>The region where the bucket is located</description>
    </property>

    <property>
        <name>fs.cosn.bucket.endpoint_suffix</name>
        <value>cos.ap-xxx.myqcloud.com</value>
        <description>COS endpoint to connect to.
        For public cloud users, it is recommended not to set this option, and only the correct area field is required.</description>
    </property>

    <property>
        <name>fs.cosn.impl</name>
        <value>org.apache.hadoop.fs.CosFileSystem</value>
        <description>The implementation class of the CosN Filesystem</description>
    </property>

    <property>
        <name>fs.AbstractFileSystem.cosn.impl</name>
        <value>org.apache.hadoop.fs.CosN</value>
        <description>The implementation class of the CosN AbstractFileSystem.</description>
    </property>

    <property>
        <name>fs.cosn.tmp.dir</name>
        <value>/tmp/hadoop_cos</value>
        <description>Temporary files will be placed here.</description>
    </property>

    <property>
        <name>fs.cosn.upload.buffer</name>
        <value>mapped_disk</value>
        <description>The type of upload buffer. Available values: non_direct_memory, direct_memory, mapped_disk</description>
    </property>

    <property>
        <name>fs.cosn.upload.buffer.size</name>
        <value>33554432</value>
        <description>The total size of the upload buffer pool. -1 means unlimited.</description>
    </property>

    <property>
        <name>fs.cosn.upload.part.size</name>
        <value>8388608</value>
        <description>The part size for MultipartUpload.
        Considering the COS supports up to 10000 blocks, user should estimate the maximum size of a single file.
        For example, 8MB part size can allow  writing a 78GB single file.</description>
    </property>

    <property>
        <name>fs.cosn.maxRetries</name>
        <value>3</value>
        <description>The maximum number of retries for reading or writing files to
    COS, before we signal failure to the application.</description>
    </property>

    <property>
        <name>fs.cosn.retry.interval.seconds</name>
        <value>3</value>
        <description>The number of seconds to sleep between each COS retry.</description>
    </property>

    <property>
        <name>fs.cosn.read.ahead.block.size</name>
        <value>‭1048576‬</value>
        <description>
            Bytes to read ahead during a seek() before closing and
            re-opening the cosn HTTP connection.
        </description>
    </property>

    <property>
        <name>fs.cosn.read.ahead.queue.size</name>
        <value>8</value>
        <description>The length of the pre-read queue.</description>
    </property>

    <property>
        <name>fs.cosn.customer.domain</name>
        <value></value>
        <description>The customer domain.</description>
    </property>

    <property>
        <name>fs.cosn.server-side-encryption.algorithm</name>
        <value></value>
        <description>The server side encryption algorithm.</description>
    </property>

     <property>
        <name>fs.cosn.server-side-encryption.key</name>
        <value></value>
        <description>The SSE-C server side encryption key.</description>
    </property>

     <property>
        <name>fs.cosn.client-side-encryption.enabled</name>
        <value></value>
        <description>Enable or disable the client encryption function</description>
    </property>

    <property>
        <name>fs.cosn.client-side-encryption.public.key.path</name>
        <value>/xxx/xxx.key</value>
        <description>The absolute path to the public key</description>
    </property>

      <property>
        <name>fs.cosn.client-side-encryption.private.key.path</name>
        <value>/xxx/xxx.key</value>
        <description>The absolute path to the private key</description>
    </property>

</configuration>

配置项说明：

开始使用

命令格式为：hadoop fs -ls -R cosn://examplebucket-1250000000000/<路径>或hadoop fs -ls -R /<路径>(配置了fs.defaultFS选项为 cosn:// 后) ，下例中以名称为 hdfs-test-1252681929 的 bucket 为例，可在其后面加上具体路径。

hadoop fs -ls -R cosn://hdfs-test-1252681929/
-rw-rw-rw-   1 root root       1087 2018-06-11 07:49 cosn://hdfs-test-1252681929/LICENSE
drwxrwxrwx   - root root          0 1970-01-01 00:00 cosn://hdfs-test-1252681929/hdfs
drwxrwxrwx   - root root          0 1970-01-01 00:00 cosn://hdfs-test-1252681929/hdfs/2018
-rw-rw-rw-   1 root root       1087 2018-06-12 03:26 cosn://hdfs-test-1252681929/hdfs/2018/LICENSE
-rw-rw-rw-   1 root root       2386 2018-06-12 03:26 cosn://hdfs-test-1252681929/hdfs/2018/ReadMe
drwxrwxrwx   - root root          0 1970-01-01 00:00 cosn://hdfs-test-1252681929/hdfs/test
-rw-rw-rw-   1 root root       1087 2018-06-11 07:32 cosn://hdfs-test-1252681929/hdfs/test/LICENSE
-rw-rw-rw-   1 root root       2386 2018-06-11 07:29 cosn://hdfs-test-1252681929/hdfs/test/ReadMe

运行 MapReduce 自带的 wordcount

注意： 以下命令中 hadoop-mapreduce-examples-2.7.2.jar 是以 2.7.2 版本为例，如版本不同，请修改成对应的版本号。

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount cosn://examplebucket-1250000000000/mr/input cosn://examplebucket-1250000000000/mr/output3

执行成功会返回统计信息，示例如下：

File System Counters
        COSN: Number of bytes read=72
        COSN: Number of bytes written=40
        COSN: Number of read operations=0
        COSN: Number of large read operations=0
        COSN: Number of write operations=0
        FILE: Number of bytes read=547350
        FILE: Number of bytes written=1155616
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=0
        HDFS: Number of bytes written=0
        HDFS: Number of read operations=0
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=0
    Map-Reduce Framework
        Map input records=5
        Map output records=7
        Map output bytes=59
        Map output materialized bytes=70
        Input split bytes=99
        Combine input records=7
        Combine output records=6
        Reduce input groups=6
        Reduce shuffle bytes=70
        Reduce input records=6
        Reduce output records=6
        Spilled Records=12
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=0
        Total committed heap usage (bytes)=653262848
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters
        Bytes Read=36
    File Output Format Counters
        Bytes Written=40

Bucket独立配置

背景： 跨园区访问不同的bucket，每个bucket的配置项不同，为此支持了一次配置，访问多个bucket的需求。 核心： 单独设置bucket维度的配置优先走独立配置项，没有设置独立配置项的走原配置，原配置没有设置的走默认配置。 例如： fs.cosn.upload.buffer的subkey（trim fs.cosn. prefix)是 upload.buffer. 如果bucket-appid是 test-123456, 那么bucket的独立配置项就是fs.cosn.bucket.test-123456.upload.buffer (格式fs.cosn.bucket.{bucket-appid}.{subkey}) 使用： hadoop fs -ls cosn://test-123456/ 注意：

1) 如果设置了fs.cosn.userinfo.appid则可以通过设置fs.cosn.bucket.{bucket}.{subkey} 使用方式hadoop fs -ls cosn://test/ 2) 如果使用了元数据加速桶，想使用hadoop fs -ls cosn//test/方式访问 除了需要配置fs.cosn.userinfo.appid外需要设置fs.cosn.trsf.fs.ofs.use.short.bucketname为true

简化Bucket独立配置

背景： 由于旧版本OFS JAVA SDK对挂载点格式做了访问限制，只支持bucket-appid方式进行访问， 新版本（1.1.8+）支持bucket方式进行访问，不需要带appid。hadoopcos 8.3.0开始支持 如需cosn://bucket/ posix方式访问元数据加速桶则添加添加配置项： fs.cosn.trsf.fs.ofs.use.short.bucketname true fs.cosn.userinfo.appid 12345678 同理可以简化上一节各Bucket独立配置项通过添加如下配置做到cosn://test/得方式进行访问 fs.cosn.bucket.test.trsf.fs.ofs.use.short.bucketname true fs.cosn.bucket.test.upload.buffer 1024 fs.cosn.bucket.test. *

插件配置优先级说明

插件配置优先级：Bucket独立配置项大于元数据加速桶配置项（fs.cosn.trsf.) 大于默认配置。 插件配置转化顺序： Bucket独立配置项-> 显式配置fs.cosn.trsf. 相关配置则不进行转换-> 没有显式配置fs.cosn.trsf.* 的则会自动转换，例如如下配置项：

1. fs.cosn.userinfo.appid
2. fs.cosn.bucket.region
3. fs.cosn.userinfo.region
4. fs.cosn.server-side-encryption.algorithm
5. fs.cosn.server-side-encryption.context