[Bug][Worker] Sqoop task create file permission denied

EtachGu commented 3 years ago

For better global communication, Please describe it in English. If you feel the description in English is not clear, then you can append description in Chinese(just for Mandarin(CN)), thx! Describe the bug A clear and concise description of what the bug is.

A sqoop task create file permission denied

To Reproduce log file, mysql url has been desensitization

[INFO] 2021-06-28 20:06:42.543  - [taskAppId=TASK-3-66-67]:[115] - create dir success /tmp/dolphinscheduler/exec/process/1/3/66/67
[INFO] 2021-06-28 20:06:42.599  - [taskAppId=TASK-3-66-67]:[115] - create linux os user : tenet1
[INFO] 2021-06-28 20:06:42.600  - [taskAppId=TASK-3-66-67]:[115] - execute cmd : sudo useradd -g root
 tenet1
[INFO] 2021-06-28 20:06:42.761  - [taskAppId=TASK-3-66-67]:[115] - create user name success tenet1
[INFO] 2021-06-28 20:06:42.885  - [taskAppId=TASK-3-66-67]:[57] - sqoop task params {"customShell":"/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin/sqoop import-all-tables \\\n       --connect \"jdbc:mysql://xxx:3306/record?allowMultiQueries=true&useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=GMT%2B8&allowPublicKeyRetrieval=true\" \\\n       --username root \\\n       --password xxx \\\n       --hive-import \\\n       --hive-database ods_record2 \\\n       --hive-overwrite \\\n       --create-hive-table \\\n       -m 5","jobType":"CUSTOM","localParams":[]}
[INFO] 2021-06-28 20:06:42.903  - [taskAppId=TASK-3-66-67]:[84] - sqoop script: /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin/sqoop import-all-tables \
       --connect "jdbc:mysql://xxx:3306/platform_record?allowMultiQueries=true&useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=GMT%2B8&allowPublicKeyRetrieval=true" \
       --username root \
       --password xxx \
       --hive-import \
       --hive-database ods_record2 \
       --hive-overwrite \
       --create-hive-table \
       -m 5
[INFO] 2021-06-28 20:06:42.905  - [taskAppId=TASK-3-66-67]:[87] - tenantCode user:tenet1, task dir:3_66_67
[INFO] 2021-06-28 20:06:42.909  - [taskAppId=TASK-3-66-67]:[92] - create command file:/tmp/dolphinscheduler/exec/process/1/3/66/67/3_66_67.command
[INFO] 2021-06-28 20:06:42.910  - [taskAppId=TASK-3-66-67]:[111] - command : #!/bin/sh
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
source /opt/apache-dolphinscheduler-1.3.6-bin/conf/env/dolphinscheduler_env.sh
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin/sqoop import-all-tables \
       --connect "jdbc:mysql://xxx:3306/platform_record?allowMultiQueries=true&useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=GMT%2B8&allowPublicKeyRetrieval=true" \
       --username root \
       --password xxx \
       --hive-import \
       --hive-database ods_record2 \
       --hive-overwrite \
       --create-hive-table \
       -m 5
[INFO] 2021-06-28 20:06:42.920  - [taskAppId=TASK-3-66-67]:[327] - task run command:
sudo -u tenet1 sh /tmp/dolphinscheduler/exec/process/1/3/66/67/3_66_67.command
[INFO] 2021-06-28 20:06:42.926  - [taskAppId=TASK-3-66-67]:[208] - process start, process id is: 153
[INFO] 2021-06-28 20:06:43.934  - [taskAppId=TASK-3-66-67]:[129] -  -> Warning: /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
    Please set $ACCUMULO_HOME to the root of your Accumulo installation.
    WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
[INFO] 2021-06-28 20:06:44.937  - [taskAppId=TASK-3-66-67]:[129] -  -> WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
    2021-06-28 20:06:44,872 INFO  [main] sqoop.Sqoop (Sqoop.java:<init>(94)) - Running Sqoop version: 1.4.7-cdh6.3.2
[INFO] 2021-06-28 20:06:45.939  - [taskAppId=TASK-3-66-67]:[129] -  -> 2021-06-28 20:06:45,022 WARN  [main] tool.BaseSqoopTool (BaseSqoopTool.java:applyCredentialsOptions(1179)) - Setting your password on the command-line is insecure. Consider using -P instead.
    2021-06-28 20:06:45,031 INFO  [main] tool.BaseSqoopTool (BaseSqoopTool.java:validateOutputFormatOptions(1550)) - Using Hive-specific delimiters for output. You can override
    2021-06-28 20:06:45,031 INFO  [main] tool.BaseSqoopTool (BaseSqoopTool.java:validateOutputFormatOptions(1551)) - delimiters with --fields-terminated-by, etc.
    2021-06-28 20:06:45,163 INFO  [main] manager.MySQLManager (MySQLManager.java:initOptionDefaults(68)) - Preparing to use a MySQL streaming resultset.
    Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
    2021-06-28 20:06:45,701 INFO  [main] tool.CodeGenTool (CodeGenTool.java:generateORM(91)) - Beginning code generation
    2021-06-28 20:06:45,733 INFO  [main] manager.SqlManager (SqlManager.java:execute(775)) - Executing SQL statement: SELECT t.* FROM `api_time_statistics` AS t LIMIT 1
    2021-06-28 20:06:45,785 INFO  [main] manager.SqlManager (SqlManager.java:execute(775)) - Executing SQL statement: SELECT t.* FROM `api_time_statistics` AS t LIMIT 1
    2021-06-28 20:06:45,799 INFO  [main] orm.CompilationManager (CompilationManager.java:findHadoopJars(100)) - HADOOP_MAPRED_HOME is /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce
[INFO] 2021-06-28 20:06:47.943  - [taskAppId=TASK-3-66-67]:[129] -  -> 2021-06-28 20:06:47,874 ERROR [main] orm.CompilationManager (CompilationManager.java:compile(255)) - Could not rename /tmp/sqoop-tenet1/compile/c7a2c509fe2a239a24844297b4d03f00/api_time_statistics.java to /tmp/dolphinscheduler/exec/process/1/3/66/67/./api_time_statistics.java. Error: /tmp/dolphinscheduler/exec/process/1/3/66/67/./api_time_statistics.java (Permission denied)
    2021-06-28 20:06:47,874 INFO  [main] orm.CompilationManager (CompilationManager.java:jar(363)) - Writing jar file: /tmp/sqoop-tenet1/compile/c7a2c509fe2a239a24844297b4d03f00/api_time_statistics.jar
    2021-06-28 20:06:47,906 WARN  [main] manager.MySQLManager (MySQLManager.java:importTable(113)) - It looks like you are importing from mysql.
    2021-06-28 20:06:47,906 WARN  [main] manager.MySQLManager (MySQLManager.java:importTable(114)) - This transfer can be faster! Use the --direct
    2021-06-28 20:06:47,906 WARN  [main] manager.MySQLManager (MySQLManager.java:importTable(115)) - option to exercise a MySQL-specific fast path.
    2021-06-28 20:06:47,907 INFO  [main] manager.MySQLManager (MySQLManager.java:checkDateTimeBehavior(204)) - Setting zero DATETIME behavior to convertToNull (mysql)
    2021-06-28 20:06:47,919 INFO  [main] mapreduce.ImportJobBase (ImportJobBase.java:runImport(241)) - Beginning import of api_time_statistics
[INFO] 2021-06-28 20:06:48.944  - [taskAppId=TASK-3-66-67]:[129] -  -> 2021-06-28 20:06:48,312 INFO  [main] Configuration.deprecation (Configuration.java:logDeprecation(1381)) - mapred.jar is deprecated. Instead, use mapreduce.job.jar
    2021-06-28 20:06:48,627 WARN  [main] impl.MetricsConfig (MetricsConfig.java:loadFirst(133)) - Cannot locate configuration: tried hadoop-metrics2-jobtracker.properties,hadoop-metrics2.properties
    2021-06-28 20:06:48,717 INFO  [main] impl.MetricsSystemImpl (MetricsSystemImpl.java:startTimer(374)) - Scheduled Metric snapshot period at 10 second(s).
    2021-06-28 20:06:48,720 INFO  [main] impl.MetricsSystemImpl (MetricsSystemImpl.java:start(191)) - JobTracker metrics system started
    2021-06-28 20:06:48,866 INFO  [main] db.DBInputFormat (DBInputFormat.java:setTxIsolation(192)) - Using read commited transaction isolation
    2021-06-28 20:06:48,868 INFO  [main] db.DataDrivenDBInputFormat (DataDrivenDBInputFormat.java:getSplits(167)) - BoundingValsQuery: SELECT MIN(`id`), MAX(`id`) FROM `api_time_statistics`
    2021-06-28 20:06:48,873 INFO  [main] db.IntegerSplitter (IntegerSplitter.java:split(145)) - Split size: 23; Num splits: 5 from: 1 to: 117
    2021-06-28 20:06:48,894 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(202)) - number of splits:5
[INFO] 2021-06-28 20:06:49.946  - [taskAppId=TASK-3-66-67]:[129] -  -> 2021-06-28 20:06:49,033 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(298)) - Submitting tokens for job: job_local2104280907_0001
    2021-06-28 20:06:49,035 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(299)) - Executing with tokens: []
    2021-06-28 20:06:49,307 INFO  [main] mapred.LocalDistributedCacheManager (LocalDistributedCacheManager.java:symlink(201)) - Creating symlink: /tmp/hadoop-tenet1/mapred/local/1624882009141/libjars <- /tmp/dolphinscheduler/exec/process/1/3/66/67/libjars/*
    2021-06-28 20:06:49,340 WARN  [main] fs.FileUtil (FileUtil.java:symLink(1054)) - Command 'ln -s /tmp/hadoop-tenet1/mapred/local/1624882009141/libjars /tmp/dolphinscheduler/exec/process/1/3/66/67/libjars/*' failed 1 with: ln: failed to create symbolic link '/tmp/dolphinscheduler/exec/process/1/3/66/67/libjars/*': No such file or directory

    2021-06-28 20:06:49,341 WARN  [main] mapred.LocalDistributedCacheManager (LocalDistributedCacheManager.java:symlink(203)) - Failed to create symlink: /tmp/hadoop-tenet1/mapred/local/1624882009141/libjars <- /tmp/dolphinscheduler/exec/process/1/3/66/67/libjars/*
    2021-06-28 20:06:49,341 INFO  [main] mapred.LocalDistributedCacheManager (LocalDistributedCacheManager.java:setup(165)) - Localized file:/tmp/hadoop/mapred/staging/tenet12104280907/.staging/job_local2104280907_0001/libjars as file:/tmp/hadoop-tenet1/mapred/local/1624882009141/libjars
    2021-06-28 20:06:49,483 INFO  [main] mapreduce.Job (Job.java:submit(1574)) - The url to track the job: http://localhost:8080/
    2021-06-28 20:06:49,484 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1619)) - Running job: job_local2104280907_0001
    2021-06-28 20:06:49,491 INFO  [Thread-6] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(501)) - OutputCommitter set in config null
    2021-06-28 20:06:49,506 INFO  [Thread-6] output.FileOutputCommitter (FileOutputCommitter.java:<init>(129)) - File Output Committer Algorithm version is 2
    2021-06-28 20:06:49,506 INFO  [Thread-6] output.FileOutputCommitter (FileOutputCommitter.java:<init>(144)) - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
    2021-06-28 20:06:49,507 INFO  [Thread-6] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(519)) - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
    2021-06-28 20:06:49,511 ERROR [Thread-6] output.FileOutputCommitter (FileOutputCommitter.java:setupJob(344)) - Mkdirs failed to create file:/tmp/dolphinscheduler/exec/process/1/3/66/67/api_time_statistics/_temporary/0
    2021-06-28 20:06:49,561 INFO  [Thread-6] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(478)) - Waiting for map tasks
    2021-06-28 20:06:49,564 INFO  [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(252)) - Starting task: attempt_local2104280907_0001_m_000000_0
    2021-06-28 20:06:49,602 INFO  [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(129)) - File Output Committer Algorithm version is 2
    2021-06-28 20:06:49,602 INFO  [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(144)) - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
    2021-06-28 20:06:49,637 INFO  [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(625)) -  Using ResourceCalculatorProcessTree : [ ]
    2021-06-28 20:06:49,648 INFO  [LocalJobRunner Map Task Executor #0] db.DBInputFormat (DBInputFormat.java:setTxIsolation(192)) - Using read commited transaction isolation
    2021-06-28 20:06:49,657 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(768)) - Processing split: `id` >= 1 AND `id` < 25
    2021-06-28 20:06:49,673 INFO  [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(252)) - Starting task: attempt_local2104280907_0001_m_000001_0
    2021-06-28 20:06:49,677 INFO  [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(129)) - File Output Committer Algorithm version is 2
    2021-06-28 20:06:49,677 INFO  [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(144)) - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
    2021-06-28 20:06:49,678 INFO  [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(625)) -  Using ResourceCalculatorProcessTree : [ ]
    2021-06-28 20:06:49,684 INFO  [LocalJobRunner Map Task Executor #0] db.DBInputFormat (DBInputFormat.java:setTxIsolation(192)) - Using read commited transaction isolation
    2021-06-28 20:06:49,687 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(768)) - Processing split: `id` >= 25 AND `id` < 48
    2021-06-28 20:06:49,689 INFO  [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(252)) - Starting task: attempt_local2104280907_0001_m_000002_0
    2021-06-28 20:06:49,693 INFO  [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(129)) - File Output Committer Algorithm version is 2
    2021-06-28 20:06:49,694 INFO  [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(144)) - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
    2021-06-28 20:06:49,694 INFO  [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(625)) -  Using ResourceCalculatorProcessTree : [ ]
    2021-06-28 20:06:49,702 INFO  [LocalJobRunner Map Task Executor #0] db.DBInputFormat (DBInputFormat.java:setTxIsolation(192)) - Using read commited transaction isolation
    2021-06-28 20:06:49,704 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(768)) - Processing split: `id` >= 48 AND `id` < 71
    2021-06-28 20:06:49,706 INFO  [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(252)) - Starting task: attempt_local2104280907_0001_m_000003_0
    2021-06-28 20:06:49,708 INFO  [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(129)) - File Output Committer Algorithm version is 2
    2021-06-28 20:06:49,708 INFO  [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(144)) - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
    2021-06-28 20:06:49,709 INFO  [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(625)) -  Using ResourceCalculatorProcessTree : [ ]
    2021-06-28 20:06:49,716 INFO  [LocalJobRunner Map Task Executor #0] db.DBInputFormat (DBInputFormat.java:setTxIsolation(192)) - Using read commited transaction isolation
    2021-06-28 20:06:49,719 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(768)) - Processing split: `id` >= 71 AND `id` < 94
    2021-06-28 20:06:49,723 INFO  [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(252)) - Starting task: attempt_local2104280907_0001_m_000004_0
    2021-06-28 20:06:49,726 INFO  [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(129)) - File Output Committer Algorithm version is 2
    2021-06-28 20:06:49,726 INFO  [LocalJobRunner Map Task Executor #0] output.FileOutputCommitter (FileOutputCommitter.java:<init>(144)) - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
    2021-06-28 20:06:49,726 INFO  [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(625)) -  Using ResourceCalculatorProcessTree : [ ]
    2021-06-28 20:06:49,733 INFO  [LocalJobRunner Map Task Executor #0] db.DBInputFormat (DBInputFormat.java:setTxIsolation(192)) - Using read commited transaction isolation
    2021-06-28 20:06:49,735 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(768)) - Processing split: `id` >= 94 AND `id` <= 117
    2021-06-28 20:06:49,738 INFO  [Thread-6] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(486)) - map task executor complete.
    2021-06-28 20:06:49,739 WARN  [Thread-6] mapred.LocalJobRunner (LocalJobRunner.java:run(590)) - job_local2104280907_0001
    java.lang.Exception: java.io.IOException: Mkdirs failed to create file:/tmp/dolphinscheduler/exec/process/1/3/66/67/api_time_statistics/_temporary/0/_temporary/attempt_local2104280907_0001_m_000000_0 (exists=false, cwd=file:/tmp/dolphinscheduler/exec/process/1/3/66/67)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
    Caused by: java.io.IOException: Mkdirs failed to create file:/tmp/dolphinscheduler/exec/process/1/3/66/67/api_time_statistics/_temporary/0/_temporary/attempt_local2104280907_0001_m_000000_0 (exists=false, cwd=file:/tmp/dolphinscheduler/exec/process/1/3/66/67)
        at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:458)
        at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:443)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1052)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1032)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:921)
        at org.apache.sqoop.mapreduce.RawKeyTextOutputFormat.getFSDataOutputStream(RawKeyTextOutputFormat.java:44)
        at org.apache.sqoop.mapreduce.RawKeyTextOutputFormat.getOutputStream(RawKeyTextOutputFormat.java:63)
        at org.apache.sqoop.mapreduce.RawKeyTextOutputFormat.getRecordWriter(RawKeyTextOutputFormat.java:74)
        at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:659)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:779)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
        at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[INFO] 2021-06-28 20:06:50.605  - [taskAppId=TASK-3-66-67]:[217] - process has exited, execute path:/tmp/dolphinscheduler/exec/process/1/3/66/67, processId:153 ,exitStatusCode:0
[INFO] 2021-06-28 20:06:50.949  - [taskAppId=TASK-3-66-67]:[129] -  -> 2021-06-28 20:06:50,493 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1640)) - Job job_local2104280907_0001 running in uber mode : false
    2021-06-28 20:06:50,496 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1647)) -  map 0% reduce 0%
    2021-06-28 20:06:50,499 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1660)) - Job job_local2104280907_0001 failed with state FAILED due to: NA
    2021-06-28 20:06:50,519 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1665)) - Counters: 0
    2021-06-28 20:06:50,528 WARN  [main] mapreduce.Counters (AbstractCounters.java:getGroup(235)) - Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
    2021-06-28 20:06:50,530 INFO  [main] mapreduce.ImportJobBase (ImportJobBase.java:runJob(190)) - Transferred 0 bytes in 1.9577 seconds (0 bytes/sec)
    2021-06-28 20:06:50,531 WARN  [main] mapreduce.Counters (AbstractCounters.java:getGroup(235)) - Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
    2021-06-28 20:06:50,531 INFO  [main] mapreduce.ImportJobBase (ImportJobBase.java:runJob(192)) - Retrieved 0 records.
    2021-06-28 20:06:50,533 ERROR [main] tool.ImportAllTablesTool (ImportAllTablesTool.java:run(111)) - Error during import: Import job failed!

Expected behavior It should work successed

Which version of Dolphin Scheduler: -[1.3.6]

Additional context

I tried to add umask 0002 in /etc/bash.bashrc, the root user umask -p is 0002, but the permission of directory created by taskAppId=TASK-3-66-67 still drwxr-xr-x

which cause above error, please tell how to solve this problem，Thank you in advance

Requirement or improvement

The process directory should created by tenet， the owner should be the tenet who exec command.

github-actions[bot] commented 3 years ago

Hi:

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/zh-cn/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.

EtachGu commented 3 years ago

有没有人看一下，要不用中文说一遍，Sqoop任务出错，提示无权限修改创建文件或修改文件。定位原因： FileUtils.java

/**
     * create directory and user
     * @param execLocalPath execute local path
     * @param userName user name
     * @throws IOException errors
     */
    public static void createWorkDirAndUserIfAbsent(String execLocalPath, String userName) throws IOException {
        //if work dir exists, first delete
        File execLocalPathFile = new File(execLocalPath);

        if (execLocalPathFile.exists()) {
            org.apache.commons.io.FileUtils.forceDelete(execLocalPathFile);
        }

        //create work dir
        org.apache.commons.io.FileUtils.forceMkdir(execLocalPathFile);  // 这个地方创建的文件夹默认权限  drwxr-xr-x
        String mkdirLog = "create dir success " + execLocalPath;
        LoggerUtils.logInfo(Optional.ofNullable(logger), mkdirLog);
        LoggerUtils.logInfo(Optional.ofNullable(taskLoggerThreadLocal.get()), mkdirLog);

        //if not exists this user,then create
        OSUtils.taskLoggerThreadLocal.set(taskLoggerThreadLocal.get());
        try {
            if (!OSUtils.getUserList().contains(userName)) {
                boolean isSuccessCreateUser = OSUtils.createUser(userName);

                String infoLog;
                if (isSuccessCreateUser) {
                    infoLog = String.format("create user name success %s", userName);
                } else {
                    infoLog = String.format("create user name fail %s", userName);
                }
                LoggerUtils.logInfo(Optional.ofNullable(logger), infoLog);
                LoggerUtils.logInfo(Optional.ofNullable(taskLoggerThreadLocal.get()), infoLog);
            }
        } catch (Throwable e) {
            LoggerUtils.logError(Optional.ofNullable(logger), e);
            LoggerUtils.logError(Optional.ofNullable(taskLoggerThreadLocal.get()), e);
        }
        OSUtils.taskLoggerThreadLocal.remove();
    }

EtachGu commented 3 years ago

Finally， I solved this problem，we must change the supervisor.ini configuration of Worker server， add umask=002

[program:worker]
command=%(ENV_DOLPHINSCHEDULER_BIN)s/dolphinscheduler-daemon.sh start worker-server
directory=%(ENV_DOLPHINSCHEDULER_HOME)s
priority=999
autostart=%(ENV_WORKER_START_ENABLED)s
autorestart=true
startsecs=10
stopwaitsecs=3
exitcodes=0
stopasgroup=true
killasgroup=true
redirect_stderr=true
stdout_logfile=/dev/fd/1
stdout_logfile_maxbytes=0
umask=002

and it works, GOOD

apache / dolphinscheduler

[Bug][Worker] Sqoop task create file permission denied #5709