search

flowerfine / scaleph

Open data platform based on Kubernetes. Scaleph supports SeaTunnel、Flink and Doris backended by SeaTunnel on Flink engine、Flink Kubernetes Operator and Doris operator.
https://flowerfine.github.io/scaleph-repress-site/
Apache License 2.0
367 stars 103 forks source link

[Bug] [docker] upload jar on project resource management #172

Closed kalencaya closed 2 years ago

kalencaya commented 2 years ago

Already searched before asking?

Scaleph Version or Branch

dev

What happened

The latest docker image encountered failure when upload jar on project resource management.

Error Exception

192.168.192.1 - - [13/Aug/2022:00:29:05 +0000] "POST /api/datadev/resource/upload HTTP/1.1" 504 569 "http://localhost/scaleph/datadev/project" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"

2022/08/13 00:29:05 [error] 11#11: *1 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.192.1, server: localhost, request: "POST /api/datadev/resource/upload HTTP/1.1", upstream: "http://192.168.192.7:8080/scaleph/api/datadev/resource/upload", host: "localhost", referrer: "http://localhost/scaleph/datadev/project"

Screenshots

image

Are you willing to submit PR?

Code of Conduct

kalencaya commented 2 years ago

Try to replace fs.s3a.enpoint by fs.s3a.proxy.host and fs.s3a.proxy.port, connect timeout exception disappears successfully, which refer https://github.com/minio/minio/issues/10715#issuecomment-915217088.

Unfortunately, new exception encountered:

org.apache.hadoop.fs.s3a.UnknownStoreException: `user/root/1': PUT 0-byte object  on user/root/1: com.amazonaws.services.s3.model.AmazonS3Exception: The specified bucket does not exist (Service: Amazon S3; Status Code: 404; Error Code: NoSuchBucket; Request ID: 170B5AC57C799AF0; S3 Extended Request ID: e0cbcdba-8163-4887-b442-cbfc633116e2; Proxy: minio), S3 Extended Request ID: e0cbcdba-8163-4887-b442-cbfc633116e2:NoSuchBucket: The specified bucket does not exist (Service: Amazon S3; Status Code: 404; Error Code: NoSuchBucket; Request ID: 170B5AC57C799AF0; S3 Extended Request ID: e0cbcdba-8163-4887-b442-cbfc633116e2; Proxy: minio)
kalencaya commented 2 years ago

It is difficult for me to find out the reason why hadoop-aws can't resolve minio container name or throw com.amazonaws.services.s3.model.AmazonS3Exception: The specified bucket does not exist on scaleph-api image.

When I have to fall back to flink FileSystem, strange thing appear: The same maven dependency but flink HadoopConfigLoader work differently:

{"hadoop.security.groups.cache.secs":"300","io.erasurecode.codec.rs.rawcoders":"rs_native,rs_java","hadoop.security.group.mapping.ldap.posix.attr.uid.name":"uidNumber","hadoop.security.group.mapping.ldap.directory.search.timeout":"10000","fs.s3a.committer.threads":"8","ipc.client.low-latency":"false","fs.ftp.host.port":"21","fs.s3a.block.size":"32M","hadoop.security.kms.client.encrypted.key.cache.expiry":"43200000","ipc.client.connection.maxidletime":"10000","hadoop.registry.zk.session.timeout.ms":"60000","tfile.io.chunk.size":"1048576","fs.azure.sas.expiry.period":"90d","fs.automatic.close":"true","fs.s3a.secret.key":"password","ha.health-monitor.sleep-after-disconnect.ms":"1000","io.map.index.interval":"128","hadoop.util.hash.type":"murmur","fs.s3a.committer.staging.abort.pending.uploads":"true","hadoop.security.groups.cache.background.reload.threads":"3","fs.s3a.access-key":"admin","fs.azure.user.agent.prefix":"unknown","hadoop.security.group.mapping.ldap.posix.attr.gid.name":"gidNumber","fs.s3a.path.style.access":"true","fs.s3a.metadatastore.authoritative":"false","fs.AbstractFileSystem.file.impl":"org.apache.hadoop.fs.local.LocalFs","net.topology.script.number.args":"100","hadoop.http.authentication.token.validity":"36000","ha.failover-controller.graceful-fence.rpc-timeout.ms":"5000","hadoop.security.group.mapping":"org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback","hadoop.security.groups.cache.warn.after.ms":"5000","io.serializations":"org.apache.hadoop.io.serializer.WritableSerialization, org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization, org.apache.hadoop.io.serializer.avro.AvroReflectSerialization","fs.s3a.access.key":"admin","hadoop.security.crypto.buffer.size":"8192","fs.ftp.transfer.mode":"BLOCK_TRANSFER_MODE","hadoop.http.cross-origin.allowed-methods":"GET,POST,HEAD","hadoop.registry.zk.retry.interval.ms":"1000","hadoop.zk.acl":"world:anyone:rwcda","hadoop.registry.secure":"false","fs.wasb.impl":"org.apache.hadoop.fs.azure.NativeAzureFileSystem","hadoop.kerberos.kinit.command":"kinit","fs.trash.interval":"0","fs.s3a.multiobjectdelete.enable":"true","ipc.client.connect.max.retries.on.timeouts":"45","fs.viewfs.rename.strategy":"SAME_MOUNTPOINT","seq.io.sort.factor":"100","fs.client.resolve.topology.enabled":"false","adl.feature.ownerandgroup.enableupn":"false","fs.s3a.fast.upload.buffer":"disk","seq.io.sort.mb":"100","nfs.exports.allowed.hosts":"* rw","ha.health-monitor.connect-retry-interval.ms":"1000","hadoop.security.instrumentation.requires.admin":"false","hadoop.registry.zk.retry.ceiling.ms":"60000","io.file.buffer.size":"4096","io.mapfile.bloom.size":"1048576","hadoop.security.kms.client.authentication.retry-count":"1","fs.ftp.data.connection.mode":"ACTIVE_LOCAL_DATA_CONNECTION_MODE","ipc.client.connect.retry.interval":"1000","fs.s3a.committer.staging.conflict-mode":"fail","ha.failover-controller.graceful-fence.connection.retries":"1","fs.swift.impl":"org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem","fs.s3a.s3guard.ddb.table.capacity.write":"100","hadoop.security.key.default.cipher":"AES/CTR/NoPadding","fs.AbstractFileSystem.adl.impl":"org.apache.hadoop.fs.adl.Adl","hadoop.registry.zk.connection.timeout.ms":"15000","hadoop.security.java.secure.random.algorithm":"SHA1PRNG","ftp.blocksize":"67108864","file.stream-buffer-size":"4096","fs.s3a.max.total.tasks":"5","fs.azure.secure.mode":"false","ipc.client.idlethreshold":"4000","hadoop.security.group.mapping.ldap.search.group.hierarchy.levels":"0","io.skip.checksum.errors":"false","ftp.stream-buffer-size":"4096","file.blocksize":"67108864","ftp.replication":"3","hadoop.security.authorization":"false","hadoop.http.authentication.simple.anonymous.allowed":"true","hadoop.rpc.socket.factory.class.default":"org.apache.hadoop.net.StandardSocketFactory","file.bytes-per-checksum":"512","fs.har.impl.disable.cache":"true","io.erasurecode.codec.xor.rawcoders":"xor_native,xor_java","fs.AbstractFileSystem.swebhdfs.impl":"org.apache.hadoop.fs.SWebHdfs","hadoop.security.dns.log-slow-lookups.threshold.ms":"1000","hadoop.common.configuration.version":"3.0.0","hadoop.security.authentication":"simple","file.replication":"1","hadoop.security.group.mapping.ldap.conversion.rule":"none","io.mapfile.bloom.error.rate":"0.005","hadoop.security.kms.client.encrypted.key.cache.size":"500","fs.s3a.committer.staging.unique-filenames":"true","fs.s3a.multipart.purge":"false","fs.s3a.connection.establish.timeout":"5000","hadoop.security.group.mapping.ldap.search.attr.member":"member","hadoop.ssl.client.conf":"ssl-client.xml","fs.s3a.multipart.purge.age":"86400","hadoop.security.groups.cache.background.reload":"false","hadoop.security.kms.client.encrypted.key.cache.low-watermark":"0.3f","hadoop.user.group.static.mapping.overrides":"dr.who=;","fs.s3a.retry.throttle.interval":"1000ms","hadoop.security.credential.clear-text-fallback":"true","ipc.maximum.data.length":"67108864","tfile.fs.input.buffer.size":"262144","ha.failover-controller.new-active.rpc-timeout.ms":"60000","io.erasurecode.codec.rs-legacy.rawcoders":"rs-legacy_java","hadoop.ssl.hostname.verifier":"DEFAULT","hadoop.http.authentication.type":"simple","hadoop.system.tags":"YARN,HDFS,NAMENODE,DATANODE,REQUIRED,SECURITY,KERBEROS,PERFORMANCE,CLIENT\n      ,SERVER,DEBUG,DEPRICATED,COMMON,OPTIONAL","net.topology.impl":"org.apache.hadoop.net.NetworkTopology","io.seqfile.compress.blocksize":"1000000","fs.AbstractFileSystem.webhdfs.impl":"org.apache.hadoop.fs.WebHdfs","hadoop.registry.zk.retry.times":"5","ftp.bytes-per-checksum":"512","fs.AbstractFileSystem.ftp.impl":"org.apache.hadoop.fs.ftp.FtpFs","hadoop.workaround.non.threadsafe.getpwuid":"true","ipc.client.fallback-to-simple-auth-allowed":"false","hadoop.rpc.protection":"authentication","fs.permissions.umask-mode":"022","ha.health-monitor.rpc-timeout.ms":"45000","hadoop.http.staticuser.user":"dr.who","fs.s3a.connection.maximum":"15","fs.s3a.list.version":"2","fs.AbstractFileSystem.viewfs.impl":"org.apache.hadoop.fs.viewfs.ViewFs","fs.s3a.paging.maximum":"5000","fs.s3a.s3guard.cli.prune.age":"86400000","fs.ftp.host":"0.0.0.0","fs.adl.oauth2.access.token.provider.type":"ClientCredential","ipc.maximum.response.length":"134217728","hadoop.shell.missing.defaultFs.warning":"false","hadoop.registry.zk.root":"/registry","fs.s3a.impl":"org.apache.hadoop.fs.s3a.S3AFileSystem","hadoop.http.authentication.kerberos.keytab":"${user.home}/hadoop.keytab","fs.s3a.s3guard.ddb.background.sleep":"25","hadoop.jetty.logs.serve.aliases":"true","fs.azure.saskey.usecontainersaskeyforallaccess":"true","hadoop.caller.context.max.size":"128","hadoop.http.cross-origin.max-age":"1800","io.compression.codec.bzip2.library":"system-native","ipc.ping.interval":"60000","net.topology.node.switch.mapping.impl":"org.apache.hadoop.net.ScriptBasedMapping","fs.df.interval":"60000","fs.s3a.assumed.role.session.duration":"30m","fs.s3a.readahead.range":"64K","ha.zookeeper.parent-znode":"/hadoop-ha","hadoop.http.cross-origin.allowed-headers":"X-Requested-With,Content-Type,Accept,Origin","fs.s3a.retry.limit":"${fs.s3a.attempts.maximum}","fs.s3a.attempts.maximum":"20","fs.azure.authorization":"false","io.seqfile.local.dir":"${hadoop.tmp.dir}/io/local","fs.s3a.assumed.role.credentials.provider":"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider","hadoop.security.kms.client.failover.sleep.max.millis":"2000","fs.azure.authorization.caching.enable":"true","fs.wasbs.impl":"org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure","hadoop.ssl.keystores.factory.class":"org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory","hadoop.zk.num-retries":"1000","fs.AbstractFileSystem.hdfs.impl":"org.apache.hadoop.fs.Hdfs","hadoop.security.random.device.file.path":"/dev/urandom","hadoop.http.filter.initializers":"org.apache.hadoop.http.lib.StaticUserWebFilter","ipc.client.rpc-timeout.ms":"0","fs.client.resolve.remote.symlinks":"true","fs.s3a.committer.magic.enabled":"false","fs.s3a.s3guard.ddb.max.retries":"9","hadoop.ssl.enabled.protocols":"TLSv1,SSLv2Hello,TLSv1.1,TLSv1.2","fs.s3a.committer.name":"file","rpc.metrics.quantile.enable":"false","hadoop.ssl.enabled":"false","io.bytes.per.checksum":"512","fs.azure.local.sas.key.mode":"false","ipc.client.kill.max":"10","ipc.server.listen.queue.size":"128","fs.s3a.threads.max":"10","fs.ftp.impl":"org.apache.hadoop.fs.ftp.FTPFileSystem","fs.s3a.threads.keepalivetime":"60","hadoop.caller.context.signature.max.size":"40","fs.s3a.connection.timeout":"200000","hadoop.security.dns.log-slow-lookups.enabled":"false","hadoop.security.group.mapping.ldap.connection.timeout.ms":"60000","file.client-write-packet-size":"65536","fs.s3a.s3guard.ddb.table.capacity.read":"500","hadoop.security.groups.shell.command.timeout":"0s","ha.failover-controller.cli-check.rpc-timeout.ms":"20000","ha.zookeeper.acl":"world:anyone:rwcda","ipc.client.connect.max.retries":"10","ipc.client.ping":"true","hadoop.security.kms.client.timeout":"60","hadoop.tmp.dir":"/tmp/hadoop-${user.name}","fs.s3a.etag.checksum.enabled":"false","hadoop.security.kms.client.failover.sleep.base.millis":"100","fs.s3a.retry.interval":"500ms","hadoop.ssl.require.client.cert":"false","hadoop.security.uid.cache.secs":"14400","hadoop.registry.rm.enabled":"false","fs.trash.checkpoint.interval":"0","hadoop.zk.timeout-ms":"10000","ha.health-monitor.check-interval.ms":"1000","fs.s3a.path-style-access":"true","fs.AbstractFileSystem.s3a.impl":"org.apache.hadoop.fs.s3a.S3A","ipc.client.tcpnodelay":"true","ipc.client.connect.timeout":"20000","fs.s3a.multipart.threshold":"2147483647","io.map.index.skip":"0","hadoop.http.cross-origin.enabled":"false","fs.s3a.committer.staging.tmp.path":"tmp/staging","hadoop.security.group.mapping.providers.combined":"true","fs.AbstractFileSystem.har.impl":"org.apache.hadoop.fs.HarFs","hadoop.kerberos.min.seconds.before.relogin":"60","hadoop.security.kms.client.encrypted.key.cache.num.refill.threads":"2","tfile.fs.output.buffer.size":"262144","fs.du.interval":"600000","hadoop.zk.retry-interval-ms":"1000","hadoop.security.group.mapping.ldap.ssl":"false","fs.s3a.buffer.dir":"${hadoop.tmp.dir}/s3a","fs.defaultFS":"file:///","fs.s3a.multipart.size":"100M","hadoop.security.group.mapping.ldap.search.attr.group.name":"cn","dfs.ha.fencing.ssh.connect-timeout":"30000","fs.s3a.socket.send.buffer":"8192","hadoop.registry.zk.quorum":"localhost:2181","hadoop.http.cross-origin.allowed-origins":"*","hadoop.registry.system.acls":"sasl:yarn@, sasl:mapred@, sasl:hdfs@","hadoop.security.crypto.cipher.suite":"AES/CTR/NoPadding","fs.s3a.fast.upload.active.blocks":"4","hadoop.security.crypto.codec.classes.aes.ctr.nopadding":"org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec, org.apache.hadoop.crypto.JceAesCtrCryptoCodec","fs.s3a.metadatastore.impl":"org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore","hadoop.security.groups.negative-cache.secs":"30","hadoop.registry.jaas.context":"Client","hadoop.ssl.server.conf":"ssl-server.xml","hadoop.security.group.mapping.ldap.search.filter.group":"(objectClass=group)","hadoop.caller.context.enabled":"false","hadoop.http.authentication.kerberos.principal":"HTTP/_HOST@LOCALHOST","hadoop.shell.safely.delete.limit.num.files":"100","fs.s3a.secret-key":"password","fs.s3a.s3guard.ddb.table.create":"false","hadoop.security.group.mapping.ldap.search.filter.user":"(&(objectClass=user)(sAMAccountName={0}))","ftp.client-write-packet-size":"65536","fs.s3a.socket.recv.buffer":"8192","fs.adl.impl":"org.apache.hadoop.fs.adl.AdlFileSystem","hadoop.security.sensitive-config-keys":"\n      secret$\n      password$\n      ssl.keystore.pass$\n      fs.s3.*[Ss]ecret.?[Kk]ey\n      fs.s3a.*.server-side-encryption.key\n      fs.azure.account.key.*\n      credential$\n      oauth.*token$\n      hadoop.security.sensitive-config-keys\n  ","ipc.server.log.slow.rpc":"false","hadoop.security.group.mapping.ldap.read.timeout.ms":"60000","hadoop.http.logs.enabled":"true","ha.zookeeper.session-timeout.ms":"5000","fs.s3a.connection.ssl.enabled":"true","fs.s3a.retry.throttle.limit":"${fs.s3a.attempts.maximum}","hadoop.security.key.default.bitlength":"128","hadoop.http.authentication.signature.secret.file":"${user.home}/hadoop-http-auth-signature-secret","ipc.server.max.connections":"0","fs.s3a.endpoint":"http://127.0.0.1:9000"}

and 

{"hadoop.security.groups.cache.secs":"300","io.erasurecode.codec.rs.rawcoders":"rs_native,rs_java","fs.s3a.select.output.csv.record.delimiter":"\\n","hadoop.security.group.mapping.ldap.posix.attr.uid.name":"uidNumber","hadoop.security.group.mapping.ldap.directory.search.timeout":"10000","fs.s3a.committer.threads":"8","ipc.client.low-latency":"false","fs.s3a.select.input.csv.quote.character":"\"","fs.s3a.block.size":"32M","fs.ftp.host.port":"21","hadoop.security.kms.client.encrypted.key.cache.expiry":"43200000","ipc.client.connection.maxidletime":"10000","fs.AbstractFileSystem.abfs.impl":"org.apache.hadoop.fs.azurebfs.Abfs","hadoop.registry.zk.session.timeout.ms":"60000","tfile.io.chunk.size":"1048576","fs.azure.sas.expiry.period":"90d","fs.automatic.close":"true","ha.health-monitor.sleep-after-disconnect.ms":"1000","fs.s3a.secret.key":"password","io.map.index.interval":"128","hadoop.util.hash.type":"murmur","hadoop.security.groups.cache.background.reload.threads":"3","fs.azure.user.agent.prefix":"unknown","hadoop.security.group.mapping.ldap.posix.attr.gid.name":"gidNumber","fs.s3a.path.style.access":"true","fs.s3a.metadatastore.authoritative":"false","fs.AbstractFileSystem.file.impl":"org.apache.hadoop.fs.local.LocalFs","net.topology.script.number.args":"100","hadoop.http.authentication.token.validity":"36000","ha.failover-controller.graceful-fence.rpc-timeout.ms":"5000","hadoop.security.group.mapping":"org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback","fs.s3a.s3guard.consistency.retry.interval":"2s","hadoop.security.groups.cache.warn.after.ms":"5000","hadoop.http.sni.host.check.enabled":"false","hadoop.security.group.mapping.ldap.num.attempts":"3","io.serializations":"org.apache.hadoop.io.serializer.WritableSerialization, org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization, org.apache.hadoop.io.serializer.avro.AvroReflectSerialization","hadoop.security.auth_to_local.mechanism":"hadoop","fs.viewfs.overload.scheme.target.o3fs.impl":"org.apache.hadoop.fs.ozone.OzoneFileSystem","hadoop.security.crypto.buffer.size":"8192","fs.s3a.access.key":"admin","fs.ftp.transfer.mode":"BLOCK_TRANSFER_MODE","hadoop.http.cross-origin.allowed-methods":"GET,POST,HEAD","fs.AbstractFileSystem.abfss.impl":"org.apache.hadoop.fs.azurebfs.Abfss","hadoop.registry.zk.retry.interval.ms":"1000","fs.s3a.executor.capacity":"16","hadoop.zk.acl":"world:anyone:rwcda","hadoop.registry.secure":"false","fs.s3a.metadatastore.fail.on.write.error":"true","ipc.[port_number].cost-provider.impl":"org.apache.hadoop.ipc.DefaultCostProvider","fs.wasb.impl":"org.apache.hadoop.fs.azure.NativeAzureFileSystem","fs.viewfs.overload.scheme.target.http.impl":"org.apache.hadoop.fs.http.HttpFileSystem","hadoop.kerberos.kinit.command":"kinit","fs.viewfs.overload.scheme.target.abfss.impl":"org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem","hadoop.prometheus.endpoint.enabled":"false","ipc.client.bind.wildcard.addr":"false","fs.trash.interval":"0","fs.s3a.multiobjectdelete.enable":"true","fs.viewfs.overload.scheme.target.gs.impl":"com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS","fs.s3a.change.detection.version.required":"true","ipc.[port_number].decay-scheduler.metrics.top.user.count":"10","ipc.client.connect.max.retries.on.timeouts":"45","fs.viewfs.rename.strategy":"SAME_MOUNTPOINT","ipc.[port_number].weighted-cost.handler":"1","fs.s3a.s3guard.ddb.throttle.retry.interval":"100ms","fs.client.resolve.topology.enabled":"false","seq.io.sort.factor":"100","fs.viewfs.overload.scheme.target.ofs.impl":"org.apache.hadoop.fs.ozone.RootedOzoneFileSystem","fs.s3a.select.output.csv.field.delimiter":",","adl.feature.ownerandgroup.enableupn":"false","fs.s3a.s3guard.consistency.retry.limit":"7","seq.io.sort.mb":"100","fs.s3a.fast.upload.buffer":"disk","nfs.exports.allowed.hosts":"* rw","ha.health-monitor.connect-retry-interval.ms":"1000","hadoop.security.instrumentation.requires.admin":"false","hadoop.registry.zk.retry.ceiling.ms":"60000","io.file.buffer.size":"4096","io.mapfile.bloom.size":"1048576","ipc.client.connect.retry.interval":"1000","hadoop.security.secure.random.impl":"org.apache.hadoop.crypto.random.OpensslSecureRandom","hadoop.security.kms.client.authentication.retry-count":"1","fs.ftp.data.connection.mode":"ACTIVE_LOCAL_DATA_CONNECTION_MODE","fs.viewfs.overload.scheme.target.hdfs.impl":"org.apache.hadoop.hdfs.DistributedFileSystem","fs.s3a.committer.staging.conflict-mode":"append","fs.s3a.s3guard.ddb.table.capacity.write":"0","hadoop.security.key.default.cipher":"AES/CTR/NoPadding","fs.swift.impl":"org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem","ha.failover-controller.graceful-fence.connection.retries":"1","hadoop.registry.zk.connection.timeout.ms":"15000","fs.AbstractFileSystem.adl.impl":"org.apache.hadoop.fs.adl.Adl","fs.AbstractFileSystem.wasb.impl":"org.apache.hadoop.fs.azure.Wasb","fs.s3a.select.input.compression":"none","hadoop.security.java.secure.random.algorithm":"SHA1PRNG","ftp.blocksize":"67108864","file.stream-buffer-size":"4096","fs.s3a.max.total.tasks":"32","ipc.[port_number].identity-provider.impl":"org.apache.hadoop.ipc.UserIdentityProvider","fs.azure.secure.mode":"false","hadoop.security.group.mapping.ldap.search.group.hierarchy.levels":"0","ipc.client.idlethreshold":"4000","fs.abfss.impl":"org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem","io.skip.checksum.errors":"false","ftp.stream-buffer-size":"4096","ipc.[port_number].weighted-cost.response":"1","file.blocksize":"67108864","ftp.replication":"3","fs.s3a.select.output.csv.quote.character":"\"","fs.ftp.timeout":"0","hadoop.security.authorization":"false","hadoop.http.authentication.simple.anonymous.allowed":"true","hadoop.rpc.socket.factory.class.default":"org.apache.hadoop.net.StandardSocketFactory","file.bytes-per-checksum":"512","fs.har.impl.disable.cache":"true","fs.AbstractFileSystem.swebhdfs.impl":"org.apache.hadoop.fs.SWebHdfs","io.erasurecode.codec.xor.rawcoders":"xor_native,xor_java","hadoop.security.dns.log-slow-lookups.threshold.ms":"1000","hadoop.common.configuration.version":"3.0.0","hadoop.security.authentication":"simple","fs.azure.buffer.dir":"${hadoop.tmp.dir}/abfs","ipc.[port_number].callqueue.impl":"java.util.concurrent.LinkedBlockingQueue","file.replication":"1","ipc.server.reuseaddr":"true","hadoop.security.group.mapping.ldap.conversion.rule":"none","fs.s3a.select.output.csv.quote.fields":"always","io.mapfile.bloom.error.rate":"0.005","hadoop.security.kms.client.encrypted.key.cache.size":"500","fs.s3a.committer.staging.unique-filenames":"true","fs.s3a.multipart.purge":"false","fs.getspaceused.jitterMillis":"60000","fs.s3a.connection.establish.timeout":"5000","hadoop.security.group.mapping.ldap.search.attr.member":"member","fs.viewfs.overload.scheme.target.webhdfs.impl":"org.apache.hadoop.hdfs.web.WebHdfsFileSystem","fs.viewfs.overload.scheme.target.https.impl":"org.apache.hadoop.fs.http.HttpsFileSystem","hadoop.ssl.client.conf":"ssl-client.xml","ipc.[port_number].backoff.enable":"false","fs.s3a.select.output.csv.quote.escape.character":"\\\\","fs.s3a.downgrade.syncable.exceptions":"true","fs.s3a.multipart.purge.age":"86400","hadoop.security.groups.cache.background.reload":"false","hadoop.security.kms.client.encrypted.key.cache.low-watermark":"0.3f","hadoop.user.group.static.mapping.overrides":"dr.who=;","hadoop.security.credential.clear-text-fallback":"true","fs.s3a.retry.throttle.interval":"100ms","ipc.maximum.data.length":"134217728","tfile.fs.input.buffer.size":"262144","ha.failover-controller.new-active.rpc-timeout.ms":"60000","io.erasurecode.codec.rs-legacy.rawcoders":"rs-legacy_java","hadoop.ssl.hostname.verifier":"DEFAULT","fs.viewfs.overload.scheme.target.s3a.impl":"org.apache.hadoop.fs.s3a.S3AFileSystem","hadoop.http.authentication.type":"simple","hadoop.system.tags":"YARN,HDFS,NAMENODE,DATANODE,REQUIRED,SECURITY,KERBEROS,PERFORMANCE,CLIENT\n      ,SERVER,DEBUG,DEPRECATED,COMMON,OPTIONAL","net.topology.impl":"org.apache.hadoop.net.NetworkTopology","ipc.[port_number].weighted-cost.lockfree":"1","io.seqfile.compress.blocksize":"1000000","fs.AbstractFileSystem.webhdfs.impl":"org.apache.hadoop.fs.WebHdfs","hadoop.registry.zk.retry.times":"5","ha.failover-controller.active-standby-elector.zk.op.retries":"3","fs.AbstractFileSystem.ftp.impl":"org.apache.hadoop.fs.ftp.FtpFs","ftp.bytes-per-checksum":"512","hadoop.workaround.non.threadsafe.getpwuid":"true","fs.s3a.select.input.csv.header":"none","ipc.client.fallback-to-simple-auth-allowed":"false","ipc.[port_number].scheduler.priority.levels":"4","ipc.[port_number].faircallqueue.multiplexer.weights":"8,4,2,1","hadoop.rpc.protection":"authentication","fs.abfs.impl":"org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem","fs.permissions.umask-mode":"022","ipc.[port_number].decay-scheduler.decay-factor":"0.5","ha.health-monitor.rpc-timeout.ms":"45000","hadoop.http.staticuser.user":"dr.who","fs.s3a.connection.maximum":"96","fs.viewfs.overload.scheme.target.swift.impl":"org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem","fs.s3a.list.version":"2","fs.s3a.paging.maximum":"5000","fs.AbstractFileSystem.viewfs.impl":"org.apache.hadoop.fs.viewfs.ViewFs","fs.s3a.s3guard.cli.prune.age":"86400000","fs.ftp.host":"0.0.0.0","dfs.client.ignore.namenode.default.kms.uri":"false","ipc.maximum.response.length":"134217728","fs.adl.oauth2.access.token.provider.type":"ClientCredential","hadoop.shell.missing.defaultFs.warning":"false","ipc.[port_number].scheduler.impl":"org.apache.hadoop.ipc.DefaultRpcScheduler","hadoop.http.authentication.kerberos.keytab":"${user.home}/hadoop.keytab","fs.s3a.impl":"org.apache.hadoop.fs.s3a.S3AFileSystem","hadoop.registry.zk.root":"/registry","hadoop.jetty.logs.serve.aliases":"true","fs.s3a.s3guard.ddb.background.sleep":"25ms","fs.azure.saskey.usecontainersaskeyforallaccess":"true","hadoop.caller.context.max.size":"128","hadoop.http.cross-origin.max-age":"1800","io.compression.codec.bzip2.library":"system-native","ipc.ping.interval":"60000","net.topology.node.switch.mapping.impl":"org.apache.hadoop.net.ScriptBasedMapping","hadoop.kerberos.keytab.login.autorenewal.enabled":"false","fs.s3a.aws.credentials.provider":"\n    org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider,\n    org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,\n    com.amazonaws.auth.EnvironmentVariableCredentialsProvider,\n    org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider\n  ","fs.df.interval":"60000","hadoop.metrics.jvm.use-thread-mxbean":"false","fs.s3a.select.input.csv.record.delimiter":"\\n","fs.s3a.assumed.role.session.duration":"30m","fs.viewfs.overload.scheme.target.oss.impl":"org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem","fs.s3a.readahead.range":"64K","fs.s3a.metadatastore.metadata.ttl":"15m","fs.s3a.accesspoint.required":"false","ha.zookeeper.parent-znode":"/hadoop-ha","hadoop.http.cross-origin.allowed-headers":"X-Requested-With,Content-Type,Accept,Origin","fs.s3a.retry.limit":"7","fs.s3a.attempts.maximum":"20","fs.azure.authorization":"false","io.seqfile.local.dir":"${hadoop.tmp.dir}/io/local","fs.viewfs.overload.scheme.target.wasb.impl":"org.apache.hadoop.fs.azure.NativeAzureFileSystem","fs.s3a.assumed.role.credentials.provider":"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider","fs.azure.authorization.caching.enable":"true","hadoop.security.kms.client.failover.sleep.max.millis":"2000","ha.health-monitor.rpc.connect.max.retries":"1","fs.s3a.committer.abort.pending.uploads":"true","fs.wasbs.impl":"org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure","hadoop.ssl.keystores.factory.class":"org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory","hadoop.zk.num-retries":"1000","fs.AbstractFileSystem.hdfs.impl":"org.apache.hadoop.fs.Hdfs","hadoop.security.random.device.file.path":"/dev/urandom","hadoop.http.filter.initializers":"org.apache.hadoop.http.lib.StaticUserWebFilter","fs.s3a.s3guard.ddb.table.sse.enabled":"false","ipc.[port_number].decay-scheduler.period-ms":"5000","ipc.client.rpc-timeout.ms":"0","fs.client.resolve.remote.symlinks":"true","ipc.[port_number].decay-scheduler.thresholds":"13,25,50","fs.AbstractFileSystem.wasbs.impl":"org.apache.hadoop.fs.azure.Wasbs","fs.s3a.committer.magic.enabled":"true","fs.s3a.s3guard.ddb.max.retries":"9","hadoop.ssl.enabled.protocols":"TLSv1.2","fs.s3a.committer.name":"file","rpc.metrics.quantile.enable":"false","io.bytes.per.checksum":"512","fs.azure.local.sas.key.mode":"false","ipc.client.kill.max":"10","fs.s3a.threads.max":"64","ipc.server.listen.queue.size":"256","fs.s3a.select.input.csv.comment.marker":"#","rpc.metrics.timeunit":"MILLISECONDS","adl.http.timeout":"-1","fs.ftp.impl":"org.apache.hadoop.fs.ftp.FTPFileSystem","fs.s3a.connection.timeout":"200000","fs.s3a.threads.keepalivetime":"60","hadoop.caller.context.signature.max.size":"40","hadoop.security.dns.log-slow-lookups.enabled":"false","hadoop.security.group.mapping.ldap.connection.timeout.ms":"60000","hadoop.security.groups.shell.command.timeout":"0s","fs.s3a.s3guard.ddb.table.capacity.read":"0","file.client-write-packet-size":"65536","ha.failover-controller.cli-check.rpc-timeout.ms":"20000","ha.zookeeper.acl":"world:anyone:rwcda","ipc.client.connect.max.retries":"10","ipc.client.ping":"true","fs.s3a.change.detection.source":"etag","hadoop.security.kms.client.timeout":"60","hadoop.tmp.dir":"/tmp/hadoop-${user.name}","ipc.server.purge.interval":"15","fs.s3a.etag.checksum.enabled":"false","hadoop.security.kms.client.failover.sleep.base.millis":"100","fs.s3a.retry.interval":"500ms","ipc.[port_number].decay-scheduler.backoff.responsetime.thresholds":"10s,20s,30s,40s","hadoop.http.idle_timeout.ms":"60000","hadoop.ssl.require.client.cert":"false","hadoop.security.uid.cache.secs":"14400","hadoop.service.shutdown.timeout":"30s","fs.s3a.select.errors.include.sql":"false","fs.viewfs.overload.scheme.target.file.impl":"org.apache.hadoop.fs.LocalFileSystem","fs.trash.checkpoint.interval":"0","hadoop.zk.timeout-ms":"10000","fs.viewfs.overload.scheme.target.abfs.impl":"org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem","fs.s3a.select.input.csv.quote.escape.character":"\\\\","ha.health-monitor.check-interval.ms":"1000","fs.AbstractFileSystem.s3a.impl":"org.apache.hadoop.fs.s3a.S3A","hadoop.tags.system":"YARN,HDFS,NAMENODE,DATANODE,REQUIRED,SECURITY,KERBEROS,PERFORMANCE,CLIENT\n      ,SERVER,DEBUG,DEPRECATED,COMMON,OPTIONAL","fs.s3a.path-style-access":"true","ipc.client.tcpnodelay":"true","fs.s3a.select.enabled":"true","ipc.client.connect.timeout":"20000","fs.s3a.multipart.threshold":"128M","ipc.[port_number].weighted-cost.lockexclusive":"100","hadoop.http.cross-origin.enabled":"false","io.map.index.skip":"0","fs.s3a.committer.staging.tmp.path":"tmp/staging","hadoop.security.group.mapping.providers.combined":"true","fs.AbstractFileSystem.har.impl":"org.apache.hadoop.fs.HarFs","hadoop.security.token.service.use_ip":"true","hadoop.security.kms.client.encrypted.key.cache.num.refill.threads":"2","hadoop.kerberos.min.seconds.before.relogin":"60","fs.s3a.ssl.channel.mode":"default_jsse","hadoop.security.group.mapping.ldap.num.attempts.before.failover":"3","tfile.fs.output.buffer.size":"262144","fs.du.interval":"600000","fs.s3a.change.detection.mode":"server","hadoop.security.group.mapping.ldap.ssl":"false","hadoop.zk.retry-interval-ms":"1000","fs.s3a.buffer.dir":"${hadoop.tmp.dir}/s3a","fs.defaultFS":"file:///","fs.s3a.multipart.size":"64M","hadoop.security.group.mapping.ldap.search.attr.group.name":"cn","fs.s3a.socket.send.buffer":"8192","dfs.ha.fencing.ssh.connect-timeout":"30000","hadoop.registry.zk.quorum":"localhost:2181","hadoop.http.cross-origin.allowed-origins":"*","fs.viewfs.overload.scheme.target.swebhdfs.impl":"org.apache.hadoop.hdfs.web.SWebHdfsFileSystem","hadoop.registry.system.acls":"sasl:yarn@, sasl:mapred@, sasl:hdfs@","hadoop.security.crypto.cipher.suite":"AES/CTR/NoPadding","fs.s3a.fast.upload.active.blocks":"4","hadoop.security.crypto.codec.classes.aes.ctr.nopadding":"org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec, org.apache.hadoop.crypto.JceAesCtrCryptoCodec","fs.s3a.metadatastore.impl":"org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore","hadoop.security.groups.negative-cache.secs":"30","hadoop.ssl.server.conf":"ssl-server.xml","hadoop.registry.jaas.context":"Client","hadoop.security.group.mapping.ldap.search.filter.group":"(objectClass=group)","ipc.[port_number].weighted-cost.lockshared":"10","hadoop.http.authentication.kerberos.principal":"HTTP/_HOST@LOCALHOST","hadoop.caller.context.enabled":"false","fs.s3a.connection.request.timeout":"0","hadoop.shell.safely.delete.limit.num.files":"100","fs.s3a.s3guard.ddb.table.create":"false","hadoop.security.group.mapping.ldap.search.filter.user":"(&(objectClass=user)(sAMAccountName={0}))","fs.s3a.select.input.csv.field.delimiter":",","ftp.client-write-packet-size":"65536","fs.s3a.socket.recv.buffer":"8192","fs.adl.impl":"org.apache.hadoop.fs.adl.AdlFileSystem","fs.AbstractFileSystem.gs.impl":"com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS","hadoop.security.sensitive-config-keys":"\n      secret$\n      password$\n      ssl.keystore.pass$\n      fs.s3a.server-side-encryption.key\n      fs.s3a.*.server-side-encryption.key\n      fs.s3a.encryption.algorithm\n      fs.s3a.encryption.key\n      fs.s3a.secret.key\n      fs.s3a.*.secret.key\n      fs.s3a.session.key\n      fs.s3a.*.session.key\n      fs.s3a.session.token\n      fs.s3a.*.session.token\n      fs.azure.account.key.*\n      fs.azure.oauth2.*\n      fs.adl.oauth2.*\n      fs.gs.encryption.*\n      fs.gs.proxy.*\n      fs.gs.auth.*\n      credential$\n      oauth.*secret\n      oauth.*password\n      oauth.*token\n      hadoop.security.sensitive-config-keys\n  ","ipc.server.log.slow.rpc":"false","hadoop.http.logs.enabled":"true","hadoop.security.group.mapping.ldap.read.timeout.ms":"60000","ha.zookeeper.session-timeout.ms":"10000","fs.s3a.connection.ssl.enabled":"true","fs.viewfs.overload.scheme.target.ftp.impl":"org.apache.hadoop.fs.ftp.FTPFileSystem","fs.s3a.retry.throttle.limit":"20","hadoop.security.key.default.bitlength":"128","ipc.[port_number].decay-scheduler.backoff.responsetime.enable":"false","hadoop.http.authentication.signature.secret.file":"${user.home}/hadoop-http-auth-signature-secret","ipc.server.max.connections":"0","hadoop.domainname.resolver.impl":"org.apache.hadoop.net.DNSDomainNameResolver","fs.s3a.endpoint":"http://127.0.0.1:9000"}
kalencaya commented 2 years ago

The confused appearance has leading me to a unknown direction and I have to think from the original problem and find out what happened.

Inside scaleph-ui container, scaleph-ui can't read response from upload api and time out finally, so there must be an exception catched by scaleph-api global exception handler, but I had tryed to find exception log from scaleph-api container and failed. I assume that hadoop-aws didn't response immediately, which resulting in scaleph-ui read response timeout, hadoop-aws may be in wait response or retry status. After wait a long time, scaleph-api output an exception log which explained that hadoop-aws can't resolve scaleph.minio host and raised an exception. Luckily, I got the reason why hadoop-aws not work.

The reason is that I had used a config options supported by flink FileSystem which will be translated to hadoop-aws fs.s3a.path.style.access:

    @Bean
    @ConditionalOnProperty(value = "file-system.type", havingValue = "s3")
    public FileSystem s3FileSystem(S3FileSystemProperties s3FileSystemProperties) throws URISyntaxException, IOException {
        org.apache.hadoop.conf.Configuration conf = HadoopUtil.getHadoopConfiguration(s3FileSystemProperties.getHadoopConfPath());
        conf.set("fs.s3a.endpoint", s3FileSystemProperties.getEndpoint());
        conf.set("fs.s3a.access.key", s3FileSystemProperties.getAccessKey());
        conf.set("fs.s3a.secret.key", s3FileSystemProperties.getSecretKey());
        conf.setBoolean("fs.s3a.path-style-access", true);
        URI uri = new URI(FileSystemType.S3.getSchema() + s3FileSystemProperties.getBucket());
        return FileSystem.get(uri, conf);
    }

When I had changed flink FileSystem to hadoop FileSystem, I didn't rename fs.s3a.path-style-access to fs.s3a.path.style.access and exception raised.