[Bug] [Seatunnel Web; S3 File] S3 file does not render all the required options in the UI

neehar18 commented 9 months ago

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened

I am trying to deploy a S3 connector (as source) using the UI and the UI does not render all the required options. I am using the following config file to deploy the same connector using the command line and it works fine.

# Defining the runtime environment
env {
  # You can set flink configuration here
  execution.parallelism = 1
  job.mode = "BATCH"
}

source {
  S3File {
    path = "/test/sample.json"
    fs.s3a.endpoint="s3.us-west-2.amazonaws.com"
    fs.s3a.aws.credentials.provider = "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
    access_key = "##########"
    secret_key = "#########"
    bucket = "s3a://s3-dashdb"
    file_format_type = "json"
    schema {
      fields {
        id = int 
        name = string
      }
    }
  }
}

transform {
  # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
    # please go to https://seatunnel.apache.org/docs/category/transform-v2
}

sink {
  Console {}
}

In the screenshot below, you can notice that the UI does not have options for path, file_format_type, schema. Can you please help look into this issue? Thank you.

SeaTunnel Version

Seatunnel engine: 2.3.3 Seatunnel web: 1.0.0

SeaTunnel Config

# Defining the runtime environment
env {
  # You can set flink configuration here
  execution.parallelism = 1
  job.mode = "BATCH"
}

source {
  S3File {
    path = "/test/sample.json"
    fs.s3a.endpoint="s3.us-west-2.amazonaws.com"
    fs.s3a.aws.credentials.provider = "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
    access_key = "##########"
    secret_key = "#########"
    bucket = "s3a://s3-dashdb"
    file_format_type = "json"
    schema {
      fields {
        id = int 
        name = string
      }
    }
  }
}

transform {
  # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
    # please go to https://seatunnel.apache.org/docs/category/transform-v2
}

sink {
  Console {}
}


### Running Command

```shell
Not running any command, deploying from the UI.

Error Exception

Not enough options rendered in the UI.

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

neehar18 commented 9 months ago

@Carl-Zhou-CN are you able to assist with this issue?

Carl-Zhou-CN commented 9 months ago

@raheen1 May refer to https://mp.weixin.qq.com/s/eNWGP_09Oh4pHdoQkmGPzg ，DataSource ust save the connection information and really use it in the task editing phase

neehar18 commented 9 months ago

@Carl-Zhou-CN the link says "No results matched your search."

Carl-Zhou-CN commented 9 months ago

https://mp.weixin.qq.com/s/eNWGP_09Oh4pHdoQkmGPzg

neehar18 commented 9 months ago

Not able to reach the site. But I understand what you mean. So, first, I need to establish a connection and then I will be able to pass the other parameters like path, file_format_type, schema.

But when I try to test the connection, I get the following error and the UI error (Unknown Exception) in the screenshot :

2023-12-05 06:30:19.826 seatunnel 53dabe219ae9 INFO [tr:,sp:] [qtp2121754508-18] [DatasourceClassLoader.loadClass():43] - load class for name : org.apache.hadoop.fs.s3a.S3AFileSystem
2023-12-05 06:30:19.827 seatunnel 53dabe219ae9 INFO [tr:,sp:] [qtp2121754508-18] [DatasourceClassLoader.loadClass():53] - load class from parentClassLoader : org.apache.hadoop.fs.s3a.S3AFileSystem
2023-12-05 06:30:19.828 seatunnel 53dabe219ae9 ERROR [tr:,sp:] [qtp2121754508-18] [DatasourceClassLoader.loadClass():58] - parentClassLoader load class is error : java.lang.ClassNotFoundException: org.apache.hadoop.fs.s3a.S3AFileSystem
    at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
    at org.apache.seatunnel.datasource.classloader.DatasourceClassLoader.loadClass(DatasourceClassLoader.java:51)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:398)
    at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2575)
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2540)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2636)
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3269)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3301)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
    at org.apache.seatunnel.datasource.plugin.s3.S3DatasourceChannel.checkDataSourceConnectivity(S3DatasourceChannel.java:65)
    at org.apache.seatunnel.datasource.AbstractDataSourceClient.checkDataSourceConnectivity(AbstractDataSourceClient.java:117)
    at org.apache.seatunnel.app.service.impl.DatasourceServiceImpl.testDatasourceConnectionAble(DatasourceServiceImpl.java:212)
    at org.apache.seatunnel.app.controller.SeatunnelDatasourceController.testConnect(SeatunnelDatasourceController.java:152)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:517)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:584)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
    at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1631)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.Server.handle(Server.java:516)
    at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
    at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
    at java.base/java.lang.Thread.run(Thread.java:829)

2023-12-05 06:30:19.832 seatunnel 53dabe219ae9 ERROR [tr:,sp:] [qtp2121754508-18] [GlobalExceptionHandler.logError():83] - java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2638)
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3269)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3301)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
    at org.apache.seatunnel.datasource.plugin.s3.S3DatasourceChannel.checkDataSourceConnectivity(S3DatasourceChannel.java:65)
    at org.apache.seatunnel.datasource.AbstractDataSourceClient.checkDataSourceConnectivity(AbstractDataSourceClient.java:117)
    at org.apache.seatunnel.app.service.impl.DatasourceServiceImpl.testDatasourceConnectionAble(DatasourceServiceImpl.java:212)
    at org.apache.seatunnel.app.controller.SeatunnelDatasourceController.testConnect(SeatunnelDatasourceController.java:152)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:517)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:584)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
    at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1631)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.Server.handle(Server.java:516)
    at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
    at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2542)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2636)
    ... 68 common frames omitted

neehar18 commented 9 months ago

But the same config file works when I run from the command line. I think the error is stating that hadoop libraries might be missing, but I have all the libraries in my pom.xml file.

Carl-Zhou-CN commented 9 months ago

Try doing this，metadata and data source jar package also need to copy to/opt/seatunnel/backend/apache - seatunnel - 2.3.3 / lib

neehar18 commented 9 months ago

I already have this jar file in the seatunnel/lib folder

Carl-Zhou-CN commented 9 months ago

Try to put in/opt/seatunnel/web/apache-seatunnel-web-1.0.0- bin/libs

Carl-Zhou-CN commented 9 months ago

There are still many unsolved problems in the web section

neehar18 commented 9 months ago

Okay, I will try this and update you if it works.

neehar18 commented 9 months ago

I tried adding the seatunnel-hadoop3-3.1.4-uber.jar to the apache-seatunnel-web-1.0.0- bin/libs folder. Still getting the same error.

neehar18 commented 9 months ago

Should I build the project again after adding the above library to the libs folder?

neehar18 commented 9 months ago

Or will the changes get reflected in the UI automatically?

neehar18 commented 9 months ago

@Carl-Zhou-CN I also don't see hadoop-aws and aws-java-sdk-bundle jars in the seatunnel-web. I only see them in the seatunnel lib folder. Should I include those two packages as well in the web libs folder as well?

ls seatunnel-web/libs/
HikariCP-4.0.3.jar             jcl-over-slf4j-1.7.25.jar            mybatis-plus-extension-3.5.3.1.jar          spring-beans-5.3.20.jar
apiguardian-api-1.1.2.jar          jcommander-1.81.jar              mybatis-spring-2.0.7.jar                spring-boot-2.6.8.jar
aspectjweaver-1.9.7.jar            jetty-continuation-9.4.46.v20220331.jar  mysql-connector-java-8.0.28.jar         spring-boot-autoconfigure-2.6.8.jar
auto-service-annotations-1.0.1.jar     jetty-http-9.4.46.v20220331.jar      ojdbc8-21.5.0.0.jar                 spring-boot-starter-2.6.8.jar
checker-qual-3.10.0.jar            jetty-io-9.4.46.v20220331.jar        opentest4j-1.2.0.jar                spring-boot-starter-aop-2.6.8.jar
checkpoint-storage-api-2.3.3.jar       jetty-security-9.4.46.v20220331.jar      postgresql-42.4.3.jar               spring-boot-starter-jdbc-2.6.8.jar
checkpoint-storage-local-file-2.3.3.jar    jetty-server-9.4.46.v20220331.jar        protobuf-java-3.11.4.jar                spring-boot-starter-jetty-2.6.8.jar
classmate-1.3.1.jar            jetty-servlet-9.4.46.v20220331.jar       protostuff-api-1.8.0.jar                spring-boot-starter-json-2.6.8.jar
commons-codec-1.13.jar             jetty-servlets-9.4.46.v20220331.jar      protostuff-collectionschema-1.8.0.jar       spring-boot-starter-web-2.6.8.jar
commons-collections4-4.4.jar           jetty-util-9.4.46.v20220331.jar      protostuff-core-1.8.0.jar               spring-context-5.3.20.jar
commons-compress-1.20.jar          jetty-util-ajax-9.4.46.v20220331.jar     protostuff-runtime-1.8.0.jar            spring-core-5.3.20.jar
commons-io-2.11.0.jar              jetty-webapp-9.4.46.v20220331.jar        scala-library-2.11.12.jar               spring-expression-5.3.20.jar
commons-lang3-3.4.jar              jetty-xml-9.4.46.v20220331.jar       seatunnel-api-2.3.3.jar             spring-jcl-5.3.20.jar
config-1.3.3.jar               jjwt-api-0.10.7.jar              seatunnel-app-1.0.0-SNAPSHOT.jar            spring-jdbc-5.3.20.jar
cron-utils-9.1.6.jar               jjwt-impl-0.10.7.jar             seatunnel-common-2.3.3.jar              spring-plugin-core-1.2.0.RELEASE.jar
gson-2.8.6.jar                 jjwt-jackson-0.10.7.jar          seatunnel-config-base-2.3.3.jar         spring-plugin-metadata-1.2.0.RELEASE.jar
guava-19.0.jar                 jsqlparser-4.4.jar               seatunnel-config-shade-2.3.3.jar            spring-tx-5.3.20.jar
h2-2.1.214.jar                 jsr305-3.0.0.jar             seatunnel-core-starter-2.3.3.jar            spring-web-5.3.20.jar
hazelcast-5.1.jar              junit-jupiter-api-5.9.0.jar          seatunnel-datasource-client-1.0.0-SNAPSHOT.jar  spring-webmvc-5.3.20.jar
hibernate-validator-6.2.2.Final.jar    junit-platform-commons-1.9.0.jar     seatunnel-dynamicform-1.0.0-SNAPSHOT.jar        springfox-core-2.6.1.jar
hutool-all-5.8.16.jar              log4j-api-2.17.1.jar             seatunnel-engine-client-2.3.3.jar           springfox-schema-2.6.1.jar
jackson-annotations-2.12.6.jar         log4j-over-slf4j-1.7.25.jar          seatunnel-engine-common-2.3.3.jar           springfox-spi-2.6.1.jar
jackson-core-2.12.6.jar            log4j-to-slf4j-2.17.1.jar            seatunnel-engine-core-2.3.3.jar         springfox-spring-web-2.6.1.jar
jackson-databind-2.12.6.jar        logback-classic-1.2.3.jar            seatunnel-guava-2.3.3-optional.jar          springfox-swagger-common-2.6.1.jar
jackson-datatype-jdk8-2.13.3.jar       logback-core-1.2.3.jar           seatunnel-jackson-2.3.3-optional.jar        springfox-swagger-ui-2.6.1.jar
jackson-datatype-jsr310-2.13.3.jar     lombok-1.18.24.jar               seatunnel-plugin-discovery-2.3.3.jar        springfox-swagger2-2.6.1.jar
jackson-module-parameter-names-2.13.3.jar  mapstruct-1.0.0.Final.jar            seatunnel-server-common-1.0.0-SNAPSHOT.jar      swagger-annotations-1.5.10.jar
jakarta.annotation-api-1.3.5.jar       mssql-jdbc-9.2.1.jre8.jar            seatunnel-transforms-v2-2.3.3.jar           swagger-annotations-2.2.14.jar
jakarta.servlet-api-4.0.4.jar          mybatis-3.5.10.jar               serializer-api-2.3.3.jar                swagger-models-1.5.10.jar
jakarta.validation-api-2.0.2.jar       mybatis-plus-3.5.3.1.jar         serializer-protobuf-2.3.3.jar           tomcat-embed-el-9.0.63.jar
jakarta.websocket-api-1.1.2.jar        mybatis-plus-annotation-3.5.3.1.jar      slf4j-api-1.7.25.jar
javax.el-3.0.0.jar             mybatis-plus-boot-starter-3.5.3.1.jar    snakeyaml-1.29.jar
jboss-logging-3.4.1.Final.jar          mybatis-plus-core-3.5.3.1.jar        spring-aop-5.3.20.jar

Carl-Zhou-CN commented 9 months ago

@raheen1 Sorry, may not be able to help you, this may be a bug, the future will be improved

neehar18 commented 9 months ago

@Carl-Zhou-CN thanks, I will try and debug this issue myself and update you if it works.

Question: if I want to add a jar file to the seatunnel-web/libs folder, which pom file should I add the jar file to?

Carl-Zhou-CN commented 9 months ago

@raheen1 ,Thank you very much, seatunnel-web/libs is packaged using maven-assembly-plugin, which you can check out https://github.com/apache/seatunnel-web/blob/main/seatunnel-web-dist/src/main/assembly/seatunnel-web-ci.xml

neehar18 commented 9 months ago

@Carl-Zhou-CN Earlier today, I was able to fix the "Unknown Exception" error while trying to test connect. I had to add hadoop-aws, aws-java-sdk seatunnel-hadoop3-3.1.4-uber jars in the following files:

The guava package was not compatible with the hadoop version so, I had to upgrade the guava jar package from 19.0 to 26.0-jre here: https://github.com/apache/seatunnel-web/blob/a4f8eebef0c1dddaf9102a628138e481d5f544e0/pom.xml#L71

After making the above changes, I got a successful connection test:

neehar18 commented 9 months ago

But after creating the S3 datasource, I am unable to perform any source to sink tasks from S3 because, there is no support in the seatunnel-web code:

neehar18 commented 9 months ago

I get the following "Unknown Exception" when trying to create a task using the S3 datasource:

neehar18 commented 9 months ago

The logs show this error:

2023-12-07 11:12:13.901 seatunnel 2e5fb308b26f ERROR [tr:,sp:] [qtp106787773-20] [GlobalExceptionHandler.logError():83] - getDatabases is not supported for S3 datasource
java.lang.UnsupportedOperationException: getDatabases is not supported for S3 datasource

neehar18 commented 9 months ago

@Carl-Zhou-CN How should I create a task with S3 datasource?

Carl-Zhou-CN commented 9 months ago

@EricJoy2048 @hailin0 hi, can you help me look at this problem?

neehar18 commented 9 months ago

@EricJoy2048 @hailin0 can you help understand if this is a limitation with the seatunnel web or am I doing anything wrong in creating a S3 task?

neehar18 commented 9 months ago

@Carl-Zhou-CN do you have information on this issue?

Carl-Zhou-CN commented 9 months ago

Sorry, I'm not familiar with this web

neehar18 commented 9 months ago

@hailin0 @EricJoy2048 can you guys clarify if this is a UI limitation or if I am doing something wrong in task creation

github-actions[bot] commented 8 months ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

apache / seatunnel