search

apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
8.04k stars 1.82k forks source link

[Bug] [Seatunnel Web; S3 File] S3 file does not render all the required options in the UI #5961

Open neehar18 opened 11 months ago

neehar18 commented 11 months ago

Search before asking

What happened

I am trying to deploy a S3 connector (as source) using the UI and the UI does not render all the required options. I am using the following config file to deploy the same connector using the command line and it works fine.

# Defining the runtime environment
env {
  # You can set flink configuration here
  execution.parallelism = 1
  job.mode = "BATCH"
}

source {
  S3File {
    path = "/test/sample.json"
    fs.s3a.endpoint="s3.us-west-2.amazonaws.com"
    fs.s3a.aws.credentials.provider = "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
    access_key = "##########"
    secret_key = "#########"
    bucket = "s3a://s3-dashdb"
    file_format_type = "json"
    schema {
      fields {
        id = int 
        name = string
      }
    }
  }
}

transform {
  # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
    # please go to https://seatunnel.apache.org/docs/category/transform-v2
}

sink {
  Console {}
}

In the screenshot below, you can notice that the UI does not have options for path, file_format_type, schema. Can you please help look into this issue? Thank you.

Screenshot 2023-12-05 at 10 16 44 AM

SeaTunnel Version

Seatunnel engine: 2.3.3 Seatunnel web: 1.0.0

SeaTunnel Config

# Defining the runtime environment
env {
  # You can set flink configuration here
  execution.parallelism = 1
  job.mode = "BATCH"
}

source {
  S3File {
    path = "/test/sample.json"
    fs.s3a.endpoint="s3.us-west-2.amazonaws.com"
    fs.s3a.aws.credentials.provider = "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
    access_key = "##########"
    secret_key = "#########"
    bucket = "s3a://s3-dashdb"
    file_format_type = "json"
    schema {
      fields {
        id = int 
        name = string
      }
    }
  }
}

transform {
  # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
    # please go to https://seatunnel.apache.org/docs/category/transform-v2
}

sink {
  Console {}
}

### Running Command

```shell
Not running any command, deploying from the UI.

Error Exception

Not enough options rendered in the UI.

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

neehar18 commented 11 months ago

@Carl-Zhou-CN are you able to assist with this issue?

Carl-Zhou-CN commented 11 months ago

@raheen1 May refer to https://mp.weixin.qq.com/s/eNWGP_09Oh4pHdoQkmGPzg ，DataSource ust save the connection information and really use it in the task editing phase

neehar18 commented 11 months ago

@Carl-Zhou-CN the link says "No results matched your search."

Screenshot 2023-12-05 at 11 53 22 AM
Carl-Zhou-CN commented 11 months ago

https://mp.weixin.qq.com/s/eNWGP_09Oh4pHdoQkmGPzg

neehar18 commented 11 months ago

Not able to reach the site. But I understand what you mean. So, first, I need to establish a connection and then I will be able to pass the other parameters like path, file_format_type, schema.

But when I try to test the connection, I get the following error and the UI error (Unknown Exception) in the screenshot :

2023-12-05 06:30:19.826 seatunnel 53dabe219ae9 INFO [tr:,sp:] [qtp2121754508-18] [DatasourceClassLoader.loadClass():43] - load class for name : org.apache.hadoop.fs.s3a.S3AFileSystem
2023-12-05 06:30:19.827 seatunnel 53dabe219ae9 INFO [tr:,sp:] [qtp2121754508-18] [DatasourceClassLoader.loadClass():53] - load class from parentClassLoader : org.apache.hadoop.fs.s3a.S3AFileSystem
2023-12-05 06:30:19.828 seatunnel 53dabe219ae9 ERROR [tr:,sp:] [qtp2121754508-18] [DatasourceClassLoader.loadClass():58] - parentClassLoader load class is error : java.lang.ClassNotFoundException: org.apache.hadoop.fs.s3a.S3AFileSystem
    at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
    at org.apache.seatunnel.datasource.classloader.DatasourceClassLoader.loadClass(DatasourceClassLoader.java:51)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:398)
    at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2575)
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2540)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2636)
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3269)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3301)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
    at org.apache.seatunnel.datasource.plugin.s3.S3DatasourceChannel.checkDataSourceConnectivity(S3DatasourceChannel.java:65)
    at org.apache.seatunnel.datasource.AbstractDataSourceClient.checkDataSourceConnectivity(AbstractDataSourceClient.java:117)
    at org.apache.seatunnel.app.service.impl.DatasourceServiceImpl.testDatasourceConnectionAble(DatasourceServiceImpl.java:212)
    at org.apache.seatunnel.app.controller.SeatunnelDatasourceController.testConnect(SeatunnelDatasourceController.java:152)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:517)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:584)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
    at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1631)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.Server.handle(Server.java:516)
    at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
    at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
    at java.base/java.lang.Thread.run(Thread.java:829)

2023-12-05 06:30:19.832 seatunnel 53dabe219ae9 ERROR [tr:,sp:] [qtp2121754508-18] [GlobalExceptionHandler.logError():83] - java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2638)
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3269)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3301)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
    at org.apache.seatunnel.datasource.plugin.s3.S3DatasourceChannel.checkDataSourceConnectivity(S3DatasourceChannel.java:65)
    at org.apache.seatunnel.datasource.AbstractDataSourceClient.checkDataSourceConnectivity(AbstractDataSourceClient.java:117)
    at org.apache.seatunnel.app.service.impl.DatasourceServiceImpl.testDatasourceConnectionAble(DatasourceServiceImpl.java:212)
    at org.apache.seatunnel.app.controller.SeatunnelDatasourceController.testConnect(SeatunnelDatasourceController.java:152)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:517)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:584)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
    at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1631)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.Server.handle(Server.java:516)
    at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
    at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2542)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2636)
    ... 68 common frames omitted
neehar18 commented 11 months ago

But the same config file works when I run from the command line. I think the error is stating that hadoop libraries might be missing, but I have all the libraries in my pom.xml file.

Carl-Zhou-CN commented 11 months ago

image

Try doing this，metadata and data source jar package also need to copy to/opt/seatunnel/backend/apache - seatunnel - 2.3.3 / lib

neehar18 commented 11 months ago

I already have this jar file in the seatunnel/lib folder

Screenshot 2023-12-05 at 12 30 25 PM
Carl-Zhou-CN commented 11 months ago

Try to put in/opt/seatunnel/web/apache-seatunnel-web-1.0.0- bin/libs

Carl-Zhou-CN commented 11 months ago

There are still many unsolved problems in the web section

neehar18 commented 11 months ago

Okay, I will try this and update you if it works.

neehar18 commented 11 months ago

I tried adding the seatunnel-hadoop3-3.1.4-uber.jar to the apache-seatunnel-web-1.0.0- bin/libs folder. Still getting the same error.

neehar18 commented 11 months ago

Should I build the project again after adding the above library to the libs folder?

neehar18 commented 11 months ago

Or will the changes get reflected in the UI automatically?

neehar18 commented 11 months ago

@Carl-Zhou-CN I also don't see hadoop-aws and aws-java-sdk-bundle jars in the seatunnel-web. I only see them in the seatunnel lib folder. Should I include those two packages as well in the web libs folder as well?

ls seatunnel-web/libs/
HikariCP-4.0.3.jar             jcl-over-slf4j-1.7.25.jar            mybatis-plus-extension-3.5.3.1.jar          spring-beans-5.3.20.jar
apiguardian-api-1.1.2.jar          jcommander-1.81.jar              mybatis-spring-2.0.7.jar                spring-boot-2.6.8.jar
aspectjweaver-1.9.7.jar            jetty-continuation-9.4.46.v20220331.jar  mysql-connector-java-8.0.28.jar         spring-boot-autoconfigure-2.6.8.jar
auto-service-annotations-1.0.1.jar     jetty-http-9.4.46.v20220331.jar      ojdbc8-21.5.0.0.jar                 spring-boot-starter-2.6.8.jar
checker-qual-3.10.0.jar            jetty-io-9.4.46.v20220331.jar        opentest4j-1.2.0.jar                spring-boot-starter-aop-2.6.8.jar
checkpoint-storage-api-2.3.3.jar       jetty-security-9.4.46.v20220331.jar      postgresql-42.4.3.jar               spring-boot-starter-jdbc-2.6.8.jar
checkpoint-storage-local-file-2.3.3.jar    jetty-server-9.4.46.v20220331.jar        protobuf-java-3.11.4.jar                spring-boot-starter-jetty-2.6.8.jar
classmate-1.3.1.jar            jetty-servlet-9.4.46.v20220331.jar       protostuff-api-1.8.0.jar                spring-boot-starter-json-2.6.8.jar
commons-codec-1.13.jar             jetty-servlets-9.4.46.v20220331.jar      protostuff-collectionschema-1.8.0.jar       spring-boot-starter-web-2.6.8.jar
commons-collections4-4.4.jar           jetty-util-9.4.46.v20220331.jar      protostuff-core-1.8.0.jar               spring-context-5.3.20.jar
commons-compress-1.20.jar          jetty-util-ajax-9.4.46.v20220331.jar     protostuff-runtime-1.8.0.jar            spring-core-5.3.20.jar
commons-io-2.11.0.jar              jetty-webapp-9.4.46.v20220331.jar        scala-library-2.11.12.jar               spring-expression-5.3.20.jar
commons-lang3-3.4.jar              jetty-xml-9.4.46.v20220331.jar       seatunnel-api-2.3.3.jar             spring-jcl-5.3.20.jar
config-1.3.3.jar               jjwt-api-0.10.7.jar              seatunnel-app-1.0.0-SNAPSHOT.jar            spring-jdbc-5.3.20.jar
cron-utils-9.1.6.jar               jjwt-impl-0.10.7.jar             seatunnel-common-2.3.3.jar              spring-plugin-core-1.2.0.RELEASE.jar
gson-2.8.6.jar                 jjwt-jackson-0.10.7.jar          seatunnel-config-base-2.3.3.jar         spring-plugin-metadata-1.2.0.RELEASE.jar
guava-19.0.jar                 jsqlparser-4.4.jar               seatunnel-config-shade-2.3.3.jar            spring-tx-5.3.20.jar
h2-2.1.214.jar                 jsr305-3.0.0.jar             seatunnel-core-starter-2.3.3.jar            spring-web-5.3.20.jar
hazelcast-5.1.jar              junit-jupiter-api-5.9.0.jar          seatunnel-datasource-client-1.0.0-SNAPSHOT.jar  spring-webmvc-5.3.20.jar
hibernate-validator-6.2.2.Final.jar    junit-platform-commons-1.9.0.jar     seatunnel-dynamicform-1.0.0-SNAPSHOT.jar        springfox-core-2.6.1.jar
hutool-all-5.8.16.jar              log4j-api-2.17.1.jar             seatunnel-engine-client-2.3.3.jar           springfox-schema-2.6.1.jar
jackson-annotations-2.12.6.jar         log4j-over-slf4j-1.7.25.jar          seatunnel-engine-common-2.3.3.jar           springfox-spi-2.6.1.jar
jackson-core-2.12.6.jar            log4j-to-slf4j-2.17.1.jar            seatunnel-engine-core-2.3.3.jar         springfox-spring-web-2.6.1.jar
jackson-databind-2.12.6.jar        logback-classic-1.2.3.jar            seatunnel-guava-2.3.3-optional.jar          springfox-swagger-common-2.6.1.jar
jackson-datatype-jdk8-2.13.3.jar       logback-core-1.2.3.jar           seatunnel-jackson-2.3.3-optional.jar        springfox-swagger-ui-2.6.1.jar
jackson-datatype-jsr310-2.13.3.jar     lombok-1.18.24.jar               seatunnel-plugin-discovery-2.3.3.jar        springfox-swagger2-2.6.1.jar
jackson-module-parameter-names-2.13.3.jar  mapstruct-1.0.0.Final.jar            seatunnel-server-common-1.0.0-SNAPSHOT.jar      swagger-annotations-1.5.10.jar
jakarta.annotation-api-1.3.5.jar       mssql-jdbc-9.2.1.jre8.jar            seatunnel-transforms-v2-2.3.3.jar           swagger-annotations-2.2.14.jar
jakarta.servlet-api-4.0.4.jar          mybatis-3.5.10.jar               serializer-api-2.3.3.jar                swagger-models-1.5.10.jar
jakarta.validation-api-2.0.2.jar       mybatis-plus-3.5.3.1.jar         serializer-protobuf-2.3.3.jar           tomcat-embed-el-9.0.63.jar
jakarta.websocket-api-1.1.2.jar        mybatis-plus-annotation-3.5.3.1.jar      slf4j-api-1.7.25.jar
javax.el-3.0.0.jar             mybatis-plus-boot-starter-3.5.3.1.jar    snakeyaml-1.29.jar
jboss-logging-3.4.1.Final.jar          mybatis-plus-core-3.5.3.1.jar        spring-aop-5.3.20.jar
Carl-Zhou-CN commented 11 months ago

@raheen1 Sorry, may not be able to help you, this may be a bug, the future will be improved

neehar18 commented 11 months ago

@Carl-Zhou-CN thanks, I will try and debug this issue myself and update you if it works.

Question: if I want to add a jar file to the seatunnel-web/libs folder, which pom file should I add the jar file to?

Carl-Zhou-CN commented 11 months ago

@raheen1 ,Thank you very much, seatunnel-web/libs is packaged using maven-assembly-plugin, which you can check out https://github.com/apache/seatunnel-web/blob/main/seatunnel-web-dist/src/main/assembly/seatunnel-web-ci.xml

neehar18 commented 11 months ago

@Carl-Zhou-CN Earlier today, I was able to fix the "Unknown Exception" error while trying to test connect. I had to add hadoop-aws, aws-java-sdk seatunnel-hadoop3-3.1.4-uber jars in the following files:

The guava package was not compatible with the hadoop version so, I had to upgrade the guava jar package from 19.0 to 26.0-jre here: https://github.com/apache/seatunnel-web/blob/a4f8eebef0c1dddaf9102a628138e481d5f544e0/pom.xml#L71

After making the above changes, I got a successful connection test:

Screenshot 2023-12-07 at 4 30 48 PM
neehar18 commented 11 months ago

But after creating the S3 datasource, I am unable to perform any source to sink tasks from S3 because, there is no support in the seatunnel-web code:

neehar18 commented 11 months ago

I get the following "Unknown Exception" when trying to create a task using the S3 datasource:

Screenshot 2023-12-07 at 4 42 17 PM
neehar18 commented 11 months ago

The logs show this error: 

2023-12-07 11:12:13.901 seatunnel 2e5fb308b26f ERROR [tr:,sp:] [qtp106787773-20] [GlobalExceptionHandler.logError():83] - getDatabases is not supported for S3 datasource
java.lang.UnsupportedOperationException: getDatabases is not supported for S3 datasource
neehar18 commented 11 months ago

@Carl-Zhou-CN How should I create a task with S3 datasource?

Carl-Zhou-CN commented 11 months ago

@EricJoy2048 @hailin0 hi, can you help me look at this problem?

neehar18 commented 11 months ago

@EricJoy2048 @hailin0 can you help understand if this is a limitation with the seatunnel web or am I doing anything wrong in creating a S3 task?

neehar18 commented 11 months ago

@Carl-Zhou-CN do you have information on this issue?

Carl-Zhou-CN commented 11 months ago

Sorry, I'm not familiar with this web

neehar18 commented 11 months ago

@hailin0 @EricJoy2048 can you guys clarify if this is a UI limitation or if I am doing something wrong in task creation

github-actions[bot] commented 10 months ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.