search

streamsets / tutorials

StreamSets Tutorials
Apache License 2.0
345 stars 189 forks source link

the tutorial compilation issue #118

Open cuiwei-github opened 4 years ago

cuiwei-github commented 4 years ago

Hi, I git cloned datacollector 1.2.2.0, datacollector-api 1.2.2.0, can you tell me which version of datacollector-plugin-api I should get? it has no 1.2.2.0.

While compiling the datacollector: mvn clean install -DskipTests, the container failed. the log:

[INFO] Building StreamSets Data Collector Container 1.2.2.0 [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ streamsets-datacollector-container --- [INFO] Deleting C:\Users\g706872\git\datacollector\container\target [INFO] [INFO] --- hadoop-maven-plugins:2.5.0:version-info (version-info) @ streamsets-datacollector-container --- [WARNING] [svn, info] failed with error code 1 [INFO] SCM: GIT [INFO] Computed MD5: 62ac5d1ee141cfb7921c48f46b28db [INFO] [INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ streamsets-datacollector-container --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 9 resources [INFO] Copying 1 resource [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ streamsets-datacollector-container --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 364 source files to C:\Users\g706872\git\datacollector\container\target\classes [INFO] ------------------------------------------------------------- [WARNING] COMPILATION WARNING : [INFO] ------------------------------------------------------------- [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/memory/MultipleMonitorLocker.java:[24,16] sun.misc.Unsafe is internal proprietary API and may be removed in a future release [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/memory/MultipleMonitorLocker.java:[24,16] sun.misc.Unsafe is internal proprietary API and may be removed in a future release [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/memory/MultipleMonitorLocker.java:[24,16] sun.misc.Unsafe is internal proprietary API and may be removed in a future release [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/memory/MultipleMonitorLocker.java:[24,16] sun.misc.Unsafe is internal proprietary API and may be removed in a future release [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/memory/MultipleMonitorLocker.java:[33,24] sun.misc.Unsafe is internal proprietary API and may be removed in a future release [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/memory/MultipleMonitorLocker.java:[35,5] sun.misc.Unsafe is internal proprietary API and may be removed in a future release [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/memory/MultipleMonitorLocker.java:[37,27] sun.misc.Unsafe is internal proprietary API and may be removed in a future release [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/memory/MultipleMonitorLocker.java:[39,17] sun.misc.Unsafe is internal proprietary API and may be removed in a future release [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/execution/runner/cluster/ClusterRunner.java: Some input files use or override a deprecated API. [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/execution/runner/cluster/ClusterRunner.java: Recompile with -Xlint:deprecation for details. [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/config/StageDefinition.java: Some input files use unchecked or unsafe operations. [WARNING] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/config/StageDefinition.java: Recompile with -Xlint:unchecked for details. [INFO] 12 warnings [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /C:/Users/g706872/git/datacollector/container/src/main/java/com/streamsets/datacollector/definition/ConfigDefinitionExtractor.java:[391,12] an enum switch case label must be the unqualified name of an enumeration constant [INFO] 1 error [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------

metadaddy commented 4 years ago

Is there a specific reason you're trying to use such an old version? We are currently on 3.13.0.

cuiwei-github commented 4 years ago

Hi, Mr. Pat,

I picked the 1.2.2.0 because it is the version used in this tutorial I am trying to follow. I am not sure the difference of building new version and the old version. and I am not sure the steps/code in the tutorial work with new version of streamsets datacollector. If you have equivalent tutorial on how to create custom java destination, that will be awesome. Can you share with me if you have?

if i want to try the new version, which version number do you recommend? and all three modules are with the same version number?

Thanks,

Wei

On Wed, Mar 4, 2020 at 2:14 PM Pat Patterson notifications@github.com wrote:

Is there a specific reason you're trying to use such an old version? We are currently on 3.13.0.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/streamsets/tutorials/issues/118?email_source=notifications&email_token=AOXQL4H2RPMD4GL3IRDS2TLRF2SC7A5CNFSM4LBQXRV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENZWNGA#issuecomment-594765464, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOXQL4AG6KNE34ZJDRDI25LRF2SC7ANCNFSM4LBQXRVQ .

metadaddy commented 4 years ago

Check the version of StreamSets Data Collector that you are using. It needs to be the same as that. The same instructions should still work.

cuiwei-github commented 4 years ago

Hi, Mr. Pat,

I got all three project from master and I am trying to build following the order. I am still having issue with building datacollector. I managed to solve as many as I can and I am stuck at this:

(very similar warning about the same line 201, column 19 for many dependencies) [WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: com.streamsets:streamsets-datacollector-kafka_multiso urce-protolib:test-jar -> duplicate declaration of version ${project.version} @ com.streamsets:streamsets-datacollector-root-lib:3.15.0-SNAPSHOT, C:\Users\g706872\ git\datacollector\root-lib\pom.xml, line 201, column 19 [WARNING] [WARNING] Some problems were encountered while building the effective model for com.streamsets:streamsets-datacollector-mongodb_3-lib:jar:3.15.0-SNAPSHOT [WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: com.streamsets:streamsets-datacollector-kafka_multiso urce-protolib:test-jar -> duplicate declaration of version ${project.version} @ com.streamsets:streamsets-datacollector-root-lib:3.15.0-SNAPSHOT, C:\Users\g706872\ git\datacollector\root-lib\pom.xml, line 201, column 19 [WARNING] [WARNING] Some problems were encountered while building the effective model for com.streamsets:streamsets-datacollector-databricks-ml_2-lib:jar:3.15.0-SNAPSHOT [WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: com.streamsets:streamsets-datacollector-kafka_multiso urce-protolib:test-jar -> duplicate declaration of version ${project.version} @ com.streamsets:streamsets-datacollector-root-lib:3.15.0-SNAPSHOT, C:\Users\g706872\ git\datacollector\root-lib\pom.xml, line 201, column 19 [WARNING] [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build. [WARNING] [WARNING] For this reason, future Maven versions might no longer support building such malformed projects. [WARNING] [INFO]

[INFO] Reactor Build Order: [INFO] [INFO] StreamSets Data Collector Jython 2.7.0 Lib [INFO] StreamSets Data Collector MapR 6.1.0 Lib [INFO] StreamSets Data Collector MapR 6.1.0 MEP 6 Lib [INFO] Streamsets Data Collector MapR Spark 2.1 MEP 3.0 Lib [INFO] StreamSets Data Collector MongoDB 3.0.2 Lib [INFO] StreamSets Data Collector Databricks ML 2 Lib [INFO] [INFO]

[INFO] Building StreamSets Data Collector Jython 2.7.0 Lib 3.15.0-SNAPSHOT [INFO]

[INFO]

[INFO] Reactor Summary: [INFO] [INFO] StreamSets Data Collector Jython 2.7.0 Lib ......... FAILURE [ 1.247 s] [INFO] StreamSets Data Collector MapR 6.1.0 Lib ........... SKIPPED [INFO] StreamSets Data Collector MapR 6.1.0 MEP 6 Lib ..... SKIPPED [INFO] Streamsets Data Collector MapR Spark 2.1 MEP 3.0 Lib SKIPPED [INFO] StreamSets Data Collector MongoDB 3.0.2 Lib ........ SKIPPED [INFO] StreamSets Data Collector Databricks ML 2 Lib ...... SKIPPED [INFO]

[INFO] BUILD FAILURE [INFO]

[INFO] Total time: 12.922 s [INFO] Finished at: 2020-03-06T14:06:40-05:00 [INFO] Final Memory: 175M/917M [INFO]

[ERROR] Failed to execute goal on project streamsets-datacollector-jython_2_7-lib: Could not resolve dependencies for project com.streamsets:streamsets-datacollect or-jython_2_7-lib:jar:3.15.0-SNAPSHOT: Failure to find com.streamsets:streamsets-datacollector-jython-protolib:jar:tests:3.15.0-SNAPSHOT in http://packages.conflue nt.io/maven/ was cached in the local repository, resolution will not be reattempted until the update interval of confluent has elapsed or updates are forced -> [He lp 1]

While I was trying to build streamsets-datacollector-jython-protolib:jar:tests:3.15.0-SNAPSHOT separately, I got:

[ERROR] C:\users\g706872\git\datacollector\jython-protolib\src\test\java\com\streamsets\pipeline\stage\processor\scripting\TestJythonAbstractScriptingProcessor.jav a:[19,43] package com.streamsets.datacollector.runner does not exist [ERROR] C:\users\g706872\git\datacollector\jython-protolib\src\test\java\com\streamsets\pipeline\stage\origin\scripting\TestJythonAbstractScriptingSource.java:[19, 43] package com.streamsets.datacollector.runner does not exist

Then, I tried to rebuild the container, since I know the com.streamsets.datacollector.runner is in the container project. the container was build successfully, but still the jython-protolib test jar build failed for the same reason.

Can you help me on this compilation issue? or is it enough to create my own java custom destination if my custom destination has nothing to do with jython, MarR, MongoDB, databricks?

Thanks,

Wei

On Wed, Mar 4, 2020 at 2:53 PM Pat Patterson notifications@github.com wrote:

Check the version of StreamSets Data Collector that you are using. It needs to be the same as that. The same instructions should still work.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/streamsets/tutorials/issues/118?email_source=notifications&email_token=AOXQL4CGBJPYHKLM66SKAELRF2WSPA5CNFSM4LBQXRV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENZ5XUA#issuecomment-594795472, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOXQL4EBOXZ7OZPHPUBKCUDRF2WSPANCNFSM4LBQXRVQ .

metadaddy commented 4 years ago

Hi Wei - I'll investigate the build failure. Yes - try creating your custom destination now. I don't think those specific issues will be a problem.