search

spring-cloud / stream-applications

Functions and Spring Cloud Stream Applications for data driven microservices
https://spring.io/projects/spring-cloud-stream-applications
249 stars 104 forks source link

Project builds fail (both tests and builds itself) #149

Closed alpreu closed 3 years ago

alpreu commented 3 years ago

Hi, we have cloned the repo and want to maintain a private version with some functions still in development. We are currently following the Readme on how to build the whole project (first running the setup-build.sh, then ./mvn clean install). However, for us the builds are constantly failing. There are some modules where the build itself fails (the object detection modules, due to a problem with a shared library) and some where the tests fail (e.g. aggregator-function, the mqtt and mongodb modules).

Can you give us any pointers what could be the reason we run into these problems? What JDK version are you using when building, and what Docker version?

artembilan commented 3 years ago

Hi @alpreu !

Thank you for reporting this one!

You know I fixed yesterday several Docker images issues and upgraded Testcontainers we use in the project. Would be great if you pull the latest master and check it again on your side. We probably have missed anything else yet.

The project has to be build against Java 8. We couldn't figure out yet how to upgrade Maven Javadocs plugin to make it working against higher Java version.

As a hint: as long as you have failing tests in modules you don't care so far, just mark them with @Disabled. Our fault over there should not effect your productivity.

And according to our Jenkins status it is green so far: https://jenkins.spring.io/job/stream-application-builds-master/

fiidim commented 3 years ago

I've been using <maven-javadoc-plugin.version>3.2.0</maven-javadoc-plugin.version> instead of the version in the latest 3.1.0-SNAPSHOT <maven-javadoc-plugin.version>3.1.1</maven-javadoc-plugin.version>, and its been working fine with openjdk 11.0.2. Have you tried that version?

artembilan commented 3 years ago

Thank you for your suggestion! Well, I changed it in the stream-applications-build/pom.xml and it still fails for me. Not immediately but somewhere in the middle:

[INFO] Building jar: C:\SpringIO\spring-cloud\stream-applications\applications\processor\image-recognition-processor\target\image-recognition-processor-3.1.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-javadoc-plugin:3.1.1:jar (javadoc) @ image-recognition-processor ---
[INFO]
1 error
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] stream-applications-build 1.1.0-SNAPSHOT ........... SUCCESS [  2.524 s]
[INFO] java-functions-parent 1.1.0-SNAPSHOT ............... SUCCESS [  0.663 s]
[INFO] spring-functions-parent 1.1.0-SNAPSHOT ............. SUCCESS [  0.566 s]
[INFO] aws-s3-common 1.1.0-SNAPSHOT ....................... SUCCESS [  3.311 s]
[INFO] config-common 1.1.0-SNAPSHOT ....................... SUCCESS [  2.508 s]
[INFO] file-common 1.1.0-SNAPSHOT ......................... SUCCESS [  2.712 s]
[INFO] ftp-common 1.1.0-SNAPSHOT .......................... SUCCESS [  2.182 s]
[INFO] function-test-support 1.1.0-SNAPSHOT ............... SUCCESS [  2.919 s]
[INFO] geode-common 1.1.0-SNAPSHOT ........................ SUCCESS [  2.418 s]
[INFO] metadata-store-common 1.1.0-SNAPSHOT ............... SUCCESS [  3.259 s]
[INFO] mqtt-common 1.1.0-SNAPSHOT ......................... SUCCESS [  3.129 s]
[INFO] tcp-common 1.1.0-SNAPSHOT .......................... SUCCESS [  2.311 s]
[INFO] twitter-common 1.1.0-SNAPSHOT ...................... SUCCESS [  2.625 s]
[INFO] tensorflow-common 1.1.0-SNAPSHOT ................... SUCCESS [  3.592 s]
[INFO] cdc-debezium-common 1.1.0-SNAPSHOT ................. SUCCESS [  2.653 s]
[INFO] cdc-debezium-boot-starter 1.1.0-SNAPSHOT ........... SUCCESS [  3.116 s]
[INFO] cassandra-consumer 1.1.0-SNAPSHOT .................. SUCCESS [  3.326 s]
[INFO] payload-converter-function 1.1.0-SNAPSHOT .......... SUCCESS [  2.202 s]
[INFO] elasticsearch-consumer 1.1.0-SNAPSHOT .............. SUCCESS [  2.214 s]
[INFO] analytics-consumer 1.1.0-SNAPSHOT .................. SUCCESS [  2.228 s]
[INFO] file-consumer 1.1.0-SNAPSHOT ....................... SUCCESS [  2.258 s]
[INFO] ftp-consumer 1.1.0-SNAPSHOT ........................ SUCCESS [  1.928 s]
[INFO] geode-consumer 1.1.0-SNAPSHOT ...................... SUCCESS [  2.060 s]
[INFO] jdbc-consumer 1.1.0-SNAPSHOT ....................... SUCCESS [  2.367 s]
[INFO] log-consumer 1.1.0-SNAPSHOT ........................ SUCCESS [  2.327 s]
[INFO] mongodb-consumer 1.1.0-SNAPSHOT .................... SUCCESS [  1.865 s]
[INFO] mqtt-consumer 1.1.0-SNAPSHOT ....................... SUCCESS [  1.828 s]
[INFO] rabbit-consumer 1.1.0-SNAPSHOT ..................... SUCCESS [  2.433 s]
[INFO] redis-consumer 1.1.0-SNAPSHOT ...................... SUCCESS [  2.634 s]
[INFO] sftp-consumer 1.1.0-SNAPSHOT ....................... SUCCESS [  2.072 s]
[INFO] tcp-consumer 1.1.0-SNAPSHOT ........................ SUCCESS [  1.834 s]
[INFO] websocket-consumer 1.1.0-SNAPSHOT .................. SUCCESS [  2.573 s]
[INFO] s3-consumer 1.1.0-SNAPSHOT ......................... SUCCESS [  2.691 s]
[INFO] twitter-consumer 1.1.0-SNAPSHOT .................... SUCCESS [  2.107 s]
[INFO] wavefront-consumer 1.1.0-SNAPSHOT .................. SUCCESS [  2.098 s]
[INFO] rsocket-consumer 1.1.0-SNAPSHOT .................... SUCCESS [  2.598 s]
[INFO] zeromq-consumer 1.1.0-SNAPSHOT ..................... SUCCESS [  2.347 s]
[INFO] aggregator-function 1.1.0-SNAPSHOT ................. SUCCESS [  2.343 s]
[INFO] filter-function 1.1.0-SNAPSHOT ..................... SUCCESS [  1.783 s]
[INFO] header-enricher-function 1.1.0-SNAPSHOT ............ SUCCESS [  2.626 s]
[INFO] http-request-function 1.1.0-SNAPSHOT ............... SUCCESS [  2.382 s]
[INFO] spel-function 1.1.0-SNAPSHOT ....................... SUCCESS [  1.728 s]
[INFO] splitter-function 1.1.0-SNAPSHOT ................... SUCCESS [  1.839 s]
[INFO] tasklauncher-function 1.1.0-SNAPSHOT ............... SUCCESS [  3.343 s]
[INFO] task-launch-request-function 1.1.0-SNAPSHOT ........ SUCCESS [  2.464 s]
[INFO] twitter-function 1.1.0-SNAPSHOT .................... SUCCESS [  2.189 s]
[INFO] image-recognition-function 1.1.0-SNAPSHOT .......... SUCCESS [  2.606 s]
[INFO] object-detection-function 1.1.0-SNAPSHOT ........... SUCCESS [  5.953 s]
[INFO] semantic-segmentation-function 1.1.0-SNAPSHOT ...... SUCCESS [  2.508 s]
[INFO] file-supplier 1.1.0-SNAPSHOT ....................... SUCCESS [  2.525 s]
[INFO] ftp-supplier 1.1.0-SNAPSHOT ........................ SUCCESS [  1.871 s]
[INFO] geode-supplier 1.1.0-SNAPSHOT ...................... SUCCESS [  2.267 s]
[INFO] http-supplier 1.1.0-SNAPSHOT ....................... SUCCESS [  2.439 s]
[INFO] jdbc-supplier 1.1.0-SNAPSHOT ....................... SUCCESS [  2.328 s]
[INFO] jms-supplier 1.1.0-SNAPSHOT ........................ SUCCESS [  1.834 s]
[INFO] mail-supplier 1.1.0-SNAPSHOT ....................... SUCCESS [  1.951 s]
[INFO] mongodb-supplier 1.1.0-SNAPSHOT .................... SUCCESS [  2.349 s]
[INFO] mqtt-supplier 1.1.0-SNAPSHOT ....................... SUCCESS [  2.561 s]
[INFO] sftp-supplier 1.1.0-SNAPSHOT ....................... SUCCESS [  2.195 s]
[INFO] tcp-supplier 1.1.0-SNAPSHOT ........................ SUCCESS [  1.795 s]
[INFO] time-supplier 1.1.0-SNAPSHOT ....................... SUCCESS [  2.233 s]
[INFO] rabbit-supplier 1.1.0-SNAPSHOT ..................... SUCCESS [  2.629 s]
[INFO] websocket-supplier 1.1.0-SNAPSHOT .................. SUCCESS [  1.930 s]
[INFO] s3-supplier 1.1.0-SNAPSHOT ......................... SUCCESS [  2.043 s]
[INFO] twitter-supplier 1.1.0-SNAPSHOT .................... SUCCESS [  2.544 s]
[INFO] cdc-debezium-supplier 1.1.0-SNAPSHOT ............... SUCCESS [  2.463 s]
[INFO] syslog-supplier 1.1.0-SNAPSHOT ..................... SUCCESS [  2.024 s]
[INFO] zeromq-supplier 1.1.0-SNAPSHOT ..................... SUCCESS [  1.864 s]
[INFO] function-dependencies 1.1.0-SNAPSHOT ............... SUCCESS [  0.381 s]
[INFO] stream-applications-core 3.1.0-SNAPSHOT ............ SUCCESS [  0.414 s]
[INFO] stream-applications-test-support 3.1.0-SNAPSHOT .... SUCCESS [  3.867 s]
[INFO] stream-applications-postprocessor-common 3.1.0-SNAPSHOT SUCCESS [  1.787 s]
[INFO] stream-applications-micrometer-common 3.1.0-SNAPSHOT SUCCESS [  1.789 s]
[INFO] stream-applications-security-common 3.1.0-SNAPSHOT . SUCCESS [  2.451 s]
[INFO] stream-applications-composite-function-support 3.1.0-SNAPSHOT SUCCESS [  2.710 s]
[INFO] geode-source 3.1.0-SNAPSHOT ........................ SUCCESS [  1.504 s]
[INFO] file-source 3.1.0-SNAPSHOT ......................... SUCCESS [  0.560 s]
[INFO] ftp-source 3.1.0-SNAPSHOT .......................... SUCCESS [  0.576 s]
[INFO] jdbc-source 3.1.0-SNAPSHOT ......................... SUCCESS [  0.538 s]
[INFO] jms-source 3.1.0-SNAPSHOT .......................... SUCCESS [  0.592 s]
[INFO] http-source 3.1.0-SNAPSHOT ......................... SUCCESS [  0.567 s]
[INFO] mqtt-source 3.1.0-SNAPSHOT ......................... SUCCESS [  0.557 s]
[INFO] time-source 3.1.0-SNAPSHOT ......................... SUCCESS [  1.561 s]
[INFO] load-generator-source 3.1.0-SNAPSHOT ............... SUCCESS [  2.450 s]
[INFO] mongodb-source 3.1.0-SNAPSHOT ...................... SUCCESS [  0.562 s]
[INFO] tcp-source 3.1.0-SNAPSHOT .......................... SUCCESS [  0.543 s]
[INFO] rabbit-source 3.1.0-SNAPSHOT ....................... SUCCESS [  0.576 s]
[INFO] s3-source 3.1.0-SNAPSHOT ........................... SUCCESS [  0.615 s]
[INFO] sftp-source 3.1.0-SNAPSHOT ......................... SUCCESS [  0.582 s]
[INFO] twitter-stream-source 3.1.0-SNAPSHOT ............... SUCCESS [  0.643 s]
[INFO] twitter-search-source 3.1.0-SNAPSHOT ............... SUCCESS [  0.562 s]
[INFO] twitter-message-source 3.1.0-SNAPSHOT .............. SUCCESS [  0.593 s]
[INFO] websocket-source 3.1.0-SNAPSHOT .................... SUCCESS [  1.178 s]
[INFO] cdc-debezium-source 3.1.0-SNAPSHOT ................. SUCCESS [  1.328 s]
[INFO] syslog-source 3.1.0-SNAPSHOT ....................... SUCCESS [  1.358 s]
[INFO] mail-source 3.1.0-SNAPSHOT ......................... SUCCESS [  0.570 s]
[INFO] zeromq-source 3.1.0-SNAPSHOT ....................... SUCCESS [  0.570 s]
[INFO] source 3.1.0-SNAPSHOT .............................. SUCCESS [  0.019 s]
[INFO] analytics-sink 3.1.0-SNAPSHOT ...................... SUCCESS [  0.567 s]
[INFO] cassandra-sink 3.1.0-SNAPSHOT ...................... SUCCESS [  0.601 s]
[INFO] file-sink 3.1.0-SNAPSHOT ........................... SUCCESS [  0.540 s]
[INFO] ftp-sink 3.1.0-SNAPSHOT ............................ SUCCESS [  0.588 s]
[INFO] geode-sink 3.1.0-SNAPSHOT .......................... SUCCESS [  0.579 s]
[INFO] log-sink 3.1.0-SNAPSHOT ............................ SUCCESS [  0.583 s]
[INFO] mongodb-sink 3.1.0-SNAPSHOT ........................ SUCCESS [  0.556 s]
[INFO] mqtt-sink 3.1.0-SNAPSHOT ........................... SUCCESS [  1.267 s]
[INFO] jdbc-sink 3.1.0-SNAPSHOT ........................... SUCCESS [  1.177 s]
[INFO] redis-sink 3.1.0-SNAPSHOT .......................... SUCCESS [  1.341 s]
[INFO] rabbit-sink 3.1.0-SNAPSHOT ......................... SUCCESS [  0.598 s]
[INFO] router-sink 3.1.0-SNAPSHOT ......................... SUCCESS [  2.005 s]
[INFO] sftp-sink 3.1.0-SNAPSHOT ........................... SUCCESS [  0.608 s]
[INFO] tasklauncher-sink 3.1.0-SNAPSHOT ................... SUCCESS [  2.710 s]
[INFO] s3-sink 3.1.0-SNAPSHOT ............................. SUCCESS [  1.191 s]
[INFO] tcp-sink 3.1.0-SNAPSHOT ............................ SUCCESS [  1.336 s]
[INFO] throughput-sink 3.1.0-SNAPSHOT ..................... SUCCESS [  2.453 s]
[INFO] websocket-sink 3.1.0-SNAPSHOT ...................... SUCCESS [  0.668 s]
[INFO] twitter-update-sink 3.1.0-SNAPSHOT ................. SUCCESS [  0.679 s]
[INFO] twitter-message-sink 3.1.0-SNAPSHOT ................ SUCCESS [  0.644 s]
[INFO] wavefront-sink 3.1.0-SNAPSHOT ...................... SUCCESS [  0.661 s]
[INFO] pgcopy-sink 3.1.0-SNAPSHOT ......................... SUCCESS [  2.714 s]
[INFO] rsocket-sink 3.1.0-SNAPSHOT ........................ SUCCESS [  1.175 s]
[INFO] elasticsearch-sink 3.1.0-SNAPSHOT .................. SUCCESS [  0.597 s]
[INFO] zeromq-sink 3.1.0-SNAPSHOT ......................... SUCCESS [  0.567 s]
[INFO] sink 3.1.0-SNAPSHOT ................................ SUCCESS [  0.013 s]
[INFO] aggregator-processor 3.1.0-SNAPSHOT ................ SUCCESS [  0.625 s]
[INFO] bridge-processor 3.1.0-SNAPSHOT .................... SUCCESS [  1.879 s]
[INFO] filter-processor 3.1.0-SNAPSHOT .................... SUCCESS [  0.571 s]
[INFO] groovy-processor 3.1.0-SNAPSHOT .................... SUCCESS [  2.157 s]
[INFO] header-enricher-processor 3.1.0-SNAPSHOT ........... SUCCESS [  1.186 s]
[INFO] http-request-processor 3.1.0-SNAPSHOT .............. SUCCESS [  1.342 s]
[INFO] splitter-processor 3.1.0-SNAPSHOT .................. SUCCESS [  0.533 s]
[INFO] transform-processor 3.1.0-SNAPSHOT ................. SUCCESS [  0.597 s]
[INFO] script-processor 3.1.0-SNAPSHOT .................... SUCCESS [  2.255 s]
[INFO] twitter-trend-processor 3.1.0-SNAPSHOT ............. SUCCESS [  0.733 s]
[INFO] image-recognition-processor 3.1.0-SNAPSHOT ......... FAILURE [  1.724 s]
[INFO] object-detection-processor 3.1.0-SNAPSHOT .......... SKIPPED
[INFO] semantic-segmentation-processor 3.1.0-SNAPSHOT ..... SKIPPED
[INFO] processor 3.1.0-SNAPSHOT ........................... SKIPPED
[INFO] applications 3.1.0-SNAPSHOT ........................ SKIPPED
[INFO] stream-applications-release-train 2021.0.0-SNAPSHOT  SKIPPED
[INFO] stream-applications-docs 2021.0.0-SNAPSHOT ......... SKIPPED
[INFO] stream-applications-descriptor 2021.0.0-SNAPSHOT ... SKIPPED
[INFO] stream-applications 3.1.0-SNAPSHOT ................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  03:56 min
[INFO] Finished at: 2021-06-10T15:09:49-04:00
[INFO] ------------------------------------------------------------------------
[WARNING] The requested profile "spring" could not be activated because it does not exist.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:3.1.1:jar (javadoc) on project image-recognition-processor: MavenReportException: Error while generating Javadoc: 
[ERROR] Exit code: 1 - javadoc: error - The code being documented uses modules but the packages defined in https://docs.oracle.com/en/java/javase/14/docs/api/ are in the unnamed module.
[ERROR]
[ERROR] Command line was: C:\Java\jdk-14\bin\javadoc.exe @options @packages
[ERROR]
[ERROR] Refer to the generated Javadoc files in 'C:\SpringIO\spring-cloud\stream-applications\applications\processor\image-recognition-processor\target\apidocs' dir.

You can see that it still points to version 3.1.1 of the mentioned plugin.

Maybe we need to consider adding a <pluginManagement> in that parent pom?..

fiidim commented 3 years ago

Interesting. WIth the latest snapshot, I only modified stream-applications-build/pom.xml to change 3.1.1 to 3.2.0 and I get the below output for that function (no errors):

[INFO] --- maven-javadoc-plugin:3.2.0:jar (javadoc) @ image-recognition-processor ---
[INFO] No previous run data found, generating javadoc.
[INFO] Building jar: /stream-applications/applications/processor/image-recognition-processor/target/image-recognition-processor-3.1.0-SNAPSHOT-javadoc.jar
[INFO]

Maybe 3.1.1 is cached for you somewhere.

artembilan commented 3 years ago

OK! After running setup-build.sh I see this in logs so far:

[INFO] --- maven-javadoc-plugin:3.2.0:jar (javadoc) @ metadata-store-common ---
[ERROR] Error fetching link: C:\SpringIO\spring-cloud\stream-applications\functions\common\metadata-store-common\target\javadoc-bundle-options. Ignored it.

So, probably all good for now 😄

Thank you for your feedback and contribution!

fiidim commented 3 years ago

No worries. I'll do everything I can to bring this project out of 1.8 and into something more recent. After all, if this project is moving to spring-boot 2.5.0, the support level is up to 16 now.

artembilan commented 3 years ago

I have just fixed this upgrading to the latest Maven Javadoc Plugin 3.3.0: https://github.com/spring-cloud/stream-applications/commit/167ece5e9f15c5ef80107c39b64bb106e12e4d63.

Works well for me with Java 14.

Meanwhile it is really convenient even for us to have only one JDK installed and have all the projects compatible, it doesn't mean that we raise the bar. Whatever you heard about Spring compatibility with Java 16, it doesn't mean that we build it over here with that JDK. That's only means that whatever we build here can be used in the target environment starting with Java 8. The point is that we still need to provide a Java 8 byte-code keeping our eye opened on whatever could be broken if we run our solution on Java version higher than 8. And fix respectively. Like we did recently for ByteBuffer and Buffer interoperability according their changes in Java 11.

Anyway thank you for your feedback and you are welcome to come back to us for more contributions!

sobychacko commented 3 years ago

@fiidim just curious, are you trying to replicate this repo for your custom needs? Or cloning/forking it first and then adding components on top of it? In any case, as @artembilan mentioned ^^, please feel free to send us any enhancements as PR's, feature requests, etc. Thank you!

fiidim commented 3 years ago

@sobychacko @artembilan I replicate parts of this repo. The project structure you've created to allow binder-agnostic applications is exactly what I needed (and genius). I adapted your build structure (almost verbatim), plus the stream-applications-core project for my own set of SCDF apps. I use the starter applications from this repo as source components (usually sftp, and http) as-is, and the rest are custom functions/applications. Since it's not a full clone or fork, I monitor your commits from time to time to see what you guys are up to. The challenges I face in this approach are:

In general this repo is great for ideas, and it's structure is intuitive. SCDF has come such a long way in the past 6-12 months that it's hard to stay up to date sometimes. This project/repo helps bridge the gap a bit.

Thanks for all the hard work!

sobychacko commented 3 years ago

@fiidim Thank you for this great feedback! We can certainly adjust the plugin to generate apps with custom group id. If you want to PR that into the plugin, even better. I am not sure how we can address the issue you raised with stream-applications-composite-functions-support, I am afraid you have to manually decide which ones need to go there.

Versioning of the functions vs applications is an interesting one. When we started this initiative, functions were a new entity, but the apps were already in the middle of their release cycles. Therefore, we couldn't restart the apps from a 1.0.x generation. Since functions were brand new, it didn't make sense to start them at 3.0.x. That's the history behind this version mismatch with functions vs applications. The way we think about it is as if the functions are in a different repo with a different release cycle. Since we wanted to maintain all the components as part of a mono-repo structure, we didn't have a choice but to maintain them as separate version lines but still part of the same repo. The only component that resides outside of the mono-repo are the plugins.

Thanks again!

fiidim commented 3 years ago

@sobychacko PR created here.

Agreed on the composite-function comments. However, (arguably) my gut tells me that all processor functions should be eligible for composition by default. I'm not sure how to make that a reality either without manually using editing the pom.xml for that project. They need to be listed somewhere.

Understood on the versioning. The discussion goes even one more layer up. Since the application and function are so tightly coupled, that a change to a function would require the application to be re-versioned, but not vice-versa. On the other hand, your project is versioned as a suite (not per application), and the customer facing piece (the application) matches the project version, which is good. Semantically, you are right, the version of the functions are technically fresh since they post-date the original application (from another repo), but they are so tightly coupled to the application they probably should align. That's my 2 cents anyways.