BUG: Build fails when using data records and messaging steps in Spark pipelines

[ewilkins-csi]

Description

There is a build error that occurs during project generation when there are semantic data record models and Spark pipelines that use those records in messaging steps.

Steps to Reproduce

Clear, specific, and detailed steps taken to enable reproduction of the bug for investigation.

1. Create a project
2. Add a dictionary metamodel, a record metamodel, and a pipeline metamodel with Spark and at least one step with inbound/outbound type of messaging and a recordType set to the record model
3. Generate out the project following manual actions

Expected Behavior

The project can be incepted without error.

Actual Behavior

A build exception is thrown during the compile phase of the *-data-records module.

Definition of Done

• The POMs for combined data records, and the POM for core data records when using split generation are updated to include the requisite Kafka dependency, regardless of the pipeline
  • This will result in the dependency included when it isn't needed, but the extra dependency is relatively cheap, and avoids user confusion if they update their messaging steps later to include the record

Additional Context

Error:

14:49:13 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.13.0:compile (default-compile) on project semdata-messaging-bug-data-records: Compilation failure: Compilation failure: 
14:49:13 [ERROR] /Users/ewilkins/dev/aiops/init-tests/oneoff/semdata-messaging-bug/semdata-messaging-bug-shared/semdata-messaging-bug-data-records/src/generated/java/org/test/SimpleRecordDeserializerBase.java:[16,45] package org.apache.kafka.common.serialization does not exist
14:49:13 [ERROR] /Users/ewilkins/dev/aiops/init-tests/oneoff/semdata-messaging-bug/semdata-messaging-bug-shared/semdata-messaging-bug-data-records/src/generated/java/org/test/SimpleRecordDeserializerBase.java:[26,54] cannot find symbol
14:49:13 [ERROR]   symbol: class Deserializer
14:49:13 [ERROR] /Users/ewilkins/dev/aiops/init-tests/oneoff/semdata-messaging-bug/semdata-messaging-bug-shared/semdata-messaging-bug-data-records/src/generated/java/org/test/SimpleRecordSerializerBase.java:[16,45] package org.apache.kafka.common.serialization does not exist
14:49:13 [ERROR] /Users/ewilkins/dev/aiops/init-tests/oneoff/semdata-messaging-bug/semdata-messaging-bug-shared/semdata-messaging-bug-data-records/src/generated/java/org/test/SimpleRecordSerializerBase.java:[26,52] cannot find symbol
14:49:13 [ERROR]   symbol: class Serializer

Test Steps

1. Create a project from the archetype

   mvn archetype:generate '-DarchetypeGroupId=com.boozallen.aissemble' \
                          '-DarchetypeArtifactId=foundation-archetype' \
                          '-DarchetypeVersion=1.10.0-SNAPSHOT' \
                          '-DgroupId=org.test' \
                          '-Dpackage=org.test' \
                          '-DprojectGitUrl=test.org/test.git' \
                          '-DprojectName=Test Project' \
                          '-DartifactId=ots-457' \
   && cd ots-457

2. Add the following MDA models:
   • ots-457-pipeline-models/src/main/resources/pipelines/SparkPipeline.json
   • ots-457-pipeline-models/src/main/resources/dictionaries/SimpleDictionary.json
   • ots-457-pipeline-models/src/main/resources/records/SimpleRecord.json
3. Run the build to get the first set of manual actions: mvn clean install
4. Follow the manual actions output (this will include adding the data records module)
5. Run the build again and verify that the project builds successfully (with more manual actions): mvn clean install
6. Edit ots-457-shared/pom.xml to remove the <module> for ots-457-data-records and change the aissemble-data-records-combined-module Fermenter profile to aissemble-data-records-separate-module
7. Edit ots-457-pipelines/pom.xml to remove the <module> for spark-pipeline
8. Run the build again to regenerate the module manual actions: mvn clean install -Dmaven.build.cache.skipCache
9. Follow the manual actions
10. Edit ots-457-pipelines/spark-pipeline/pom.xml to change the single ots-457-data-records dependency to the split dependencies:

            <dependency>
                <groupId>${project.groupId}</groupId>
                <artifactId>ots-457-data-records-core</artifactId>
                <version>${project.version}</version>
            </dependency>
            <dependency>
                <groupId>${project.groupId}</groupId>
                <artifactId>ots-457-data-records-spark</artifactId>
                <version>${project.version}</version>
            </dependency>

11. Run the build again and verify it builds successfully: mvn clean install

[ewilkins-csi]

DoD with @carter-cundiff and @nartieri

[ewilkins-csi]

OTS passed with @carter-cundiff

[carter-cundiff]

Testing passed