lmco / streamflow

StreamFlow™ is a stream processing tool designed to help build and monitor processing workflows.
https://github.com/lmco/streamflow/wiki
Apache License 2.0
253 stars 69 forks source link

Notification system enhancements #3

Open juliencruz opened 9 years ago

juliencruz commented 9 years ago

Currently, error notifications for framework uploads and topology submissions do not provide contextual information as to the source of the error. This enhancement will improve the error notification system to provide detailed information about the problem so it can be resolved more quickly.

During framework upload the framework.yml file will be analyzed for correctness and ensure any required properties are specified. In addition, any class names referenced in the configuration will dynamically loaded to ensure all classes exists as defined and to catch a class loading error that will currently manifest during topology submission.

During topology submission, the streamflow engine will verify that all specified dependencies (e.g. frameworks, resources, files) are available and can be built into the final topology jar. In addition, all topology properties will be validated to ensure all required properties have been specified and configured properly.

In each of the above cases when an error is found, the notification will provide detailed information to correct the problem.