waleedev01 / analytics_automated_v2

Middleware layer for exposing analytics and distributed computing jobs as web services
Other
1 stars 0 forks source link

CWL Management Engine 1.0 #86

Closed waleedev01 closed 1 month ago

waleedev01 commented 1 month ago

This pull request introduces the CWL (Common Workflow Language) Management Engine 1.0 to AA. The key components of this pull request include:

  1. Command Line Tool (CLT) Handler:
    • Parses and processes CWL CommandLineTool definitions.
    • Handles environment variable requirements and dynamic input/output mappings.
    • Saves parsed tasks to the database, ensuring integrity and correctness.
    • Added error handling and logging for robustness.
  2. CWL Parser:
    • Reads and validates CWL files against the schema.
    • Differentiates between Workflows and CommandLineTools for appropriate handling.
    • Integrates with the CLT handler for seamless task creation and updating.
  3. Schema Validator:
    • Ensures CWL files adhere to the correct version and structure.
    • Manually validates key components like requirements, hints, inputs, and outputs.
    • Comprehensive error reporting to aid in debugging and fixing invalid CWL files.
  4. Workflow Handler:
    • Processes CWL workflows, managing step dependencies and task execution order.
    • Handles circular dependencies and integrates task environment variables.
    • Supports both inline CommandLineTools and external task references.
  5. Unit Tests for Schema Validator:
    • Extensive test coverage for various CWL validation scenarios.
    • Includes tests for valid workflows, missing elements, and unsupported features.
    • Ensures the validator accurately identifies issues and passes correct CWL files.
  6. REST API and Frontend Page for CWL File Upload:
    • Added a new REST API endpoint for uploading CWL files.
    • Developed a simple frontend page for users to upload multiple CWL files.
    • Displays processed files and detailed messages for user feedback.
  7. CI/CD Pipeline:
    • Integrated a simple CI/CD pipeline to automate testing and deployment.
    • Ensures code quality and functionality through continuous integration.